WebFaction
Community site: login faq

Hi,

I've got a connection error with Haystack and ElasticSearch. Strangely enough, I have the same config on another webfaction server running fine. I can't figure out what the cause is.

The app is running on Django 1.5.10 and Haystack 2.2.0

I created a "Custom app (listening on port)" which runs on port 31728.

The ElasticSearch config (elasticsearch.yml) is as follow:

# Unicast Discovery (disable multicast)
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

# Name your cluster here to whatever.
# My machine is called "Venus", so...
cluster:
  name: aesgp

network:
  host: 127.0.0.1

http:
  port: 31728

path:
  logs: /home/aesgp/webapps/aesgp_new/search/logs
  data: /home/aesgp/webapps/aesgp_new/search/data

The haystack config block in settings.py is:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
        'URL': 'http://127.0.0.1:31728/',
        'INDEX_NAME': 'haystack',
    },
}

Haystack is running, I can see it in the process list and I can reach it by doing a

curl -X GET http://127.0.0.1:31728/_cat/indices

which responds with:

red ingredients 5 1

(ingredients is the index)

However, when I run the manage.py rebuild_index, I get the following error:

WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
Are you sure you wish to continue? [y/N] y
Removing all documents from your index because you said so.
No handlers could be found for logger "elasticsearch"
Failed to clear Elasticsearch index: ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ProtocolError(('Connection aborted.', error(111, 'Connection refused')))
All documents removed.
Indexing 9546 substances for countries
ERROR:root:Error updating ingredients using default 
Traceback (most recent call last):
  File "/home/aesgp/lib/python2.7/haystack/management/commands/update_index.py", line 221, in handle_label
    self.update_backend(label, using)
  File "/home/aesgp/lib/python2.7/haystack/management/commands/update_index.py", line 266, in update_backend
    do_update(backend, index, qs, start, end, total, self.verbosity)
  File "/home/aesgp/lib/python2.7/haystack/management/commands/update_index.py", line 89, in do_update
    backend.update(index, current_qs)
  File "/home/aesgp/lib/python2.7/haystack/backends/elasticsearch_backend.py", line 184, in update
    bulk_index(self.conn, prepped_docs, index=self.index_name, doc_type='modelresult')
  File "/home/aesgp/.local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 145, in bulk
    for ok, item in streaming_bulk(client, actions, **kwargs):
  File "/home/aesgp/.local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 104, in streaming_bulk
    resp = client.bulk(bulk_actions, **kwargs)
  File "/home/aesgp/.local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/aesgp/.local/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 646, in bulk
    params=params, body=self._bulk_body(body))
  File "/home/aesgp/.local/lib/python2.7/site-packages/elasticsearch/transport.py", line 284, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/aesgp/.local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 51, in perform_request
    raise ConnectionError('N/A', str(e), e)
ConnectionError: ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ProtocolError(('Connection aborted.', error(111, 'Connection refused')))
ConnectionError: ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ProtocolError(('Connection aborted.', error(111, 'Connection refused')))

So, it seems it can't connect to the elasticsearch instance, but I can reach it from the server's command line.

Am I missing something obvious?

Thanks!

asked 23 Sep '14, 17:03

onepointzero
1313
accept rate: 0%

I just looked into this and at the moment your elasticsearch instance isn't running. Can you please start it and then ping back here to let us know?

(23 Sep '14, 19:11) seanf

Thanks. I just restarted it.

(23 Sep '14, 19:38) onepointzero

Your elasticsearch process seems to be listening only on IPv6:

[root@web466 aesgp]# lsof -p 32440 | grep TCP
java    32440 aesgp  134u  IPv6          527594740        0t0       TCP localhost.localdomain:vrace (LISTEN)
java    32440 aesgp  190u  IPv6          527599269        0t0       TCP localhost.localdomain:31728 (LISTEN)
java    32440 aesgp  191u  IPv6          527607966        0t0       TCP localhost.localdomain:50854->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  192u  IPv6          527607965        0t0       TCP localhost.localdomain:50853->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  193u  IPv6          527607967        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50853 (ESTABLISHED)
java    32440 aesgp  194u  IPv6          527607970        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50854 (ESTABLISHED)
java    32440 aesgp  195u  IPv6          527607971        0t0       TCP localhost.localdomain:50855->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  196u  IPv6          527607973        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50855 (ESTABLISHED)
java    32440 aesgp  197u  IPv6          527607974        0t0       TCP localhost.localdomain:50858->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  198u  IPv6          527607979        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50858 (ESTABLISHED)
java    32440 aesgp  199u  IPv6          527607980        0t0       TCP localhost.localdomain:50860->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  200u  IPv6          527607981        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50860 (ESTABLISHED)
java    32440 aesgp  201u  IPv6          527607982        0t0       TCP localhost.localdomain:50861->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  202u  IPv6          527607984        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50861 (ESTABLISHED)
java    32440 aesgp  203u  IPv6          527607985        0t0       TCP localhost.localdomain:50862->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  204u  IPv6          527607987        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50862 (ESTABLISHED)
java    32440 aesgp  205u  IPv6          527607988        0t0       TCP localhost.localdomain:50863->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  206u  IPv6          527607998        0t0       TCP localhost.localdomain:50867->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  207u  IPv6          527607990        0t0       TCP localhost.localdomain:50864->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  208u  IPv6          527607991        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50863 (ESTABLISHED)
java    32440 aesgp  209u  IPv6          527607992        0t0       TCP localhost.localdomain:50865->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  210u  IPv6          527607993        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50864 (ESTABLISHED)
java    32440 aesgp  211u  IPv6          527607997        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50866 (ESTABLISHED)
java    32440 aesgp  212u  IPv6          527607994        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50865 (ESTABLISHED)
java    32440 aesgp  213u  IPv6          527607995        0t0       TCP localhost.localdomain:50866->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  214u  IPv6          527608000        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50867 (ESTABLISHED)
java    32440 aesgp  215u  IPv6          527608001        0t0       TCP localhost.localdomain:50868->localhost.localdomain:vrace (ESTABLISHED)
java    32440 aesgp  216u  IPv6          527608003        0t0       TCP localhost.localdomain:vrace->localhost.localdomain:50868 (ESTABLISHED)

Try adding the following to your elasticsearch start script, then restart the process:

JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

Hope that helps!

permanent link

answered 23 Sep '14, 20:39

seanf
12.2k41836
accept rate: 37%

Thanks.

However, I added that command and I now see (via lsof) that it is listening on IPv4 but I'm still getting exactly the same error. Any ideas?

(24 Sep '14, 06:03) onepointzero

Everything looks good now. The process is listening on 127.0.0.1:31728 using IPv4:

TCP 127.0.0.1:31728 (LISTEN)

And we're even able to curl that address successfully with curl -I 'http://127.0.0.1:31728/'.

So, the problem has to be on the Django side. Can you find the exact code where it tries to connect and confirm it's connecting to the same HAYSTACK_CONNECTIONS.URL?

(24 Sep '14, 07:02) ryans ♦♦
1

Figured it out:

Due to a previous django version upgrade, the settings file had changed place but manage.py and the wsgi process were looking for them in different places.

The manage.py settings file was no longer there but the .pyc was, so it was getting erroneous confg info from there.

So from apache everything looked great, but from the command line it was running off a different and older settings(.pyc) file.

(29 Sep '14, 10:43) onepointzero
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×909
×4
×4

question asked: 23 Sep '14, 17:03

question was seen: 7,898 times

last updated: 29 Sep '14, 10:43

WEBFACTION
REACH US
SUPPORT
AFFILIATE PROGRAM
LEGAL
© COPYRIGHT 2003-2019 SWARMA LIMITED - WEBFACTION IS A SERVICE OF SWARMA LIMITED
REGISTERED IN ENGLAND AND WALES 5729350 - VAT REGISTRATION NUMBER 877397162
5TH FLOOR, THE OLD VINYL FACTORY, HAYES, UB3 1HA, UNITED KINGDOM