WebFaction
Community site: login faq

Good evening,

I am trying to set up a recurrent(it will be run with a cron task) Scrapy spider in order to parse a changing web page. The spider works properly when executing it on my local machine, however I am not able to run it on the Webfaction server. The special thing is that it has to register every parsed object into a database handled by Django. (This also works locally)

I created a dedicated django-admin command to execute this task, however it looks like when executing it there's a problem with: os.system('cd ' + SCRAPPY_DIR + '/fis/ && scrapy crawl races'). It says that there's no module named django.db. Additionnaly I found someone saying that I had to add those lines to the script, in order to make the spider "Django-aware":

sys.path.append(WEBAPPS_DIR + '/') os.environ['DJANGO_SETTINGS_MODULE'] = 'python_core.settings' from django.conf import settings

However it still doesn't work. Do you guys have any idea of how I could improve that ? Thanks a lot for your help and have a nice day, Seba-1511 PS: To have a full link to the project, check out: https://github.com/tooskich/python_core the problem is in the rankings app, more specifically with scraper.py or management/commands/updateraces.py

asked 30 Jul '14, 17:17

seba-1511
1315
accept rate: 0%

edited 30 Jul '14, 17:26

This does sound like a python path problem you suggest, but we're not sure that you're adding the correct path in the sys.path.append() line. Can you tell us the python path as it exists immediately before and after that line, and also tell us the full path to the settings.py file that you're going to be pointing DJANGO_SETTINGS_MODULE to?

(31 Jul '14, 09:35) ryans ♦♦

Hello,

Sorry for my late comment, I was traveling and had to set a few things before having access to an Internet connection. As I am adding a few path to the python path, here's exactly what I do: sys.path.append(BACKEND_DIR + '/') sys.path.append(APP_DIR + '/') sys.path.append(MODULE_DIR + '/') sys.path.append(WEBAPPS_DIR + '/')

And the path before those lines are executed: sys.path: '/home/tooski/bin', '/home/tooski/lib/python2.7/pip-1.5.4-py2.7.egg', '/home/tooski/lib/python2.7/Scrapy-0.22.2-py2.7.egg', '/home/tooski/lib/python2.7/six-1.6.1-py2.7.egg', '/home/tooski/lib/python2.7/cssselect-0.9.1-py2.7.egg', '/home/tooski/lib/python2.7/pyOpenSSL-0.14-py2.7.egg', '/home/tooski/lib/python2.7/lxml-3.3.5-py2.7-linux-x86_64.egg', '/home/tooski/lib/python2.7/queuelib-1.1.1-py2.7.egg', '/home/tooski/lib/python2.7/w3lib-1.5-py2.7.egg', '/home/tooski/lib/python2.7/Twisted-13.2.0-py2.7-linux-x86_64.egg', '/home/tooski/lib/python2.7', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/home/tooski/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages/PIL', '/home/tooski/webapps/python_core/python_core/rankings/fis'

after the lines are executed: sys.path: '/home/tooski/bin', '/home/tooski/lib/python2.7/pip-1.5.4-py2.7.egg', '/home/tooski/lib/python2.7/Scrapy-0.22.2-py2.7.egg', '/home/tooski/lib/python2.7/six-1.6.1-py2.7.egg', '/home/tooski/lib/python2.7/cssselect-0.9.1-py2.7.egg', '/home/tooski/lib/python2.7/pyOpenSSL-0.14-py2.7.egg', '/home/tooski/lib/python2.7/lxml-3.3.5-py2.7-linux-x86_64.egg', '/home/tooski/lib/python2.7/queuelib-1.1.1-py2.7.egg', '/home/tooski/lib/python2.7/w3lib-1.5-py2.7.egg', '/home/tooski/lib/python2.7/Twisted-13.2.0-py2.7-linux-x86_64.egg', '/home/tooski/lib/python2.7', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/home/tooski/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages/PIL', '/home/tooski/webapps/python_core/python_core/rankings/fis', '/home/tooski/webapps/python_core/python_core/', '/home/tooski/webapps/python_core/', '/home/tooski/webapps/python_core/python_core/rankings/', '/home/tooski/webapps/'

Note that when I try to do: import os os.environ['PYTHONPATH'] It results in a KeyError. Right after that, the import django error appears.

The DJANGO_SETTINGS_MODULE should point to the file /home/tooski/webapps/python_core/python_core/python_core/settings_server.py

Thank you for your help and have a nice day.

(04 Aug '14, 16:25) seba-1511

Your Django libraries are in /home/tooski/webapps/python_core/lib/python2.7 so you need to ensure that location is in your sys.path somewhere. You can do that either explicitly by setting the PYTHONPATH environment variable (or by adding it to sys.path in your settings, or implicitly by changing the working directory to /home/tooski/webapps/python_core before you run the command.

permanent link

answered 04 Aug '14, 17:55

seanf
12.2k41836
accept rate: 37%

Thank you very much, it seems like it solved my issue.

(11 Aug '14, 22:11) seba-1511
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×909
×3

question asked: 30 Jul '14, 17:17

question was seen: 3,741 times

last updated: 11 Aug '14, 22:19

WEBFACTION
REACH US
SUPPORT
AFFILIATE PROGRAM
LEGAL
© COPYRIGHT 2003-2019 SWARMA LIMITED - WEBFACTION IS A SERVICE OF SWARMA LIMITED
REGISTERED IN ENGLAND AND WALES 5729350 - VAT REGISTRATION NUMBER 877397162
5TH FLOOR, THE OLD VINYL FACTORY, HAYES, UB3 1HA, UNITED KINGDOM