WebFaction
Community site: login faq

I have a django site using mod_wsgi running with virtualenv. It was working well.

Then I added celery w kombu and launch celeryd using supervisor. As soon as I start celeryd my django site gets very slow, even if celery is not doing anything. As soon as I turn celery off my django site goes back to normal speed.

Anybody have any ideas of what the problem is or how to debug it?

Thanks

asked 03 Dec '11, 01:23

cwurld2
5137
accept rate: 0%

That alone should not cause any problems, what kind of tasks are you scheduling?

(03 Dec '11, 02:03) johns

I have it setup to run the celerybeat scheduler - mostly aperiodic tasks. To debug, I have no tasks scheduled or running. So django is slow even when celery is not "doing anything".

I am not sure how celery does it's timing. It seems to not use cron. It also seems to offer pretty fine grained timing precision. Is there some sort of background task that is interrupting every second? I do not need such resolution +-5minutes would be good enough. Is there a setting to adjust that?

Thanks

(03 Dec '11, 11:55) cwurld2

It seems as it uses 2 methods, either a python timedelta, or the crontab,

http://ask.github.com/celery/userguide/periodic-tasks.html

Based on that assumption it should not be causing a problem based on how often its running. At this point we need to gather more data. I have never used this django app but it looks promising in its ability to gather more data,

https://github.com/jmoiron/django-slow-log

You will also want to ask the official developers of the projects your using to see if they have any insight on the problem,

http://ask.github.com/django-celery/introduction.html#getting-help

(03 Dec '11, 17:32) johns

Thanks Johns. I was not aware of django-slow-log. I looked into installing that, but was uncomfortable with patching django. Everything works well as long as celery is off.

I found a setting for kombu that seemed promising - DJKOMBU_POLLING_INTERVAL I set that to 120 seconds. But that did not help.

When I start celery with supervisord, I get 3 processes. When I check the cpu usage w top, I see that one of those processes is long lived and consistently uses between 5% and 10% of the cpu on the shared server - it is almost always in the top 3 processes. Obviously this should not be since I am not running any tasks.

(04 Dec '11, 14:18) cwurld2

I solved the problem by switching from Kombu to Redis. For more info, see my post on stackoverflow: http://stackoverflow.com/questions/8379886/celery-slowing-down-django

permanent link

answered 18 Dec '11, 17:42

cwurld2
5137
accept rate: 0%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×906
×21

question asked: 03 Dec '11, 01:23

question was seen: 3,659 times

last updated: 18 Dec '11, 17:42

WEBFACTION
REACH US
SUPPORT
AFFILIATE PROGRAM
LEGAL
© COPYRIGHT 2003-2019 SWARMA LIMITED - WEBFACTION IS A SERVICE OF SWARMA LIMITED
REGISTERED IN ENGLAND AND WALES 5729350 - VAT REGISTRATION NUMBER 877397162
5TH FLOOR, THE OLD VINYL FACTORY, HAYES, UB3 1HA, UNITED KINGDOM