Debugging or Automatically Killing Runaway Apache Processes

You can debug Apache/mod_wsgi processes with gdb.

First, make a note of the process id and command of whatever process is using a lot of memory or CPU. You can get that info by running:

ps -u $USER -o pid,command

Next, invoke gdb with the full path to the command of that process. For example, if your username is "tclancy" and your app name is "django", you can attach to your Apache processes with gdb by running:

gdb /home/tclancy/webapps/django/apache2/bin/httpd.worker

Once you are in gdb, attach to the process you noted above. For example, if the process id was 12345:

attach 12345

Next, you need to see the active threads by running:

info threads

The threads will be shown in a numbered list - switch to the thread that has your process id. For example, if the thread was #4:

thread 4

Finally, generate a Python traceback for whatever that thread is chewing on:

call PyRun_SimpleString("import sys, traceback; sys.stderr=open('/home/tclancy/traceback.txt','w',0); traceback.print_stack()")

With any luck, /home/tclancy/traceback.txt will contain a traceback that shows you what was executing in the thread that was causing a problem.

This doesn't always work as described above, for example if Apache isn't actually spinning on a Python operation. However, you can still use gdb to poke around in Apache to try to see what's going on. More information is available here: Apache Debugging Guide.

Hope that helps!

Thanks for this awesome tip seanf. I did some more research and found a gdb script that helps automate this:

Download script and copy it to your gdb config file:

wget http://svn.python.org/projects/python/trunk/Misc/gdbinit
cat gdbinit >> ~/.gdbinit

Run gdb with the PID of your process and then run the pystack command:

gdb -p PID
pystack

Source: http://stackoverflow.com/a/618748