Debugging too many WordPress fcgi processes leading to memory limit exceeded

Question

I'm running a WordPress site using FastCGI and every week or so the number of fcgi processes goes from the usual 7 to something approaching 30 and everything ends up being killed by WebFaction because the memory usage has exceeded the maximum 1GB.

The .htaccess file has the following at the top:

<FilesMatch \.php$>
    SetHandler php56-fcgi6
</FilesMatch>

Here's an normal ps listing:

  PID   RSS                  STARTED   TIME COMMAND
  788  2176 Tue Jul 17 16:53:13 2018   0:00 -bash
 1445  1844 Tue Jul 17 16:53:54 2018   0:00 ps -u username -o pid,rss,lstart,bsdtime,command
 1446   972 Tue Jul 17 16:53:54 2018   0:00 awk {print $0}{sum+=$2} END {print "Total", sum/1024, "MB"}
14551 16388 Tue Jul 17 15:56:48 2018   0:00 /home/php-cgi/php56.fcgi
14552 77372 Tue Jul 17 15:56:48 2018   4:14 /home/php-cgi/php56.fcgi
14553 79780 Tue Jul 17 15:56:48 2018   4:14 /home/php-cgi/php56.fcgi
14554 80016 Tue Jul 17 15:56:48 2018   4:13 /home/php-cgi/php56.fcgi
14555 81012 Tue Jul 17 15:56:48 2018   4:20 /home/php-cgi/php56.fcgi
14556 77048 Tue Jul 17 15:56:48 2018   4:17 /home/php-cgi/php56.fcgi
14557 78868 Tue Jul 17 15:56:48 2018   4:22 /home/php-cgi/php56.fcgi
34635  2172 Tue Jul 17 15:36:04 2018   0:00 -bash
36505  2140 Tue Jul 17 15:37:45 2018   0:00 -bash
Total 488.074 MB

And here's an example of when it has gone wrong:

  PID   RSS                  STARTED   TIME COMMAND
  689 76352 Sun Jul  8 20:56:41 2018   0:04 /home/php-cgi/php56.fcgi
  693 33228 Sun Jul  8 20:56:41 2018   0:02 /home/php-cgi/php56.fcgi
  694 70368 Sun Jul  8 20:56:41 2018   0:03 /home/php-cgi/php56.fcgi
 8596 41760 Sun Jul  8 21:01:47 2018   0:02 /home/php-cgi/php56.fcgi
 8598 35208 Sun Jul  8 21:01:47 2018   0:02 /home/php-cgi/php56.fcgi
 8599 41812 Sun Jul  8 21:01:47 2018   0:02 /home/php-cgi/php56.fcgi
 8601 76796 Sun Jul  8 21:01:47 2018   0:02 /home/php-cgi/php56.fcgi
 8602 76712 Sun Jul  8 21:01:47 2018   0:01 /home/php-cgi/php56.fcgi
 8604 77308 Sun Jul  8 21:01:47 2018   0:01 /home/php-cgi/php56.fcgi
 8608 41448 Sun Jul  8 21:01:47 2018   0:01 /home/php-cgi/php56.fcgi
 8609 40768 Sun Jul  8 21:01:47 2018   0:03 /home/php-cgi/php56.fcgi
 8611 41520 Sun Jul  8 21:01:47 2018   0:02 /home/php-cgi/php56.fcgi
 8613 69880 Sun Jul  8 21:01:47 2018   0:02 /home/php-cgi/php56.fcgi
 8614 77040 Sun Jul  8 21:01:47 2018   0:03 /home/php-cgi/php56.fcgi
 8617 77392 Sun Jul  8 21:01:47 2018   0:02 /home/php-cgi/php56.fcgi
 8618 70404 Sun Jul  8 21:01:47 2018   0:01 /home/php-cgi/php56.fcgi
 8621 39952 Sun Jul  8 21:01:47 2018   0:02 /home/php-cgi/php56.fcgi
 8623 74988 Sun Jul  8 21:01:47 2018   0:01 /home/php-cgi/php56.fcgi
 8625 77004 Sun Jul  8 21:01:47 2018   0:02 /home/php-cgi/php56.fcgi
 9069 16444 Sun Jul  8 03:16:11 2018   0:00 /home/php-cgi/php56.fcgi
 9322  2184 Sun Jul  8 21:02:26 2018   0:00 -bash
11998  1840 Sun Jul  8 21:05:01 2018   0:00 ps -u username -o pid,rss,lstart,bsdtime,command
11999   968 Sun Jul  8 21:05:01 2018   0:00 awk {print $0}{sum+=$2} END {print "Total", sum/1024, "MB"}
15462 16444 Sun Jul  8 20:32:01 2018   0:00 /home/php-cgi/php56.fcgi
15757 16448 Sun Jul  8 20:32:19 2018   0:00 /home/php-cgi/php56.fcgi
16183 16444 Sun Jul  8 20:32:53 2018   0:00 /home/php-cgi/php56.fcgi
16647 16444 Sun Jul  8 20:33:10 2018   0:00 /home/php-cgi/php56.fcgi
35793 40872 Sun Jul  8 20:46:31 2018   0:08 /home/php-cgi/php56.fcgi
35803 33452 Sun Jul  8 20:46:31 2018   0:07 /home/php-cgi/php56.fcgi
35807 78028 Sun Jul  8 20:46:31 2018   0:08 /home/php-cgi/php56.fcgi
44304 34652 Sun Jul  8 20:51:37 2018   0:06 /home/php-cgi/php56.fcgi
44306 75664 Sun Jul  8 20:51:37 2018   0:06 /home/php-cgi/php56.fcgi
44308 80192 Sun Jul  8 20:51:37 2018   0:06 /home/php-cgi/php56.fcgi
44312 77732 Sun Jul  8 20:51:37 2018   0:06 /home/php-cgi/php56.fcgi
44313 50808 Sun Jul  8 20:51:37 2018   0:07 /home/php-cgi/php56.fcgi
44315 48976 Sun Jul  8 20:51:37 2018   0:07 /home/php-cgi/php56.fcgi
44316 87060 Sun Jul  8 20:51:37 2018   0:06 /home/php-cgi/php56.fcgi
44318 77176 Sun Jul  8 20:51:37 2018   0:07 /home/php-cgi/php56.fcgi
Total 1869.46 MB

I thought the SetHandler php56-fcgi6 was supposed to limit the number of FastCGI processes to 6?

Does anyone have any suggestions as to how we can further investigate this, and find out what is causing the explosion in processes?

Thanks in advance!

Answer 1

The thread limit set should limit the number of threads started, unless there is a forking bug or something in your PHP trying to execute a command in shell which is failing. With FCGI you should be using a task queue to manage external threads. It is hard to see if this is the case from the data above. Ideally you would want to use a debugger like strace while the problem is happening to get more info from the stuck threads.

First identify the PID of the process you want,

ps -u $USER -o pid,rss,command

Once you know the PID you can run strace, replace PID with the number from ps,

strace -tTvf -s 1024 -p PID

This will provide verbose information from which you may be able to figure out what is causing the failure.

Answer 2

0

Thank you for the suggestion, I have incorporated strace into my script that monitors memory usage and this will hopefully give me some more clues.

In the meantime, I have noticed that we have the following crontab entry, I don't know when or why this was added, or indeed if it is part of some standard WebFaction configuration:

13,33,53 * * * * /home/username/webapps/phpstack/apache2/bin/start > /dev/null 2>&1

Is this still needed these days, and what does/did it do?

Thanks!

permanent link

answered 21 Jul, 09:32

pd2018
11●2
accept rate: 0%

phpstack was an old custom apache and php that was often used on some CentOS 5 servers when customers needed something specific.

You should be able to safely delete that cron job after verifying none of your websites is still using that application.

(21 Jul, 11:54) aaront ♦♦

Debugging too many WordPress fcgi processes leading to memory limit exceeded

Follow this question

Related questions