WebFaction
Community site: login faq

I have a django site. I did a quick test, and created a url which is requested a lot (doesn't matter how for this question).

I've checked the apache frontend access log, I see a lot of 502 errors.
Some more error codes I see are 504, 499.
In the error log, I see many of these:
[alert] 12199#0: *23870231 8192 worker_connections are not enough while connecting to upstream, client...

Also, my website is somewhat freeze when all these requests arrive. Is the reason is the high traffic? If so, what can be done for a website to handle that? since I assume it does not relate to django code, since in this url there are no db queries, or anything special. just return a string.

And another question, in the frontend logs there are a lot of log files, does they somehow get cleaned/created automatically? Some explanation would be helpful.

asked 24 Nov '16, 18:46

clip
138
accept rate: 0%


The error message that you've quoted is from the front-end Nginx log. For possible errors from your Django app, you should refer to your app's error log in ~/logs/user/error_appname.log.

Regarding old logs, we rotate logs every day and keep a seven day history.

permanent link

answered 24 Nov '16, 23:18

seanf
12.2k41836
accept rate: 37%

I dont see a special error in this log. I don't see an error in my django app (in my own code). The server still hangs during the high traffic. Is it possible the server just can't handle this? Does upgrading to cloud/other plan can help?

(25 Nov '16, 01:00) clip

It could be that you simply want to increase the number workers and threads available for processing requests. mod_wsgi is designed to queue requests and process them only as threads are available, and that's a feature (so as to help control your memory consumption). You can adjust the resources available in httpd.conf and some documentation is available here.

That document describes how to reduce the number of available processes to help reduce memory consumption, but you can likewise increase the number of processes to allow more requests to be handled at once.

(25 Nov '16, 05:35) ryans ♦♦

Should I increase the number of process, threads, or both? What is a reasonable number to set for a site that's say, hit 100K visits in a few minutes? no expensive queries or anything else, just simple serving of pages that maybe based on a very simple query. If the pages are static, is there a big benefit to use just the static app instead? (this is not entirely the case though, just asking)

(25 Nov '16, 07:47) clip

The official mod_wsgi documentation suggests processes over threads unless the application is primarily I/O bound. I can't speculate as to what actual number of threads makes the most sense -- it depends on a lot of factors, including how much RAM each thread might consume on average and how long various tasks take on this particular hardware. The best way to determine this through benchmarking and measuring the effects of various configurations.

A static media application (such as "Static Only (no .htaccess)" is indeed much more efficient. Here is a simple comparison between Apache and Nginx that we did years ago. If any of those requests were actually processed through mod_wsgi rather than Apache, the benefits of a static-only application replacing them increase further.

(25 Nov '16, 10:09) ryans ♦♦

So I'll guess I'll just try to increase the process number. Since I don't know how much power (in terms of traffic) a process worth, it will be a total guess. Not even a guess for a django high traffic simple url which serves a plain text? How can I measure the performance difference when I change the numbers?

(25 Nov '16, 10:41) clip

Not even a guess for a django high traffic simple url which serves a plain text?

No, sorry.

How can I measure the performance difference when I change the numbers?

You can use any tool that generates traffic to a URL on your site. A couple of options are:

Whichever method you use, you can then look through the results and through your own logs to see how the site performed during the test.

(25 Nov '16, 17:56) seanf

How can I install the ab tool? I've tried apt-get install apache2-utils.
Not working. I don't have sudo..

(26 Nov '16, 15:39) clip

The 'ab' command is already installed on our servers, so you just need to execute it from the command line via SSH.

(26 Nov '16, 22:43) NickR ♦♦
showing 5 of 8 show 3 more comments
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×909
×337
×3
×1

question asked: 24 Nov '16, 18:46

question was seen: 1,860 times

last updated: 26 Nov '16, 22:43

WEBFACTION
REACH US
SUPPORT
AFFILIATE PROGRAM
LEGAL
© COPYRIGHT 2003-2019 SWARMA LIMITED - WEBFACTION IS A SERVICE OF SWARMA LIMITED
REGISTERED IN ENGLAND AND WALES 5729350 - VAT REGISTRATION NUMBER 877397162
5TH FLOOR, THE OLD VINYL FACTORY, HAYES, UB3 1HA, UNITED KINGDOM