--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: apache2-mpm-prefork: Stale child threads not serving requests or timing out
- From: Administrator <azhrarn@underhanded.org>
- Date: Thu, 08 May 2008 19:05:51 +0000
- Message-id: <20080508190551.22400.16877.reportbug@azbox.underhanded.org>
Package: apache2-mpm-prefork
Version: 2.2.8-3
Severity: normal
I've been coming across issues lately with Apache child threads hanging in a wait
state (and not serving connections) indefinitely with no recovery. Now, it may be
php related or something, I'm not ruling that out. But at present I'm unable to
debug it to the degree of finding the source (after much searching and doing things
like stracing bad threads).
What happens is the server is running fine in general, but certain clients seem to
be able to cause a thread to lock up while serving a request. The majority of the
offenders seem to be while serving jpg thumbnail images out of a gallery, although
I have found other examples as well where threads do not recover. The bad cases
are the same few files requested by one client, typically very rapidly in succession
and usually from country TLDs like .pl or .ar (which makes me believe it's an
intentional DoS, although I can't find a description of the attack anywhere).
The following is an strace when the server was recieving these requests on a thread
being accessed by the IP causing the lock ups:
http://underhanded.org/13098.out
The requests in that thread referencing URLs with "thumbnail" in them are the ones
in question. I was unable to derive anything useful from that particular trace,
but it's there if you should need it.
Outside of the apparently deliberate attempts to lock up (or maybe just crawl) the server,
each of these threads shares the common attribute of being flagged as (W)ait status
in apache itself, lasting until the thread is killed (20 hours + on some).
Here's an example of a grep of long wait status threads and what they were doing:
root@blond:/etc# /usr/sbin/apache2ctl fullstatus | /bin/grep -A 1 -E '[0-9]+/[0-9]+/[0-9]+[[:space:]]+W[[:space:]]+[0-9]+\.[0-9]+[[:space:]]+[0-9]{2,}'
77-0 6850 0/130/30864 W 1.19 24970 0 0.0 0.84 205.13 195.114.160.136
blond.website.com GET / HTTP/1.1
--
111-0 12167 0/13/601 W 0.00 73426 0 0.0 0.04 3.51 91.106.219.88
blond.website.com GET
--
136-0 6870 0/146/26295 W 1.14 24810 0 0.0 1.06 175.10 195.114.160.136
blond.website.com GET / HTTP/1.1
--
141-0 9486 0/0/24479 W 0.20 24970 0 0.0 0.00 156.45 195.114.160.136
blond.website.com GET /articles/category HTTP/1.1
--
164-0 9496 0/26/21884 W 0.22 24804 0 0.0 0.07 141.98 195.114.160.136
blond.website.com GET / HTTP/1.1
--
332-0 9607 0/19/5332 W 0.33 24809 0 0.0 0.19 30.86 195.114.160.136
blond.website.com GET /articles/category HTTP/1.1
--
376-0 13058 0/186/655 W 0.75 72737 0 0.0 1.22 3.95 128.180.192.139
blond.website.com GET / HTTP/1.1
--
405-0 7313 0/169/831 W 0.85 25003 0 0.0 0.69 4.55 195.114.160.136
blond.website.com GET /articles/category HTTP/1.1
--
424-0 8346 0/78/1624 W 0.28 24939 0 0.0 0.30 10.43 195.114.160.136
blond.website.com GET / HTTP/1.1
--
480-0 7355 1/134/175 W 0.75 24933 0 41.7 1.24 1.49 195.114.160.136
blond.website.com GET /articles/category HTTP/1.1
Any suggestions as to rectifying this would be GREATLY appreciated, as I am at
my wits' end trying to track down what exactly is going on. Currently I am writing
a script to take teh results frmo the above grep and kill off the processes.
If you need any additional information or need me to test anything to help resolve
this or reproduce, please do not hesitate to let me know.
Here some relevant parts of the apache config (this happens with KeepAlives off as well):
------
HostnameLookups Off
Timeout 5
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 4
<IfModule mpm_prefork_module>
ServerLimit 700
StartServers 100
MinSpareServers 5
MaxSpareServers 10
MaxClients 700
MaxRequestsPerChild 10000
GracefulShutdownTimeout 15
</IfModule>
-----
-- Package-specific info:
List of enabled modules from 'apache2 -M':
actions* alias auth_basic auth_mysql authn_alias authn_file
authnz_ldap authz_default authz_groupfile authz_host authz_user
autoindex cgi dav_fs dav_lock dav deflate dir env include ldap mime
negotiation php5 rewrite setenvif status suexec unique_id
vhost_alias
(A * means that the .conf file for that module is not enabled in
/etc/apache2/mods-enabled/)
-- System Information:
Debian Release: lenny/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: i386 (i686)
Kernel: Linux 2.6.22-1-686 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages apache2-mpm-prefork depends on:
ii apache2.2-common 2.2.8-3 Next generation, scalable, extenda
ii libapr1 1.2.12-1 The Apache Portable Runtime Librar
ii libaprutil1 1.2.12+dfsg-3 The Apache Portable Runtime Utilit
ii libc6 2.7-10 GNU C Library: Shared libraries
ii libpcre3 7.6-2 Perl 5 Compatible Regular Expressi
apache2-mpm-prefork recommends no packages.
-- no debconf information
--- End Message ---