[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#480210: apache2-mpm-prefork: Stale child threads not serving requests or timing out

Package: apache2-mpm-prefork
Version: 2.2.8-3
Severity: normal

I've been coming across issues lately with Apache child threads hanging in a wait
state (and not serving connections) indefinitely with no recovery.  Now, it may be
php related or something, I'm not ruling that out.  But at present I'm unable to 
debug it to the degree of finding the source (after much searching and doing things
like stracing bad threads).

What happens is the server is running fine in general, but certain clients seem to
be able to cause a thread to lock up while serving a request.  The majority of the
offenders seem to be while serving jpg thumbnail images out of a gallery, although
I have found other examples as well where threads do not recover.  The bad cases
are the same few files requested by one client, typically very rapidly in succession
and usually from country TLDs like .pl or .ar (which makes me believe it's an 
intentional DoS, although I can't find a description of the attack anywhere).

The following is an strace when the server was recieving these requests on a thread
being accessed by the IP causing the lock ups:


The requests in that thread referencing URLs with "thumbnail" in them are the ones
in question.  I was unable to derive anything useful from that particular trace, 
but it's there if you should need it.

Outside of the apparently deliberate attempts to lock up (or maybe just crawl) the server,
each of these threads shares the common attribute of being flagged as (W)ait status
in apache itself, lasting until the thread is killed (20 hours + on some).

Here's an example of a grep of long wait status threads and what they were doing:

root@blond:/etc# /usr/sbin/apache2ctl fullstatus | /bin/grep -A 1 -E '[0-9]+/[0-9]+/[0-9]+[[:space:]]+W[[:space:]]+[0-9]+\.[0-9]+[[:space:]]+[0-9]{2,}' 

    77-0 6850 0/130/30864 W 1.19 24970 0 0.0 0.84 205.13 
    blond.website.com GET / HTTP/1.1 
    111-0 12167 0/13/601 W 0.00 73426 0 0.0 0.04 3.51 
    blond.website.com GET 
    136-0 6870 0/146/26295 W 1.14 24810 0 0.0 1.06 175.10 
    blond.website.com GET / HTTP/1.1 
    141-0 9486 0/0/24479 W 0.20 24970 0 0.0 0.00 156.45 
    blond.website.com GET /articles/category HTTP/1.1 
    164-0 9496 0/26/21884 W 0.22 24804 0 0.0 0.07 141.98 
    blond.website.com GET / HTTP/1.1 
    332-0 9607 0/19/5332 W 0.33 24809 0 0.0 0.19 30.86 
    blond.website.com GET /articles/category HTTP/1.1 
    376-0 13058 0/186/655 W 0.75 72737 0 0.0 1.22 3.95 
    blond.website.com GET / HTTP/1.1 
    405-0 7313 0/169/831 W 0.85 25003 0 0.0 0.69 4.55 
    blond.website.com GET /articles/category HTTP/1.1 
    424-0 8346 0/78/1624 W 0.28 24939 0 0.0 0.30 10.43 
    blond.website.com GET / HTTP/1.1 
    480-0 7355 1/134/175 W 0.75 24933 0 41.7 1.24 1.49 
    blond.website.com GET /articles/category HTTP/1.1 

Any suggestions as to rectifying this would be GREATLY appreciated, as I am at
my wits' end trying to track down what exactly is going on.  Currently I am writing
a script to take teh results frmo the above grep and kill off the processes.

If you need any additional information or need me to test anything to help resolve
this or reproduce, please do not hesitate to let me know.

Here some relevant parts of the apache config (this happens with KeepAlives off as well):

HostnameLookups Off
Timeout 5
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 4
<IfModule mpm_prefork_module>
        ServerLimit             700
        StartServers            100
        MinSpareServers         5
        MaxSpareServers         10
        MaxClients              700
        MaxRequestsPerChild     10000
        GracefulShutdownTimeout 15

-- Package-specific info:
List of enabled modules from 'apache2 -M':
  actions* alias auth_basic auth_mysql authn_alias authn_file
  authnz_ldap authz_default authz_groupfile authz_host authz_user
  autoindex cgi dav_fs dav_lock dav deflate dir env include ldap mime
  negotiation php5 rewrite setenvif status suexec unique_id
  (A * means that the .conf file for that module is not enabled in

-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: i386 (i686)

Kernel: Linux 2.6.22-1-686 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages apache2-mpm-prefork depends on:
ii  apache2.2-common           2.2.8-3       Next generation, scalable, extenda
ii  libapr1                    1.2.12-1      The Apache Portable Runtime Librar
ii  libaprutil1                1.2.12+dfsg-3 The Apache Portable Runtime Utilit
ii  libc6                      2.7-10        GNU C Library: Shared libraries
ii  libpcre3                   7.6-2         Perl 5 Compatible Regular Expressi

apache2-mpm-prefork recommends no packages.

-- no debconf information

Reply to: