Bug#1067104: server stalls: AH00046: child process 2876749 still did not exit, sending a SIGKILL
Package: apache2
Version: 2.4.57-2
Severity: important
Server was working just fine for years and recently started to stall
completely after 3-7 days of functioning normally. error logs get filled up
first with AH03490 and then eventually with AH00045 messages:
[Sun Mar 17 02:26:01.353381 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
...
[Sun Mar 17 22:00:42.201774 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Sun Mar 17 22:00:42.995574 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Sun Mar 17 22:00:42.998488 2024] [mpm_event:notice] [pid 2649373:tid 139846579189632] AH00492: caught SIGWINCH, shutting down gracefully
[Sun Mar 17 22:00:46.358981 2024] [core:warn] [pid 2649373:tid 139846579189632] AH00045: child process 2649375 still did not exit, sending a SIGTERM
[Sun Mar 17 22:00:46.359064 2024] [core:warn] [pid 2649373:tid 139846579189632] AH00045: child process 2649376 still did not exit, sending a SIGTERM
until I restart the beast.
$> grep AH03490 error.log | wc -l
70404
$> grep AH00045 error.log | wc -l
48
Server has a number of virtualserver's configured.
Seems has started about a month ago
$> for e in error.log*; do zgrep AH03490 $e| head -n 1 ; done
[Sun Mar 17 02:26:01.353381 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Mon Mar 11 16:47:41.181900 2024] [mpm_event:error] [pid 1172065:tid 140192799893376] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Tue Mar 05 00:00:12.307813 2024] [mpm_event:error] [pid 2686718:tid 139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Sun Feb 25 03:23:33.382200 2024] [mpm_event:error] [pid 2686718:tid 139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Sat Feb 24 01:02:29.148887 2024] [mpm_event:error] [pid 2686718:tid 139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Tue Feb 13 14:28:00.653754 2024] [mpm_event:error] [pid 2434335:tid 140300052350848] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
and likely after I configured some wsgi
$> zgrep apache /var/log/dpkg.log.* | grep 2024
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 install libapache2-mod-python:amd64 <none> 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status half-installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status unpacked libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 configure libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 <none>
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status unpacked libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status half-configured libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:34:25 status installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:51:18 status installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:51:19 remove libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 <none>
/var/log/dpkg.log.2.gz:2024-02-02 12:51:19 status half-configured libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:51:21 status half-installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:51:21 status config-files libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 install libapache2-mod-wsgi-py3:amd64 <none> 4.9.4-1+b2
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status half-installed libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status unpacked libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 configure libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2 <none>
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status unpacked libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status half-configured libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
/var/log/dpkg.log.2.gz:2024-02-02 12:52:14 status installed libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
wsgi is still in use but not mod-python (was enabled and then disabled and
uninstalled) so might be relating to that. It is in use so can't just
disable wsgi ATM.
-- Package-specific info:
-- System Information:
Debian Release: 12.5
APT prefers stable-security
APT policy: (100, 'stable-security'), (100, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 6.1.0-10-amd64 (SMP w/8 CPU threads; PREEMPT)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
Locale: LANG=en_US, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages apache2 depends on:
ii apache2-bin 2.4.57-2
ii apache2-data 2.4.57-2
ii apache2-utils 2.4.57-2
ii init-system-helpers 1.65.2
ii lsb-base 11.6
ii media-types 10.0.0
ii perl 5.36.0-7+deb12u1
ii procps 2:4.0.2-3
ii sysvinit-utils [lsb-base] 3.06-4
Versions of packages apache2 recommends:
ii ssl-cert 1.1.2
Versions of packages apache2 suggests:
pn apache2-doc <none>
pn apache2-suexec-pristine | apache2-suexec-custom <none>
ii chromium [www-browser] 121.0.6167.160-1~deb12u1
ii links [www-browser] 2.28-1+b2
ii lynx [www-browser] 2.9.0dev.12-1
ii w3m [www-browser] 0.5.3+git20230121-2
Versions of packages apache2-bin depends on:
ii libapr1 1.7.2-3
ii libaprutil1 1.6.3-1
ii libaprutil1-dbd-sqlite3 1.6.3-1
ii libaprutil1-ldap 1.6.3-1
ii libbrotli1 1.0.9-2+b6
ii libc6 2.36-9+deb12u4
ii libcrypt1 1:4.4.33-2
ii libcurl4 7.88.1-10+deb12u5
ii libjansson4 2.14-2
ii libldap-2.5-0 2.5.13+dfsg-5
ii liblua5.3-0 5.3.6-2
ii libnghttp2-14 1.52.0-1+deb12u1
ii libpcre2-8-0 10.42-1
ii libssl3 3.0.11-1~deb12u2
ii libxml2 2.9.14+dfsg-1.3~deb12u1
ii perl 5.36.0-7+deb12u1
ii zlib1g 1:1.2.13.dfsg-1
Versions of packages apache2-bin suggests:
pn apache2-doc <none>
pn apache2-suexec-pristine | apache2-suexec-custom <none>
ii chromium [www-browser] 121.0.6167.160-1~deb12u1
ii links [www-browser] 2.28-1+b2
ii lynx [www-browser] 2.9.0dev.12-1
ii w3m [www-browser] 0.5.3+git20230121-2
Versions of packages apache2 is related to:
ii apache2 2.4.57-2
ii apache2-bin 2.4.57-2
-- Configuration Files:
/etc/apache2/mods-available/cgid.conf changed:
ScriptSock ${APACHE_RUN_DIR}/socks/cgisock
ScriptLog /srv/popcon-neuro.debian.net/logs/cgi/popcon-submit.log
/etc/apache2/mods-available/mpm_event.conf changed:
StartServers 4
MinSpareThreads 25
MaxSpareThreads 225
ThreadLimit 128
ThreadsPerChild 25
MaxRequestWorkers 150
MaxConnectionsPerChild 0
/etc/apache2/sites-available/000-default.conf changed:
<VirtualHost *:80>
# The ServerName directive sets the request scheme, hostname and port that
# the server uses to identify itself. This is used when creating
# redirection URLs. In the context of virtual hosts, the ServerName
# specifies what hostname must appear in the request's Host: header to
# match this virtual host. For the default virtual host (this file) this
# value is not decisive as it is used as a last resort host regardless.
# However, you must set it for any further virtual host explicitly.
#ServerName www.example.com
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html
# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
</VirtualHost>
/etc/apache2/sites-available/default-ssl.conf changed:
<IfModule mod_ssl.c>
<VirtualHost _default_:443>
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html
# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
# SSL Engine Switch:
# Enable/Disable SSL for this virtual host.
SSLEngine on
# A self-signed (snakeoil) certificate can be created by installing
# the ssl-cert package. See
# /usr/share/doc/apache2/README.Debian.gz for more info.
# If both key and certificate are stored in the same file, only the
# SSLCertificateFile directive is needed.
SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
# Server Certificate Chain:
# Point SSLCertificateChainFile at a file containing the
# concatenation of PEM encoded CA certificates which form the
# certificate chain for the server certificate. Alternatively
# the referenced file can be the same as SSLCertificateFile
# when the CA certificates are directly appended to the server
# certificate for convinience.
#SSLCertificateChainFile /etc/apache2/ssl.crt/server-ca.crt
# Certificate Authority (CA):
# Set the CA certificate verification path where to find CA
# certificates for client authentication or alternatively one
# huge file containing all of them (file must be PEM encoded)
# Note: Inside SSLCACertificatePath you need hash symlinks
# to point to the certificate files. Use the provided
# Makefile to update the hash symlinks after changes.
#SSLCACertificatePath /etc/ssl/certs/
#SSLCACertificateFile /etc/apache2/ssl.crt/ca-bundle.crt
# Certificate Revocation Lists (CRL):
# Set the CA revocation path where to find CA CRLs for client
# authentication or alternatively one huge file containing all
# of them (file must be PEM encoded)
# Note: Inside SSLCARevocationPath you need hash symlinks
# to point to the certificate files. Use the provided
# Makefile to update the hash symlinks after changes.
#SSLCARevocationPath /etc/apache2/ssl.crl/
#SSLCARevocationFile /etc/apache2/ssl.crl/ca-bundle.crl
# Client Authentication (Type):
# Client certificate verification type and depth. Types are
# none, optional, require and optional_no_ca. Depth is a
# number which specifies how deeply to verify the certificate
# issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth 10
# SSL Engine Options:
# Set various options for the SSL engine.
# o FakeBasicAuth:
# Translate the client X.509 into a Basic Authorisation. This means that
# the standard Auth/DBMAuth methods can be used for access control. The
# user name is the `one line' version of the client's X.509 certificate.
# Note that no password is obtained from the user. Every entry in the user
# file needs this password: `xxj31ZMTZzkVA'.
# o ExportCertData:
# This exports two additional environment variables: SSL_CLIENT_CERT and
# SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
# server (always existing) and the client (only existing when client
# authentication is used). This can be used to import the certificates
# into CGI scripts.
# o StdEnvVars:
# This exports the standard SSL/TLS related `SSL_*' environment variables.
# Per default this exportation is switched off for performance reasons,
# because the extraction step is an expensive operation and is usually
# useless for serving static content. So one usually enables the
# exportation for CGI and SSI requests only.
# o OptRenegotiate:
# This enables optimized SSL connection renegotiation handling when SSL
# directives are used in per-directory context.
#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory /usr/lib/cgi-bin>
SSLOptions +StdEnvVars
</Directory>
# SSL Protocol Adjustments:
# The safe and default but still SSL/TLS standard compliant shutdown
# approach is that mod_ssl sends the close notify alert but doesn't wait for
# the close notify alert from client. When you need a different shutdown
# approach you can use one of the following variables:
# o ssl-unclean-shutdown:
# This forces an unclean shutdown when the connection is closed, i.e. no
# SSL close notify alert is send or allowed to received. This violates
# the SSL/TLS standard but is needed for some brain-dead browsers. Use
# this when you receive I/O errors because of the standard approach where
# mod_ssl sends the close notify alert.
# o ssl-accurate-shutdown:
# This forces an accurate shutdown when the connection is closed, i.e. a
# SSL close notify alert is send and mod_ssl waits for the close notify
# alert of the client. This is 100% SSL/TLS standard compliant, but in
# practice often causes hanging connections with brain-dead browsers. Use
# this only for browsers where you know that their SSL implementation
# works correctly.
# Notice: Most problems of broken clients are also related to the HTTP
# keep-alive facility, so you usually additionally want to disable
# keep-alive for those clients, too. Use variable "nokeepalive" for this.
# Similarly, one has to force some clients to use HTTP/1.0 to workaround
# their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
# "force-response-1.0" for this.
BrowserMatch "MSIE [2-6]" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
# MSIE 7 and newer should be able to use keepalive
BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>
</IfModule>
/etc/logrotate.d/apache2 changed:
/var/log/apache2/*.log {
weekly
missingok
rotate 10000
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
if /etc/init.d/apache2 status > /dev/null ; then \
/etc/init.d/apache2 reload > /dev/null; \
fi;
endscript
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi; \
endscript
}
-- no debconf information
Reply to: