[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#896090: AH02611: create: apr_shm_create failed (No space left on device)

Package: apache2
Version: 2.4.25-3+deb9u4
Severity: normal

Dear Maintainers,

We're running into a problem on a graceful restart in combination with a
front proxy and mod_balancer:

[Mon Apr 09 09:10:17.169139 2018] [slotmem_shm:error] [pid 9268:tid
140349331453824] (28)No space left on device: AH02611: create:

I found the No Space error bizarre, so I have verified if there is room in the limits:
> $ ipcs -lm
> ------ Semaphore Limits --------
> max number of arrays = 32000
> max semaphores per array = 32000
> max semaphores system wide = 1024000000
> max ops per semop call = 500
> semaphore max value = 32767

The amount of semaphores in use while running is around 2118.

I can reliably reproduce it with:
 - 45 vhosts that use mod_balancer, with in total 138 BalancerMembers.
 - a long running connection (keep alive) over one of the configured
 - both on jessie and stretch

I observed that:
 - Disabling "a few" vhosts makes the problem go away
 - Stopping the long lived connection makes the problem go away

While browsing the apache mailinglist for this error, I found a
similar bug report which was fixed in 2.4.30 [1].

[1]: https://bz.apache.org/bugzilla/show_bug.cgi?id=62044

Let me know if I provided enough details.


-- Package-specific info:

-- System Information:
Debian Release: 9.4
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apache2 depends on:
ii  apache2-bin          2.4.25-3+deb9u4
ii  apache2-data         2.4.25-3+deb9u4
ii  apache2-utils        2.4.25-3+deb9u4
ii  dpkg                 1.18.24
ii  init-system-helpers  1.48
ii  lsb-base             9.20161125
ii  mime-support         3.60
ii  perl                 5.24.1-3+deb9u3
ii  procps               2:3.3.12-3

Versions of packages apache2 recommends:
ii  ssl-cert  1.0.39

Versions of packages apache2 suggests:
pn  apache2-doc                                      <none>
pn  apache2-suexec-pristine | apache2-suexec-custom  <none>
pn  www-browser                                      <none>

Versions of packages apache2-bin depends on:
ii  libapr1                  1.5.2-5
ii  libaprutil1              1.5.4-3
ii  libaprutil1-dbd-sqlite3  1.5.4-3
ii  libaprutil1-ldap         1.5.4-3
ii  libc6                    2.24-11+deb9u3
ii  libldap-2.4-2            2.4.44+dfsg-5+deb9u1
ii  liblua5.2-0              5.2.4-1.1+b2
ii  libnghttp2-14            1.18.1-1
ii  libpcre3                 2:8.39-3
ii  libssl1.0.2              1.0.2l-2+deb9u3
ii  libxml2                  2.9.4+dfsg1-2.2+deb9u2
ii  perl                     5.24.1-3+deb9u3
ii  zlib1g                   1:1.2.8.dfsg-5

Versions of packages apache2-bin suggests:
pn  apache2-doc                                      <none>
pn  apache2-suexec-pristine | apache2-suexec-custom  <none>
pn  www-browser                                      <none>

Versions of packages apache2 is related to:
ii  apache2      2.4.25-3+deb9u4
ii  apache2-bin  2.4.25-3+deb9u4

-- Configuration Files:
/etc/apache2/apache2.conf changed:
ServerTokens Prod
ServerSignature Off
TraceEnable On
ServerName "xswebfront7.xs4all.net"
ServerRoot "/etc/apache2"
Timeout 120
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
LimitRequestFieldSize 8190
User www-data
Group www-data
AccessFileName .htaccess
<FilesMatch "^\.ht">
    Require all denied
<Directory />
  Options FollowSymLinks
  AllowOverride None
HostnameLookups Off
ErrorLog "/var/log/apache2/error.log"
LogLevel warn 
EnableSendfile On
Include "/etc/apache2/mods-enabled/*.load"
Include "/etc/apache2/mods-enabled/*.conf"
Include "/etc/apache2/ports.conf"
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" forwarded
IncludeOptional "/etc/apache2/conf.d/*.conf"
IncludeOptional "/etc/apache2/sites-enabled/*"

/etc/apache2/mods-available/access_compat.load changed:
LoadModule access_compat_module /usr/lib/apache2/modules/mod_access_compat.so

/etc/apache2/mods-available/alias.conf changed:
<IfModule alias_module>
Alias /icons/ "/usr/share/apache2/icons/"
<Directory "/usr/share/apache2/icons">
    Options Indexes MultiViews
    AllowOverride None
    Require all granted

/etc/apache2/mods-available/authz_host.load changed:
LoadModule authz_host_module /usr/lib/apache2/modules/mod_authz_host.so

/etc/apache2/mods-available/deflate.conf changed:
AddOutputFilterByType DEFLATE application/json
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/html text/plain text/xml
DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio

/etc/apache2/mods-available/deflate.load changed:
LoadModule deflate_module /usr/lib/apache2/modules/mod_deflate.so

/etc/apache2/mods-available/lbmethod_byrequests.load changed:
LoadModule lbmethod_byrequests_module /usr/lib/apache2/modules/mod_lbmethod_byrequests.so

/etc/apache2/mods-available/mime.conf changed:
TypesConfig /etc/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType application/x-bzip2 .bz2
AddLanguage ca .ca
AddLanguage cs .cz .cs
AddLanguage da .dk
AddLanguage de .de
AddLanguage el .el
AddLanguage en .en
AddLanguage eo .eo
AddLanguage es .es
AddLanguage et .et
AddLanguage fr .fr
AddLanguage he .he
AddLanguage hr .hr
AddLanguage it .it
AddLanguage ja .ja
AddLanguage ko .ko
AddLanguage ltz .ltz
AddLanguage nl .nl
AddLanguage nn .nn
AddLanguage no .no
AddLanguage pl .po
AddLanguage pt .pt
AddLanguage pt-BR .pt-br
AddLanguage ru .ru
AddLanguage sv .sv
AddLanguage zh-CN .zh-cn
AddLanguage zh-TW .zh-tw
AddHandler type-map var
AddOutputFilter INCLUDES .shtml
AddType text/html .shtml

/etc/apache2/mods-available/proxy.conf changed:
<IfModule mod_proxy.c>
  # Do not enable proxying with ProxyRequests until you have secured your
  # server.  Open proxy servers are dangerous both to your network and to the
  # Internet at large.
  ProxyRequests Off
  # Enable/disable the handling of HTTP/1.1 "Via:" headers.
  # ("Full" adds the server version; "Block" removes all outgoing Via: headers)
  # Set to one of: Off | On | Full | Block
  ProxyVia On

/etc/apache2/mods-available/proxy_balancer.load changed:
LoadModule proxy_balancer_module /usr/lib/apache2/modules/mod_proxy_balancer.so

/etc/apache2/mods-available/proxy_http.load changed:
LoadModule proxy_http_module /usr/lib/apache2/modules/mod_proxy_http.so

/etc/apache2/mods-available/setenvif.conf changed:
BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0
BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
BrowserMatch "MS FrontPage" redirect-carefully
BrowserMatch "^WebDrive" redirect-carefully
BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully
BrowserMatch "^gnome-vfs/1.0" redirect-carefully
BrowserMatch "^gvfs/1" redirect-carefully
BrowserMatch "^XML Spy" redirect-carefully
BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully
BrowserMatch " Konqueror/4" redirect-carefully
<IfModule mod_ssl.c>
  BrowserMatch "MSIE [2-6]" \
    nokeepalive ssl-unclean-shutdown \
    downgrade-1.0 force-response-1.0
  # MSIE 7 and newer should be able to use keepalive
  BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown

/etc/apache2/mods-available/ssl.conf changed:
<IfModule mod_ssl.c>
  SSLRandomSeed startup builtin
  SSLRandomSeed startup file:/dev/urandom 512
  SSLRandomSeed connect builtin
  SSLRandomSeed connect file:/dev/urandom 512
  AddType application/x-x509-ca-cert .crt
  AddType application/x-pkcs7-crl    .crl
  SSLPassPhraseDialog builtin
  SSLSessionCache "shmcb:${APACHE_RUN_DIR}/ssl_scache(512000)"
  SSLSessionCacheTimeout 300
  Mutex default
  SSLCryptoDevice builtin
  SSLHonorCipherOrder On
  SSLUseStapling Off
  SSLStaplingCache "shmcb:${APACHE_RUN_DIR}/ocsp(32768)"
  SSLProtocol all -SSLv2 -SSLv3
  SSLOptions StdEnvVars

/etc/apache2/mods-available/ssl.load changed:
LoadModule ssl_module /usr/lib/apache2/modules/mod_ssl.so

/etc/apache2/mods-available/status.conf changed:
<Location /server-status>
    SetHandler server-status
    Require ip ::1
ExtendedStatus On
<IfModule mod_proxy.c>
    # Show Proxy LoadBalancer status in mod_status
    ProxyStatus On

/etc/apache2/ports.conf changed:
Listen 443
Listen 80

/etc/apache2/sites-available/000-default.conf [Errno 2] No such file or directory: '/etc/apache2/sites-available/000-default.conf'
/etc/apache2/sites-available/default-ssl.conf [Errno 2] No such file or directory: '/etc/apache2/sites-available/default-ssl.conf'

-- no debconf information

Reply to: