Bug#493573: Backtrace on 'hung' apache2 (while trying to stop service)
Package: apache2.2-common
Version: 2.2.9-6
Followup-For: Bug #493573
Stopping apache2 via /etc/init.d/apache2 stop does not work. All of the
worker threads are stopped, but the main '/usr/sbin/apache2 -k start'
process (the one in the PID file) lingers on. After the 60
second timeout the PID file is removed even though the main process has
not stopped; I have to use 'kill -9'.
A sample lingering process ('ps aux'):
root 21290 0.0 0.3 25464 7544 ? Ss 21:30 0:00
/usr/sbin/apache2 -k start
To re-create normal situation:
1. /etc/init.d/apache2 start
2. /etc/init.d/apache2 stop
3. apache2 is now hung
To re-create debug situation:
1. In shell #1, start apache2 with 'apache2 -X'
2. In shell #2, execute 'apache2ctl stop' or 'source
/etc/apache2/envvars; apache2 -k stop'
3. apache2 is now hung
4. run gdb and perform backtrace
-----
(gdb) bt full
#0 0xb7e630ee in __lll_lock_wait_private () from /lib/libpthread.so.0
No symbol table info available.
#1 0xb7e606e1 in _L_lock_23 () from /lib/libpthread.so.0
No symbol table info available.
#2 0xb7e604a8 in pthread_cond_destroy@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
No symbol table info available.
#3 0xb79d9383 in my_thread_global_end () from
/usr/lib/libmysqlclient_r.so.15
No symbol table info available.
#4 0xb79d477a in my_end () from /usr/lib/libmysqlclient_r.so.15
No symbol table info available.
#5 0xb79d3281 in mysql_server_end () from
/usr/lib/libmysqlclient_r.so.15
No symbol table info available.
#6 0xb6c69dd7 in zm_shutdown_mysql (type=1, module_number=42) at
/build/buildd/php5-5.2.6/ext/mysql/php_mysql.c:426
module_number = -512
#7 0xb721a74a in module_destructor (module=0x826ac28) at
/build/buildd/php5-5.2.6/Zend/zend_API.c:1921
module = (zend_module_entry *) 0x826ac28
#8 0xb7220d7a in zend_hash_apply_deleter (ht=0x0, p=0x826abf8) at
/build/buildd/php5-5.2.6/Zend/zend_hash.c:805
retval = (Bucket *) 0xb7220cef
#9 0xb7221047 in zend_hash_graceful_reverse_destroy (ht=0xb7496dc0) at
/build/buildd/php5-5.2.6/Zend/zend_hash.c:840
p = (Bucket *) 0xb7b6d4e0
ht = (HashTable *) 0x826abf8
#10 0xb72167dc in zend_shutdown () at
/build/buildd/php5-5.2.6/Zend/zend.c:814
No locals.
#11 0xb71cbebd in php_module_shutdown () at
/build/buildd/php5-5.2.6/main/main.c:1906
No locals.
#12 0xb71cbf87 in php_module_shutdown_wrapper (sapi_globals=0xb7491240)
at /build/buildd/php5-5.2.6/main/main.c:1877
No locals.
#13 0xb728caa1 in php_apache_child_shutdown (tmp=0x0) at
/build/buildd/php5-5.2.6/sapi/apache2handler/sapi_apache2.c:362
No locals.
#14 0xb7e86691 in run_cleanups () from /usr/lib/libapr-1.so.0
No locals.
#15 0xb7e85b5d in apr_pool_destroy () from /usr/lib/libapr-1.so.0
No locals.
#16 0x0808f884 in clean_child_exit (code=0) at
/tmp/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:198
No locals.
#17 0x0808fd65 in child_main (child_num_arg=<value optimized out>) at
/tmp/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:214
current_conn = (conn_rec *) 0x4
csd = (void *) 0x0
ptrans = (apr_pool_t *) 0x8363dd0
allocator = (apr_allocator_t *) 0x8361d40
status = 4
i = <value optimized out>
lr = (ap_listen_rec *) 0x4
pollset = (apr_pollset_t *) 0x8361e58
sbh = (ap_sb_handle_t *) 0x8361e50
bucket_alloc = (apr_bucket_alloc_t *) 0x8367fc8
last_poll_idx = -1219925568
#18 0x0808ff47 in make_child (s=0x80ab908, slot=0) at
/tmp/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:713
No locals.
#19 0x08090652 in ap_mpm_run (_pconf=0x80a70c8, plog=0x80d9190,
s=0x80ab908)
at /tmp/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:989
index = <value optimized out>
remaining_children_to_start = <value optimized out>
rv = <value optimized out>
#20 0x08066f10 in main (argc=Cannot access memory at address 0x0
) at /tmp/buildd/apache2-2.2.9/server/main.c:732
c = 88 'X'
configtestonly = 0
confname = 0x8092886 "/etc/apache2/apache2.conf"
def_server_root = 0x809a9b0 ""
temp_error_log = 0x0
error = <value optimized out>
process = (process_rec *) 0x80a5140
server_conf = (server_rec *) 0x80ab908
pglobal = (apr_pool_t *) 0x80a50c0
pconf = (apr_pool_t *) 0x80a70c8
plog = (apr_pool_t *) 0x80d9190
ptemp = (apr_pool_t *) 0x80e01a8
pcommands = (apr_pool_t *) 0x80a90d0
opt = (apr_getopt_t *) 0x80a9168
rv = <value optimized out>
mod = <value optimized out>
optarg = 0x0
-------------------------
from /var/log/error.log
<some more SIGTERM lines snipped out>
[Thu Aug 14 21:31:16 2008] [warn] child process 21297 still did not
exit, sending a SIGTERM
[Thu Aug 14 21:31:16 2008] [warn] child process 21298 still did not
exit, sending a SIGTERM
[Thu Aug 14 21:31:16 2008] [warn] child process 21299 still did not
exit, sending a SIGTERM
[Thu Aug 14 21:31:18 2008] [error] child process 21295 still did not
exit, sending a SIGKILL
[Thu Aug 14 21:31:18 2008] [error] child process 21296 still did not
exit, sending a SIGKILL
[Thu Aug 14 21:31:18 2008] [error] child process 21297 still did not
exit, sending a SIGKILL
[Thu Aug 14 21:31:18 2008] [error] child process 21298 still did not
exit, sending a SIGKILL
[Thu Aug 14 21:31:18 2008] [error] child process 21299 still did not
exit, sending a SIGKILL
[Thu Aug 14 21:31:19 2008] [info] removed PID file /var/run/apache2.pid
(pid=21290)
[Thu Aug 14 21:31:19 2008] [notice] caught SIGTERM, shutting down
-- Package-specific info:
List of enabled modules from 'apache2 -M':
alias auth_basic authn_file authz_default authz_groupfile
authz_host authz_user autoindex cgi dir env layout mime negotiation
php5 rewrite setenvif status
-- System Information:
Debian Release: 4.0
APT prefers stable
APT policy: (990, 'stable')
Architecture: i386 (i686)
Kernel: Linux 2.6.26 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages apache2 depends on:
ii apache2-mpm-prefork 2.2.9-6 Apache HTTP Server - traditional n
apache2 recommends no packages.
apache2 suggests no packages.
Versions of packages apache2.2-common depends on:
ii apache2-utils 2.2.9-6 utility programs for webservers
ii libapr1 1.2.12-4 The Apache Portable Runtime Librar
ii libaprutil1 1.2.12+dfsg-7 The Apache Portable Runtime Utilit
ii libc6 2.7-3 GNU C Library: Shared libraries
ii libmagic1 4.17-5etch3 File type determination library us
ii libssl0.9.8 0.9.8g-3 SSL shared libraries
ii lsb-base 3.1-23.2etch1 Linux Standard Base 3.1 init scrip
ii mime-support 3.39-1 MIME files 'mime.types' & 'mailcap
ii net-tools 1.60-17 The NET-3 networking toolkit
ii perl 5.10.0-11 Larry Wall's Practical Extraction
ii procps 1:3.2.7-3 /proc file system utilities
ii zlib1g 1:1.2.3.3.dfsg-5 compression library - runtime
-- no debconf information
Reply to: