logrotate can't restart apache2
Hi everyboy.
recently I meet this problem in my web server.
Software version:
apache2: 2.2.3-3 0
md-jk: 1:1.2.18-2 0
every morning,at 6:25,logrotate restart apache2 to rotate apache's logs.I find apache2 restart failed.
Here is output of error.log:
[Thu Nov 23 14:07:46 2006] [notice] Apache/2.2.3 (Debian) mod_jk/1.2.18 configured -- resuming normal operations
[Thu Nov 23 15:29:58 2006] [notice] child pid 27593 exit signal Segmentation fault (11)
[Thu Nov 23 18:18:13 2006] [notice] child pid 1852 exit signal Segmentation fault (11)
*** glibc detected *** double free or corruption (!prev): 0x0000000000ba6650 ***
[Thu Nov 23 18:30:50 2006] [notice] child pid 1850 exit signal Aborted (6)
*** glibc detected *** double free or corruption (!prev): 0x0000000000c2cde0 ***
[Thu Nov 23 18:37:54 2006] [notice] child pid 1709 exit signal Aborted (6)
*** glibc detected *** corrupted double-linked list: 0x00002aaaaac93410 ***
[Thu Nov 23 18:46:22 2006] [notice] child pid 3549 exit signal Aborted (6)
[Thu Nov 23 20:21:34 2006] [notice] child pid 6114 exit signal Segmentation fault (11)
*** glibc detected *** double free or corruption (!prev): 0x0000000000c2ed30 ***
[Thu Nov 23 20:27:49 2006] [notice] child pid 7900 exit signal Aborted (6)
*** glibc detected *** double free or corruption (!prev): 0x0000000000c2a8a0 ***
[Thu Nov 23 21:48:06 2006] [notice] child pid 10554 exit signal Aborted (6)
*** glibc detected *** double free or corruption (!prev): 0x0000000000c2e990 ***
[Thu Nov 23 23:26:27 2006] [notice] child pid 17614 exit signal Aborted (6)
*** glibc detected *** double free or corruption (out): 0x0000000000c86b80 ***
[Thu Nov 23 23:40:42 2006] [notice] child pid 19552 exit signal Aborted (6)
[Fri Nov 24 00:32:49 2006] [notice] child pid 19790 exit signal Segmentation fault (11)
*** glibc detected *** double free or corruption (!prev): 0x0000000000b8a570 ***
[Fri Nov 24 01:05:37 2006] [notice] child pid 23364 exit signal Aborted (6)
[Fri Nov 24 01:31:16 2006] [notice] child pid 23756 exit signal Segmentation fault (11)
[Fri Nov 24 01:50:53 2006] [notice] child pid 23506 exit signal Segmentation fault (11)
*** glibc detected *** corrupted double-linked list: 0x0000000000bde700 ***
[Fri Nov 24 01:51:40 2006] [notice] child pid 23969 exit signal Aborted (6)
*** glibc detected *** corrupted double-linked list: 0x0000000000be6cb0 ***
[Fri Nov 24 01:56:26 2006] [notice] child pid 23683 exit signal Aborted (6)
*** glibc detected *** corrupted double-linked list: 0x00002aaaaac56f30 ***
[Fri Nov 24 02:17:32 2006] [notice] child pid 24188 exit signal Aborted (6)
*** glibc detected *** corrupted double-linked list: 0x0000000000ba0a20 ***
[Fri Nov 24 03:52:37 2006] [notice] child pid 26292 exit signal Aborted (6)
[Fri Nov 24 05:50:09 2006] [notice] child pid 32388 exit signal Segmentation fault (11)
[Fri Nov 24 06:25:05 2006] [warn] child process 1714 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 1867 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 32214 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 1949 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 1789 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 21663 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 32315 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 17689 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 1633 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 30460 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 32461 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 28432 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 32538 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 30208 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 30283 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 30539 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 30393 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 30612 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 3786 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 30131 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:05 2006] [warn] child process 19307 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 1714 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 1867 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 32214 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 1949 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 1789 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 21663 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 32315 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 17689 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 1633 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 30460 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 32461 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 28432 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 32538 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 30208 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 30283 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 30539 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 30393 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 30612 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 3786 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 30131 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:07 2006] [warn] child process 19307 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 1714 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 1867 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 32214 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 1949 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 1789 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 21663 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 32315 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 17689 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 1633 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 30460 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 32461 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 28432 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 32538 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 30208 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 30283 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 30539 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 30393 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 30612 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 3786 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 30131 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:09 2006] [warn] child process 19307 still did not exit, sending a SIGTERM
[Fri Nov 24 06:25:11 2006] [error] child process 1714 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 1867 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 32214 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 1949 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 1789 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 21663 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 32315 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 17689 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 1633 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 30460 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 32461 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 28432 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 32538 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 30208 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 30283 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 30539 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 30393 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 30612 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 3786 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 30131 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:11 2006] [error] child process 19307 still did not exit, sending a SIGKILL
[Fri Nov 24 06:25:12 2006] [notice] caught SIGTERM, shutting down
-----
It seems something wrong in glibc.
-------
#uname -a
Linux xxx.yyy.163.org 2.6.16-2-em64t-p4-smp #1 SMP Fri Aug 18 21:39:53 CEST 2006 x86_64 GNU/Linux
-----------
And some errors in mod_jk.log:
[Sun Nov 26 06:24:52 2006] [1206:0000] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (192.168.164.27:8009), err=-104
[Sun Nov 26 06:24:52 2006] [1206:0000] [error] ajp_get_reply::jk_ajp_common.c (1531): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 26 06:24:54 2006] [1188:0000] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (192.168.164.27:8009), err=-104
[Sun Nov 26 06:24:54 2006] [1188:0000] [error] ajp_get_reply::jk_ajp_common.c (1531): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 26 06:24:57 2006] [1984:0000] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (192.168.164.27:8009), err=-104
[Sun Nov 26 06:24:57 2006] [1984:0000] [error] ajp_get_reply::jk_ajp_common.c (1531): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 26 06:24:59 2006] [1186:0000] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (192.168.164.27:8009), err=-104
[Sun Nov 26 06:24:59 2006] [1186:0000] [error] ajp_get_reply::jk_ajp_common.c (1531): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 26 06:25:01 2006] [3581:0000] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (192.168.164.27:8009), err=-104
[Sun Nov 26 06:25:01 2006] [3581:0000] [error] ajp_get_reply::jk_ajp_common.c (1531): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 26 06:25:03 2006] [3581:0000] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (192.168.164.27:8009), err=-104
[Sun Nov 26 06:25:03 2006] [3581:0000] [error] ajp_get_reply::jk_ajp_common.c (1531): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 26 06:25:05 2006] [1984:0000] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (192.168.164.27:8009), err=-104
[Sun Nov 26 06:25:05 2006] [1984:0000] [error] ajp_get_reply::jk_ajp_common.c (1531): Tomcat is down or refused connection. No response has been sent to the client (yet)
------------
But why? My tomcat5 run well in that time and no network problem.
My server is in strict testing.and keep 160+ connection in every min.
--
Best Regards!
Reply to: