automatically restarting dying daemons?
problem: xinetd, after working just fine and dandy for weeks at
a time, gets dozens of "unexpected signal" (source unknown)
and gives up the ghost.
questions:
1) what's the best way (e.g. debian way) to monitor active
daemons and restart them when necessary? maybe some
utility already exists for this? or /proc/something?
or `ps ax`?
2) how can i track down the source of the signals specific
to this case and make it stop?
xinetd chugs along nicely for the most part, and then -- poof!
-- it dies a sudden death:
root@boss# cd /var/log
root@boss# grep xinetd daemon.log
Jun 30 13:39:13 boss xinetd[21953]: {general_handler} (21953) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:13 boss xinetd[21953]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:13 boss xinetd[4873]: Resetting...
Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:13 boss xinetd[4873]: Resetting...
Jun 30 13:39:25 boss xinetd[21964]: {general_handler} (21964) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:25 boss xinetd[21964]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:25 boss xinetd[4873]: Resetting...
Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:39:25 boss xinetd[4873]: Resetting...
Jun 30 13:41:27 boss xinetd[21990]: {general_handler} (21990) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:41:27 boss xinetd[21990]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
Jun 30 13:41:27 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:41:27 boss xinetd[4873]: Resetting...
Jun 30 13:41:27 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:41:27 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
Jun 30 13:41:27 boss xinetd[4873]: {bad_signal} Received 50 bad signals. Exiting...
Jun 30 15:51:56 boss xinetd[23062]: xinetd Version 2.3.4 started with libwrap loadavg options compiled in.
Jun 30 15:51:56 boss xinetd[23062]: Started working: 8 available services
so that shows when xinetd died. here's all the activity at
that time plus-or-minus a second or two (13:39:12 - :14 and
13:39:24 - :28):
root@boss# find . -type f \
| xargs grep 'Jun 30 13:39' \
| egrep ':1[234] |:2[45678] '
./syslog:Jun 30 13:39:13 boss xinetd[21953]: {general_handler} (21953) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:13 boss last message repeated 9 times
./syslog:Jun 30 13:39:13 boss xinetd[21953]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
./syslog:Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:13 boss last message repeated 9 times
./syslog:Jun 30 13:39:13 boss xinetd[4873]: Resetting...
./syslog:Jun 30 13:39:13 boss postgres[21954]: [1] DEBUG: pq_recvbuf: recv() failed: Connection reset by peer
./syslog:Jun 30 13:39:13 boss postgres[21954]: [2] DEBUG: incomplete startup packet
./syslog:Jun 30 13:39:13 boss -f[21958]: (v4.0.4) Unable to get canonical name of client 127.12.21.44: Unknown host (1) [pop_init.c:1075]
./syslog:Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:13 boss last message repeated 8 times
./syslog:Jun 30 13:39:13 boss xinetd[4873]: Resetting...
./syslog:Jun 30 13:39:13 boss -f[21958]: (null) at 127.12.21.44 (127.12.21.44): -ERR POP EOF or I/O Error [popper.c:820]
./syslog:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./syslog:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./syslog:Jun 30 13:39:25 boss postgres[21962]: [1] DEBUG: pq_recvbuf: recv() failed: Connection reset by peer
./syslog:Jun 30 13:39:25 boss postgres[21962]: [2] DEBUG: incomplete startup packet
./syslog:Jun 30 13:39:25 boss xinetd[21964]: {general_handler} (21964) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:25 boss last message repeated 9 times
./syslog:Jun 30 13:39:25 boss xinetd[21964]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
./syslog:Jun 30 13:39:25 boss spamd[371]: connection from localhost [127.0.0.1] at port 4706
./syslog:Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:25 boss last message repeated 9 times
./syslog:Jun 30 13:39:25 boss xinetd[4873]: Resetting...
./syslog:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./syslog:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./syslog:Jun 30 13:39:25 boss spamd[21967]: SIGPIPE received - reopening log socket
./syslog:Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:25 boss last message repeated 8 times
./syslog:Jun 30 13:39:25 boss xinetd[4873]: Resetting...
./syslog:Jun 30 13:39:26 boss -f[21968]: (null) at localhost (127.0.0.1): -ERR POP EOF or I/O Error [popper.c:820]
./syslog:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./syslog:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./auth.log:Jun 30 13:39:25 boss sshd[21963]: warning: can't get client address: Connection reset by peer
./auth.log:Jun 30 13:39:25 boss sshd[21963]: Could not write ident string to 127.0.0.1
./daemon.log:Jun 30 13:39:13 boss xinetd[21953]: {general_handler} (21953) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:13 boss last message repeated 9 times
./daemon.log:Jun 30 13:39:13 boss xinetd[21953]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
./daemon.log:Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:13 boss last message repeated 9 times
./daemon.log:Jun 30 13:39:13 boss xinetd[4873]: Resetting...
./daemon.log:Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:13 boss last message repeated 8 times
./daemon.log:Jun 30 13:39:13 boss xinetd[4873]: Resetting...
./daemon.log:Jun 30 13:39:25 boss xinetd[21964]: {general_handler} (21964) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:25 boss last message repeated 9 times
./daemon.log:Jun 30 13:39:25 boss xinetd[21964]: {bad_signal} Received 10 signals in 1 seconds. Exiting...
./daemon.log:Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:25 boss last message repeated 9 times
./daemon.log:Jun 30 13:39:25 boss xinetd[4873]: Resetting...
./daemon.log:Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11 (Segmentation fault)
./daemon.log:Jun 30 13:39:25 boss last message repeated 8 times
./daemon.log:Jun 30 13:39:25 boss xinetd[4873]: Resetting...
./mail.log:Jun 30 13:39:13 boss -f[21958]: (v4.0.4) Unable to get canonical name of client 127.12.21.44: Unknown host (1) [pop_init.c:1075]
./mail.log:Jun 30 13:39:13 boss -f[21958]: (null) at 127.12.21.44 (127.12.21.44): -ERR POP EOF or I/O Error [popper.c:820]
./mail.log:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./mail.log:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./mail.log:Jun 30 13:39:25 boss spamd[371]: connection from localhost [127.0.0.1] at port 4706
./mail.log:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./mail.log:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./mail.log:Jun 30 13:39:25 boss spamd[21967]: SIGPIPE received - reopening log socket
./mail.log:Jun 30 13:39:26 boss -f[21968]: (null) at localhost (127.0.0.1): -ERR POP EOF or I/O Error [popper.c:820]
./mail.log:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./mail.log:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./mail.info:Jun 30 13:39:13 boss -f[21958]: (null) at 127.12.21.44 (127.12.21.44): -ERR POP EOF or I/O Error [popper.c:820]
./mail.info:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./mail.info:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at 127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./mail.info:Jun 30 13:39:25 boss spamd[371]: connection from localhost [127.0.0.1] at port 4706
./mail.info:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./mail.info:Jun 30 13:39:25 boss spamd[21967]: bad protocol: header error: (closed before headers)
./mail.info:Jun 30 13:39:25 boss spamd[21967]: SIGPIPE received - reopening log socket
./mail.info:Jun 30 13:39:26 boss -f[21968]: (null) at localhost (127.0.0.1): -ERR POP EOF or I/O Error [popper.c:820]
./mail.info:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./mail.info:Jun 30 13:39:26 boss -f[21968]: I/O error flushing output to client at localhost [127.0.0.1]: Operation not permitted (1) [pop_send.c:689]
./mail.warn:Jun 30 13:39:25 boss spamd[21967]: SIGPIPE received - reopening log socket
./debug:Jun 30 13:39:13 boss postgres[21954]: [1] DEBUG: pq_recvbuf: recv() failed: Connection reset by peer
./debug:Jun 30 13:39:13 boss postgres[21954]: [2] DEBUG: incomplete startup packet
./debug:Jun 30 13:39:25 boss postgres[21962]: [1] DEBUG: pq_recvbuf: recv() failed: Connection reset by peer
./debug:Jun 30 13:39:25 boss postgres[21962]: [2] DEBUG: incomplete startup packet
as you can see below, i'm using kernelt 2.4.18-bf2.4...
are there any likely suspects in there? are there any likely
suspects to be found elsewhere?
--
I use Debian/GNU Linux version 3.0;
Linux boss 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i586 unknown
DEBIAN NEWBIE TIP #51 from Will Trillich <will@serensoft.com>
:
Interested in CUSTOMIZING MUTT to work the way you'd like?
Visit Tom Gilbert's site at http://linuxbrit.co.uk/mutt/ and
download his .muttrc to your home directory (save it under a
different name if you're paranoid like I am, then tell mutt
":source file/path/here" to give it a whirl). Wow!
Also see http://newbieDoc.sourceForge.net/ ...
Reply to: