Hi Björn,
Thanks for the leads, so far, though....
Björn Ballard wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Miles,
Although I've not come across anything like this before
if you suspect an infection try looking at the output
from something like:
ps -aef | less
nothing suspicious
for anything obviously out of place. If nothing seems
obviously wrong compare the number of processes
returned by:
ps -ax | wc -l
and:
ls -d /proc/* | grep [0-9] | wc -l
same number of processes
Could you post extracts from the log files for around
the time of the spontaneous reset?
ok, another look and I do find some suspicious stuff -- I've been having
a number of people try to crack the machine for a while, but (I thought)
to no avail
from auth.log on both machines:
a whole slew of these, and similar entries with different user names
(both 8/7 and 8/21 logs)
Aug 7 08:49:07 server2 sshd[11271]: Illegal user diamond from
::ffff:60.28.24.84
Aug 7 08:49:10 server2 sshd[11273]: Illegal user heaven from
::ffff:60.28.24.84
Aug 7 08:49:12 server2 sshd[11275]: Illegal user guadalupe from
::ffff:60.28.24.84
and these (only the earlier log):
Aug 7 06:48:02 server2 sshd[6567]: Address 66.132.182.28 maps to
goldcrownresor
t.com, but this does not map back to the address - POSSIBLE BREAKIN
ATTEMPT!
Aug 7 06:48:03 server2 sshd[6569]: Address 66.132.182.28 maps to
goldcrownresor
t.com, but this does not map back to the address - POSSIBLE BREAKIN
ATTEMPT!
Aug 7 06:48:03 server2 sshd[6571]: Address 66.132.182.28 maps to
goldcrownresor
t.com, but this does not map back to the address - POSSIBLE BREAKIN
ATTEMPT!
Aug 7 06:48:04 server2 sshd[6573]: Address 66.132.182.28 maps to
goldcrownresor
t.com, but this does not map back to the address - POSSIBLE BREAKIN
ATTEMPT!
BUT... the events stopped a couple of hours before the reboot
from auth.log on 1st server, today:
Aug 21 11:50:01 server1 CRON[27533]: (pam_unix) session closed for user
root
Aug 21 11:55:01 server1 CRON[27540]: (pam_unix) session opened for user
root by
(uid=0)
Aug 21 11:55:01 server1 CRON[27540]: (pam_unix) session closed for user
root
Aug 21 12:00:01 server1 CRON[27550]: (pam_unix) session opened for user
root by
(uid=0)
Aug 21 12:00:01 server1 CRON[27550]: (pam_unix) session closed for user
root
Aug 21 12:02:01 server1 CRON[27556]: (pam_unix) session opened for user
logcheck
by (uid=0)
Aug 21 12:02:04 server1 CRON[27556]: (pam_unix) session closed for user
logcheck
Aug 21 12:05:43 server1 sshd[3166]: Server listening on :: port 22.
Aug 21 12:05:44 server1 perl: (pam_unix) authentication failure;
logname= uid=0
euid=0 tty= ruser= rhost= user=root
Aug 21 12:05:46 server1 webmin[3187]: Webmin starting
Aug 21 12:05:46 server1 CRON[3296]: (pam_unix) session opened for user
logcheck
by (uid=0)
Aug 21 12:05:50 server1 CRON[3296]: (pam_unix) session closed for user
logcheck
Aug 21 12:09:01 server1 CRON[4163]: (pam_unix) session opened for user
root by (
uid=0)
the two lines that caught my eye are:
Aug 21 12:05:43 server1 sshd[3166]: Server listening on :: port 22.
Aug 21 12:05:44 server1 perl: (pam_unix) authentication failure;
logname= uid=0
euid=0 tty= ruser= rhost= user=root
I know that I wasn't trying to log on then. Perhaps somebody was trying
to break in with something that crashed the machine, immediately
followed by an attempt to log in as root. But I can't find anything like
this on the other server or for the previous crashes, and it looks like
it failed.
From syslog on the more built up machine (note the artifacts of the
partial amavisd/clam install):
Aug 21 11:55:18 server1 postfix/smtp[27539]: connect to
example.com[192.0.34.166
]: Connection timed out (port 25)
Aug 21 11:55:18 server1 postfix/smtp[27539]: 702D8B64492:
to=<postmaster@example
.com>, relay=none, delay=9984, status=deferred (connect to
example.com[192.0.34.
166]: Connection timed out)
Aug 21 12:00:01 server1 /USR/SBIN/CRON[27551]: (root) CMD (if [ -x
/usr/bin/vnst
at ] && [ `ls /var/lib/vnstat/ | wc -l` -ge 1 ]; then /usr/bin/vnstat
-u; fi)
Aug 21 12:02:01 server1 /USR/SBIN/CRON[27557]: (logcheck) CMD ( if [
-x /usr/s
bin/logcheck ]; then nice -n10 /usr/sbin/logcheck; fi)
Aug 21 12:02:04 server1 postfix/pickup[27547]: 4E67BB6465F: uid=104
from=<logche
ck>
Aug 21 12:02:04 server1 postfix/cleanup[28308]: 4E67BB6465F:
message-id=<2006082
1160204.4E67BB6465F@server1.neighborhoods.net>
Aug 21 12:02:04 server1 postfix/qmgr[25448]: 4E67BB6465F:
from=<logcheck@server1
.neighborhoods.net>, size=8749, nrcpt=1 (queue active)
Aug 21 12:02:04 server1 amavis[26522]: (26522-02) Clam Antivirus-clamd
FAILED -
unknown status: /var/lib/amavis/amavis-20060821T101517-26522/parts:
Access denie
d. ERROR\n
Aug 21 12:02:04 server1 amavis[26522]: (26522-02) WARN: all primary
virus scanne
rs failed, considering backups
Aug 21 12:02:06 server1 postfix/smtpd[28316]: connect from
localhost.localdomain
[127.0.0.1]
Aug 21 12:02:06 server1 postfix/smtpd[28316]: 9BB38B6465E:
client=localhost.loca
ldomain[127.0.0.1]
Aug 21 12:02:06 server1 postfix/cleanup[28308]: 9BB38B6465E:
message-id=<2006082
1160204.4E67BB6465F@server1.neighborhoods.net>
Aug 21 12:02:06 server1 postfix/smtpd[28316]: disconnect from
localhost.localdom
ain[127.0.0.1]
Aug 21 12:02:06 server1 amavis[26522]: (26522-02) Passed,
<logcheck@server1.neig
hborhoods.net> -> <root@server1.neighborhoods.net>, Message-ID:
<20060821160204.
4E67BB6465F@server1.neighborhoods.net>, Hits: -
Aug 21 12:02:06 server1 postfix/lmtp[28312]: 4E67BB6465F:
to=<root@server1.neigh
borhoods.net>, orig_to=<root>, relay=127.0.0.1[127.0.0.1], delay=2,
status=sent
(250 2.6.0 Ok, id=26522-02, from MTA: 250 Ok: queued as 9BB38B6465E)
Aug 21 12:02:06 server1 postfix/qmgr[25448]: 9BB38B6465E:
from=<logcheck@server1
.neighborhoods.net>, size=9247, nrcpt=1 (queue active)
Aug 21 12:02:06 server1 postfix/qmgr[25448]: 4E67BB6465F: removed
Aug 21 12:02:06 server1 postfix/local[28317]: 9BB38B6465E:
to=<milesf@server1.ne
ighborhoods.net>, orig_to=<root@server1.neighborhoods.net>, relay=local,
delay=0
, status=sent (delivered to command: procmail -a "$EXTENSION")
Aug 21 12:02:06 server1 postfix/qmgr[25448]: 9BB38B6465E: removed
Aug 21 12:05:35 server1 syslogd 1.4.1#17: restart.
Aug 21 12:05:35 server1 kernel: klogd 1.4.1#17, log source = /proc/kmsg
started.
Aug 21 12:05:35 server1 kernel: Inspecting /boot/System.map-2.6.8-3-686
Aug 21 12:05:35 server1 kernel: Loaded 27395 symbols from
/boot/System.map-2.6.8
syslog from the less built-up machine:
Aug 21 10:17:01 server2 /USR/SBIN/CRON[1534]: (root) CMD ( run-parts
--report
/etc/cron.hourly)
Aug 21 10:46:05 server2 -- MARK --
Aug 21 11:06:05 server2 -- MARK --
Aug 21 11:17:01 server2 /USR/SBIN/CRON[1538]: (root) CMD ( run-parts
--report
/etc/cron.hourly)
Aug 21 11:46:05 server2 -- MARK --
Aug 21 12:06:08 server2 syslogd 1.4.1#17: restart.
Aug 21 12:06:08 server2 kernel: klogd 1.4.1#17, log source = /proc/kmsg
started.
Aug 21 12:06:08 server2 kernel: Inspecting /boot/System.map-2.6.8-2-386
Aug 21 12:06:08 server2 kernel: Loaded 28183 symbols from
/boot/System.map-2.6.8
-2-386.
Aug 21 12:06:08 server2 kernel: Symbols match kernel version 2.6.8.
Aug 21 12:06:08 server2 kernel: No module symbols loaded - kernel
modules not en
abled.
Aug 21 12:06:08 server2 kernel: \_SB_.PCI0.PEX1._PRT]
Aug 21 12:06:08 server2 kernel: ACPI: PCI Interrupt Routing Table
[\_SB_.PCI0.HU
Any further thoughts?
Thanks again,
Miles Fidelman