Hacked stable system?
I am running an up-to-date stable distribution. It looks like it may
have been hacked yesterday, but I am not sure how.
Yesterday, I noticed that I could no longer login using ssh or telnet.
ssh logins would hang indefinitely whether I entered correct or
incorrect passwords. Telnet logins would time-out after 60s. I tried
different users with the same result. However, I could ftp and imap
in.
I had the server reboot this morning (it is co-located) and seems to
be functioning fine. Looking at /var/logs, most of the logs seem to
be empty, with .0 version having a strange ctime, e.g.
-rw-r----- 1 root root 4432 Nov 7 15:49 auth.log
-rw-r----- 1 root adm 43014 Jun 10 04:25 auth.log.0
-rw-r----- 1 root root 31 Oct 28 02:26 auth.log.1.gz
-rw-r----- 1 root root 31 Oct 21 02:27 auth.log.2.gz
-rw-r----- 1 root root 31 Oct 14 02:26 auth.log.3.gz
-rw-r----- 1 root root 1416 Oct 8 19:19 auth.log.4.gz
OTOH, Jun 10th is around the time this system was set up, so perhaps
these logs somehow got excluded from the rotation. Apache logs are
intact. I see no other sings of break-in. Also, unfortunately, I
don't know if fsck printed out any messages for /var/ filesystem when
the system was rebooted.
/etc/passwd seems intact
So, what could have caused ssh/telnet to hang like this while ftp
worked fine? What else should I check for break-in signs? I am
thinking I should reinstall the system from scratch. However, same
exploit could be used again.
Gleb
PS I'll include current ps aux:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.3 1020 460 ? S 08:22 0:04 init [2]
root 2 0.0 0.0 0 0 ? SW 08:22 0:00 [kflushd]
root 3 0.0 0.0 0 0 ? SW 08:22 0:01 [kupdate]
root 4 0.0 0.0 0 0 ? SW 08:22 0:01 [kswapd]
root 5 0.0 0.0 0 0 ? SW 08:22 0:00 [keventd]
daemon 92 0.0 0.3 1132 492 ? S 08:23 0:00 /sbin/portmap
root 157 0.0 0.4 1204 560 ? S 08:23 0:01 /sbin/syslog-ng
root 159 0.0 0.6 1424 844 ? S 08:23 0:00 /sbin/klogd
root 163 0.0 0.8 1056 1056 ? SL 08:23 0:00 /usr/sbin/watchdog
root 169 0.0 0.5 1148 644 ? S 08:23 0:00 /sbin/rpc.statd
root 174 0.0 0.2 1024 332 ? S 08:23 0:00 svscan
daemon 175 0.0 0.2 1000 296 ? S 08:23 0:00 multilog t /var/log/svscan
root 178 0.0 0.2 988 304 ? S 08:23 0:00 supervise dnscache
root 179 0.0 0.2 988 304 ? S 08:23 0:00 supervise log
root 180 0.0 0.2 988 304 ? S 08:23 0:00 supervise tinydns
root 181 0.0 0.2 988 304 ? S 08:23 0:00 supervise log
dnscache 184 0.0 1.0 2296 1380 ? S 08:23 0:01 /usr/bin/dnscache
dnslog 185 0.0 0.2 1004 348 ? S 08:23 0:00 multilog t ./main
tinydns 186 0.0 0.2 1108 344 ? S 08:23 0:00 /usr/bin/tinydns
root 187 0.0 0.2 988 304 ? S 08:23 0:00 supervise axfrdns
root 188 0.0 0.2 988 304 ? S 08:23 0:00 supervise log
dnslog 189 0.0 0.2 1000 300 ? S 08:23 0:00 multilog t ./main
root 190 0.0 0.3 1072 420 ? S 08:23 0:00 tcpserver -hvDrl0 -x tcp.cdb -- 208.47.211.42 53 /usr/bin/axfrdns
dnslog 192 0.0 0.2 1000 300 ? S 08:23 0:00 multilog t ./main
root 193 0.0 0.4 1300 552 ? S 08:23 0:00 /usr/sbin/inetd
root 201 0.0 0.4 1352 560 ? S 08:23 0:00 /usr/sbin/lpd
root 209 0.0 0.6 1740 828 ? S 08:23 0:00 sh /usr/bin/safe_mysqld
mysql 222 0.0 1.2 19172 1536 ? S 08:23 0:00 /usr/sbin/mysqld --pid-file=/var/run/mysqld/mysqld.pid
mysql 224 0.0 1.2 19172 1536 ? S 08:23 0:00 /usr/sbin/mysqld --pid-file=/var/run/mysqld/mysqld.pid
mysql 225 0.0 1.2 19172 1536 ? S 08:23 0:00 /usr/sbin/mysqld --pid-file=/var/run/mysqld/mysqld.pid
qmails 232 0.2 0.2 1044 380 ? S 08:23 1:01 qmail-send
perforce 234 0.0 0.4 2032 564 ? S 08:23 0:00 /usr/sbin/p4d
qmaill 236 0.0 0.3 1012 404 ? S 08:23 0:00 splogger qmail
root 237 0.0 0.2 1000 324 ? S 08:23 0:00 qmail-lspawn |/usr/sbin/qmail-procmail
qmailr 238 0.0 0.2 1000 328 ? S 08:23 0:00 qmail-rspawn
qmailq 239 0.0 0.2 992 336 ? S 08:23 0:00 qmail-clean
qmaild 241 0.0 0.4 1376 580 ? S 08:23 0:00 /usr/bin/tcpserver -u 64011 -g 65534 -x /etc/tcp.smtp.cdb 0 smtp /usr/sbin/rblsmtpd -rblackholes.mail-abuse.org /usr/sbin/rblsmtpd -rdialups.mail-abuse.org /usr/sbin/rblsmtpd -rrelays.mail-abuse.org /usr/sbin/qmail-smtpd
root 242 0.0 0.2 1000 276 ? S 08:23 0:00 splogger qmail -t qmail -p mail.notice
root 245 0.0 0.7 1596 956 ? S 08:23 0:05 /usr/sbin/snort -D -S HOME_NET 208.47.211.42/32 -h 208.47.211.42/32 -c /etc/snort/snort-lib -l /var/log/snort/ -s -b -i eth0
root 253 0.0 0.7 2240 932 ? S 08:23 0:05 /usr/sbin/sshd
nobody 256 0.0 2.0 3616 2596 ? S 08:23 0:00 /usr/bin/X11/xfs-xtt -user nobody
root 260 0.0 1.2 1556 1548 ? SL 08:23 0:00 /usr/sbin/ntpd
daemon 265 0.0 0.4 1140 544 ? S 08:23 0:00 /usr/sbin/atd
root 268 0.0 0.4 1168 616 ? S 08:23 0:00 /usr/sbin/cron
root 273 0.0 2.8 39328 3644 ? S 08:23 0:01 /usr/sbin/apache
root 276 0.0 0.3 1004 440 tty1 S 08:23 0:00 /sbin/getty 38400 tty1
root 277 0.0 0.3 1004 440 tty2 S 08:23 0:00 /sbin/getty 38400 tty2
root 278 0.0 0.3 1004 440 tty3 S 08:23 0:00 /sbin/getty 38400 tty3
root 279 0.0 0.3 1004 440 tty4 S 08:23 0:00 /sbin/getty 38400 tty4
root 280 0.0 0.3 1004 440 tty5 S 08:23 0:00 /sbin/getty 38400 tty5
root 281 0.0 0.3 1004 440 tty6 S 08:23 0:00 /sbin/getty 38400 tty6
www-data 282 0.0 2.9 39368 3776 ? S 08:23 0:00 /usr/sbin/apache
www-data 283 0.0 2.8 39328 3652 ? S 08:23 0:00 /usr/sbin/apache
www-data 284 0.0 2.8 39328 3652 ? S 08:23 0:00 /usr/sbin/apache
www-data 285 0.0 2.9 39368 3776 ? S 08:23 0:00 /usr/sbin/apache
www-data 286 0.0 2.8 39328 3652 ? S 08:23 0:00 /usr/sbin/apache
www-data 1176 0.0 2.8 39328 3656 ? S 09:40 0:00 /usr/sbin/apache
alexg 1991 0.0 1.1 2332 1432 ? S 10:29 0:01 SCREEN
alexg 1992 0.0 1.0 2244 1360 pts/2 S 10:29 0:00 /usr/bin/zsh
alexg 1998 0.1 8.9 14256 11440 pts/2 S 10:29 0:23 xemacs
alexg 2041 0.0 1.2 2432 1568 pts/4 S 10:32 0:00 /usr/bin/zsh
root 2079 0.0 1.2 2424 1568 pts/4 S 10:34 0:00 zsh
alexg 2258 0.0 1.0 2244 1368 pts/3 S 10:39 0:00 /usr/bin/zsh
root 2434 0.0 0.4 1328 616 pts/4 S 10:43 0:00 less mail.log.1.gz
root 3793 0.0 1.1 2896 1492 ? S 12:24 0:09 /usr/sbin/sshd
gleb 3795 0.0 0.9 2000 1204 pts/0 S 12:24 0:00 -bash
gleb 3804 0.0 0.4 1220 632 pts/0 S 12:24 0:00 newmail /home/gleb/Mail/new/default
gleb 4431 0.0 0.6 1880 776 pts/0 S 13:18 0:00 screen -e^}]
gleb 4432 0.2 0.9 2100 1188 ? S 13:18 0:26 SCREEN -e^}]
gleb 4433 0.0 0.9 1992 1204 pts/1 S 13:18 0:00 /bin/bash
gleb 4436 0.8 12.2 18384 15568 pts/1 S 13:18 1:27 xemacs
gleb 5407 0.0 0.9 2012 1232 pts/5 S 14:05 0:00 /bin/bash
gleb 5414 0.0 0.9 1992 1208 pts/6 S 14:05 0:00 /bin/bash
gleb 5417 2.4 1.0 2076 1316 pts/6 S 14:05 3:01 top
gleb 8324 0.0 0.9 2032 1252 pts/8 S 14:12 0:01 /bin/bash
gleb 11003 0.0 0.9 2008 1228 pts/7 S 15:53 0:00 /bin/bash
gleb 11099 1.6 1.3 3588 1672 ? S 15:59 0:09 imapd
gleb 11277 0.5 0.9 1996 1212 pts/9 S 16:08 0:00 /bin/bash
gleb 11282 0.0 0.9 2916 1204 pts/9 R 16:08 0:00 ps aux
Reply to: