Long Exim break-in analysis
Hello all,
first, I apologize for a long mail. Don't read if you don't like long e-mails.
But as Thorsten was already affected by exim exploit I thought this might be
interesting for all debian-exim users:
one of my friends asked me for help with his server, and I discovered that it
was rooted through unpatched exim. System is being reinstalled now, and I
decided to write something about this exploit. I hope you will find the info
interesting. It won't be anything exact, because the machine is offline now,
but anyway here it goes:
First sign was that mail did not get through. Server was overloaded and a
process named syslogd was using most of CPU. On the first sight "top" was
looking a bit different than usual. ps showed processes /sbin/syslogd and
syslogd (without path). First one was ok, the second one was doing something
nasty and using the CPU. /proc/PID/exe was symlink to perl...
After I killed (-9) this rogue syslog, exim spawned new one! So I killed them
both. There were some interesting files in /var/spool/exim4/ - two configs
that download binary named setuid into /var/spool/exim4/ and make it setuid
and try to run it. The other config did the sam ine /var/spool/exim/.
I think it was the same as shown on exim mailing list.
However /var/ was mounted nosuid so it failed (few days ago). But the bad guy
was able to get shell as debian-exim user, and compiled another binary. He
left us the source ;) - it was supposed to install his public key
into /root/.ssh/authorized_keys. I checked this file and found there a public
key but it was different then the one in /var/spool/exim/. Removed.
It seems that the first attack was uncuccessfull, but then some other attacker
found that /tmp was not on separate partition, and setuid worked there. He
left some evidence in /var/spool/exim/.bash_history - downloading and running
some rootkit. Further search for suspicious processes found sshd on port
above 55000. Killed immediately.
Then I started to get annoyed by ls, because it was spewing errors. It was
because I have alias l='ls --color=auto'. Pure ls was ok. So I started
looking for modified binaries, and found that some are owned by UID=122 which
was not present in /etc/passwd:
find /bin/ /sbin/ /usr/bin/ /usr/sbin/ -not -user root -ls
-rwxr-xr-x 1 122 114 54152 Dec 4 2005 /bin/netstat
-rwxr-xr-x 1 122 114 39696 Jan 30 2007 /bin/ls
-rwxr-xr-x 1 122 114 62920 Sep 13 2006 /bin/ps
-rwxr-xr-x 1 122 114 212747 Jan 30 2007 /sbin/ttyload
-rwxrwxr-x 1 122 114 93476 Jan 30 2007 /sbin/ttymon
-rwxr-xr-x 1 122 114 31504 Dec 4 2005 /sbin/ifconfig
-rwxr-xr-x 1 122 114 33992 Sep 13 2006 /usr/bin/top
-rwxr-xr-x 1 122 114 31452 Jan 30 2007 /usr/bin/md5sum
-rwxr-xr-x 1 122 114 12340 Aug 9 2006 /usr/bin/pstree
-rwxr-xr-x 1 122 114 59536 Jul 30 2007 /usr/bin/find
so now it explained why ls and top behaved differently than usual. Of course
we cannot trust these results because ls and find are modified as well...
Further idea was, they must have done something to start after reboot,
check /etc/inittab and there was something like this:
# standard tty stuff
0:2345:respawn:/sbin/ttyload
nice comment eh? Intersting is that mtime was probably preserved, but ctime
was recent (few hours).
ps did not show that ttyload is running, but killall killed something
anyway ;) because on first run it did not complain, but second time it said:
no process killed. Then I compared netstat (hacked) with nmap from outside,
and found that lots of ports are missing. Apache is running but not listening
according to netstat... so there might be further backdoors hidden.
Thats almost all. Machine is now offline, replaced by another one. I'll try to
get the hacked machine booted from live-cd, so I can examine it with
trustworthy tools, and if i find more interesting thing i'll post a follow
up.
Lessons learned:
1. subscribe to DSA and run apt-get
2. /var/spool, /var/tmp, /tmp and other places where unprivileged users can
write, should be mounted nosuid and even better noexec. It seems that this
could prevent the attack, or at least make it much more difficult.
As for point 2. it's a pity that dpkg is using /tmp and /var/lib/dpkg/ to run
scripts during installation and removal of packages. It would be nice if
whole /var could be mounted noexec.
That's all folks
--
Regards
Vladislav Kurz
Reply to: