[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Long Exim break-in analysis

Hello all,

first, I apologize for a long mail. Don't read if you don't like long e-mails.
But as Thorsten was already affected by exim exploit I thought this might be 
interesting for all debian-exim users:

one of my friends asked me for help with his server, and I discovered that it 
was rooted through unpatched exim. System is being reinstalled now, and I 
decided to write something about this exploit. I hope you will find the info 
interesting. It won't be anything exact, because the machine is offline now, 
but anyway here it goes:

First sign was that mail did not get through. Server was overloaded and a 
process named syslogd was using most of CPU. On the first sight "top" was 
looking a bit different than usual. ps showed processes /sbin/syslogd and 
syslogd (without path). First one was ok, the second one was doing something 
nasty and using the CPU. /proc/PID/exe was symlink to perl...

After I killed (-9) this rogue syslog, exim spawned new one! So I killed them 
both. There were some interesting files in /var/spool/exim4/ - two configs 
that download binary named setuid into /var/spool/exim4/ and make it setuid 
and try to run it. The other config did the sam ine /var/spool/exim/.
I think it was the same as shown on exim mailing list.

However /var/ was mounted nosuid so it failed (few days ago). But the bad guy 
was able to get shell as debian-exim user, and compiled another binary. He 
left us the source ;) - it was supposed to install his public key 
into /root/.ssh/authorized_keys. I checked this file and found there a public 
key but it was different then the one in /var/spool/exim/. Removed.

It seems that the first attack was uncuccessfull, but then some other attacker 
found that /tmp was not on separate partition, and setuid worked there. He 
left some evidence in /var/spool/exim/.bash_history - downloading and running 
some rootkit. Further search for suspicious processes found sshd on port 
above 55000. Killed immediately.

Then I started to get annoyed by ls, because it was spewing errors. It was 
because I have alias l='ls --color=auto'. Pure ls was ok. So I started 
looking for modified binaries, and found that some are owned by UID=122 which 
was not present in /etc/passwd:

find /bin/ /sbin/ /usr/bin/ /usr/sbin/ -not -user root -ls

-rwxr-xr-x   1 122      114         54152 Dec  4  2005 /bin/netstat
-rwxr-xr-x   1 122      114         39696 Jan 30  2007 /bin/ls
-rwxr-xr-x   1 122      114         62920 Sep 13  2006 /bin/ps
-rwxr-xr-x   1 122      114        212747 Jan 30  2007 /sbin/ttyload
-rwxrwxr-x   1 122      114         93476 Jan 30  2007 /sbin/ttymon
-rwxr-xr-x   1 122      114         31504 Dec  4  2005 /sbin/ifconfig
-rwxr-xr-x   1 122      114         33992 Sep 13  2006 /usr/bin/top
-rwxr-xr-x   1 122      114         31452 Jan 30  2007 /usr/bin/md5sum
-rwxr-xr-x   1 122      114         12340 Aug  9  2006 /usr/bin/pstree
-rwxr-xr-x   1 122      114         59536 Jul 30  2007 /usr/bin/find

so now it explained why ls and top behaved differently than usual. Of course 
we cannot trust these results because ls and find are modified as well...

Further idea was, they must have done something to start after reboot, 
check /etc/inittab and there was something like this:

# standard tty stuff

nice comment eh? Intersting is that mtime was probably preserved, but ctime 
was recent (few hours).

ps did not show that ttyload is running, but killall killed something 
anyway ;) because on first run it did not complain, but second time it said: 
no process killed. Then I compared netstat (hacked) with nmap from outside, 
and found that lots of ports are missing. Apache is running but not listening 
according to netstat... so there might be further backdoors hidden.

Thats almost all. Machine is now offline, replaced by another one. I'll try to 
get the hacked machine booted from live-cd, so I can examine it with 
trustworthy tools, and if i find more interesting thing i'll post a follow 

Lessons learned:
1. subscribe to DSA and run apt-get 
2. /var/spool, /var/tmp, /tmp and other places where unprivileged users can 
write, should be mounted nosuid and even better noexec. It seems that this 
could prevent the attack, or at least make it much more difficult. 

As for point 2. it's a pity that dpkg is using /tmp and /var/lib/dpkg/ to run 
scripts during installation and removal of packages. It would be nice if 
whole /var could be mounted noexec.

That's all folks
	Vladislav Kurz

Reply to: