Re: Server hacked - next...?

[Christofer Algotsson <royce@sparklet.com>]

On Sun, Jun 29, 2003 at 01:00:57PM +0800, Jason Lim wrote:
> 
> One of our servers was hacked (woody)... badly, from what I can see. A
> whole bunch of binaries have been modified, and strange processes are
> running on the server. The hack date appears to be jun 6.
> 
> Is there a document somewhere, or procedure, to recover after this? This
> is a working and running system, so somehow need to be able to recover
> from this with minimal impact to end-users.
> 
> Some things like:
> 
> www-data 17451  0.0  0.0  2164  928 ?        S    02:31   0:00 /bin/sh
> www-data 21550  0.0  0.0  1232  236 ?        S    05:02   0:00 ./x
> www-data 21551  0.0  0.0     0    0 ?        Z    05:02   0:00 [x
> <defunct>]
> root     21552  0.0  0.0     0    0 ?        Z    05:02   0:00 [modprobe
> <defunc
> root     21554  0.0  0.0  2148  912 ?        S    05:02   0:00 /bin/sh
> root     21755  0.0  0.0  2164  948 ?        S    05:02   0:00 /bin/sh
> root     21801  0.0  0.0  2180  964 ?        S    05:03   0:00 /bin/bash
> ./troja
> root     22010  0.0  0.0  1244  204 ?        S    05:03   0:00 ./siz
> ifconfigx /
> root     12267  0.0  0.0     0    0 ?        Z    07:15   0:00 [date
> <defunct>]
> root     12266  0.0  0.0  1264  252 ?        T    07:15   0:00 date +%d


Hi! 

I'm no expert in this at all... 
Here are some basic try-to-solve-it hints.

In most cases its not possible to reinstall the whole system, as in this
case. I mean. A home-server/workstation is no problem to reinstall but
a high SLA 60k -user cluster is quite boring and time consuming.

I'd do it like this.

First. We need some fresh & clean tools;

kill, killall, ps, more, netstat, ls, dpkg, apt-tools, chattr, lsattr,  bash (or whatever shell you prefer).


Replace your shell with the clean one (the /etc/passwd -race).

Killing the procs right off is almost
impossible unless you find the master process (often protected and hidden in a patched
ps or proctable and chattr'ed away on your filesystem).

since you're using the >2.4.20 kernels modprobe-bug exists, so get rid
of that bug first.

echo "blah" > /proc/sys/kernel/modprobe

then I'd; lsattr -a /*|more  to see weither we have some hidden and/or
write protected files that we dont know.

I bet you'll get some interesting output here. 

chattr this files and move them to some secret place so you can check
them out later... Dont forget to check .history-files, logs etc. Most
hacks are done in a rush and there are always pices of information left
every here and there.

After the filesystem looks nice and clean I would try find and
kill the processes. 

As stated abowe it's quite hard to kill processes that are not meant to
be killed. netstat -anp is a good tool here aswell as kill and ps. 

I've seen cases when the mastersprocess is hidden within sshd, init,
various daemons such as ftp, telnet, ldap, gpm etc. Kill all processes you dont
need. Look for respawning ones. 

A reboot might help, but dont reboot until you've checked the
startup rc-files, stuff needed to boot etc...

then i'd apt-getted the base system and then all debian-packages.

Now try find out how he did it and try fix it before it happens again.
 

 Hope this helps... 

-- 
__
Yours sincerely,
Christofer Algotsson - royce@sparklet.com