My machine was hacked - possibly via sshd?
All,
My machine was cracked on Thursday evening. I'm trying to understand
how it happened so that it doesn't go down again.
Machine was running Debian 3.0 and was behind a NAT box with ports
forwarded for SMTP, HTTP and SSH. It hadn't been rebooted for 430
days. I was using a 2.4 kernel with MPPE builtin.
Early on the 25th, my logcheck emails indicated increasing messages in
syslog concerning failed login attempts against ssh. At some point
though I see ssh authentication failures for valid user names - how?
Somehow they were being enumerated in the hack attempt, and I think that
one person had a weak password. Finally I see an attempt to load
net-pf-14 and other modprobe errors. At some point there are also
messages about the ethernet card entering promiscuous mode.
When I logged on I discovered two outgoing connections to port ircd on
the foreign hosts, and some thing listening on port 48744 TCP. No PID
associated with them. I also discovered that a bunch binaries were
failing: gzip seg faulted; man couldn't load any man pages; any commands
caused new messages to appear in the syslog concerning kernel modules
loading and eth0 going in to promis mode. I'm guessing (maybe I read it
somewhere in a log) that a packet sniffer was running.
So what can I do to prevent it? My best guess is that ssh failed, but
this is based on the log messages. Exim or Apache could have been the
point of failure too though. Seeing as it was so long since I rebooted,
perhaps the exploit was coupled with a kernel vulnerability. Any
thoughts? I was up to date on all security patches. My kernel came from:
deb http://www.vanadac.com/~dajhorn/projects/debian-pptp woody main
Somehow the usernames were enumerated and weak password was discovered.
There must have then been a local elevation of privileges attack at
which point it was definitely all over.
I've rebuilt the machine. The biggest changes so far have been
partitioning. I no longer have a single partition, but about 10,
including read-only ones for /usr and /boot. I'm also running the
Debian stock 2.4.18-1-586tsc 2.4.18-1-586tsc (I don't need to create
PPTP tunnels anymore). I have Exim up and running and exposed to the
internet. I need to open up ssh to external connections too soon, and
of course I will be reinstalling Apache within a week.
Sample logcheck messages:
Mar 25 01:24:19 erin-and-malc sshd[23707]: Did not receive
identification string from 193.170.65.132
Mar 25 01:31:02 erin-and-malc sshd[23661]: Did not receive
identification string from 203.228.120.102
Mar 25 02:23:12 erin-and-malc PAM_unix[24756]: authentication failure;
(uid=0) -> backup for ssh service
Mar 25 02:23:14 erin-and-malc sshd[24756]: Failed password for backup
from 193.170.65.132 port 4128 ssh2
Mar 25 02:24:24 erin-and-malc PAM_unix[24884]: authentication failure;
(uid=0) -> erin for ssh service
Mar 25 02:24:26 erin-and-malc sshd[24884]: Failed password for erin from
193.170.65.132 port 5776 ssh2
Mar 25 02:40:57 erin-and-malc sshd[26053]: warning: /etc/hosts.deny,
line 15: can't verify hostname:
gethostbyname(17.red-82-158-1.user.auna.net) failed
Mar 25 02:40:57 erin-and-malc sshd[26053]: refused connect from 82.158.1.17
Mar 25 02:43:53 erin-and-malc kernel: request_module[net-pf-14]:
waitpid(26279,...) failed, errno 512
Mar 25 02:43:55 erin-and-malc kernel: request_module[net-pf-14]:
waitpid(26284,...) failed, errno 512
There are hundreds of these:
Mar 25 02:40:48 erin-and-malc sshd[26038]: Could not reverse map address
193.170.65.132.
Mar 25 02:40:50 erin-and-malc sshd[26040]: Could not reverse map address
193.170.65.132.
Mar 25 02:40:52 erin-and-malc sshd[26042]: Could not reverse map address
193.170.65.132.
Mar 25 02:40:53 erin-and-malc sshd[26044]: Could not reverse map address
193.170.65.132.
Mar 25 02:40:55 erin-and-malc sshd[26046]: Could not reverse map address
193.170.65.132.
Access gained to a normal user:
Mar 25 02:44:03 erin-and-malc newgrp[26309]: user `steve' switched to
group `steve'
Mar 25 02:47:42 erin-and-malc PAM_unix[26416]: Password for steve was
changed
And finally:
Possible Security Violations
=-=-=-=-=-=-=-=-=-=
Mar 25 04:05:18 erin-and-malc kernel: request_module[ppp0]: fork failed,
errno 1
Unusual System Events
=-=-=-=-=-=-=-=-=-=-=
Mar 25 04:05:14 erin-and-malc modprobe: modprobe: Can't locate module ppp0
Mar 25 04:05:18 erin-and-malc kernel: request_module[ppp0]: fork failed,
errno 1
Mar 25 05:02:04 erin-and-malc kernel: eth0: Promiscuous mode enabled.
Mar 25 05:05:13 erin-and-malc kernel: eth0: Promiscuous mode enabled.
Reply to: