Am I compromised
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello World,
I've got a severe problem. It looks like my webserver has been compromised.
I have a webserver running apache2 (Debian Sarge). My webserver's load is
always remaining around 1.5 and the cpu utilization is 95%.
My webserver is not accepting web connections at the moment.
The top reports show that a perl process is eating up all the cpu cycles
which is executed by www-data user.
Following is a result of `ps aux`:
www-data 15855 0.0 2.8 22564 14808 ? S 05:05
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 15919 0.0 2.8 22592 14840 ? S 05:06
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 15929 0.0 2.3 20076 12176 ? S 05:07
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 15959 0.0 0.0 0 0 ? Z 05:07 0:00 [sh]
<defunct>
www-data 15963 0.0 0.6 5352 3408 ? S 05:07
0:01 /usr/sbin/httpd
www-data 15964 0.0 0.6 5352 3408 ? S 05:07
0:01 /usr/sbin/httpd
www-data 15994 0.0 0.0 0 0 ? Z 05:07 0:00 [sh]
<defunct>
www-data 15998 0.0 0.6 5352 3408 ? S 05:07
0:01 /usr/sbin/httpd
www-data 15999 0.0 0.6 5352 3408 ? S 05:07
0:01 /usr/sbin/httpd
www-data 16229 0.0 2.8 22596 14828 ? S 05:09
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 16302 0.0 0.0 0 0 ? Z 05:10 0:00 [sh]
<defunct>
www-data 16306 0.0 0.6 5352 3404 ? S 05:10
0:01 /usr/sbin/httpd
www-data 16307 0.0 0.6 5352 3408 ? S 05:10
0:01 /usr/sbin/httpd
www-data 16375 0.0 2.3 20124 12224 ? S 05:13
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 16411 0.0 0.0 0 0 ? Z 05:13 0:00 [sh]
<defunct>
www-data 16415 0.0 0.6 5352 3404 ? S 05:13
0:01 /usr/sbin/httpd
www-data 16416 0.0 0.6 5352 3404 ? S 05:13
0:01 /usr/sbin/httpd
www-data 16629 0.0 0.0 0 0 ? Z 05:14 0:00 [sh]
<defunct>
www-data 16633 0.0 0.6 5352 3400 ? S 05:14
0:01 /usr/sbin/httpd
www-data 16634 0.0 0.6 5352 3400 ? S 05:14
0:01 /usr/sbin/httpd
www-data 16930 0.0 2.3 20100 12200 ? S 05:15
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 16963 0.0 0.0 0 0 ? Z 05:15 0:00 [sh]
<defunct>
www-data 16967 0.0 0.6 5352 3400 ? S 05:15
0:01 /usr/sbin/httpd
www-data 16968 0.0 0.6 5352 3400 ? S 05:15
0:01 /usr/sbin/httpd
www-data 17087 0.0 2.8 22564 14800 ? S 05:16
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 17089 0.0 2.3 20100 12204 ? S 05:17
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 17121 0.0 0.0 0 0 ? Z 05:17 0:00 [sh]
<defunct>
www-data 17125 0.0 0.6 5352 3400 ? S 05:17
0:01 /usr/sbin/httpd
www-data 17126 0.0 0.6 5352 3400 ? S 05:17
0:01 /usr/sbin/httpd
www-data 17176 0.0 2.3 20148 12192 ? S 05:17
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 17179 0.0 2.8 22588 14832 ? S 05:17
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 17267 0.0 0.0 0 0 ? Z 05:17 0:00 [sh]
<defunct>
www-data 17271 0.0 0.6 5352 3396 ? S 05:17
0:01 /usr/sbin/httpd
www-data 17272 0.0 0.6 5352 3400 ? S 05:17
0:01 /usr/sbin/httpd
www-data 17362 0.0 0.0 0 0 ? Z 05:18 0:00 [sh]
<defunct>
www-data 17366 0.0 0.6 5352 3396 ? S 05:18
0:01 /usr/sbin/httpd
www-data 17367 0.0 0.6 5352 3396 ? S 05:18
0:01 /usr/sbin/httpd
www-data 17599 0.0 0.0 0 0 ? Z 05:18 0:00 [sh]
<defunct>
www-data 17604 0.0 0.6 5352 3400 ? S 05:18
0:01 /usr/sbin/httpd
www-data 17605 0.0 0.6 5352 3396 ? S 05:18
0:01 /usr/sbin/httpd
www-data 18252 0.0 2.8 22588 14836 ? S 05:21
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 18356 0.0 0.0 0 0 ? Z 05:22 0:00 [sh]
<defunct>
www-data 18360 0.0 0.6 5232 3392 ? S 05:22
0:01 /usr/sbin/httpd
www-data 18361 0.0 0.6 5232 3392 ? S 05:22
0:01 /usr/sbin/httpd
www-data 18742 0.0 2.7 21820 13960 ? S 05:23
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 18956 0.0 0.0 0 0 ? Z 05:24 0:00 [sh]
<defunct>
www-data 18960 0.0 0.6 5232 3388 ? S 05:24
0:01 /usr/sbin/httpd
www-data 18961 0.0 0.6 5232 3392 ? S 05:24
0:01 /usr/sbin/httpd
www-data 18979 0.0 2.8 22564 14800 ? S 05:24
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 19855 0.0 2.8 22544 14776 ? S 05:31
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 19982 0.0 0.0 0 0 ? Z 05:31 0:00 [sh]
<defunct>
www-data 19987 0.0 0.6 5232 3380 ? S 05:31
0:01 /usr/sbin/httpd
www-data 19988 0.0 0.6 5232 3384 ? S 05:31
0:01 /usr/sbin/httpd
www-data 20021 0.0 2.8 22604 14852 ? S 05:31
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 20496 0.0 2.8 22572 14804 ? S 05:32
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 20497 0.0 2.8 22564 14804 ? S 05:32
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 20856 0.0 0.0 0 0 ? Z 05:33 0:00 [sh]
<defunct>
www-data 20860 0.0 0.6 5232 3380 ? S 05:33
0:01 /usr/sbin/httpd
www-data 20861 0.0 0.6 5232 3380 ? S 05:33
0:01 /usr/sbin/httpd
www-data 20922 0.0 2.3 20148 12184 ? S 05:33
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 21049 0.0 0.0 0 0 ? Z 05:33 0:00 [sh]
<defunct>
www-data 21057 0.0 0.6 5232 3380 ? S 05:33
0:01 /usr/sbin/httpd
www-data 21058 0.0 0.6 5232 3380 ? S 05:33
0:01 /usr/sbin/httpd
www-data 21494 0.0 0.0 0 0 ? Z 05:33 0:00 [sh]
<defunct>
www-data 21498 0.0 0.6 5232 3380 ? S 05:33
0:01 /usr/sbin/httpd
www-data 21499 0.0 0.6 5232 3380 ? S 05:33
0:01 /usr/sbin/httpd
www-data 21589 0.0 0.0 0 0 ? Z 05:34 0:00 [sh]
<defunct>
www-data 21596 0.0 0.6 5232 3380 ? S 05:34
0:01 /usr/sbin/httpd
www-data 21597 0.0 0.6 5232 3380 ? S 05:34
0:01 /usr/sbin/httpd
www-data 22509 0.0 2.8 22564 14800 ? S 05:35
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 22545 0.0 0.0 0 0 ? Z 05:36 0:00 [sh]
<defunct>
www-data 22549 0.0 0.6 5232 3376 ? S 05:36
0:01 /usr/sbin/httpd
www-data 22550 0.0 0.6 5232 3376 ? S 05:36
0:01 /usr/sbin/httpd
www-data 22554 0.0 2.8 22564 14812 ? S 05:36
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 22705 0.0 2.8 22560 14800 ? S 05:37
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 22789 0.0 0.0 0 0 ? Z 05:39 0:00 [sh]
<defunct>
www-data 22793 0.0 0.6 5232 3372 ? S 05:39
0:01 /usr/sbin/httpd
www-data 22794 0.0 0.6 5232 3372 ? S 05:39
0:01 /usr/sbin/httpd
www-data 23037 0.0 2.3 20048 12144 ? S 05:40
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 23042 0.0 0.0 0 0 ? Z 05:40 0:00 [sh]
<defunct>
www-data 23047 0.0 0.6 5232 3376 ? S 05:40
0:01 /usr/sbin/httpd
www-data 23048 0.0 0.6 5232 3372 ? S 05:40
0:01 /usr/sbin/httpd
www-data 23072 0.0 0.0 0 0 ? Z 05:42 0:00 [sh]
<defunct>
www-data 23076 0.0 0.6 5232 3372 ? S 05:42
0:01 /usr/sbin/httpd
www-data 23077 0.0 0.6 5232 3372 ? S 05:42
0:01 /usr/sbin/httpd
www-data 23079 0.0 2.8 22564 14792 ? S 05:42
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 23088 0.0 2.8 22564 14788 ? S 05:42
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 23107 0.0 2.8 22564 14804 ? S 05:42
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 23240 0.0 0.0 0 0 ? Z 05:42 0:00 [sh]
<defunct>
www-data 23244 0.0 0.6 5232 3372 ? S 05:42
0:01 /usr/sbin/httpd
www-data 23245 0.0 0.6 5232 3372 ? S 05:42
0:01 /usr/sbin/httpd
www-data 23259 0.0 0.0 0 0 ? Z 05:42 0:00 [sh]
<defunct>
www-data 23263 0.0 0.6 5232 3368 ? S 05:42
0:01 /usr/sbin/httpd
www-data 23264 0.0 0.6 5232 3372 ? S 05:42
0:01 /usr/sbin/httpd
www-data 23265 0.0 2.3 20064 12160 ? S 05:42
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 23366 0.0 0.0 0 0 ? Z 05:42 0:00 [sh]
<defunct>
www-data 23373 0.0 0.6 5232 3372 ? S 05:42
0:01 /usr/sbin/httpd
www-data 23374 0.0 0.6 5232 3368 ? S 05:42
0:01 /usr/sbin/httpd
www-data 23907 0.0 2.8 22564 14784 ? S 05:43
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 23971 0.0 0.0 0 0 ? Z 05:45 0:00 [sh]
<defunct>
www-data 23975 0.0 0.6 5232 3368 ? S 05:45
0:01 /usr/sbin/httpd
www-data 23976 0.0 0.6 5232 3368 ? S 05:45
0:01 /usr/sbin/httpd
www-data 24006 0.0 2.8 22564 14796 ? S 05:45
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 24093 0.0 2.3 20100 12200 ? S 05:45
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 24153 0.0 0.0 0 0 ? Z 05:45 0:00 [sh]
<defunct>
www-data 24157 0.0 0.6 5232 3368 ? S 05:45
0:01 /usr/sbin/httpd
www-data 24158 0.0 0.6 5232 3364 ? S 05:45
0:01 /usr/sbin/httpd
www-data 24244 0.0 0.0 0 0 ? Z 05:46 0:00 [sh]
<defunct>
www-data 24248 0.0 0.6 5232 3368 ? S 05:46
0:01 /usr/sbin/httpd
www-data 24249 0.0 0.6 5232 3368 ? S 05:46
0:01 /usr/sbin/httpd
www-data 24417 0.0 0.0 0 0 ? Z 05:46 0:00 [sh]
<defunct>
www-data 24425 0.0 0.6 5232 3364 ? S 05:46
0:01 /usr/sbin/httpd
www-data 24426 0.0 0.6 5232 3364 ? S 05:46
0:01 /usr/sbin/httpd
www-data 24633 0.0 0.0 0 0 ? Z 05:46 0:00 [sh]
<defunct>
www-data 24637 0.0 0.6 5232 3364 ? S 05:46
0:01 /usr/sbin/httpd
www-data 24638 0.0 0.6 5232 3364 ? S 05:46
0:01 /usr/sbin/httpd
www-data 24650 0.0 2.3 20120 12232 ? S 05:47
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 24783 0.0 0.0 0 0 ? Z 05:47 0:00 [sh]
<defunct>
www-data 24789 0.0 0.6 5232 3364 ? S 05:47
0:01 /usr/sbin/httpd
www-data 24790 0.0 0.6 5232 3364 ? S 05:47
0:01 /usr/sbin/httpd
www-data 24880 0.0 2.8 22564 14788 ? S 05:48
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 24924 0.0 0.0 0 0 ? Z 05:49 0:00 [sh]
<defunct>
www-data 24929 0.0 0.6 5232 3360 ? S 05:49
0:01 /usr/sbin/httpd
www-data 24930 0.0 0.6 5232 3364 ? S 05:49
0:01 /usr/sbin/httpd
www-data 24932 0.0 2.3 20196 12288 ? S 05:49
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 25096 0.0 0.0 0 0 ? Z 05:49 0:00 [sh]
<defunct>
www-data 25100 0.0 0.6 5232 3364 ? S 05:49
0:01 /usr/sbin/httpd
www-data 25101 0.0 0.6 5232 3360 ? S 05:49
0:01 /usr/sbin/httpd
www-data 25415 0.0 2.8 22608 14884 ? S 05:50
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 25478 0.0 0.0 0 0 ? Z 05:51 0:00 [sh]
<defunct>
www-data 25482 0.0 0.6 5232 3360 ? S 05:51
0:01 /usr/sbin/httpd
www-data 25483 0.0 0.6 5232 3360 ? S 05:51
0:01 /usr/sbin/httpd
www-data 25598 0.0 2.8 22580 14784 ? S 05:51
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 25748 0.0 0.0 0 0 ? Z 05:52 0:00 [sh]
<defunct>
www-data 25752 0.0 0.6 5232 3360 ? S 05:52
0:01 /usr/sbin/httpd
www-data 25753 0.0 0.6 5232 3360 ? S 05:52
0:01 /usr/sbin/httpd
www-data 25795 0.0 2.3 20140 12236 ? S 05:52
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 26003 0.0 0.0 0 0 ? Z 05:52 0:00 [sh]
<defunct>
www-data 26008 0.0 0.6 5232 3360 ? S 05:52
0:01 /usr/sbin/httpd
www-data 26009 0.0 0.6 5232 3356 ? S 05:52
0:01 /usr/sbin/httpd
www-data 26306 0.0 2.8 22564 14788 ? S 05:53
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 26319 0.0 2.3 20080 12192 ? S 05:54
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 26325 0.0 0.0 0 0 ? Z 05:54 0:00 [sh]
<defunct>
www-data 26329 0.0 0.6 5232 3356 ? S 05:54
0:01 /usr/sbin/httpd
www-data 26330 0.0 0.6 5232 3356 ? S 05:54
0:01 /usr/sbin/httpd
www-data 26351 0.0 2.3 20184 12280 ? S 05:55
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 26669 0.0 0.0 0 0 ? Z 05:55 0:00 [sh]
<defunct>
www-data 26673 0.0 0.6 5232 3356 ? S 05:55
0:01 /usr/sbin/httpd
www-data 26674 0.0 0.6 5232 3356 ? S 05:55
0:01 /usr/sbin/httpd
www-data 26694 0.0 0.0 0 0 ? Z 05:55 0:00 [sh]
<defunct>
www-data 26698 0.0 0.6 5232 3356 ? S 05:55
0:01 /usr/sbin/httpd
www-data 26699 0.0 0.6 5232 3356 ? S 05:55
0:01 /usr/sbin/httpd
www-data 27008 0.0 2.8 22564 14776 ? S 05:55
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 27066 0.0 0.0 0 0 ? Z 05:55 0:00 [sh]
<defunct>
www-data 27070 0.0 0.6 5232 3356 ? S 05:55
0:01 /usr/sbin/httpd
www-data 27071 0.0 0.6 5232 3356 ? S 05:55
0:01 /usr/sbin/httpd
www-data 28373 0.0 2.3 20196 12304 ? S 06:08
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 28375 0.0 0.0 0 0 ? Z 06:08 0:00 [sh]
<defunct>
www-data 28379 0.0 0.6 5232 3344 ? S 06:08
0:01 /usr/sbin/httpd
www-data 28380 0.0 0.6 5232 3340 ? S 06:08
0:01 /usr/sbin/httpd
www-data 28382 0.0 2.3 20260 12308 ? S 06:08
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 28384 0.0 0.0 0 0 ? Z 06:08 0:00 [sh]
<defunct>
www-data 28390 27.7 0.7 5760 3808 ? R 06:08
23:19 /usr/sbin/httpd
identd 28391 0.0 0.1 51948 1032 ? S 06:08 0:00 identd
www-data 32753 0.0 2.7 22240 14348 ? S 07:20
0:00 /usr/sbin/apache2 -k start -DSSL
root 307 0.0 0.3 6224 1956 ? Ss 07:21 0:00 sshd: rrs
[priv]
rrs 310 0.0 0.3 6388 2060 ? S 07:21 0:00 sshd:
rrs@pts/0
rrs 311 0.0 0.4 3728 2356 pts/0 Ss 07:21 0:00 -bash
root 348 0.0 0.2 2592 1476 pts/0 S 07:22 0:00 -su
www-data 368 0.1 2.7 22240 14340 ? S 07:23
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 376 0.0 2.7 22240 14348 ? S 07:24
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 394 0.1 2.7 22428 14412 ? S 07:26
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 420 0.1 2.7 21836 13968 ? S 07:29
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 426 0.0 0.0 0 0 ? Z 07:29 0:00 [perl]
<defunct>
www-data 453 0.1 2.7 22420 14396 ? S 07:30
0:00 /usr/sbin/apache2 -k start -DSSL
root 462 0.0 0.1 2052 932 ? S 07:30
0:00 /USR/SBIN/CRON
root 463 0.0 0.2 2696 1200 ? Ss 07:30 0:00 /bin/sh
- -c /usr/local/bin/update-data.sh > /dev/null
root 464 0.0 0.2 2696 1248 ? S 07:30
0:00 /bin/sh /usr/local/bin/update-data.sh
root 493 0.0 0.2 3376 1496 ? S 07:30 0:00 wget -q
- -O /etc/tinydns/root/data.srv-1 http://127.0.0.1/veg
root 495 0.0 0.1 1512 624 ? Ss 07:30
0:00 /usr/sbin/anacron -s
www-data 546 5.2 2.7 22440 14424 ? S 07:31
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 548 37.5 2.7 22464 14448 ? S 07:32
0:00 /usr/sbin/apache2 -k start -DSSL
www-data 552 0.0 2.7 22240 14348 ? S 07:32
0:00 /usr/sbin/apache2 -k start -DSSL
root 553 0.0 0.1 2496 848 pts/0 R+ 07:32 0:00 ps aux
ns1:~# cd /etc/cron.d
Interesting part is that it shows that "/usr/sbin/httpd" process is being
run where as there's no "/usr/sbin/httpd" on my machine.
ns1:/etc/cron.d# file /usr/sbin/httpd
/usr/sbin/httpd: ERROR: cannot open `/usr/sbin/httpd' (No such file or
directory)
I installed "chkrootkit" to see if any rootkit was installed but chkrootkit
reports that the system is not infected.
Can anyone help me if my system is compromised or is it a system related
issue ? What steps should I follow to get my webserver usable again ? It's
a machine under production usage.
Regards,
rrs
- --
Ritesh Raj Sarraf
RESEARCHUT -- http://www.researchut.com
Gnupg Key ID: 04F130BC
"Stealing logic from one person is plagiarism, stealing from many is
research."
"Necessity is the mother of invention."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iD8DBQFDhzPm4Rhi6gTxMLwRAo99AKCFmpZv26CwQkYiIk+NqHNImoyI0wCfSflH
uq+SfrUwvjpF9VNYkZgYGyY=
=30xh
-----END PGP SIGNATURE-----
Reply to: