Strange segmentation faults and Zombies
Hello!
I've seen some strange things on my (stable with security-updates)
server: the last apt-get update didn't work because gzip segfaultet.
I've copied gzip from another server over the version on this server,
but it also crashed. Interesting was that the executable was bigger
after the segfault.
In a ps I can see a lot of Zombies (rm, ln, readlink, grep) and I've no
idea where they come from.
I thougt I should try chkrootkit downloaded and compiled on an external
computer (because on the server there are no development programs) and
scp'ed it over. After running I see the following in the ps aux output:
root 23029 0.2 0.1 2320 1300 pts/0 S 18:53 0:00 /bin/sh ./chkrootkit
root 23088 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep (^|[^A-Za-z0-9_])biff([^A-Za-z0-9_]|$)
root 23089 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23093 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep c
root 23094 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23113 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep (^|[^A-Za-z0-9_])chsh([^A-Za-z0-9_]|$)
root 23117 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23118 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep c
root 23119 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23134 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep c
root 23136 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23150 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep c
root 23151 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23170 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep c
root 23171 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23191 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep ^/bin/.*sh$|bash|elite$|vejeta|\.ark
root 23194 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep ^...s
root 23195 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23198 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep (^|[^A-Za-z0-9_])echo([^A-Za-z0-9_]|$)
root 23203 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23204 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23216 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep ^...s
root 23220 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep (^|[^A-Za-z0-9_])egrep([^A-Za-z0-9_]|$)
root 23221 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23225 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23226 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep c
root 23227 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23240 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep c
root 23245 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep ^/bin/.*sh$|bash|elite$|vejeta|\.ark
root 23258 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep c
root 23259 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23260 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23261 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23287 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep c
root 23288 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23299 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep c
root 23304 0.0 0.0 1220 216 pts/0 T 18:53 0:00 /bin/egrep givemer
root 23306 0.0 0.0 1272 412 pts/0 S 18:53 0:00 /bin/egrep ^...s
root 23307 0.0 0.0 1604 308 pts/0 T 18:53 0:00 /bin/ls -l /bin/grep
root 23308 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [ls <defunct>]
root 23309 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23311 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
root 23313 0.0 0.0 0 0 pts/0 Z 18:53 0:00 [egrep <defunct>]
As you can see there's a lot of Zombies. That output started when
chkrootkit analysed grep (it stopped there and continued only after I
removed all processes in T state), then the same with inetd and after
that I gave up.
You think the server got hacked? Are there any other things that can
lead to this? man also behaves strange, it says either "No manual entry
for...", "What manual page do you want?" or nothing.
regards
Markus
Reply to: