Status field in ps output for threaded programs in 2.6 kernel
Hello.
Today I faced a situation that confused me.
One moment I found that on our server there is almost 100% CPU load in
system state. However, ps and top and other tools could not locate any
process eating CPU.
The situation was permanent - several minutes passed, still almost 100% CPU
in system state, still no visible running processes.
I executed chkrootkit and it found lots of "hidden processes" - ones for
which /proc/N/ exists (i.e. it is possible to cd there), but N is not in
ps output (and even not in ls /proc output).
I checked /proc/N/cmdline for some of such "hidden" pids; it looks like
they represent threads.
Well, maybe it is how NPTL threads work: for each thread a hidden /proc
entry exists.
And it is chkrootkit's problem that is reports this (normal) case as
possible LKM installed.
However, in my case, CPU-eating entities were found among thise hidden
pids! They belonged to the same process (locally developed app, probably
hanged), and this process was listed in ps output with status "S". After
process was killed, CPU load went to 0%.
Looks like kernel bug: displayed status of threaded process should not be
"S" if some threads are active. And CPU usage statistics should correspond
to all threads.
This happened on a Debian testing/unstable system, running kernel package
kernel-image-2.6.5-1-k7-smp 2.6.5-4.
Reply to: