Detecting threads vs processes with ps or /proc
Hello, all.
Quoted 1 line
From searching through mail archives, etc., I'm pretty sure I know the answer
already, but I'm going to post it anyway.
Our software (Condor) and some related software (Globus) is running on a number of systems around the world. Condor attempts to monitor the RAM usage of it's "user" (maybe "client" is a better word here) processes. If the client forks, we need to monitor the client and all of it's children, which really isn't difficult. The _problem_ is that if the client creates threads, it's impossible, from what we can tell, to tell the difference between separate threads and processes.
So my question, I guess, is this. How can you tell, from user space, whether a group of processes are related as threads or through a "normal" child / parent relationship? The systems that we're running on currently are 2.2.19 and 2.4.18/19.
Quoted 1 line
From what else I've read, it seems that the new threading model in 2.5/2.6 is
changing to a more POSIX friendly model, which will effect this answer, but we're not running 2.5 and really can't force such an upgrade -- hell, right now we're having problems getting a switch from 2.2 pushed through.
Thanks _very_ much in advance. I'd be tickled pink if the answer is something like "just look at the foo flag in ps", or "upgrade to version 1.2.3.4 of procps and do xyzzy", but my intuition tells me otherwise.
Thanks,
-Nick
Quoted 4 lines · Nick LeRoy
Quoted 1 line
From what else I've read, it seems that the new threading model in 2.5/2.6 is
changing to a more POSIX friendly model, which will effect this answer, but we're not running 2.5 and really can't force such an upgrade -- hell, right now we're having problems getting a switch from 2.2 pushed through.
Yep, you should get what you want with 2.5 + NPTL. We need to add a few bits, though, to make it complete.
Quoted 3 lines
Thanks _very_ much in advance. I'd be tickled pink if the answer is something like "just look at the foo flag in ps", or "upgrade to version 1.2.3.4 of procps and do xyzzy", but my intuition tells me otherwise.
See http://tech9.net/rml/procps
and "upgrade to version 2.0.8 or later of procps" :)
It is just a heuristic, though. A hack in fact. We look at a process's children and compare RSS, VM size, and the process image they are running. If they are the same, we label them threads.
It is the default behavior. Flag `-m' turns it off.
See thread_group() and flag_threads().
Robert Love
Quoted 15 lines · Robert Love
Quoted 6 lines · Nick LeRoy
Quoted 2 lines
From what else I've read, it seems that the new threading model in 2.5/2.6 is
changing to a more POSIX friendly model, which will effect this answer, but we're not running 2.5 and really can't force such an upgrade -- hell, right now we're having problems getting a switch from 2.2 pushed through.
Yep, you should get what you want with 2.5 + NPTL. We need to add a few bits, though, to make it complete.
Quoted 3 lines
Thanks _very_ much in advance. I'd be tickled pink if the answer is something like "just look at the foo flag in ps", or "upgrade to version 1.2.3.4 of procps and do xyzzy", but my intuition tells me otherwise.
Quoted 7 lines
See http://tech9.net/rml/procps
and "upgrade to version 2.0.8 or later of procps" :)
It is just a heuristic, though. A hack in fact. We look at a process's children and compare RSS, VM size, and the process image they are running. If they are the same, we label them threads.
I was considerring doing something like this as well. From your experience, does it work reliably? Do you need to apply a small 'fudge factor' (aka VMsize.1 ~= VMsize.2)?
Quoted 3 lines
It is the default behavior. Flag `-m' turns it off.
See
thread_group()andflag_threads().
I assume these are functions in the tools themselves?
Quoted 1 line
Robert Love
Thanks
-Nick
Quoted 2 lines · Nick LeRoy
I was considerring doing something like this as well. From your experience, does it work reliably?
It never fails to detect threads (no false negatives).
It does sometimes detect child processes that forked but did not exec as threads (false positives). The failure case is when a program forks, does not call exec, and the children go on to execute the exact same code (so they look and act just like threads, but they have unique addresses spaces).
One thing to note: if you can modify the kernel and procps, you can just export the value of task->mm out of /proc. It is a gross hack, and perhaps a security issue, but that will work 100%. Same ->mm implies thread.
Quoted 2 lines
Do you need to apply a small 'fudge factor' (aka
VMsize.1 ~= VMsize.2)?
No, they need to be exact. The stats are pulled from the same structure in the kernel so they need to be perfect.
Quoted 5 lines
Quoted 3 lines
It is the default behavior. Flag `-m' turns it off.
See
thread_group()andflag_threads().I assume these are functions in the tools themselves?
Yep. Both ps and top have support. There is a patch on the website that added the feature, so you can take a look at that.
Red Hat 8.0 has support in their version of procps 2.0.7, too. Otherwise it was added in 2.0.8. The current version is 2.0.10.
We have a mailing list at [email protected] where this sort of discussion is welcome.
Robert Love
Quoted 4 lines · Robert Love
One thing to note: if you can modify the kernel and procps, you can just export the value of
task->mmout of /proc. It is a gross hack, and perhaps a security issue, but that will work 100%. Same ->mm implies thread.
It isn't a terribly gross hack. I have a patch (somewhere...) which adds an ASID: field to /proc/<pid>/status, which simply contains the mm pointer (as an opaque identifier token). If you were worried about exposing (yet another) kernel pointer value, I suppose you could mush it about a bit, but I think that would give the illusion of obscurity rather than any actual increase in security.
J
Quoted 9 lines · Nick LeRoy
Our software (Condor) and some related software (Globus) is running on a number of systems around the world. Condor attempts to monitor the RAM usage of it's "user" (maybe "client" is a better word here) processes. If the client forks, we need to monitor the client and all of it's children, which really isn't difficult. The _problem_ is that if the client creates threads, it's impossible, from what we can tell, to tell the difference between separate threads and processes.
Quoted 4 lines
So my question, I guess, is this. How can you tell, from user space, whether a group of processes are related as threads or through a "normal" child / parent relationship? The systems that we're running on currently are 2.2.19 and 2.4.18/19.
There is another approach save for the one already discussed here, which I have no idea how applicable it is in your case, but will produce 100% reliable results without additional kernel support - track the processes forks.
There are several ways you can go about it - there's the expensive (in terms of CPU cycles) approach of using ptrace(2), the relativly painless way of overriding the "default" calls to fork and friends via ld.so(8) magical LD_PRELOAD that nevertheless requires you control the execution of the "client" programs and even using a system call tracking module such as syscall-tracker (http://syscalltrack.sf.net) which might not be quite ready for use on a production system as of yet.
Hope this helps,
Gilad
Signature · 3 lines
Gilad Ben-Yossef <[email protected]> http://benyossef.com "Denial really is a river in Eygept."