Thread and process dentifiers (CPU affinity, kill)
It seems that the thread ids in Linux are unique within the complete operating system, and not only within their corresponding processes. This is explicitely allowed by the POSIX standard (it also states that applications shall no rely on it).
Apparently some system calls which normally require a process id also work with thread ids.
- a system call requiring a PID can have the same effect if a thread id of the same process was given.
Example: kill(tid,SIGTERM) will kill the entire process the thread
belongs to. I think that this is not POSIX compliant. It shall trigger
ESRCH!
Sometimes, the system call has another effect, potentialy providing
additional functionality.
Example: sched_setaffinity(). The man page and the prototype (which requires a pid_t) both show that a process id is required. Nothing indicates that it works with threads, and AFAIK there is no documented way to set affinity for a specific thread. But if you give a TID to sched_setaffinity, it will put the *thread* on the given cpu set.
If you give a PID to sched_setaffinity, it will put the *main thread* on
the given cpu set. The other threads won't be impacted.
Even if sched_setaffinity() is no standard, I find it confusing to give
it a pid_t and that it affects only threads!
Some open questions:
- is it a guaranted behaviour within Linux that thread ids and process ids do not overlap? is it documented anywhere?
- is there a real confusion at API level within Linux between threads and processes or are kill() and sched_setaffinity() isolated examples? or bugs?
- is Linux kill() POSIX compliant in this regard?
- do we want to limit the sched_setaffinity() functionality to correspond to its documentation, or do we want to update the documentation so that its covers all the functionality?
Regards
Olivier
Quoted 7 lines · Olivier Croquette
It seems that the thread ids in Linux are unique within the complete operating system, and not only within their corresponding processes. This is explicitely allowed by the POSIX standard (it also states that applications shall no rely on it).
Apparently some system calls which normally require a process id also work with thread ids.
They work with numbers, this is C, there is no real serious type checking. Don't rely on it and just pretend it isn't that way.
Quoted 5 lines
- a system call requiring a PID can have the same effect if a thread id of the same process was given.
Example:kill(tid,SIGTERM)will kill the entire process the thread belongs to. I think that this is not POSIX compliant. It shall trigger ESRCH!
How should kill know if you are sending a threadid or processid? If the integer matches a running pid then it should kill that one. If there is no such process id it should return ESRCH. Just because every threadid is a processid too doesn't mean it is broken, and you aren't supposed to pass threadids to kill anyhow.
Quoted 14 lines
Sometimes, the system call has another effect, potentialy providing additional functionality.
Example:sched_setaffinity(). The man page and the prototype (which requires apid_t) both show that a process id is required. Nothing indicates that it works with threads, and AFAIK there is no documented way to set affinity for a specific thread.
But if you give a TID tosched_setaffinity, it will put the *thread* on the given cpu set.
If you give a PID tosched_setaffinity, it will put the *main thread* on the given cpu set. The other threads won't be impacted.
Even ifsched_setaffinity()is no standard, I find it confusing to give it apid_tand that it affects only threads!
Quoted 8 lines
Some open questions:
- is it a guaranted behaviour within Linux that thread ids and process ids do not overlap? is it documented anywhere?
- is there a real confusion at API level within Linux between threads and processes or are
kill()andsched_setaffinity()isolated examples? or bugs?
Just pretend they are different things even if currently they are implemented so threads are processes and your code should be safe.
Quoted 1 line
- is Linux
kill()POSIX compliant in this regard?
Does posix say that a process can't be allocated multiple PIDs? What should kill do when sent to anyone of the PIDs beloging to a process? It should probably do the same thing as if it was sent to the PID of the main thread (whatever main thread means in a threaded program). I would think it is posix compliant even if it isn't the common way to represent threads on posix compliant systems.
Quoted 3 lines
- do we want to limit the
sched_setaffinity()functionality to correspond to its documentation, or do we want to update the documentation so that its covers all the functionality?
I believe Linux currently implements threads as seperate processes (at least top and ps sees them that way). Of course I would never recomend assuming things will always work that way, since after all someone is perfectly allowed to implement threading in user space with a posix thread compliant interface and link a program against their own thread library which doesn't use the kernel to manage the threads (using linuxthreads). A safe programmer makes no assumptions about anything if it isn't documented in the specs to work a specific way. if it is states as undefined, expect the behaviour to potentially change.
Now given linux runs threads as seperate processes, it makes sense that thread ids and process ids are the same thing and hence currently unique, and that kill would work on any thread's pid within a given process. I am not sure how the process handles signals to a thread in terms of signal handling, although I would think the kernel probably knows it is a thread and passes it to the parent process instead.
Doesn't sched_setaffinity do what it says it will? Since each thread is treated as a process then sched_setaffinity should work on it I would think since it is a process after all as far as the scheduler is concerned.
Len Sorensen
Quoted 10 lines · Lennart Sorensen
Quoted 5 lines · Olivier Croquette
- a system call requiring a PID can have the same effect if a thread id of the same process was given.
Example:kill(tid,SIGTERM)will kill the entire process the thread belongs to. I think that this is not POSIX compliant. It shall trigger ESRCH!How should kill know if you are sending a threadid or processid?
Doesn't matter. From a userspace point of view there is no process with that PID, so kill() should return ESRCH. In the kernel, I think this means that kill() should actually be looking up tgids rather than pids.
Quoted 3 lines
Quoted 1 line
- is Linux
kill()POSIX compliant in this regard?Does posix say that a process can't be allocated multiple PIDs?
PID="process ID"
You have one PID per process.
Quoted 5 lines
Quoted 3 lines
- do we want to limit the
sched_setaffinity()functionality to correspond to its documentation, or do we want to update the documentation so that its covers all the functionality?I believe Linux currently implements threads as seperate processes
No, they are implemented as separately schedulable entities with lots of shared state. "process" and "thread" are POSIX terms that don't really mean anything in the kernel.
Quoted 4 lines
Now given linux runs threads as seperate processes, it makes sense that thread ids and process ids are the same thing and hence currently unique, and that kill would work on any thread's pid within a given process.
Pthreads define signal handling. Signals are delivered to the process as a whole, not to any particular thread. If you specify a TID that is not a valid PID, then the kernel should return an error.
Quoted 4 lines
Doesn't
sched_setaffinitydo what it says it will? Since each thread is treated as a process thensched_setaffinityshould work on it I would think since it is a process after all as far as the scheduler is concerned.
If the syscall is supposed to operate on processes, it should operate on all threads within a process. It would be nice to have a way to specify affinity for threads. POSIX doesn't define one though.
Chris
Quoted 3 lines · Chris Friesen
Doesn't matter. From a userspace point of view there is no process with that PID, so
kill()should return ESRCH. In the kernel, I think this means thatkill()should actually be looking up tgids rather than pids.
If you look in the list of processes running, you WILL see that PID in the list. ERSCH should only be returned if you ask for a thread that either never existed or doesn't exist anymore. Since a thread is a process to the kernel (at least as far as cheduling and PIDs are concerned) you can send a kill to the thread, which will probably be sent to the parent process id instead.
Quoted 3 lines
PID="process ID"
You have one PID per process.
No, you have at least one PID per process. I have never heard anyone claim before that implementing threads as extra processes in the kernel is violating posix. It sure makes the scheduler simpler to implement. Much more efficient than user space threading.
Quoted 3 lines
No, they are implemented as separately schedulable entities with lots of shared state. "process" and "thread" are POSIX terms that don't really mean anything in the kernel.
Certainly process and thread does have meanings in the kernel.
Quoted 3 lines
Pthreads define signal handling. Signals are delivered to the process as a whole, not to any particular thread. If you specify a TID that is not a valid PID, then the kernel should return an error.
Well as long as the kernel send the signals sent to any of the PIDs of a multithreaded process, to that process, then that seems fine to me.
Quoted 3 lines
If the syscall is supposed to operate on processes, it should operate on all threads within a process. It would be nice to have a way to specify affinity for threads. POSIX doesn't define one though.
Hmm, well I guess the current way it works you can set the affinity per thread since you had a PID per thread to operate on. If you want to do it for the whole process, perhaps if you set it on the starting thread before it creates more threads they would probably inherit the affinity of the original thread.
Have you tried NPTL (native posix threading library) which is supposed to become the threading standard on linux in the future (if it works out)? I was under the impression amd64 systems with 2.6 kernels at least tend to use that by default, but I might be remembering something else unrelated. I wonder if NPTL doesn't do more the way you want than the way linuxthreads have worked so far.
Len Sorensen
Quoted 6 lines · Lennart Sorensen
I believe Linux currently implements threads as seperate processes (at least top and ps sees them that way).
Have you tried NPTL (native posix threading library) which is supposed to become the threading standard on linux in the future (if it works out)?
Lennart,
From the beginning we are talking about the present GNU/Linux systems, which do already use NTPL in standard. NPTL is no future standard, it is present standard.
This means basicly that 50% of your assertions (like the above) are wrong, and your conclusions "suffer" from that :)
The point is that if you make a ps on a decent Linux based system, you will *NOT* see one process for each thread. Nor they do appear in /proc.
This means there are *NOT* userland processes.
And therefore, you shall *NOT* be able to reference them as such where a process ID is required.
Quoted 3 lines · Olivier Croquette
From the beginning we are talking about the present GNU/Linux systems, which do already use NTPL in standard. NPTL is no future standard, it is present standard.
Hmm, I just noticed the page I found about NPTL had 2005 date one place and 2002 in another. Yeah that is what people are using already.
Most current i386 systems do NOT use NPTL yet by default since it only works on 2.6 kernels, and even then probably only if glibc was compiled that way.
Quoted 10 lines
This means basicly that 50% of your assertions (like the above) are wrong, and your conclusions "suffer" from that :)
The point is that if you make a ps on a decent Linux based system, you will *NOT* see one process for each thread. Nor they do appear in /proc.
This means there are *NOT* userland processes.
And therefore, you shall *NOT* be able to reference them as such where a process ID is required.
Hmm, I just ran a pthread program and every thread shows up as a process in ps. Of course that is on a system with a 2.4 kernel compatible glibc, so that is probably not a valid test. Running on 2.6.11 on an amd64 with glibc compiled for 2.6 kernels only, I only see one pid for the multithreaded program.
Doing kill on the threadid on the amd64 does return ESRCH.
Make sure your test is on a 2.6 kernel system with glibc compiled to only use nptl (so not 2.4 kernel compatible at all). It appears to work as you want it to now.
Len Sorensen
In article <[email protected]>,
Quoted 11 lines · Lennart Sorensen
Quoted 3 lines · Olivier Croquette
From the beginning we are talking about the present GNU/Linux systems, which do already use NTPL in standard. NPTL is no future standard, it is present standard.
Hmm, I just noticed the page I found about NPTL had 2005 date one place and 2002 in another. Yeah that is what people are using already.
Most current i386 systems do NOT use NPTL yet by default since it only works on 2.6 kernels, and even then probably only if glibc was compiled that way.
No. On modern systems, glibc contains both LinuxThreads and NPTL. They have the same ABI. At runtime one of the two is selected, depending on if the currently running kernel supports NTPL. You can also force it by setting the LD_ASSUME_KERNEL environment variable to 2.4 or 2.6.
Mike.
Quoted 5 lines · Miquel van Smoorenburg
No. On modern systems, glibc contains both LinuxThreads and NPTL. They have the same ABI. At runtime one of the two is selected, depending on if the currently running kernel supports NTPL. You can also force it by setting the
LD_ASSUME_KERNELenvironment variable to 2.4 or 2.6.
Well so far my tests show that glibc 2.3.2.ds1-21 on Debian Sarge when running 2.6.11 kernel on i386 uses LinuxThreads, while on amd64 version of Sarge it uses NPTL (and won't run with 2.4 kernel at all either).
Maybe Debian compiled their glibc to not do NPTL on i386 yet. Not sure.
Hmm, after checking, it turns out if you use errno in your program, it drops to linuxthreads, while using #include <errno.h> makes it able to use NPTL when using 2.6 kernel. Now my program works the same on i386 as on amd64 (I had to fix the errno to make it run on amd64 so that does make some sense). Well I learned something new.
Len Sorensen
On 20 May 2005, Lennart Sorensen prattled cheerily:
Quoted 1 line
Maybe Debian compiled their glibc to not do NPTL on i386 yet. Not sure.
This is not the case. Proof from ps -FT output:
mysql 8473 8473 8472 0 29110 14056 0 May22 pts/1 /usr/sbin/mysqld mysql 8473 8475 8472 0 29110 14056 0 May22 pts/1 /usr/sbin/mysqld mysql 8473 8476 8472 0 29110 14056 0 May22 pts/1 /usr/sbin/mysqld mysql 8473 8477 8472 0 29110 14056 0 May22 pts/1 /usr/sbin/mysqld mysql 8473 8478 8472 0 29110 14056 0 May22 pts/1 /usr/sbin/mysqld mysql 8473 8479 8472 0 29110 14056 0 May22 pts/1 /usr/sbin/mysqld mysql 8473 8480 8472 0 29110 14056 0 May22 pts/1 /usr/sbin/mysqld mysql 8473 8481 8472 0 29110 14056 0 May22 pts/1 /usr/sbin/mysqld mysql 8473 8482 8472 0 29110 14056 0 May22 pts/1 /usr/sbin/mysqld
Quoted 3 lines
Hmm, after checking, it turns out if you use errno in your program, it drops to linuxthreads, while using #include <
errno.h> makes it able to use NPTL when using 2.6 kernel.
This is a distribution-specific patch. glibc as shipped by the FSF simply refuses to run programs that reference the errno symbol: errno is no longer an exported symbol at all. (This is reasonable, as such programs would fail to work on a multithreaded NPTL program in any case.)
The only valid way to gain access to the errno symbol is to #include <errno.h>. This has been true for as long as glibc2 has existed.
Signature · 2 lines
`Once again, I must remark on the far-reaching extent of my ladylike nature.' --- Rosie Taylor
Quoted 5 lines · Miquel van Smoorenburg
No. On modern systems, glibc contains both LinuxThreads and NPTL. They have the same ABI. At runtime one of the two is selected, depending on if the currently running kernel supports NTPL. You can also force it by setting the
LD_ASSUME_KERNELenvironment variable to 2.4 or 2.6.
More info about that from:
http://linuxdevices.com/articles/AT6753699732.html
Some Linux vendors, such as later versions of Red Hat Linux, have backported NPTL to earlier kernels and have even made the threading environment for specific processes selectable through an environment variable (LD_ASSUME_KERNEL). On systems that support this feature, the variable is set via a command such as the following:
# export LD_ASSUME_KERNEL=2.4.1 This is a clever way to enable some existing applications that rely on LinuxThreads to continue to work in an NPTL environment, but is a short-term solution. To make the most of the design and performance benefits provided by NPTL, you should update the code for any existing applications that use threading.
[...]
Simply using a 2.6 based kernel does not mean that you are automatically using the NPTL. To determine the threading library that a system uses, you can execute the getconf command (part of the glibc package), to examine the GNU_LIBPTHREAD_VERSION environment variable, as in the following command example:
# getconf GNU_LIBPTHREAD_VERSION linuxthreads-0.10 If your system uses the NPTL, the command would return the value of NPTL that your system was using, as in the following example: # getconf GNU_LIBPTHREAD_VERSION nptl-0.60
Quoted 6 lines · Olivier Croquette
# export LD_ASSUME_KERNEL=2.4.1 This is a clever way to enable some existing applications that rely on LinuxThreads to continue to work in an NPTL environment, but is a short-term solution. To make the most of the design and performance benefits provided by NPTL, you should update the code for any existing applications that use threading.
Applications that rely on linuxthreads, heh, that's a good one.
The most common use of LD_ASSUME_KERNEL is to force Linuxthreads to be used in order to work around a bad bug in NPTL 0.60, often present on Debian systems. Ubuntu still reports NPTL 0.60, but they at least fixed the bug for the Hoary release. The Debian people refuse to.
The issue is very well known to JACK users.
Lee