NPTL: stack limit limiting number of threads
Hi everybody,
I recently had a chance to benchmark Linux/NPTL vs FreeBSD to
compare thread creation performance on both the systems. Let me not disclose the results as I am sure my methodology is not the best one ;-). However, I did notice something queer which I would like to discuss.
I noticed that on my system (which has a 3G:1G split linux-2.6.1
NPTL gentoo), with the (possibly default) stack size limit of 8M, I couldn't create more than around 380 threads in a process. On searching the web, I found that a reason for this could be that in 3G of userspace, there is space for exactly 384 8M chunks of memory. Thus, because it is not possible to create more than 384 thread stacks in the user space, the NPTL disallows creation of threads greater than around 380. When I reduced the stack limit (using ulimit in bash), I could create enough number of threads for my benchmarks to run properly.
However, I was most amazed to see that the limit on stack size on FreeBSD (5.3 Release) was 64M by default! I was just wondering, how is FreeBSD able to create about a 1000 threads with that kind of a stack limit. Also, is there anything specific in its implementation which makes it difficult to incorporate in Linux? Wouldn't it be a good idea to remove this "trade-off" between stack limit and number of threads and fail thread creation only when we have run out of address space being *actually used* in the stacks in a process.
Ritesh
Signature · 1 line
http://www.cs.unc.edu/~ritesh/
Quoted 8 lines · Ritesh Kumar
However, I was most amazed to see that the limit on stack size on FreeBSD (5.3 Release) was 64M by default! I was just wondering, how is FreeBSD able to create about a 1000 threads with that kind of a stack limit. Also, is there anything specific in its implementation which makes it difficult to incorporate in Linux? Wouldn't it be a good idea to remove this "trade-off" between stack limit and number of threads and fail thread creation only when we have run out of address space being *actually used* in the stacks in a process.
On FreeBSD the default thread stack size is not computed from ulimit -s, but is constant. They apparently only recently increased it to 1MB (resp. 2MB on 64-bit arches), from 64K.
On Linux, the default thread stack size (except with fixed stack LinuxThreads) is determined from ulimit -s (with a constant default if ulimit -s is unlimited).
If your threaded application has specific needs for stack sizes, it can always pthread_attr_setstacksize to whatever you find appropriate.
The thread library needs to know the stack size limit before creating the thread, that can't be changed dynamically.
Jakub
Quoted 18 lines · Jakub Jelinek
Quoted 8 lines · Ritesh Kumar
However, I was most amazed to see that the limit on stack size on FreeBSD (5.3 Release) was 64M by default! I was just wondering, how is FreeBSD able to create about a 1000 threads with that kind of a stack limit. Also, is there anything specific in its implementation which makes it difficult to incorporate in Linux? Wouldn't it be a good idea to remove this "trade-off" between stack limit and number of threads and fail thread creation only when we have run out of address space being *actually used* in the stacks in a process.
On FreeBSD the default thread stack size is not computed from ulimit -s, but is constant. They apparently only recently increased it to 1MB (resp. 2MB on 64-bit arches), from 64K.
On Linux, the default thread stack size (except with fixed stack LinuxThreads) is determined from ulimit -s (with a constant default if ulimit -s is unlimited).
Quoted 8 lines
If your threaded application has specific needs for stack sizes, it can always
pthread_attr_setstacksizeto whatever you find appropriate.The thread library needs to know the stack size limit before creating the thread, that can't be changed dynamically.
Jakub
Hi Jakub,
Thanks for your reply. I actually went ahead after getting your
mail and coded up a small program to check the stack limit deliberately. The program is shown inline.
#include <stdio.h>
#define BUF_SIZE 1024000
void recurse(int n){
char ch[BUF_SIZE];
if(n<=0)
return;
else
recurse(n-1);
}int main(argc, argv)
int argc;
char **argv;
{
if(argc!=2){
printf("Usage: %s <n (megabytes)>\n", argv[0]);
return 1;
}
printf("Checking for %dMB\n", atoi(argv[1]));
recurse(atoi(argv[1]));
}Its a fairly crude way to find out the actual stack limit. Basically, the resurse function recurses each time allocating ~1MB of space on the stack. The program segfaults exactly at the ulimit -s value of stack size on both linux and freebsd. So it does seem that the ulimit -s is the value of stack limit used on FreeBSD.
Ritesh
Signature · 1 line
http://www.cs.unc.edu/~ritesh/
Quoted 28 lines · Ritesh Kumar
Thanks for your reply. I actually went ahead after getting your
mail and coded up a small program to check the stack limit deliberately. The program is shown inline.#include <
stdio.h>#define
BUF_SIZE1024000void recurse(int n){ char ch[BUF_SIZE]; if(n<=0) return; else recurse(n-1); }int
main(argc, argv)int argc; char **argv; { if(argc!=2){ printf("Usage: %s <n (megabytes)>\n", argv[0]); return 1; } printf("Checking for %dMB\n", atoi(argv[1])); recurse(atoi(argv[1])); }
Quoted 5 lines
Its a fairly crude way to find out the actual stack limit. Basically, the resurse function recurses each time allocating ~1MB of space on the stack. The program segfaults exactly at the ulimit -s value of stack size on both linux and freebsd. So it does seem that the ulimit -s is the value of stack limit used on FreeBSD.
For the main stack sure. But now try to call that recurse in some other thread.
Jakub
Quoted 35 lines · Jakub Jelinek
Quoted 28 lines · Ritesh Kumar
Thanks for your reply. I actually went ahead after getting your
mail and coded up a small program to check the stack limit deliberately. The program is shown inline.#include <
stdio.h>#define
BUF_SIZE1024000void recurse(int n){ char ch[BUF_SIZE]; if(n<=0) return; else recurse(n-1); }int
main(argc, argv)int argc; char **argv; { if(argc!=2){ printf("Usage: %s <n (megabytes)>\n", argv[0]); return 1; } printf("Checking for %dMB\n", atoi(argv[1])); recurse(atoi(argv[1])); }Quoted 5 lines
Its a fairly crude way to find out the actual stack limit. Basically, the resurse function recurses each time allocating ~1MB of space on the stack. The program segfaults exactly at the ulimit -s value of stack size on both linux and freebsd. So it does seem that the ulimit -s is the value of stack limit used on FreeBSD.
Quoted 5 lines
For the main stack sure. But now try to call that recurse in some other thread.
Jakub
Oh great... I see your point now. I called the same recurse in an alternative thread and found the limit to be 8M on Linux ( and Mac OS X (Panther)) but around 1MB on FreeBSD. Thanks for the clarification!
Ritesh
Signature · 1 line
http://www.cs.unc.edu/~ritesh/