Linux 2.4 VS 2.6 fork VS thread creation time test
Subsystems (3)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello everyone,
Today morning I've made a test about the thead and child process creation(fork) time on both 2.4 and 2.6 kernels.
The test systems
================
Box A:
- kernel: Linux 2.4.22-xfs
- CPU: Intel P3-700 MHz (slot1)
- Ram: 384MB SDR SDRAM
Box B:
- kernel: Linux 2.6.6-mm5
- CPU: Intel P4-2.6Ghz
- Ram: 512 DDR SDRAM
Box B is a lot better, but it doesn't metter according to the test, because the aim was to get the ratio of the two times with different sample rates.
The sample rates
================
I've chosen a low, a middle and a higher sample rate for the test, they was 20,128 and 255 samples.
Notice in the attached source that every child processes and threads are terminated right after creation, and only the creation function call is timed. It was interesting that when I've chosed a _very_ high sample rate, about 10^8 there was a lot of failures, the kernel could only create about 4K child processes, and after this all the thread creations failed. As I told it above all the processes are teminated right after creation, but there were a lot of defunct processes in the system, and they were only gone when the parent termineted.
With a few number of processes I wasn't able to go over 255 threads, after the 255th every creation attempt simply failed.
The test outputs
================
Box A, 20 samples:
- ------------------
Fork failures: 0 (success: 20)
Avarage fork time: 0.326800 msecs
Total fork time: 6.536000 msecs
Thread failures: 0 (success: 20)
Avarage thread time: 0.075550 msecs
Total thread time: 1.511000 msecs
Avarage fork/thread ratio: 4.325612
Total fork/thread ratio: 4.325612
Box A, 128 samples:
- ------------------
Fork failures: 0 (success: 128)
Avarage fork time: 0.332773 msecs
Total fork time: 42.595000 msecs
Thread failures: 0 (success: 128)
Avarage thread time: 0.084859 msecs
Total thread time: 10.862000 msecs
Avarage fork/thread ratio: 3.921469
Total fork/thread ratio: 3.921469
Box A, 255 samples:
- -------------------
Fork failures: 0 (success: 255)
Avarage fork time: 0.344827 msecs
Total fork time: 87.931000 msecs
Thread failures: 0 (success: 255)
Avarage thread time: 0.209718 msecs
Total thread time: 53.478000 msecs
Avarage fork/thread ratio: 1.644246
Total fork/thread ratio: 1.644246
Box B, 20 samples:
- ------------------
Fork failures: 0 (success: 20)
Avarage fork time: 0.257900 msecs
Total fork time: 5.158000 msecs
Thread failures: 0 (success: 20)
Avarage thread time: 0.026850 msecs
Total thread time: 0.537000 msecs
Avarage fork/thread ratio: 9.605214
Total fork/thread ratio: 9.605214
Box B, 128 samples:
- -------------------
Fork failures: 0 (success: 128)
Avarage fork time: 0.242633 msecs
Total fork time: 31.057000 msecs
Thread failures: 0 (success: 128)
Avarage thread time: 0.025859 msecs
Total thread time: 3.310000 msecs
Avarage fork/thread ratio: 9.382779
Total fork/thread ratio: 9.382779
Box B, 255 samples:
- -------------------
Fork failures: 0 (success: 255)
Avarage fork time: 0.241678 msecs
Total fork time: 61.628000 msecs
Thread failures: 0 (success: 255)
Avarage thread time: 0.025729 msecs
Total thread time: 6.561000 msecs
Avarage fork/thread ratio: 9.393080
Total fork/thread ratio: 9.393080
Conlusion
=========
It's easy to notice that in case of 2.4 the ratios of the creation times are converges to 1, so it depends on the load, while in case of a 2.6 kernel the ratios are mostly fix, about 9. This means that creating a new child process takes much more time than creating a new thread.
Bye,
Gergely Czuczy
mailto: [email protected]
PGP: http://phoemix.harmless.hu/phoemix.pgp
"Wish a god, a star to believe in,
With the realm of king of fantasy..."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFAsFmObBsEN0U7BV0RAiawAJ93KmDqYwksMNGci11hOdLbr2rXWwCfdcZR cv1WcImKLpNOfNr9pdqyfm0=
=2yxd
-----END PGP SIGNATURE-----
/*
Test how much time is needed for a process/thread creation
Author: Gergely Czuczy <[email protected]>
*/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#include <time.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/time.h>
double timeval2double(struct timeval *_tv);
void *threadfunc(void *);
/*
================================
int main(int argc, char *argv[])
================================
*/
int main(int argc, char *argv[]) {
unsigned int i;
unsigned int samples;
unsigned int forkfailures(0), threadfailures(0); double totalforktime(0),totalthreadtime(0), forktime, threadtime; struct timeval begin,end,diff;
pid_t childpid;
pthread_t thread;
if ( argc != 2 ) {
printf("Usage:\n\t%s <number of samples>\n", argv[0]);
return EXIT_FAILURE;
} // we get the number of samples to take
samples = atoi(argv[1]);
// test the forks
printf("\n");
for ( i=0; i<samples; ++i ) {
printf("\rFork test: %u/%u (%3.2f%%)", i+1, samples, ((i+1.0)/samples)*100);fflush(0); gettimeofday(&begin, 0);
childpid = fork();
gettimeofday(&end, 0);
if ( !childpid ) _exit(0);
if ( childpid==-1 ) ++forkfailures;
timersub(&end, &begin, &diff);
totalforktime += timeval2double(&diff);
usleep(1); }
printf("\n");
usleep(10000);
// test the threads
printf("\n");
for ( i=0; i<samples; ++i ) {
int ret; printf("\rThread test: %u/%u (%3.2f%%)", i+1, samples, ((i+1.0)/samples)*100);fflush(0);
gettimeofday(&begin, 0);
ret = pthread_create(&thread, 0, &threadfunc, 0);
gettimeofday(&end, 0);
if ( ret ) ++threadfailures;
timersub(&end, &begin, &diff);
totalthreadtime += timeval2double(&diff);
usleep(1); }
printf("\n\n");
// print the results
forktime = totalforktime / (samples-forkfailures);
printf("Fork failures: %u (succes: %u)\n", forkfailures, samples-forkfailures);
printf("Avarage fork time: %5f msecs\n", forktime);
printf("Total fork time: %5f msecs\n", totalforktime);
threadtime = totalthreadtime / (samples-threadfailures);
printf("Thread failures: %u (success: %u)\n", threadfailures, samples-threadfailures);
printf("Avarage thread time: %5f msecs\n", threadtime);
printf("Total thread time: %5f msecs\n", totalthreadtime);
printf("Avarage fork/thread ratio: %2f\n", forktime/threadtime);
printf("Total fork/thread ratio: %2f\n", totalforktime/totalthreadtime); return EXIT_SUCCESS;
}
/*
============================
void *threadfunc(void *_arg)
============================
*/
void *threadfunc(void *) {
pthread_exit(0);
return 0;
}
/*
==========================================
double timeval2double(struct timeval *_tv)
==========================================
*/
double timeval2double(struct timeval *_tv) {
double ret;
ret = _tv->tv_usec;
ret /= 1000000;
ret += _tv->tv_sec;
ret *= 1000;
return ret;
}
Quoted 16 lines · Gergely Czuczy
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Hello everyone,
Today morning I've made a test about the thead and child process creation(fork) time on both 2.4 and 2.6 kernels.
The test systems
================Box A:
- kernel: Linux 2.4.22-xfs
- CPU: Intel P3-700 MHz (slot1)
- Ram: 384MB SDR SDRAM
Quoted 9 lines
Box B:
- kernel: Linux 2.6.6-mm5
- CPU: Intel P4-2.6Ghz
- Ram: 512 DDR SDRAMBox B is a lot better, but it doesn't metter according to the test, because the aim was to get the ratio of the two times with different sample rates.
Even so, you really should be using the same system if you want comparable results. The pentium 4 in particular can do worse at specific things in microbenchmarks.
Your problem with not being able large numbers of threads and processes could be ulimits or sysctl limits. Look at ulimit and /proc/sys/kernel/threads-max and pid_max.
[ snip numbers ]
Quoted 8 lines
Conlusion
=========It's easy to notice that in case of 2.4 the ratios of the creation times are converges to 1, so it depends on the load, while in case of a 2.6 kernel the ratios are mostly fix, about 9. This means that creating a new child process takes much more time than creating a new thread.
I tried your code on a P3-1000. 2.4.23 is not 100% comparable because it was built with an old compiler and possibly different options.
Anyway, results with 256 samples were as follows:
processes 256
2.4.23:
Total fork time: 48.224 msecs
Total thread time: 30.180 msecs
Total fork/thread ratio: 1.598
2.6.6-mm5:
Total fork time: 40.795 msecs
Total thread time: 17.615 msecs
Total fork/thread ratio: 2.316
2.6.6-mm5+child-runs-last:
Total fork time: 16.080 msecs
Total thread time: 8.600 msecs
Total fork/thread ratio: 1.870
2.6 introduced an optimisation called child-runs-first. Apparently it is nice for many common things, but it also causes a context switch at every fork(), so sucks for these microbenchmarks. Child- runs-last simply reverts that. Patch is attached if anyone is interested.
linux-2.6-npiggin/kernel/fork.c | 7 +------ linux-2.6-npiggin/kernel/sched.c | 20 ++------------------ 2 files changed, 3 insertions(+), 24 deletions(-)
kernel/sched.c renamed from kernel/sched.c~no-child-run-first +2 -18
@@ -1073,15 +1073,7 @@ void fastcall wake_up_forked_process(tas p->prio =effective_prio (p);set_task_cpu (p,smp_processor_id ()); -if (unlikely (!current->array)) -__activate_task (p, rq); -else { - p->prio = current->prio; -list_add_tail (&p->run_list, ¤t->run_list); - p->array = current->array; - p->array->nr_active++; - rq->nr_running++; - } +__activate_task (p, rq);task_rq_unlock (rq, &flags); }
@@ -1395,15 +1387,7 @@ lock_again:set_task_cpu (p, cpu);if (cpu == this_cpu) { -if (unlikely (!current->array)) -__activate_task (p, rq); -else { - p->prio = current->prio; -list_add_tail (&p->run_list, ¤t->run_list); - p->array = current->array; - p->array->nr_active++; - rq->nr_running++; - } +__activate_task (p, rq); }else {schedstat_inc (sd, sbc_pushed);/* Not the local CPU - must adjust timestamp */
kernel/fork.c renamed from kernel/fork.c~no-child-run-first +1 -6
@@ -1239,12 +1239,7 @@ long do_fork(unsigned long clone_flags,wait_for_completion (&vfork);if (unlikely (current->ptrace &PT_TRACE_VFORK_DONE ))ptrace_notify ((PTRACE_EVENT_VFORK_DONE <<8 ) |SIGTRAP ); - }else -/* -* Let the child process run first, to avoid most of the -* COW overhead when the child exec()s afterwards. -*/ -set_need_resched (); + } }return pid; }
_
Quoted 22 lines · Nick Piggin
Quoted 21 lines · Gergely Czuczy
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Hello everyone,
Today morning I've made a test about the thead and child process creation(fork) time on both 2.4 and 2.6 kernels.
The test systems
================Box A:
- kernel: Linux 2.4.22-xfs
- CPU: Intel P3-700 MHz (slot1)
- Ram: 384MB SDR SDRAMBox B:
- kernel: Linux 2.6.6-mm5
- CPU: Intel P4-2.6Ghz
- Ram: 512 DDR SDRAM
Quoted 8 lines
Quoted 4 lines
Box B is a lot better, but it doesn't metter according to the test, because the aim was to get the ratio of the two times with different sample rates.
Even so, you really should be using the same system if you want comparable results. The pentium 4 in particular can do worse at specific things in microbenchmarks.
It doesn't matter. That's why you should look at the "ratio" at the end and not on the pure numbers, they are just bonus "information"
Quoted 2 lines
Your problem with not being able large numbers of threads and
ohh wait, i don't have problems, i've just noticed the things!
Quoted 2 lines
processes could be ulimits or sysctl limits. Look at ulimit and
/proc/sys/kernel/threads-maxandpid_max.
threads-max is about 8179. this is why I don't see why it is limited to 255 threads.
Quoted 27 lines
[ snip numbers ]
Quoted 8 lines
Conlusion
=========It's easy to notice that in case of 2.4 the ratios of the creation times are converges to 1, so it depends on the load, while in case of a 2.6 kernel the ratios are mostly fix, about 9. This means that creating a new child process takes much more time than creating a new thread.
I tried your code on a P3-1000. 2.4.23 is not 100% comparable because it was built with an old compiler and possibly different options.
Anyway, results with 256 samples were as follows:
processes 256
2.4.23:
Total fork time: 48.224 msecs
Total thread time: 30.180 msecs
Total fork/thread ratio: 1.5982.6.6-mm5:
Total fork time: 40.795 msecs
Total thread time: 17.615 msecs
Total fork/thread ratio: 2.316
Quoted 4 lines
2.6.6-mm5+child-runs-last:
Total fork time: 16.080 msecs
Total thread time: 8.600 msecs
Total fork/thread ratio: 1.870
This is very nice, but notice the difference between the total and the avarage time. Total time includes all the time that taken by the test, and The avarage/relative time reflect he time taken by the successive calls, eg it not includes failed pthread_create()s and fork()s, which ones makes the test false. After posting the test I've realized that the measurement of the realitve time is not good, so I will patch this problem.
Quoted 7 lines
2.6 introduced an optimisation called child-runs-first. Apparently it is nice for many common things, but it also causes a context switch at every
fork(), so sucks for these microbenchmarks. Child- runs-last simply reverts that. Patch is attached if anyone is interested.
Bye,
Gergely Czuczy
mailto: [email protected]
PGP: http://phoemix.harmless.hu/phoemix.pgp
"Wish a god, a star, to believe in,
With the realm of king of fantasy..."
Quoted 11 lines · Gergely Czuczy
Quoted 8 lines · Nick Piggin
Quoted 4 lines · Gergely Czuczy
Box B is a lot better, but it doesn't metter according to the test, because the aim was to get the ratio of the two times with different sample rates.
Quoted 7 lines
Quoted 3 lines
Even so, you really should be using the same system if you want comparable results. The pentium 4 in particular can do worse at specific things in microbenchmarks.
It doesn't matter. That's why you should look at the "ratio" at the end and not on the pure numbers, they are just bonus "information"
I understand that... but the P4 might execute the fork test relatively slower than the thread test compared to the P3.
Different CPU architectures, cache types and sizes, memory to CPU speeds, etc. might all have some influcene on the ratio.
Quoted 3 lines
It doesn't matter. That's why you should look at the "ratio" at the end and not on the pure numbers, they are just bonus "information"
Your experimental method leaves a _lot_ to be desired. It matters a great deal. eg It would be worth reading up how wildly different atomic operations are on different x86 chips:
http://www.labs.fujitsu.com/en/techinfo/linux/lse-0211/lse-0211.pdf
Anton
Quoted 3 lines · Gergely Czuczy
failed. As I told it above all the processes are teminated right after creation, but there were a lot of defunct processes in the system, and they were only gone when the parent termineted.
Have you heard of wait, waitpid and pthread_join?
Quoted 2 lines
With a few number of processes I wasn't able to go over 255 threads, after the 255th every creation attempt simply failed.
Your 255 thread limit is propably because you have a stack size of 8192 kbytes. (see ulimit). As all threads share the same address space, this address space is the limiting factor. Try ulimit -s 1024 for example.
Quoted 4 lines
It's easy to notice that in case of 2.4 the ratios of the creation times are converges to 1, so it depends on the load, while in case of a 2.6 kernel the ratios are mostly fix, about 9. This means that creating a new child process takes much more time than creating a new thread.
Well, the other way around is the correct answer. Processes didnt get slower. Threads are faster than processes in 2.6 because of the NPTL. If you want to slow down threads to process level, just do LD_ASSUME_KERNEL=2.2.5 before running your test program.
cheers
Christian
Quoted 6 lines · Christian Borntraeger
Date: Sun, 23 May 2004 11:39:41 +0200
From: Christian Borntraeger <[email protected]>
To: [email protected]
Cc: Gergely Czuczy <[email protected]>, [email protected]
Subject: Re: Linux 2.4 VS 2.6 fork VS thread creation time test
Quoted 6 lines
Quoted 3 lines · Gergely Czuczy
failed. As I told it above all the processes are teminated right after creation, but there were a lot of defunct processes in the system, and they were only gone when the parent termineted.
Have you heard of wait, waitpid and
pthread_join?
there really is some sort of problem with 2.6.6 in this area. I have an app that I am trying to stress test on a dual opteron system and under a heavy load something goes haywire and the children become zombies. on a dual athlon the test manages 2500 forks/sec and can continue forever (Ok, I only tested it to 11M forks at full speed :-), but the dual opteron box manages 3500 connections/sec for a few thousand connections and then stops reaping the children. if I attach strace to the parent at this point the logjam is broken and strace shows the wait calls receiving and handleing the sigchild
the prarent deals with sigchild by
handler{
while ( wait(...) >0);signal(SIGCHLD, handler);
}
unfortunantly trying to leave strace attached while running the test slows it down to ~1200 forks/sec and the problem never forms.
David Lang
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 7 lines · David Lang
Quoted 5 lines · Christian Borntraeger
Quoted 3 lines · Gergely Czuczy
failed. As I told it above all the processes are teminated right after creation, but there were a lot of defunct processes in the system, and they were only gone when the parent termineted.
Have you heard of wait, waitpid and
pthread_join?there really is some sort of problem with 2.6.6 in this area. I have an
Well in the example given by Gergely there was no wait call at all. Therefore I believe your problem is not related to his one.
What do you mean by with 2.6.6. Does this testcase behaves differently with other kernel versions? Which version is the first with this problem?
Quoted 5 lines
the prarent deals with sigchild by
handler{
while ( wait(...) >0);signal(SIGCHLD, handler);
}
You run signal within the signal handler. This is not necessary, although this should cause no problems. Nevertheless, can you try your test without signal in the signal handler?
cheers
Christian
Quoted 11 lines · Christian Borntraeger
Quoted 7 lines · David Lang
Quoted 5 lines · Christian Borntraeger
Quoted 3 lines · Gergely Czuczy
failed. As I told it above all the processes are teminated right after creation, but there were a lot of defunct processes in the system, and they were only gone when the parent termineted.
Have you heard of wait, waitpid and
pthread_join?there really is some sort of problem with 2.6.6 in this area. I have an
Well in the example given by Gergely there was no wait call at all. Therefore I believe your problem is not related to his one.
Ok, very possible
Quoted 2 lines
What do you mean by with 2.6.6. Does this testcase behaves differently with other kernel versions? Which version is the first with this problem?
I started doing this testing with 2.6.4 and had problems with 2.6.[456], but only on the opterons
Quoted 9 lines
Quoted 5 lines
the prarent deals with sigchild by
handler{
while ( wait(...) >0);signal(SIGCHLD, handler);
}You run signal within the signal handler. This is not necessary, although this should cause no problems. Nevertheless, can you try your test without signal in the signal handler?
I will do this, but as I read the signal man page this will cause signal to be reset after the first signal hits and nothing will set it to handle any future signals, where should the handler get set instead?
Quoted 4 lines
cheers
Christian
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
A message that is not in the archive (sent off-list or not yet received): 1085325156.622.0.camel@boxen
A message that is not in the archive (sent off-list or not yet received): [email protected]
I just found part of a thread from early april about zombie problems in both 2.4 and 2.6
thread titles are
Exploring sigqueue leak (was: Strange 'zombie' problem both in 2.4 and 2.6)
Local DoS (was: Strange 'zombie' problem both in 2.4 and 2.6)
one piece of info that those threads were looking for is the sigqueue info
on my box I get
baseline:/proc/sys/kernel# grep sigqueue /proc/slabinfo sigqueue 22 24 160 24 1 : tunables 120 60 8 : slabdata 1 1 0 baseline:/proc/sys/kernel# cat /proc/sys/kernel/rtsig-* 2048 1
default rtsig-max was 1024, I bumped it up to 2048 but it did not affect the problem
one interesting point is that rtsig-nr is normally 0 but when it is in the error condition that causes all children to become zombies it sits at 1
since my last message didn't go out to linux-kernel and I didn't notice it I am attaching the source for a program that also generates this problem
David Lang
Quoted 23 lines · David Lang
Date: Sun, 23 May 2004 17:46:40 -0700 (PDT)
From: David Lang <[email protected]>
To: Alexander Nyberg <[email protected]>
Subject: Re: Linux 2.4 VS 2.6 fork VS thread creation time testQuoted 7 lines
Quoted 3 lines
unfortunantly trying to leave strace attached while running the test slows it down to ~1200 forks/sec and the problem never forms.
Could you please show me this program? I wrote a small one together myself but I cannot reproduce this problem, maybe because of UP, don't know.
Ok, here is another program that has the same problem.
David Lang
--
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan