Skip to contentSkip to conversation

Linux 2.4 VS 2.6 fork VS thread creation time test

DISCUSSION Gergely Czuczy 6 participants 10 messages May 23, 2004 – May 24, 2004 2 messages not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello everyone,

Today morning I've made a test about the thead and child process creation(fork) time on both 2.4 and 2.6 kernels.

The test systems
================

Box A:
- kernel: Linux 2.4.22-xfs
- CPU: Intel P3-700 MHz (slot1)
- Ram: 384MB SDR SDRAM

Box B:
- kernel: Linux 2.6.6-mm5
- CPU: Intel P4-2.6Ghz
- Ram: 512 DDR SDRAM

Box B is a lot better, but it doesn't metter according to the test, because the aim was to get the ratio of the two times with different sample rates.

The sample rates
================

I've chosen a low, a middle and a higher sample rate for the test, they was 20,128 and 255 samples.
Notice in the attached source that every child processes and threads are terminated right after creation, and only the creation function call is timed. It was interesting that when I've chosed a _very_ high sample rate, about 10^8 there was a lot of failures, the kernel could only create about 4K child processes, and after this all the thread creations failed. As I told it above all the processes are teminated right after creation, but there were a lot of defunct processes in the system, and they were only gone when the parent termineted.
With a few number of processes I wasn't able to go over 255 threads, after the 255th every creation attempt simply failed.

The test outputs
================

Box A, 20 samples:
- ------------------
Fork failures: 0 (success: 20)
Avarage fork time: 0.326800 msecs
Total fork time: 6.536000 msecs
Thread failures: 0 (success: 20)
Avarage thread time: 0.075550 msecs
Total thread time: 1.511000 msecs
Avarage fork/thread ratio: 4.325612
Total fork/thread ratio: 4.325612

Box A, 128 samples:
- ------------------
Fork failures: 0 (success: 128)
Avarage fork time: 0.332773 msecs
Total fork time: 42.595000 msecs
Thread failures: 0 (success: 128)
Avarage thread time: 0.084859 msecs
Total thread time: 10.862000 msecs
Avarage fork/thread ratio: 3.921469
Total fork/thread ratio: 3.921469

Box A, 255 samples:
- -------------------
Fork failures: 0 (success: 255)
Avarage fork time: 0.344827 msecs
Total fork time: 87.931000 msecs
Thread failures: 0 (success: 255)
Avarage thread time: 0.209718 msecs
Total thread time: 53.478000 msecs
Avarage fork/thread ratio: 1.644246
Total fork/thread ratio: 1.644246

Box B, 20 samples:
- ------------------
Fork failures: 0 (success: 20)
Avarage fork time: 0.257900 msecs
Total fork time: 5.158000 msecs
Thread failures: 0 (success: 20)
Avarage thread time: 0.026850 msecs
Total thread time: 0.537000 msecs
Avarage fork/thread ratio: 9.605214
Total fork/thread ratio: 9.605214

Box B, 128 samples:
- -------------------
Fork failures: 0 (success: 128)
Avarage fork time: 0.242633 msecs
Total fork time: 31.057000 msecs
Thread failures: 0 (success: 128)
Avarage thread time: 0.025859 msecs
Total thread time: 3.310000 msecs
Avarage fork/thread ratio: 9.382779
Total fork/thread ratio: 9.382779

Box B, 255 samples:
- -------------------
Fork failures: 0 (success: 255)
Avarage fork time: 0.241678 msecs
Total fork time: 61.628000 msecs
Thread failures: 0 (success: 255)
Avarage thread time: 0.025729 msecs
Total thread time: 6.561000 msecs
Avarage fork/thread ratio: 9.393080
Total fork/thread ratio: 9.393080

Conlusion
=========

It's easy to notice that in case of 2.4 the ratios of the creation times are converges to 1, so it depends on the load, while in case of a 2.6 kernel the ratios are mostly fix, about 9. This means that creating a new child process takes much more time than creating a new thread.

Bye,

Gergely Czuczy
mailto: [email protected]
PGP: http://phoemix.harmless.hu/phoemix.pgp

"Wish a god, a star to believe in,
With the realm of king of fantasy..."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFAsFmObBsEN0U7BV0RAiawAJ93KmDqYwksMNGci11hOdLbr2rXWwCfdcZR cv1WcImKLpNOfNr9pdqyfm0=
=2yxd
-----END PGP SIGNATURE-----

/*
Test how much time is needed for a process/thread creation
Author: Gergely Czuczy <[email protected]>
*/

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#include <time.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/time.h>

double timeval2double(struct timeval *_tv);
void *threadfunc(void *);

/*
================================
int main(int argc, char *argv[])
================================
*/
int main(int argc, char *argv[]) {
unsigned int i;
unsigned int samples;
unsigned int forkfailures(0), threadfailures(0); double totalforktime(0),totalthreadtime(0), forktime, threadtime; struct timeval begin,end,diff;
pid_t childpid;
pthread_t thread;

  if ( argc != 2 ) {
    printf("Usage:\n\t%s <number of samples>\n", argv[0]);
    return EXIT_FAILURE;
  }

// we get the number of samples to take
samples = atoi(argv[1]);

  // test the forks
  printf("\n");
  for ( i=0; i<samples; ++i ) {
    printf("\rFork test: %u/%u (%3.2f%%)", i+1, samples, ((i+1.0)/samples)*100);fflush(0);
    gettimeofday(&begin, 0);
    childpid = fork();
    gettimeofday(&end, 0);

    if ( !childpid ) _exit(0);
    if ( childpid==-1 ) ++forkfailures;

    timersub(&end, &begin, &diff);
    totalforktime += timeval2double(&diff);
    usleep(1);

}
printf("\n");

  usleep(10000);

  // test the threads
  printf("\n");
  for ( i=0; i<samples; ++i ) {
    int ret;
    printf("\rThread test: %u/%u (%3.2f%%)", i+1, samples, ((i+1.0)/samples)*100);fflush(0);

    gettimeofday(&begin, 0);
    ret = pthread_create(&thread, 0, &threadfunc, 0);
    gettimeofday(&end, 0);

    if ( ret ) ++threadfailures;

    timersub(&end, &begin, &diff);
    totalthreadtime += timeval2double(&diff);
    usleep(1);

}
printf("\n\n");

  // print the results
  forktime = totalforktime / (samples-forkfailures);
  printf("Fork failures: %u (succes: %u)\n", forkfailures, samples-forkfailures);
  printf("Avarage fork time: %5f msecs\n", forktime);
  printf("Total fork time: %5f msecs\n", totalforktime);
  threadtime = totalthreadtime / (samples-threadfailures);
  printf("Thread failures: %u (success: %u)\n", threadfailures, samples-threadfailures);
  printf("Avarage thread time: %5f msecs\n", threadtime);
  printf("Total thread time: %5f msecs\n", totalthreadtime);
  printf("Avarage fork/thread ratio: %2f\n", forktime/threadtime);
  printf("Total fork/thread ratio: %2f\n", totalforktime/totalthreadtime);

return EXIT_SUCCESS;
}

/*
============================
void *threadfunc(void *_arg)
============================
*/
void *threadfunc(void *) {
pthread_exit(0);
return 0;
}

/*
==========================================
double timeval2double(struct timeval *_tv)
==========================================
*/
double timeval2double(struct timeval *_tv) {
double ret;
ret = _tv->tv_usec;
ret /= 1000000;
ret += _tv->tv_sec;
ret *= 1000;
return ret;
}

#1 lore raw
Quoted 16 lines · Gergely Czuczy

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello everyone,

Today morning I've made a test about the thead and child process creation(fork) time on both 2.4 and 2.6 kernels.

The test systems
================

Box A:
- kernel: Linux 2.4.22-xfs
- CPU: Intel P3-700 MHz (slot1)
- Ram: 384MB SDR SDRAM

Quoted 9 lines

Box B:
- kernel: Linux 2.6.6-mm5
- CPU: Intel P4-2.6Ghz
- Ram: 512 DDR SDRAM

Box B is a lot better, but it doesn't metter according to the test, because the aim was to get the ratio of the two times with different sample rates.

Even so, you really should be using the same system if you want comparable results. The pentium 4 in particular can do worse at specific things in microbenchmarks.

Your problem with not being able large numbers of threads and processes could be ulimits or sysctl limits. Look at ulimit and /proc/sys/kernel/threads-max and pid_max.

[ snip numbers ]

Quoted 8 lines

Conlusion
=========

It's easy to notice that in case of 2.4 the ratios of the creation times are converges to 1, so it depends on the load, while in case of a 2.6 kernel the ratios are mostly fix, about 9. This means that creating a new child process takes much more time than creating a new thread.

I tried your code on a P3-1000. 2.4.23 is not 100% comparable because it was built with an old compiler and possibly different options.

Anyway, results with 256 samples were as follows:
processes 256
2.4.23:
Total fork time: 48.224 msecs
Total thread time: 30.180 msecs
Total fork/thread ratio: 1.598

2.6.6-mm5:
Total fork time: 40.795 msecs
Total thread time: 17.615 msecs
Total fork/thread ratio: 2.316

2.6.6-mm5+child-runs-last:
Total fork time: 16.080 msecs
Total thread time: 8.600 msecs
Total fork/thread ratio: 1.870

2.6 introduced an optimisation called child-runs-first. Apparently it is nice for many common things, but it also causes a context switch at every fork(), so sucks for these microbenchmarks. Child- runs-last simply reverts that. Patch is attached if anyone is interested.

 linux-2.6-npiggin/kernel/fork.c  |    7 +------
 linux-2.6-npiggin/kernel/sched.c |   20 ++------------------
 2 files changed, 3 insertions(+), 24 deletions(-)
kernel/sched.c renamed from kernel/sched.c~no-child-run-first +2 -18
diff -puN kernel/sched.c~no-child-run-first kernel/sched.c
@@ -1073,15 +1073,7 @@ void fastcall wake_up_forked_process(tas
 	p->prio = effective_prio(p);
 	set_task_cpu(p, smp_processor_id());
 
-	if (unlikely(!current->array))
-		__activate_task(p, rq);
-	else {
-		p->prio = current->prio;
-		list_add_tail(&p->run_list, &current->run_list);
-		p->array = current->array;
-		p->array->nr_active++;
-		rq->nr_running++;
-	}
+	__activate_task(p, rq);
 	task_rq_unlock(rq, &flags);
 }
 
@@ -1395,15 +1387,7 @@ lock_again:
 	set_task_cpu(p, cpu);
 
 	if (cpu == this_cpu) {
-		if (unlikely(!current->array))
-			__activate_task(p, rq);
-		else {
-			p->prio = current->prio;
-			list_add_tail(&p->run_list, &current->run_list);
-			p->array = current->array;
-			p->array->nr_active++;
-			rq->nr_running++;
-		}
+		__activate_task(p, rq);
 	} else {
 		schedstat_inc(sd, sbc_pushed);
 		/* Not the local CPU - must adjust timestamp */
kernel/fork.c renamed from kernel/fork.c~no-child-run-first +1 -6
diff -puN kernel/fork.c~no-child-run-first kernel/fork.c
@@ -1239,12 +1239,7 @@ long do_fork(unsigned long clone_flags,
 			wait_for_completion(&vfork);
 			if (unlikely (current->ptrace & PT_TRACE_VFORK_DONE))
 				ptrace_notify ((PTRACE_EVENT_VFORK_DONE << 8) | SIGTRAP);
-		} else
-			/*
-			 * Let the child process run first, to avoid most of the
-			 * COW overhead when the child exec()s afterwards.
-			 */
-			set_need_resched();
+		}
 	}
 	return pid;
 }

_

#2 lore raw
Quoted 22 lines · Nick Piggin
Quoted 21 lines · Gergely Czuczy

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello everyone,

Today morning I've made a test about the thead and child process creation(fork) time on both 2.4 and 2.6 kernels.

The test systems
================

Box A:
- kernel: Linux 2.4.22-xfs
- CPU: Intel P3-700 MHz (slot1)
- Ram: 384MB SDR SDRAM

Box B:
- kernel: Linux 2.6.6-mm5
- CPU: Intel P4-2.6Ghz
- Ram: 512 DDR SDRAM

Quoted 8 lines
Quoted 4 lines

Box B is a lot better, but it doesn't metter according to the test, because the aim was to get the ratio of the two times with different sample rates.

Even so, you really should be using the same system if you want comparable results. The pentium 4 in particular can do worse at specific things in microbenchmarks.

It doesn't matter. That's why you should look at the "ratio" at the end and not on the pure numbers, they are just bonus "information"

Quoted 2 lines

Your problem with not being able large numbers of threads and

ohh wait, i don't have problems, i've just noticed the things!

Quoted 2 lines

processes could be ulimits or sysctl limits. Look at ulimit and /proc/sys/kernel/threads-max and pid_max.

threads-max is about 8179. this is why I don't see why it is limited to 255 threads.

Quoted 27 lines

[ snip numbers ]

Quoted 8 lines

Conlusion
=========

It's easy to notice that in case of 2.4 the ratios of the creation times are converges to 1, so it depends on the load, while in case of a 2.6 kernel the ratios are mostly fix, about 9. This means that creating a new child process takes much more time than creating a new thread.

I tried your code on a P3-1000. 2.4.23 is not 100% comparable because it was built with an old compiler and possibly different options.

Anyway, results with 256 samples were as follows:
processes 256
2.4.23:
Total fork time: 48.224 msecs
Total thread time: 30.180 msecs
Total fork/thread ratio: 1.598

2.6.6-mm5:
Total fork time: 40.795 msecs
Total thread time: 17.615 msecs
Total fork/thread ratio: 2.316

Quoted 4 lines

2.6.6-mm5+child-runs-last:
Total fork time: 16.080 msecs
Total thread time: 8.600 msecs
Total fork/thread ratio: 1.870

This is very nice, but notice the difference between the total and the avarage time. Total time includes all the time that taken by the test, and The avarage/relative time reflect he time taken by the successive calls, eg it not includes failed pthread_create()s and fork()s, which ones makes the test false. After posting the test I've realized that the measurement of the realitve time is not good, so I will patch this problem.

Quoted 7 lines

2.6 introduced an optimisation called child-runs-first. Apparently it is nice for many common things, but it also causes a context switch at every fork(), so sucks for these microbenchmarks. Child- runs-last simply reverts that. Patch is attached if anyone is interested.

Bye,

Gergely Czuczy
mailto: [email protected]
PGP: http://phoemix.harmless.hu/phoemix.pgp

"Wish a god, a star, to believe in,
With the realm of king of fantasy..."

#3 lore raw
Quoted 11 lines · Gergely Czuczy
Quoted 8 lines · Nick Piggin
Quoted 4 lines · Gergely Czuczy

Box B is a lot better, but it doesn't metter according to the test, because the aim was to get the ratio of the two times with different sample rates.

Quoted 7 lines
Quoted 3 lines

Even so, you really should be using the same system if you want comparable results. The pentium 4 in particular can do worse at specific things in microbenchmarks.

It doesn't matter. That's why you should look at the "ratio" at the end and not on the pure numbers, they are just bonus "information"

I understand that... but the P4 might execute the fork test relatively slower than the thread test compared to the P3.

Different CPU architectures, cache types and sizes, memory to CPU speeds, etc. might all have some influcene on the ratio.

#5 lore raw
In reply to Gergely Czuczy
Quoted 3 lines · Gergely Czuczy

failed. As I told it above all the processes are teminated right after creation, but there were a lot of defunct processes in the system, and they were only gone when the parent termineted.

Have you heard of wait, waitpid and pthread_join?

Quoted 2 lines

With a few number of processes I wasn't able to go over 255 threads, after the 255th every creation attempt simply failed.

Your 255 thread limit is propably because you have a stack size of 8192 kbytes. (see ulimit). As all threads share the same address space, this address space is the limiting factor. Try ulimit -s 1024 for example.

Quoted 4 lines

It's easy to notice that in case of 2.4 the ratios of the creation times are converges to 1, so it depends on the load, while in case of a 2.6 kernel the ratios are mostly fix, about 9. This means that creating a new child process takes much more time than creating a new thread.

Well, the other way around is the correct answer. Processes didnt get slower. Threads are faster than processes in 2.6 because of the NPTL. If you want to slow down threads to process level, just do LD_ASSUME_KERNEL=2.2.5 before running your test program.

cheers

Christian

#6 lore raw
Quoted 6 lines · Christian Borntraeger

Date: Sun, 23 May 2004 11:39:41 +0200
From: Christian Borntraeger <[email protected]>
To: [email protected]
Cc: Gergely Czuczy <[email protected]>, [email protected]
Subject: Re: Linux 2.4 VS 2.6 fork VS thread creation time test

Quoted 6 lines
Quoted 3 lines · Gergely Czuczy

failed. As I told it above all the processes are teminated right after creation, but there were a lot of defunct processes in the system, and they were only gone when the parent termineted.

Have you heard of wait, waitpid and pthread_join?

there really is some sort of problem with 2.6.6 in this area. I have an app that I am trying to stress test on a dual opteron system and under a heavy load something goes haywire and the children become zombies. on a dual athlon the test manages 2500 forks/sec and can continue forever (Ok, I only tested it to 11M forks at full speed :-), but the dual opteron box manages 3500 connections/sec for a few thousand connections and then stops reaping the children. if I attach strace to the parent at this point the logjam is broken and strace shows the wait calls receiving and handleing the sigchild

the prarent deals with sigchild by
handler{
while ( wait(...) >0);
signal(SIGCHLD, handler);
}

unfortunantly trying to leave strace attached while running the test slows it down to ~1200 forks/sec and the problem never forms.

David Lang

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#7 lore raw
Quoted 7 lines · David Lang
Quoted 5 lines · Christian Borntraeger
Quoted 3 lines · Gergely Czuczy

failed. As I told it above all the processes are teminated right after creation, but there were a lot of defunct processes in the system, and they were only gone when the parent termineted.

Have you heard of wait, waitpid and pthread_join?

there really is some sort of problem with 2.6.6 in this area. I have an

Well in the example given by Gergely there was no wait call at all. Therefore I believe your problem is not related to his one.

What do you mean by with 2.6.6. Does this testcase behaves differently with other kernel versions? Which version is the first with this problem?

Quoted 5 lines

the prarent deals with sigchild by
handler{
while ( wait(...) >0);
signal(SIGCHLD, handler);
}

You run signal within the signal handler. This is not necessary, although this should cause no problems. Nevertheless, can you try your test without signal in the signal handler?

cheers

Christian

#8 lore raw
Quoted 11 lines · Christian Borntraeger
Quoted 7 lines · David Lang
Quoted 5 lines · Christian Borntraeger
Quoted 3 lines · Gergely Czuczy

failed. As I told it above all the processes are teminated right after creation, but there were a lot of defunct processes in the system, and they were only gone when the parent termineted.

Have you heard of wait, waitpid and pthread_join?

there really is some sort of problem with 2.6.6 in this area. I have an

Well in the example given by Gergely there was no wait call at all. Therefore I believe your problem is not related to his one.

Ok, very possible

Quoted 2 lines

What do you mean by with 2.6.6. Does this testcase behaves differently with other kernel versions? Which version is the first with this problem?

I started doing this testing with 2.6.4 and had problems with 2.6.[456], but only on the opterons

Quoted 9 lines
Quoted 5 lines

the prarent deals with sigchild by
handler{
while ( wait(...) >0);
signal(SIGCHLD, handler);
}

You run signal within the signal handler. This is not necessary, although this should cause no problems. Nevertheless, can you try your test without signal in the signal handler?

I will do this, but as I read the signal man page this will cause signal to be reset after the first signal hits and nothing will set it to handle any future signals, where should the handler get set instead?

Quoted 4 lines

cheers

Christian

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

A message that is not in the archive (sent off-list or not yet received): 1085325156.622.0.camel@boxen

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw

I just found part of a thread from early april about zombie problems in both 2.4 and 2.6
thread titles are
Exploring sigqueue leak (was: Strange 'zombie' problem both in 2.4 and 2.6)
Local DoS (was: Strange 'zombie' problem both in 2.4 and 2.6)

one piece of info that those threads were looking for is the sigqueue info

on my box I get

baseline:/proc/sys/kernel# grep sigqueue /proc/slabinfo
sigqueue              22     24    160   24    1 : tunables  120   60    8
: slabdata      1      1      0
baseline:/proc/sys/kernel# cat /proc/sys/kernel/rtsig-*
2048
1

default rtsig-max was 1024, I bumped it up to 2048 but it did not affect the problem

one interesting point is that rtsig-nr is normally 0 but when it is in the error condition that causes all children to become zombies it sits at 1

since my last message didn't go out to linux-kernel and I didn't notice it I am attaching the source for a program that also generates this problem

David Lang

Quoted 23 lines · David Lang

Date: Sun, 23 May 2004 17:46:40 -0700 (PDT)
From: David Lang <[email protected]>
To: Alexander Nyberg <[email protected]>
Subject: Re: Linux 2.4 VS 2.6 fork VS thread creation time test

Quoted 7 lines
Quoted 3 lines

unfortunantly trying to leave strace attached while running the test slows it down to ~1200 forks/sec and the problem never forms.

Could you please show me this program? I wrote a small one together myself but I cannot reproduce this problem, maybe because of UP, don't know.

Ok, here is another program that has the same problem.

David Lang

--
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan