linux scheduler limitations?
Hello,
I'm working on an enhanced version of Apache and I'm hitting my head against something I don't understand.
I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb).
Profiling the kernel shows that the scheduler and the interrupt handler are taking most of the CPU time.
I understand that there must be a limit to the number of processes that the scheduler can efficiently handle, but I would expect some sort of gradual performance degradation when increasing the number of tasks, instead I observe that by increasing Apache's MaxClient linit by as little as 10 can cause a sudden transition between smooth working with lots (30-40%) of CPU idle to a total lock-up.
Moreover the max number of processes is not even constant. If I increase the server load gradually then I manage to have 1500 processes running with no problem, but if the transition is sharp (the SpecWeb case) than I end-up having a lock up.
Anybody seen this before? Any clues?
- Fabio
2.2 or 2.4 kernel?
the 2.4 does a MUCH better job of dealing with large numbers of processes.
David Lang
Quoted 41 lines · Fabio Riccardi
Date: Thu, 29 Mar 2001 13:19:05 -0800
From: Fabio Riccardi <[email protected]>
To: [email protected]
Subject: linux scheduler limitations?Hello,
I'm working on an enhanced version of Apache and I'm hitting my head against something I don't understand.
I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb).
Profiling the kernel shows that the scheduler and the interrupt handler are taking most of the CPU time.
I understand that there must be a limit to the number of processes that the scheduler can efficiently handle, but I would expect some sort of gradual performance degradation when increasing the number of tasks, instead I observe that by increasing Apache's MaxClient linit by as little as 10 can cause a sudden transition between smooth working with lots (30-40%) of CPU idle to a total lock-up.
Moreover the max number of processes is not even constant. If I increase the server load gradually then I manage to have 1500 processes running with no problem, but if the transition is sharp (the SpecWeb case) than I end-up having a lock up.
Anybody seen this before? Any clues?
- Fabio
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
I'm using 2.4.2-ac26, but I've noticed the same behavior with all the 2.4 kernels I've seen so far.
I haven't even tried on 2.2
- Fabio
Quoted 49 lines · David Lang
2.2 or 2.4 kernel?
the 2.4 does a MUCH better job of dealing with large numbers of processes.
David Lang
Quoted 41 lines · Fabio Riccardi
Date: Thu, 29 Mar 2001 13:19:05 -0800
From: Fabio Riccardi <[email protected]>
To: [email protected]
Subject: linux scheduler limitations?Hello,
I'm working on an enhanced version of Apache and I'm hitting my head against something I don't understand.
I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb).
Profiling the kernel shows that the scheduler and the interrupt handler are taking most of the CPU time.
I understand that there must be a limit to the number of processes that the scheduler can efficiently handle, but I would expect some sort of gradual performance degradation when increasing the number of tasks, instead I observe that by increasing Apache's MaxClient linit by as little as 10 can cause a sudden transition between smooth working with lots (30-40%) of CPU idle to a total lock-up.
Moreover the max number of processes is not even constant. If I increase the server load gradually then I manage to have 1500 processes running with no problem, but if the transition is sharp (the SpecWeb case) than I end-up having a lock up.
Anybody seen this before? Any clues?
- Fabio
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Fabio Riccardi
I'm using 2.4.2-ac26, but I've noticed the same behavior with all the 2.4 kernels I've seen so far.
I haven't even tried on 2.2
- Fabio
Fabio,
Just for fun, you might want to try out some of our scheduler patches located at:
http://lse.sourceforge.net/scheduling/
I would be interested in your observations.
Signature · 2 lines
Mike Kravetz [email protected] IBM Linux Technology Center
Hi Mike,
somebody else on the list already pointed me at your stuff and I quickly downloaded your multiqueue patch for 2.4.1 to try it out.
It works great! I finally manage to have 100% CPU utilization and keep the machine decently responsive.
On a two 1GHz pentium box i went from 1300 specweb to 1600. That's pretty amazing.
There is a bit more overhead though, I'd say arount 5%, when the CPU is not fully loaded.
What is the status of your code? Is it going to end-up in the mainstream kernel?
Do you have a port to the 2.4.2x kernels?
In my enthousiasm I tried to port the patch to 2.4.2-ac26 but I broke something and it didn't work anymore... :)
I havent't tried the pooling patch yet, it didn't seem to make much sense on a 2-way box. I have an 8-way on which I'm planning to bench my web server enhancements, I'll try the pooling stuff on it.
BTW: interested in the fastest linux web server?
BTW2: what about the HP scheduler patches?
Thanks, ciao,
- Fabio
Quoted 20 lines · Mike Kravetz
Quoted 6 lines · Fabio Riccardi
I'm using 2.4.2-ac26, but I've noticed the same behavior with all the 2.4 kernels I've seen so far.
I haven't even tried on 2.2
- Fabio
Fabio,
Just for fun, you might want to try out some of our scheduler patches located at:
http://lse.sourceforge.net/scheduling/
I would be interested in your observations.
--
Mike Kravetz [email protected]
IBM Linux Technology Center
Quoted 6 lines · On 03.29 Fabio Riccardi wrote:
I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb).
Have you though about pthreads (when you talk about fork, I suppose you say literally 'fork()') ?
I give a course on Parallel Programming at the university and the practical work was done with POSIX threads. One of my students caught the idea and used it to modify his assignment from one other matter on Networks, and changed the traditional 'fork()' in a simple ftp server he had to implement by 'pthread_create' and got a 10-30 speedup (conns per second).
And you will get rid of some process-per-user limit. But you will fall into an threads-per-user limit, if there is any.
And you cal also control its scheduling, to make each thread fight against the whole system or only its siblings.
Signature · 4 lines
J.A. Magallon # Let the source mailto:[email protected] # be with you, Luke... Linux werewolf 2.4.2-ac28 #1 SMP Thu Mar 29 16:41:17 CEST 2001 i686
Apache uses a pre-fork "threading" mechanism, it spawns (fork()s) new instances of itself whenever it finds out that the number of idle "threads" is below a certain (configurable) threshold.
Despite of all apparences this method performs beautifully on Linux, pthreads are actually slower in many cases, since you will incur some additional overhead due to thread synchronization and scheduling.
The problem is that beyond a certain number of processes the scheduler just goes bananas, or so it seems to me.
Since Linux threads are mapped on processes, I don't think that (p)threads woud help in any way, unless it is the VM context switch overhead that is playing a role here, which I wouldn't think is the case.
- Fabio
Quoted 28 lines · J . A . Magallon
Quoted 6 lines · On 03.29 Fabio Riccardi wrote:
I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb).
Have you though about pthreads (when you talk about fork, I suppose you say literally '
fork()') ?I give a course on Parallel Programming at the university and the practical work was done with POSIX threads. One of my students caught the idea and used it to modify his assignment from one other matter on Networks, and changed the traditional '
fork()' in a simple ftp server he had to implement by 'pthread_create' and got a 10-30 speedup (conns per second).And you will get rid of some process-per-user limit. But you will fall into an threads-per-user limit, if there is any.
And you cal also control its scheduling, to make each thread fight against the whole system or only its siblings.
--
J.A. Magallon # Let the source mailto:[email protected] # be with you, Luke...Linux werewolf 2.4.2-ac28 #1 SMP Thu Mar 29 16:41:17 CEST 2001 i686
Quoted 7 lines · On 03.30 Fabio Riccardi wrote:
Despite of all apparences this method performs beautifully on Linux, pthreads are
actually slower in many cases, since you will incur some additional overhead due
to thread synchronization and scheduling.
It all depends on your app, as every parallel algorithm. In a web-ftp-whatever server, you do not need any synchro. You can start threads in free run and let them die alone.
Quoted 9 lines
The problem is that beyond a certain number of processes the scheduler just goes
bananas, or so it seems to me.Since Linux threads are mapped on processes, I don't think that (p)threads woud
help in any way, unless it is the VM context switch overhead that is playing a role here, which I wouldn't think is the case.
You said, 'mapped'.
AFAIK, that is the advantage, you can avoid the VM switch by sharing memory.
Signature · 4 lines
J.A. Magallon # Let the source mailto:[email protected] # be with you, Luke... Linux werewolf 2.4.2-ac28 #1 SMP Thu Mar 29 16:41:17 CEST 2001 i686
Quoted 3 lines · J . A . Magallon
It all depends on your app, as every parallel algorithm. In a web-ftp-whatever server, you do not need any synchro. You can start threads in free run and let them die alone.
even if you don't need synchronization you pay for it anyway, since you will have to use the pthread version of libc that is reentrant. Moreover many calls (i.e. accept) are "scheduling points" for pthreads, whenever you call them the runtime will perform quite a bit of bookeeping.
it is instructive to use a profiler on your application and see what happens when you use pthreads...
Quoted 2 lines
You said, 'mapped'.
AFAIK, that is the advantage, you can avoid the VM switch by sharing memory.
If your application uses lots of memory than I agree, Apache only uses a tiny amount of RAM per instance though, so I don't think that that is my case.
- Fabio
Quoted 9 lines · On 29-Mar-01 Fabio Riccardi wrote:
Hello,
I'm working on an enhanced version of Apache and I'm hitting my head against something I don't understand.
I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb).
Are you using 2.2.x ? I had the same problem here until I switched to 2.4.x. 2.2 internal locks are not fine grained enough.
Bye.
Giuliano Pochini ->)|(<- Shiny Network {AS6665} ->)|(<-Quoted 4 lines
I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb).
Im suprised it gets that far
Quoted 4 lines
Moreover the max number of processes is not even constant. If I increase the server load gradually then I manage to have 1500 processes running with no problem, but if the transition is sharp (the SpecWeb case) than I end-up having a lock up.
With that many servers and a sudden load you are probably causing a lot of paging. What kernel version. And while this isnt a solution to kernel issues take a look at thttpd instead (www.acme.com). If you have 1500 8K stacks thrashing in your cache you are not going to have good performance.