15000+ processes -- poor performance ?!
Dear List(s),
as part of my project I need to run a very high number of processes/threads on a linux machine. Right now I have a Dual-PIII 1.4G w/ 8GB RAM -- I am running 4000 processes w/ 2-3 threads each totaling in a process count of 15000+ processes (since Linux doesn't really distinguish between threads and processes...).
Once I pass the 10000 (+/-) pocesses load increases drastically (on startup, although it returns to normal), however the system time (on one processor) reaches for 54% (12061 procs) while the only non sleeping process is top -- the system is basically doing nothing (except scheduling the "nothing" which consumes significant system time).
Is there anything I can do to reduce that system load/time? (I haven't been able to exactly define the "line" but it definitly gets worse the more processes need to be handled.)
Does any of the patchsets address this particular problem?
BTW: The processes are all alike...
Thanks for you help!
Immanuel
forgot the kernel version (2.4.20aa1)...
Quoted 27 lines · Till Immanuel Patzschke
Dear List(s),
as part of my project I need to run a very high number of processes/threads on a linux machine. Right now I have a Dual-PIII 1.4G w/ 8GB RAM -- I am running 4000 processes w/ 2-3 threads each totaling in a process count of 15000+ processes (since Linux doesn't really distinguish between threads and processes...).
Once I pass the 10000 (+/-) pocesses load increases drastically (on startup, although it returns to normal), however the system time (on one processor) reaches for 54% (12061 procs) while the only non sleeping process is top -- the system is basically doing nothing (except scheduling the "nothing" which consumes significant system time).
Is there anything I can do to reduce that system load/time? (I haven't been able to exactly define the "line" but it definitly gets worse the more processes need to be handled.)
Does any of the patchsets address this particular problem?
BTW: The processes are all alike...Thanks for you help!
Immanuel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 1 line · Till Immanuel Patzschke
forgot the kernel version (2.4.20aa1)...
2.4.20aa1 is missing some of the infrastructure to reduce the cpu consumption under high process count loads, but that's not going to help you anyway. 150K processes is not going to be feasible in the immediate future (months or longer away) so you'll have to figure out how to take that into account.
Bill
also top is very inefficant with large numbers of processes. use vmstat or cat out the files in /proc to get the info more efficiantly (it won't get you per process info, but it son't cause the interferance with your desired load that top gives you.)
David Lang
Quoted 24 lines · William Lee Irwin III
Date: Wed, 18 Dec 2002 17:15:41 -0800
From: William Lee Irwin III <[email protected]>
To: Till Immanuel Patzschke <[email protected]>
Cc: lse-tech <[email protected]>,
"[email protected]" <[email protected]>
Subject: Re: 15000+ processes -- poor performance ?!Quoted 1 line · Till Immanuel Patzschke
forgot the kernel version (2.4.20aa1)...
2.4.20aa1 is missing some of the infrastructure to reduce the cpu consumption under high process count loads, but that's not going to help you anyway. 150K processes is not going to be feasible in the immediate future (months or longer away) so you'll have to figure out how to take that into account.
Bill
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines · David Lang
also top is very inefficant with large numbers of processes. use vmstat or cat out the files in /proc to get the info more efficiantly (it won't get you per process info, but it son't cause the interferance with your desired load that top gives you.)
It's mostly just the fact top(1) doesn't scan /proc/ incrementally and that proc_pid_readdir() is quadratic in the number of tasks.
Bill
Ok, I wasn't sure of the cause, but I've seen this as far back as 2.2 I had a machine trying to run 2000 processes under 2.2 and 2.4.0 (after upping the 2.2 kernel limit) and top would cost me ~40% throughput on the machine (while claiming it was useing ~5% of the CPU)
David Lang
Quoted 20 lines · William Lee Irwin III
Date: Wed, 18 Dec 2002 17:25:49 -0800
From: William Lee Irwin III <[email protected]>
To: David Lang <[email protected]>
Cc: Till Immanuel Patzschke <[email protected]>,
lse-tech <[email protected]>, "[email protected]" <[email protected]>
Subject: Re: 15000+ processes -- poor performance ?!Quoted 4 lines · David Lang
also top is very inefficant with large numbers of processes. use vmstat or cat out the files in /proc to get the info more efficiantly (it won't get you per process info, but it son't cause the interferance with your desired load that top gives you.)
It's mostly just the fact top(1) doesn't scan
/proc/incrementally and thatproc_pid_readdir()is quadratic in the number of tasks.Bill
Quoted 5 lines · David Lang
Ok, I wasn't sure of the cause, but I've seen this as far back as 2.2 I had a machine trying to run 2000 processes under 2.2 and 2.4.0 (after upping the 2.2 kernel limit) and top would cost me ~40% throughput on the machine (while claiming it was useing ~5% of the CPU)
David Lang
It wasn't really lying to you. The issue is that the kernel samples at regular intervals to avoid timer reprogramming overhead. Now top(1) is isochronous in nature as it's trying to periodically refresh, and so it runs in lockstep with the clock interrupt, and the kernel hands back bad numbers to top(1).
Bill
Quoted 4 lines · David Lang
Ok, I wasn't sure of the cause, but I've seen this as far back as 2.2 I had a machine trying to run 2000 processes under 2.2 and 2.4.0 (after upping the 2.2 kernel limit) and top would cost me ~40% throughput on the machine (while claiming it was useing ~5% of the CPU)
Yah a lot of it is like William is saying... you just do not want to read multiple files for each process in /proc when you have a kajillion processes, and that is what top does. Over and over.
Work has gone into 2.5 to make this a lot better.. If you use threads with NPTL in 2.5, a lot of this is resolved, since the sub-threads will not show up in as /proc/#/ entries.
Robert Love
In my case I will still be running thousands of processes, so I have to just teach everyone not to use top instead.
David Lang
Quoted 25 lines · Robert Love
Date: 18 Dec 2002 20:42:58 -0500
From: Robert Love <[email protected]>
To: David Lang <[email protected]>
Cc: William Lee Irwin III <[email protected]>,
Till Immanuel Patzschke <[email protected]>,
lse-tech <[email protected]>, "[email protected]" <[email protected]>
Subject: Re: 15000+ processes -- poor performance ?!Quoted 4 lines · David Lang
Ok, I wasn't sure of the cause, but I've seen this as far back as 2.2 I had a machine trying to run 2000 processes under 2.2 and 2.4.0 (after upping the 2.2 kernel limit) and top would cost me ~40% throughput on the machine (while claiming it was useing ~5% of the CPU)
Yah a lot of it is like William is saying... you just do not want to read multiple files for each process in /proc when you have a kajillion processes, and that is what top does. Over and over.
Work has gone into 2.5 to make this a lot better.. If you use threads with NPTL in 2.5, a lot of this is resolved, since the sub-threads will not show up in as /proc/#/ entries.
Robert Love
Quoted 3 lines · David Lang
In my case I will still be running thousands of processes, so I have to just teach everyone not to use top instead.
David Lang
Well, a better solution would be a userspace free of /proc/ dependency.
Or actually fixing the kernel. proc_pid_readdir() wants an efficiently indexable linear list, e.g. TAOCP's 6.2.3 "Linear List Representation". At that point its expense is proportional to the buffer size and "seeking" about the list as it is wont to do is O(lg(processes)).
Bill
Quoted 5 lines · William Lee Irwin III
Quoted 3 lines · David Lang
In my case I will still be running thousands of processes, so I have to just teach everyone not to use top instead. David Lang
Quoted 8 lines
Well, a better solution would be a userspace free of
/proc/dependency.Or actually fixing the kernel.
proc_pid_readdir()wants an efficiently indexable linear list, e.g. TAOCP's 6.2.3 "Linear List Representation". At that point its expense is proportional to the buffer size and "seeking" about the list as it is wont to do is O(lg(processes)).
A short-time solution: run top d 30 to make it refresh only every 30 seconds. This will greatly reduce top's own load skew.
--
vda
Quoted 7 lines · William Lee Irwin III
Quoted 7 lines
Well, a better solution would be a userspace free of
/proc/dependency.
Or actually fixing the kernel.proc_pid_readdir()wants an efficiently indexable linear list, e.g. TAOCP's 6.2.3 "Linear List Representation". At that point its expense is proportional to the buffer size and "seeking" about the list as it is wont to do is O(lg(processes)).
Quoted 2 lines · Denis Vlasenko
A short-time solution: run top d 30 to make it refresh only every 30 seconds. This will greatly reduce top's own load skew.
As userspace solutions go your suggestions is just as good. The kernel still needs to get its act together and with some urgency.
Bill
Quoted 9 lines · William Lee Irwin III
Quoted 7 lines · William Lee Irwin III
Quoted 7 lines
Well, a better solution would be a userspace free of
/proc/dependency.
Or actually fixing the kernel.proc_pid_readdir()wants an efficiently indexable linear list, e.g. TAOCP's 6.2.3 "Linear List Representation". At that point its expense is proportional to the buffer size and "seeking" about the list as it is wont to do is O(lg(processes)).
Quoted 6 lines
Quoted 2 lines · Denis Vlasenko
A short-time solution: run top d 30 to make it refresh only every 30 seconds. This will greatly reduce top's own load skew.
As userspace solutions go your suggestions is just as good. The kernel still needs to get its act together and with some urgency.
That was just a suggestion as to how to get realistic picture of system load for Till Immanuel Patzschke <[email protected]>. --
vda
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
William Lee Irwin (WLI) writes:
Quoted 6 lines · William Lee Irwin III
Quoted 6 lines
Quoted 6 lines
Well, a better solution would be a userspace free of
/proc/dependency. Or actually fixing the kernel.proc_pid_readdir()wants an efficiently indexable linear list, e.g. TAOCP's 6.2.3 "Linear List Representation". At that point its expense is proportional to the buffer size and "seeking" about the list as it is wont to do is O(lg(processes)).
Quoted 2 lines · Denis Vlasenko
Quoted 2 lines
A short-time solution: run top d 30 to make it refresh only every 30 seconds. This will greatly reduce top's own load skew.
WLI> As userspace solutions go your suggestions is just as good. The WLI> kernel still needs to get its act together and with some WLI> urgency.
what about retreiving info from /proc/kmem or something like? just to avoid binary -> text(proc) -> binary
William Lee Irwin (WLI) writes:
WLI> As userspace solutions go your suggestions is just as good. The WLI> kernel still needs to get its act together and with some WLI> urgency.
Quoted 2 lines · Alex Tomas
what about retreiving info from
/proc/kmemor something like? just to avoid binary -> text(proc) -> binary
That would also be an excellent userspace solution to this local DoS.
Bill
Quoted 6 lines
WLI> As userspace solutions go your suggestions is just as good. The WLI> kernel still needs to get its act together and with some WLI> urgency.
what about retreiving info from
/proc/kmemor something like? just to avoid binary -> text(proc) -> binary
The binary <-> text translation problem is less of an issue than all the syscall traffic, dcache hits, etc. Search linux-kernel archives for a recent thread entitiled "ps performance sucks" or something similar.
M.
Quoted 1 line · Till Immanuel Patzschke
Quoted 1 line
forgot the kernel version (2.4.20aa1)...
Quoted 5 lines · William Lee Irwin III
2.4.20aa1 is missing some of the infrastructure to reduce the cpu consumption under high process count loads, but that's not going to help you anyway. 150K processes is not going to be feasible in the immediate future (months or longer away) so you'll have to figure out how to take that into account.
Er, sorry, on a brief rereading my eyes deceived me and I thought an extra zero got in there. 15K is fine on 2.5 + patches.
Bill
Quoted 5 lines
as part of my project I need to run a very high number of processes/threads on a linux machine. Right now I have a Dual-PIII 1.4G w/ 8GB RAM -- I am running 4000 processes w/ 2-3 threads each totaling in a process count of 15000+ processes (since Linux doesn't really distinguish between threads and processes...).
Quoted 8 lines
Once I pass the 10000 (+/-) pocesses load increases drastically (on startup, although it returns to normal), however the system time (on one processor) reaches for 54% (12061 procs) while the only non sleeping process is top -- the system is basically doing nothing (except scheduling the "nothing" which consumes significant system time).
Is there anything I can do to reduce that system load/time? (I haven't been able to exactly define the "line" but it definitly gets worse the more processes need to be handled.)
You don't even specify what kernel you're using ...
Quoted 1 line
Does any of the patchsets address this particular problem?
Read the linux-kernel archives.
M.
Quoted 15 lines · Till Immanuel Patzschke
Dear List(s),
as part of my project I need to run a very high number of processes/threads on a linux machine. Right now I have a Dual-PIII 1.4G w/ 8GB RAM -- I am running 4000 processes w/ 2-3 threads each totaling in a process count of 15000+ processes (since Linux doesn't really distinguish between threads and processes...).
Once I pass the 10000 (+/-) pocesses load increases drastically (on startup, although it returns to normal), however the system time (on one processor) reaches for 54% (12061 procs) while the only non sleeping process is top -- the system is basically doing nothing (except scheduling the "nothing" which consumes significant system time).
Is there anything I can do to reduce that system load/time? (I haven't been able to exactly define the "line" but it definitly gets worse the more processes need to be handled.)
Redesign your program to not do silly things like this.
Unless you have hardware with 5000 or more CPUs...
Jeff
Quoted 5 lines · Till Immanuel Patzschke
as part of my project I need to run a very high number of processes/threads on a linux machine. Right now I have a Dual-PIII 1.4G w/ 8GB RAM -- I am running 4000 processes w/ 2-3 threads each totaling in a process count of 15000+ processes (since Linux doesn't really distinguish between threads and processes...).
You're for the most part SOL unless you can either hack the support or can wait for it to be finished. More details below.
Quoted 4 lines · Till Immanuel Patzschke
Once I pass the 10000 (+/-) pocesses load increases drastically (on startup, although it returns to normal), however the system time (on one processor) reaches for 54% (12061 procs) while the only non sleeping process is top -- the system is basically doing nothing
Quoted 8 lines
(except scheduling the "nothing" which
consumes significant system time).
Is there anything I can do to reduce that system load/time? (I haven't been able to exactly define the "line" but it definitly gets worse the more processes need to be handled.)
Does any of the patchsets address this particular problem?
BTW: The processes are all alike...
Thanks for you help!
Try 2.5.52-mm1 + 2.5.52-wli-1. The -wli bits are orthogonal but they do a small bit to reduce the cpu inefficiencies of many task loads. -wli is actually maintenance and follow-through on various early 2.5 promises.
proc_pid_readdir() is the cpu culprit, which I have not yet addressed. You are also going to have severe memory management problems due to the number of L2 and L3 pagetables created as well as kernel stacks. 2.5.52-mm1 will have 2 of 3 possible things that can be done about L3 pagetables. L2 pagetables limit you to 64K processes with more practical limits around 16K. As 16K is feasible here, you are running the wrong kernel version(s).
Bill
Quoted 7 lines · Till Immanuel Patzschke
Dear List(s),
as part of my project I need to run a very high number of processes/threads on a linux machine. Right now I have a Dual-PIII 1.4G w/ 8GB RAM -- I am running 4000 processes w/ 2-3 threads each totaling in a process count of 15000+ processes (since Linux doesn't really distinguish between threads and processes...).
BTW, can you say _what_ are you trying to do?
Quoted 11 lines
Once I pass the 10000 (+/-) pocesses load increases drastically (on startup, although it returns to normal), however the system time (on one processor) reaches for 54% (12061 procs) while the only non sleeping process is top -- the system is basically doing nothing (except scheduling the "nothing" which consumes significant system time).
Is there anything I can do to reduce that system load/time? (I haven't been able to exactly define the "line" but it definitly gets worse the more processes need to be handled.)
Does any of the patchsets address this particular problem?
BTW: The processes are all alike...
You need to collect memory info (especially lowmem and highmem situation) and maybe profile your kernel to find out where does it spend that time doing "nothing".
BTW, your .config?
--
vda