package-infra: limit the number of // jobs
PATCH v2 1/1 package-infra: limit the number of // jobs
From: "Yann E. MORIN" <[email protected]>
The current code spawns as many jobs as up to twice the number of CPUs.
On small-class machines like laptops, with a limitted amount of memory, but still a few CPUs (real or hyperthreads), the HDD becomes a bottleneck, and it becomes almost impossible to do anythiong else while there is a build in progress.
Limit the number of jobs to the number of CPUs plus one.
Even on fast machines with fast HDDs, this settings keeps the machine fully busy (for those packages that can build in parallel, of course).
For example, building qemu or the linux kernel kept my hyperthreaded hexa Core i7 with 18GiB of RAM, busy at 99% (I never ever managed to get 100% even with more jobs, not even 200); while on my hyperthreaded dual Core i5 with only 4GiB and a slow HDD, I still topped at 100% CPU, while still able to do some work involving the HDD.
If the number of processors is not available, assume one.
---
Changes v1 -> v2:
- fix the code to actually reflect the commit log
- fix the comment (Nathan)
--- package/Makefile.in | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
package/Makefile.in +3 -2
@@ -7,11 +7,12 @@ endif HOSTMAKE :=$(shell which $(HOSTMAKE) || type -p $(HOSTMAKE) || echo make) # If BR2_LEVEL is 0, scale the maximum concurrency with the number of -# CPUs. A coefficient of 2 is used in order to keep processors busy +# CPUs. An additional job is used in order to keep processors busy # while waiting on I/O. +# If the number of processors is not available, assume one. ifeq ($(BR2_JLEVEL),0) PARALLEL_JOBS:=$(shell echo \ - $$((2 * `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1`))) + $$((1 + `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1`))) else PARALLEL_JOBS:=$(BR2_JLEVEL) endif
Signature · 1 line
1.8.1.2
PATCH v2 1/1 package-infra: limit the number of // jobs
Dear Yann E. MORIN,
Quoted 20 lines · Yann E. MORIN
From: "Yann E. MORIN" <[email protected]>
The current code spawns as many jobs as up to twice the number of CPUs.
On small-class machines like laptops, with a limitted amount of memory, but still a few CPUs (real or hyperthreads), the HDD becomes a bottleneck, and it becomes almost impossible to do anythiong else while there is a build in progress.
Limit the number of jobs to the number of CPUs plus one.
Even on fast machines with fast HDDs, this settings keeps the machine fully busy (for those packages that can build in parallel, of course).
For example, building qemu or the linux kernel kept my hyperthreaded hexa Core i7 with 18GiB of RAM, busy at 99% (I never ever managed to get 100% even with more jobs, not even 200); while on my hyperthreaded dual Core i5 with only 4GiB and a slow HDD, I still topped at 100% CPU, while still able to do some work involving the HDD.
Quoted 4 lines
If the number of processors is not available, assume one.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Nathan Lynch <[email protected]>
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 1/1 package-infra: limit the number of // jobs
In reply to Yann E. MORINQuoted 20 lines · Yann E. MORIN
From: "Yann E. MORIN" <[email protected]>
The current code spawns as many jobs as up to twice the number of CPUs.
On small-class machines like laptops, with a limitted amount of memory, but still a few CPUs (real or hyperthreads), the HDD becomes a bottleneck, and it becomes almost impossible to do anythiong else while there is a build in progress.
Limit the number of jobs to the number of CPUs plus one.
Even on fast machines with fast HDDs, this settings keeps the machine fully busy (for those packages that can build in parallel, of course).
For example, building qemu or the linux kernel kept my hyperthreaded hexa Core i7 with 18GiB of RAM, busy at 99% (I never ever managed to get 100% even with more jobs, not even 200); while on my hyperthreaded dual Core i5 with only 4GiB and a slow HDD, I still topped at 100% CPU, while still able to do some work involving the HDD.
Quoted 4 lines
If the number of processors is not available, assume one.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Nathan Lynch <[email protected]>
Thanks Yann.
PATCH v2 1/1 package-infra: limit the number of // jobs
In reply to Yann E. MORINQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Yann> From: "Yann E. MORIN" <[email protected]>
Yann> The current code spawns as many jobs as up to twice the number of Yann> CPUs.
Yann> On small-class machines like laptops, with a limitted amount of Yann> memory, but still a few CPUs (real or hyperthreads), the HDD Yann> becomes a bottleneck, and it becomes almost impossible to do Yann> anythiong else while there is a build in progress.
Yann> Limit the number of jobs to the number of CPUs plus one.
Yann> Even on fast machines with fast HDDs, this settings keeps the Yann> machine fully busy (for those packages that can build in Yann> parallel, of course).
Yann> For example, building qemu or the linux kernel kept my Yann> hyperthreaded hexa Core i7 with 18GiB of RAM, busy at 99% (I Yann> never ever managed to get 100% even with more jobs, not even Yann> 200); while on my hyperthreaded dual Core i5 with only 4GiB and a Yann> slow HDD, I still topped at 100% CPU, while still able to do some Yann> work involving the HDD.
Yes, I remember the old default being pretty hard on my old laptop with spinning rust. With a SSD I don't really notice it much.
Committed, thanks.
Signature · 1 line
Bye, Peter Korsgaard