Update suggestion of values for BB_NUMBER_THREADS and PARALLEL_MAKE
cover Update suggestion of values for BB_NUMBER_THREADS and PARALLEL_MAKE
Title and patch header say it all. Just clearing some local changes out. This one based on opinion I've overheard.
Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches. If you agree with the changes, please provide your Acked-by.
The following changes since commit 2163461ec94528ecf046a04edc5db3d2dd3a6b8b:
systemtap: remove non-core COMPATIBLE_MACHINES (2011-06-16 22:14:06 +0100)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib josh/localconf
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=josh/localconf
Joshua Lock (1):
local.conf.sample: update suggestions for BB_NUMBER_THREADS and
PARALLEL_MAKE
meta-yocto/conf/local.conf.sample | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)
Signature · 1 line
1.7.5.4
RFC PATCH 1/1 local.conf.sample: update suggestions for BB_NUMBER_THREADS and PARALLEL_MAKE
It's been suggested that BB_NUMBER_THREADS should be 2 * the number of cores and PARALLEL_MAKE should be equal to the number of cores available on the build machine.
Update local.conf.sample to suggest this.
--- meta-yocto/conf/local.conf.sample | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)
meta-yocto/conf/local.conf.sample +3 -1
@@ -9,7 +9,9 @@ CONF_VERSION = "1" #SSTATE_DIR ?= "${TOPDIR}/sstate-cache" # Uncomment and set to allow bitbake to execute multiple tasks at once. -# For a quadcore, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would +# Recommended values are twice the number of processor cores for +# BB_NUMBER_THREADS and the number of processor cores for PARALLEL_MAKE +# For a quadcore, BB_NUMBER_THREADS = "8", PARALLEL_MAKE = "-j 4" would # be appropriate. # BB_NUMBER_THREADS = "4" # Also, make can be passed flags so it run parallel threads e.g.:
Signature · 1 line
1.7.5.4
Re: [RFC PATCH 1/1] local.conf.sample: update suggestions for BB_NUMBER_THREADS and PARALLEL_MAKE
Quoted 16 lines · Joshua Lock
It's been suggested that
BB_NUMBER_THREADSshould be 2 * the number of cores andPARALLEL_MAKEshould be equal to the number of cores available on the build machine.Update local.conf.sample to suggest this.
Signed-off-by Joshua Lock <[email protected]>--- meta-yocto/conf/local.conf.sample | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)meta-yocto/conf/local.conf.sample +0
-0
meta-yocto/conf/local.conf.sample CONF_VERSION = "1" · line 14
Quoted 7 lines
meta-yocto/conf/local.conf.sample +3
-1@@ -9,7 +9,9 @@ CONF_VERSION = "1" #SSTATE_DIR ?= "${TOPDIR}/sstate-cache" # Uncomment and set to allow bitbake to execute multiple tasks at once. -# For a quadcore, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would +# Recommended values are twice the number of processor cores for +# BB_NUMBER_THREADS and the number of processor cores for PARALLEL_MAKE +# For a quadcore, BB_NUMBER_THREADS = "8", PARALLEL_MAKE = "-j 4" would
Hrm, where is this coming from? In my experience it works better the other way around. We probably also need to be explicit about cores versus threads.
Quoted 3 lines
meta-yocto/conf/local.conf.sample +0
-0# be appropriate. # BB_NUMBER_THREADS = "4" # Also, make can be passed flags so it run parallel threads e.g.:
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Linux Kernel
Re: [RFC PATCH 1/1] local.conf.sample: update suggestions for BB_NUMBER_THREADS and PARALLEL_MAKE
Quoted 21 lines · Darren Hart
Quoted 18 lines · Joshua Lock
It's been suggested that
BB_NUMBER_THREADSshould be 2 * the number of cores andPARALLEL_MAKEshould be equal to the number of cores available on the build machine.Update local.conf.sample to suggest this.
Signed-off-by Joshua Lock <[email protected]>--- meta-yocto/conf/local.conf.sample | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)meta-yocto/conf/local.conf.sample +0
-0@@ -9,7 +9,9 @@ CONF_VERSION = "1" #SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
Quoted 9 lines
meta-yocto/conf/local.conf.sample CONF_VERSION = "1" · line 14
Quoted 5 lines
meta-yocto/conf/local.conf.sample +3
-1# Uncomment and set to allow bitbake to execute multiple tasks at once. -# For a quadcore, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would +# Recommended values are twice the number of processor cores for +# BB_NUMBER_THREADS and the number of processor cores for PARALLEL_MAKE +# For a quadcore, BB_NUMBER_THREADS = "8", PARALLEL_MAKE = "-j 4" wouldHrm, where is this coming from? In my experience it works better the other way around. We probably also need to be explicit about cores versus threads.
OK, let's get some real number behind this.
I'm running the attached script on a quadcore (8 thread) i7 system with 8 GB of RAM. an SSD for the OS, and a single spinning disk for the build.
We'll see how things look in... 13 2 ^ 2 * 24 / ... 14.08 days ... assuming I don't burn something up first.... maybe I should have programmed in a sleep? nah, that's what the "rm -rf tmp" is for ;-)
Would be nice to have some runtime atsar disk, sched, fault stats as well...
Quoted 4 lines
Quoted 3 lines
meta-yocto/conf/local.conf.sample +0
-0# be appropriate. # BB_NUMBER_THREADS = "4" # Also, make can be passed flags so it run parallel threads e.g.:
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Linux Kernel
Re: [RFC PATCH 1/1] local.conf.sample: update suggestions for BB_NUMBER_THREADS and PARALLEL_MAKE
In reply to Darren HartQuoted 21 lines · Darren Hart
Quoted 18 lines · Joshua Lock
It's been suggested that
BB_NUMBER_THREADSshould be 2 * the number of cores andPARALLEL_MAKEshould be equal to the number of cores available on the build machine.Update local.conf.sample to suggest this.
Signed-off-by Joshua Lock <[email protected]>--- meta-yocto/conf/local.conf.sample | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)meta-yocto/conf/local.conf.sample +0
-0@@ -9,7 +9,9 @@ CONF_VERSION = "1" #SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
Quoted 9 lines
meta-yocto/conf/local.conf.sample CONF_VERSION = "1" · line 14
Quoted 5 lines
meta-yocto/conf/local.conf.sample +3
-1# Uncomment and set to allow bitbake to execute multiple tasks at once. -# For a quadcore, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would +# Recommended values are twice the number of processor cores for +# BB_NUMBER_THREADS and the number of processor cores for PARALLEL_MAKE +# For a quadcore, BB_NUMBER_THREADS = "8", PARALLEL_MAKE = "-j 4" wouldHrm, where is this coming from? In my experience it works better the other way around. We probably also need to be explicit about cores versus threads.
On my older quad-core AMD box, -j 6 / 4 threads is where it's at, and our testing / poking around on other hardware bears that out (for example my Dell M4400 laptop is -j 3 / 2 threads). That said, for much more beefy configurations, we use -j 16, and 12 threads (on an 8 core machine with 12GB mem). I think perhaps the best change here is to keep it at 1:1 in the sample (since we've also run into older hardware too) and explain that anywhere between 1:1 and 2*core:2*core could work best depending on setup, ymmv, etc.
Signature · 2 lines
Tom Rini Mentor Graphics Corporation
Re: [RFC PATCH 1/1] local.conf.sample: update suggestions for BB_NUMBER_THREADS and PARALLEL_MAKE
Quoted 32 lines · Tom Rini
Quoted 21 lines · Darren Hart
Quoted 18 lines · Joshua Lock
It's been suggested that
BB_NUMBER_THREADSshould be 2 * the number of cores andPARALLEL_MAKEshould be equal to the number of cores available on the build machine.Update local.conf.sample to suggest this.
Signed-off-by Joshua Lock <[email protected]>--- meta-yocto/conf/local.conf.sample | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)meta-yocto/conf/local.conf.sample +0
-0@@ -9,7 +9,9 @@ CONF_VERSION = "1" #SSTATE_DIR ?= "${TOPDIR}/sstate-cache"Quoted 9 lines
meta-yocto/conf/local.conf.sample CONF_VERSION = "1" · line 14
Quoted 5 lines
meta-yocto/conf/local.conf.sample +3
-1# Uncomment and set to allow bitbake to execute multiple tasks at once. -# For a quadcore, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would +# Recommended values are twice the number of processor cores for +# BB_NUMBER_THREADS and the number of processor cores for PARALLEL_MAKE +# For a quadcore, BB_NUMBER_THREADS = "8", PARALLEL_MAKE = "-j 4" wouldHrm, where is this coming from? In my experience it works better the other way around. We probably also need to be explicit about cores versus threads.
Quoted 3 lines
On my older quad-core AMD box, -j 6 / 4 threads is where it's at, and our testing / poking around on other hardware bears that out (for example my Dell M4400 laptop is -j 3 / 2 threads).
Those ratios are closer to what I have seen as optimal as well - specifically more PARALLEL_MAKE threads than BB_NUMBER_THREADS, which is the opposite of the suggestion made above.
Quoted 6 lines
That said, for much
more beefy configurations, we use -j 16, and 12 threads (on an 8 core machine with 12GB mem). I think perhaps the best change here is to keep it at 1:1 in the sample (since we've also run into older hardware too) and explain that anywhere between 1:1 and 2*core:2*core could work best depending on setup, ymmv, etc.
I'm well into the data collection process I promised in my first response, with several days still remaining. However, here is a snapshot of the results. This is a quad-core i7.
BB PM Seconds ...
04 04 9083.43 2704.41 16073.55 206% 5079496 57424647 2377276782 51099 0 0 1910864 0 0 0 0 04 05 9032.61 2708.10 16093.60 208% 5114730 56938109 2377140907 52246 0 0 2027232 0 0 0 0 04 06 9031.50 2711.30 16095.67 208% 5154734 56737116 2377276988 52525 0 0 1937824 0 0 0 0 04 07 9022.27 2711.39 16150.86 209% 5165928 56811664 2377060362 52531 0 0 2027248 0 0 0 0 04 08 9087.11 2716.88 16103.36 207% 5061918 56763715 2376970987 52525 0 0 1940448 0 0 0 0 04 09 9310.98 2698.72 16080.94 201% 5159166 56868437 2377495925 53496 0 0 2027232 0 0 0 0 04 10 9243.03 2709.35 16077.68 203% 4983749 56880160 2377210887 53044 0 0 2027360 0 0 0 0 04 11 9139.86 2711.91 16085.86 205% 5108951 56824250 2377433181 53253 0 0 2027232 0 0 0 0 04 12 9132.83 2719.61 16066.65 205% 4995133 56904875 2377286845 53102 0 0 1857632 0 0 0 0 04 13 9169.78 2715.16 16095.05 205% 5046506 56856301 2376970111 52827 0 0 2027248 0 0 0 0 04 14 9079.85 2715.04 16082.28 207% 5075792 56804956 2377616169 53000 0 0 1912128 0 0 0 0 04 15 9133.57 2710.45 16096.39 205% 5101938 56804976 2377283916 53125 0 0 2027360 0 0 0 0 04 16 9200.60 2728.44 16129.32 204% 4959845 56852830 2381322571 53793 0 0 2027248 0 0 0 0 05 04 8685.73 2942.91 17328.96 233% 8572490 54322660 2383126091 52468 0 0 2027232 0 0 0 0 05 05 9109.86 2964.68 18033.54 230% 8637358 54222700 2380041373 54070 0 0 2027248 0 0 0 0 05 06 8681.89 2950.25 17384.38 234% 8466556 54244954 2382944050 53202 0 0 2027360 0 0 0 0 05 07 8683.63 2950.78 17380.57 234% 8420681 54277047 2382977368 52970 0 0 2027360 0 0 0 0 05 08 8845.89 2922.37 17296.46 228% 8356145 54197213 2375969732 52062 0 0 2027232 0 0 0 0 05 09 8801.84 2932.86 17509.07 232% 8508554 54179792 2383768769 52943 0 0 2027216 0 0 0 0 05 10 8742.35 2942.06 17341.79 232% 8475095 54323068 2383570357 52587 0 0 2027232 0 0 0 0 05 11 8798.83 2943.76 17428.09 231% 8821969 54234482 2375921193 53157 0 0 2027232 0 0 0 0 05 12 8805.69 2947.29 17417.49 231% 8753090 54442047 2383254479 52402 0 0 1883440 0 0 0 0 05 13 8834.71 2948.17 17435.29 230% 8794659 54271404 2382972825 53267 0 0 2027360 0 0 0 0 05 14 9549.34 2867.80 16897.12 206% 7406864 54830774 2376222762 57989 0 0 2027376 0 0 0 0 05 15 8912.57 2935.90 17301.71 227% 8389127 54353519 2383108921 52124 0 0 1883312 0 0 0 0 05 16 8780.23 2935.33 17286.67 230% 8417447 54242775 2376062987 52913 0 0 2027248 0 0 0 0 06 04 8514.25 3134.09 18582.59 255% 12856900 50543451 2375930920 52359 0 0 2027248 0 0 0 0 06 05 8502.72 3138.54 18576.75 255% 12884270 50607027 2376254598 52315 0 0 2027248 0 0 0 0 06 06 8485.14 3144.19 18611.51 256% 12892763 50365642 2376118338 53214 0 0 2027248 0 0 0 0 06 07 8452.52 3123.77 18596.88 256% 13006966 50461438 2376229877 52251 0 0 2027248 0 0 0 0 06 08 8450.55 3135.17 18586.06 257% 12926578 50394462 2375790855 52833 0 0 2027232 0 0 0 0 06 09 8473.10 3123.47 18554.24 255% 12696799 50642273 2375742839 52867 0 0 2027248 0 0 0 0 06 10 8491.59 3125.92 18580.78 255% 12931115 50381662 2375931612 51677 0 0 2027232 0 0 0 0
Here we can see diminishing returns around 7 threads or so for PARALLEL_MAKE, and continue to see improvements going from 4 to 6 threads on BB_NUMBER_THREADS. I suspect this will find an optimal build time with BB=4 and PM=6, but we'll see (it should be around Jul 1 if things proceed on track and I don't melt the machine).
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Linux Kernel
Re: [RFC PATCH 1/1] local.conf.sample: update suggestions for BB_NUMBER_THREADS and PARALLEL_MAKE
Quoted 88 lines · Darren Hart
Quoted 32 lines · Tom Rini
Quoted 21 lines · Darren Hart
Quoted 18 lines · Joshua Lock
It's been suggested that
BB_NUMBER_THREADSshould be 2 * the number of cores andPARALLEL_MAKEshould be equal to the number of cores available on the build machine.Update local.conf.sample to suggest this.
Signed-off-by Joshua Lock <[email protected]>--- meta-yocto/conf/local.conf.sample | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)meta-yocto/conf/local.conf.sample +0
-0@@ -9,7 +9,9 @@ CONF_VERSION = "1" #SSTATE_DIR ?= "${TOPDIR}/sstate-cache"Quoted 9 lines
meta-yocto/conf/local.conf.sample CONF_VERSION = "1" · line 14
Quoted 5 lines
meta-yocto/conf/local.conf.sample +3
-1# Uncomment and set to allow bitbake to execute multiple tasks at once. -# For a quadcore, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would +# Recommended values are twice the number of processor cores for +# BB_NUMBER_THREADS and the number of processor cores for PARALLEL_MAKE +# For a quadcore, BB_NUMBER_THREADS = "8", PARALLEL_MAKE = "-j 4" wouldHrm, where is this coming from? In my experience it works better the other way around. We probably also need to be explicit about cores versus threads.
Quoted 3 lines
On my older quad-core AMD box, -j 6 / 4 threads is where it's at, and our testing / poking around on other hardware bears that out (for example my Dell M4400 laptop is -j 3 / 2 threads).
Those ratios are closer to what I have seen as optimal as well - specifically more
PARALLEL_MAKEthreads thanBB_NUMBER_THREADS, which is the opposite of the suggestion made above.Quoted 6 lines
That said, for much
more beefy configurations, we use -j 16, and 12 threads (on an 8 core machine with 12GB mem). I think perhaps the best change here is to keep it at 1:1 in the sample (since we've also run into older hardware too) and explain that anywhere between 1:1 and 2*core:2*core could work best depending on setup, ymmv, etc.I'm well into the data collection process I promised in my first response, with several days still remaining. However, here is a snapshot of the results. This is a quad-core i7.
BB PM Seconds ...
04 04 9083.43 2704.41 16073.55 206% 5079496 57424647 2377276782 51099 0 0 1910864 0 0 0 0 04 05 9032.61 2708.10 16093.60 208% 5114730 56938109 2377140907 52246 0 0 2027232 0 0 0 0 04 06 9031.50 2711.30 16095.67 208% 5154734 56737116 2377276988 52525 0 0 1937824 0 0 0 0 04 07 9022.27 2711.39 16150.86 209% 5165928 56811664 2377060362 52531 0 0 2027248 0 0 0 0 04 08 9087.11 2716.88 16103.36 207% 5061918 56763715 2376970987 52525 0 0 1940448 0 0 0 0 04 09 9310.98 2698.72 16080.94 201% 5159166 56868437 2377495925 53496 0 0 2027232 0 0 0 0 04 10 9243.03 2709.35 16077.68 203% 4983749 56880160 2377210887 53044 0 0 2027360 0 0 0 0 04 11 9139.86 2711.91 16085.86 205% 5108951 56824250 2377433181 53253 0 0 2027232 0 0 0 0 04 12 9132.83 2719.61 16066.65 205% 4995133 56904875 2377286845 53102 0 0 1857632 0 0 0 0 04 13 9169.78 2715.16 16095.05 205% 5046506 56856301 2376970111 52827 0 0 2027248 0 0 0 0 04 14 9079.85 2715.04 16082.28 207% 5075792 56804956 2377616169 53000 0 0 1912128 0 0 0 0 04 15 9133.57 2710.45 16096.39 205% 5101938 56804976 2377283916 53125 0 0 2027360 0 0 0 0 04 16 9200.60 2728.44 16129.32 204% 4959845 56852830 2381322571 53793 0 0 2027248 0 0 0 0 05 04 8685.73 2942.91 17328.96 233% 8572490 54322660 2383126091 52468 0 0 2027232 0 0 0 0 05 05 9109.86 2964.68 18033.54 230% 8637358 54222700 2380041373 54070 0 0 2027248 0 0 0 0 05 06 8681.89 2950.25 17384.38 234% 8466556 54244954 2382944050 53202 0 0 2027360 0 0 0 0 05 07 8683.63 2950.78 17380.57 234% 8420681 54277047 2382977368 52970 0 0 2027360 0 0 0 0 05 08 8845.89 2922.37 17296.46 228% 8356145 54197213 2375969732 52062 0 0 2027232 0 0 0 0 05 09 8801.84 2932.86 17509.07 232% 8508554 54179792 2383768769 52943 0 0 2027216 0 0 0 0 05 10 8742.35 2942.06 17341.79 232% 8475095 54323068 2383570357 52587 0 0 2027232 0 0 0 0 05 11 8798.83 2943.76 17428.09 231% 8821969 54234482 2375921193 53157 0 0 2027232 0 0 0 0 05 12 8805.69 2947.29 17417.49 231% 8753090 54442047 2383254479 52402 0 0 1883440 0 0 0 0 05 13 8834.71 2948.17 17435.29 230% 8794659 54271404 2382972825 53267 0 0 2027360 0 0 0 0 05 14 9549.34 2867.80 16897.12 206% 7406864 54830774 2376222762 57989 0 0 2027376 0 0 0 0 05 15 8912.57 2935.90 17301.71 227% 8389127 54353519 2383108921 52124 0 0 1883312 0 0 0 0 05 16 8780.23 2935.33 17286.67 230% 8417447 54242775 2376062987 52913 0 0 2027248 0 0 0 0 06 04 8514.25 3134.09 18582.59 255% 12856900 50543451 2375930920 52359 0 0 2027248 0 0 0 0 06 05 8502.72 3138.54 18576.75 255% 12884270 50607027 2376254598 52315 0 0 2027248 0 0 0 0 06 06 8485.14 3144.19 18611.51 256% 12892763 50365642 2376118338 53214 0 0 2027248 0 0 0 0 06 07 8452.52 3123.77 18596.88 256% 13006966 50461438 2376229877 52251 0 0 2027248 0 0 0 0 06 08 8450.55 3135.17 18586.06 257% 12926578 50394462 2375790855 52833 0 0 2027232 0 0 0 0 06 09 8473.10 3123.47 18554.24 255% 12696799 50642273 2375742839 52867 0 0 2027248 0 0 0 0 06 10 8491.59 3125.92 18580.78 255% 12931115 50381662 2375931612 51677 0 0 2027232 0 0 0 0
Quoted 7 lines
Here we can see diminishing returns around 7 threads or so for
PARALLEL_MAKE, and continue to see improvements going from 4 to 6 threads onBB_NUMBER_THREADS. I suspect this will find an optimal build time with BB=4 and PM=6, but we'll see (it should be around Jul 1 if things proceed on track and I don't melt the machine).
The attached files present the data for BB [4-8] and PM [4-16]. I lost power during the 9,1 run, but I think the data is adequate as is. I see an optimal run with a BB of 7 and a PM of 6. I am not seeing the variation with PM that I thought I should.
NOTE: Some testing revealed that bitbake is ignoring PARALLEL_MAKE from the environment, this is consistent with the rather long build times I was experiencing as well. I am going to rework the script to force the value in the local.conf and kick it off again for two weeks. But, for now, the data below is still interesting assuming a constant PM.
The system under test:
CPU (1): Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz
Cores: 4
Threads: 8
Memory: 8186560 kB
OS Disk: INTEL SSDSA2M040G2GC (SSD)
Build Disk: Hitachi HDT721050SLA360 (Spinning Media)
plot.png
This is an isometrix 3D surface plot with a density map of sorts applied to the xy plane. Optimal build time occurs here around BB=7 PM=6.
plot-bb.png
Effectively a runtime (y) vs BB (x) plot. Fairly little gain from increasing BB beyond 6 (1.5 * NR_CORES).
plot-pm.png
Effectively a runtime (y) vs PM (x) plot. Some improvement is seen from increasing PM from 4 to 5 and 6 for large values of BB. After 6 there is no improvement and a trend toward diminishing returns is certainly evident beyond 6. (or this would be analysis it bitbake was paying attention to PARALLEL_MAKE from env).
You recreate these plots, run the following with the .plt and .dat attachments in the same directory:
$ gnuplot -persist < bb-pm-matrix.plt
This will regenerate the plots and display an interactive view of the plot which allows for manipulating the viewpoint.
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Linux Kernel
set xlabel "BB_NUMBER_THREADS"
set ylabel "PARALLEL_MAKE"
set zlabel "Build Time (seconds)"
set dgrid3d 13,9
set pm3d at b
set ticslevel 0.8
set term png
set output "plot.png"
splot "bb-pm-runtime.dat" u 1:2:3 with lines
set view 90,0
set output "plot-bb.png"
replot
set view 90,90
set output "plot-pm.png"
replot
set view 60,30
set term wxt
replot