local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE
PATCH 1/1 local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
---
meta/conf/local.conf.sample +6 -0
@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" # # The second option controls how many processes make should run in parallel when # running compile tasks: # #PARALLEL_MAKE ?= "-j 4" # +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example.
Quoted 8 lines · Richard Purdie
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Quoted 9 lines · Otavio Salvador
Quoted 7 lines · Richard Purdie
Its rather sad that people don't appear to read local.conf and then complain
about slow builds when they're just using a single thread. Most systems have
more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
Quoted 3 lines
Quoted 3 lines
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>
Nice! Now if we could just convince people to configure their VM guests with enough cpus -
:-)
Quoted 2 lines · David C
Nice! Now if we could just convince people to configure their VM guests with enough cpus -
Now if only we could make bitbake error out when it's running inside a VM. ;)
Ross
Op 27 jan. 2014, om 15:39 heeft Richard Purdie <[email protected]> het volgende geschreven:
Quoted 14 lines
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +0
-0
meta/conf/local.conf.sample line 23
Quoted 6 lines
meta/conf/local.conf.sample +3
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
I've noticed that after 4 threads IO becomes a big bottleneck when you have things like webkit, qt, asio etc in the buildqueue. Combine that with issues like every make -j thread taking >2GB ram with asio and webkit this default seems a bit high. I'd use 0.5*numcpu with a lower bound of 2.
regards,
Koen
Quoted 19 lines
meta/conf/local.conf.sample +3
-0# # The second option controls how many processes make should run in parallel when # running compile tasks: # #PARALLEL_MAKE ?= "-j 4" # +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example._______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 20 lines · Koen Kooi
Op 27 jan. 2014, om 15:39 heeft Richard Purdie <[email protected]> het volgende geschreven:
Quoted 18 lines
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +1
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +#
Quoted 8 lines
meta/conf/local.conf.sample line 23
Quoted 2 lines
meta/conf/local.conf.sample +2
-0+# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"I've noticed that after 4 threads IO becomes a big bottleneck when you have things like webkit, qt, asio etc in the buildqueue. Combine that with issues like every make -j thread taking >2GB ram with asio and webkit this default seems a bit high. I'd use 0.5*numcpu with a lower bound of 2.
This is one of those things I think we'll not find a perfect answer to and its why I left the defaults alone for as long.
Personally, I run with 48 threads enabled and I've yet to see it do anything too crazy. At the end of the day it all comes down to the hardware you're running it on though.
webkit is a particular case where whilst its building you'd probably want different values. Which numbers you use depends on how often you're building it...
I'm not sure we can win. If the build appears to take over a system, I'm hoping the user will look at the conf file and figure out how to make it use less of the system...
Cheers,
Richard
2014-01-28 11:08, Koen Kooi skrev:
Quoted 24 lines
Op 27 jan. 2014, om 15:39 heeft Richard Purdie <[email protected]> het volgende geschreven:
Quoted 14 lines
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +0
-0meta/conf/local.conf.sample line 23
Quoted 6 lines
meta/conf/local.conf.sample +3
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"I've noticed that after 4 threads IO becomes a big bottleneck when you have things like webkit, qt, asio etc in the buildqueue. Combine that with issues like every make -j thread taking >2GB ram with asio and webkit this default seems a bit high. I'd use 0.5*numcpu with a lower bound of 2.
Quoted 1 line
regards,
We discussed this 2.3 months ago.
Did some studies on my dual hex-core machine (24 H/W treads) while building a cloud9-gnome-image derivative. This did about 7500 tasks.
Enabled the CPU supervisors in the panel.
Everything seems to be ok with BB_NUMBER_THREADS = "24" for about 4-4500 tasks.
Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks. Then parallellism is resumed until about task 7000, and again only a few CPUs are active.
I believe that some tools use "make" within the Makefile, and they are written badly, and do not use "-j <n>" for that part of the build.
Got my build down to 83 minutes.
Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build, and copied the download and the recipes to the ram.
That shaved only 2 monutes from the build, and some stuff, still built using only a single CPU.
BR
Ulf Samuelsson
Quoted 25 lines
Koen
Quoted 14 lines
meta/conf/local.conf.sample +3
-0# # The second option controls how many processes make should run in parallel when # running compile tasks: # #PARALLEL_MAKE ?= "-j 4" # +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example.Quoted 5 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 3 lines
Best Regards Ulf Samuelsson eMagii
Quoted 5 lines · Ulf Samuelsson
We discussed this 2.3 months ago.
Did some studies on my dual hex-core machine (24 H/W treads) while building a cloud9-gnome-image derivative.
This did about 7500 tasks.
Quoted 8 lines
Enabled the CPU supervisors in the panel.
Everything seems to be ok with
BB_NUMBER_THREADS= "24" for about 4-4500 tasks.Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks. Then parallellism is resumed until about task 7000, and again only a few CPUs are active.
This is likely whilst the lib and toolchain is getting built.
Quoted 3 lines
I believe that some tools use "make" within the
Makefile, and they are written badly, and do not use "-j <n>" for that part of the build.
Which recipes were building at this point? It would be interesting to track them down.
Quoted 7 lines
Got my build down to 83 minutes.
Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build, and copied the download and the recipes to the ram.
That shaved only 2 monutes from the build, and some stuff, still built using only a single CPU.
There are certainly dependency bottlenecks in the build such as the toolchain, compiler, gettext, gtk+ and so where large numbers of things need those dependencies to get built before they can proceed. Not sure what we can do to help this though.
Cheers,
Richard
2014-01-29 13:56, Richard Purdie skrev:
Quoted 14 lines · Ulf Samuelsson
Quoted 5 lines
We discussed this 2.3 months ago.
Did some studies on my dual hex-core machine (24 H/W treads) while building a cloud9-gnome-image derivative.
This did about 7500 tasks.Quoted 8 lines
Enabled the CPU supervisors in the panel.
Everything seems to be ok with
BB_NUMBER_THREADS= "24" for about 4-4500 tasks.Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks. Then parallellism is resumed until about task 7000, and again only a few CPUs are active.
Quoted 7 lines
This is likely whilst the lib and toolchain is getting built.
Quoted 3 lines
I believe that some tools use "make" within the
Makefile, and they are written badly, and do not use "-j <n>" for that part of the build.Which recipes were building at this point? It would be interesting to track them down.
I think eglibc, node[-js] (native) and a few others
Think I remember mentioning them when it was discussed.
webkit, node are also not running in parallell in later part of the build
Quoted 6 lines
Quoted 6 lines
Got my build down to 83 minutes.
Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build, and copied the download and the recipes to the ram.
Quoted 6 lines
Quoted 2 lines
That shaved only 2 monutes from the build, and some stuff, still built using only a single CPU.
There are certainly dependency bottlenecks in the build such as the toolchain, compiler, gettext, gtk+ and so where large numbers of things need those dependencies to get built before they can proceed. Not sure what we can do to help this though.
In order to find out more, perhaps it should be possible to log info about the build.
1. How many makes are in progress when you start a task
2. How long does it take to build a recipe.
3. CPU activity (frequency scaling) when the recipe is build.
A recipe which takes very long time to build but with low CPU activity should be analyzed to find out if more parallellism can be introduced.
BR
Ulf
Quoted 5 lines
Cheers,
Richard
Signature · 4 lines
Best Regards Ulf Samuelsson [email protected] +46 722 427437
webkit, node are also not running in parallell in later part of the build
WebKit does parallel builds internally so what you’re seeing here is everything that doesn’t depend on webkit has finished, so the remaining tasks are blocking on the webkit build.
Ross
--
Ross Burton
Open Source Technology Centre
Intel Corporation
Quoted 15 lines · Richard Purdie
Quoted 5 lines · Ulf Samuelsson
We discussed this 2.3 months ago.
Did some studies on my dual hex-core machine (24 H/W treads) while building a cloud9-gnome-image derivative.
This did about 7500 tasks.Quoted 8 lines
Enabled the CPU supervisors in the panel.
Everything seems to be ok with
BB_NUMBER_THREADS= "24" for about 4-4500 tasks.Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks. Then parallellism is resumed until about task 7000, and again only a few CPUs are active.
Quoted 8 lines
This is likely whilst the lib and toolchain is getting built.
Quoted 3 lines
I believe that some tools use "make" within the
Makefile, and they are written badly, and do not use "-j <n>" for that part of the build.Which recipes were building at this point? It would be interesting to track them down.
I would suspect "recursive" makes. If you use autotools with the evil "SUDIRS=..." construction, it will WAIT for that dir to finish before doing anything else.
It's much better to construct a giant makefile in the root, autotools will happily do that once properly instructed. I've seen first time builds going from half an hour to two minutes, and incremental builds taking only one or two seconds instead of several minutes just because I removed the recursion.
Quoted 7 lines
Quoted 7 lines
Got my build down to 83 minutes.
Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build, and copied the download and the recipes to the ram.
That shaved only 2 monutes from the build, and some stuff, still built using only a single CPU.
That confirms what I have already suspected - it is pointless to buy an SSD, building OE is mostly CPU limited and hardly I/O related.
I guess the only way to really speed up the build would be to have multiple machines participate in it. Single machines aren't really getting any faster.
Quoted 4 lines
There are certainly dependency bottlenecks in the build such as the toolchain, compiler, gettext, gtk+ and so where large numbers of things need those dependencies to get built before they can proceed. Not sure what we can do to help this though.
Move them to the front as far as possible I guess. And any package they depend on as well... It should try to set up the shortest tree to be able to build the crosscompiler and build that first...
Mike.
1 feb 2014 kl. 10:21 skrev Mike Looijmans <[email protected]>:
Quoted 30 lines · Richard Purdie
Quoted 15 lines · Ulf Samuelsson
Quoted 5 lines
We discussed this 2.3 months ago.
Did some studies on my dual hex-core machine (24 H/W treads) while building a cloud9-gnome-image derivative.
This did about 7500 tasks.Quoted 8 lines
Enabled the CPU supervisors in the panel.
Everything seems to be ok with
BB_NUMBER_THREADS= "24" for about 4-4500 tasks.Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks. Then parallellism is resumed until about task 7000, and again only a few CPUs are active.
Quoted 8 lines
This is likely whilst the lib and toolchain is getting built.
Quoted 3 lines
I believe that some tools use "make" within the
Makefile, and they are written badly, and do not use "-j <n>" for that part of the build.Which recipes were building at this point? It would be interesting to track them down.
I would suspect "recursive" makes. If you use autotools with the evil "SUDIRS=..." construction, it will WAIT for that dir to finish before doing anything else.
It's much better to construct a giant makefile in the root, autotools will happily do that once properly instructed. I've seen first time builds going from half an hour to two minutes, and incremental builds taking only one or two seconds instead of several minutes just because I removed the recursion.Quoted 2 lines
Quoted 2 lines
Got my build down to 83 minutes.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build, and copied the download and the recipes to the ram.
That shaved only 2 monutes from the build, and some stuff, still built using only a single CPU.
That confirms what I have already suspected - it is pointless to buy an SSD, building OE is mostly CPU limited and hardly I/O related.
To be fair, the 83 minute build was on a RAID with two striped 600 GB, 15k RPM, 3 Gbps SAS disks (luckily on the machine I got on eBay :-) hdparm -t -T give about 390 MB/s.
If I run a single late model SATA-III disk (on a SATA-II) I get 120-150 MB/s with the same test.
Using 15k RPM SAS disks will give 2 x the number of seeks per second vs the SATA disks, (170 vs 85 IIRC)
Using RAID does not increase this value.
Quoted 10 lines
I guess the only way to really speed up the build would be to have multiple machines participate in it. Single machines aren't really getting any faster.
Quoted 4 lines
There are certainly dependency bottlenecks in the build such as the toolchain, compiler, gettext, gtk+ and so where large numbers of things need those dependencies to get built before they can proceed. Not sure what we can do to help this though.
Move them to the front as far as possible I guess. And any package they depend on as well... It should try to set up the shortest tree to be able to build the crosscompiler and build that first...
If I understand things correctly, bitbake will calculate a priority for a certain package based on how many packages are depending on this package.
If you could add a value to the priority, specified in the recipe
then you could make the build of the package occur earlier.
If bitbakes can track the loading of the CPU, then maybe some parts
in the later part of each recipe can be delayed until loading is low (maybe already doing this).
If the toolchain/native stuff could be pulled down from packages as part of a build it would be real cool.
Today you can install cross compilers for certain Linux distributions. Maybe there should be a possibility to generate automatically the .rpm/.deb etc. and make that the default way to get stuff done for those distributions.
Best Regards
Ulf Samuelsson
Quoted 9 lines
Mike.
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 6 lines · Ulf Samuelsson
1 feb 2014 kl. 10:21 skrev Mike Looijmans <[email protected]>:
Quoted 2 lines · Richard Purdie
Quoted 1 line · Ulf Samuelsson
If bitbakes can track the loading of the CPU, then maybe some parts
in the later part of each recipe can be delayed until loading is low (maybe already doing this).
It doesn't. We did try this, the Linux scheduler is in fact rather good and we only ever slowed down builds when we tried adding hints.
Quoted 2 lines
If the toolchain/native stuff could be pulled down from packages as part of a build it would be real cool.
This is called sstate ;-).
Quoted 3 lines
Today you can install cross compilers for certain Linux distributions. Maybe there should be a possibility to generate automatically the .rpm/.deb etc. and make that the default way to get stuff done for those distributions.
Its a maintenance/bug report nightmare...
Cheers,
Richard
Koen Kooi <[email protected]>
Quoted 8 lines · writes:
meta/conf/local.conf.sample line 23
Quoted 2 lines
meta/conf/local.conf.sample +2
-0+# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"I've noticed that after 4 threads IO becomes a big bottleneck when you have things like webkit, qt, asio etc in the buildqueue. Combine that with issues like every make -j thread taking >2GB ram with asio and webkit this default seems a bit high. I'd use 0.5*numcpu with a lower bound of 2.
limitting the load mitigates this (high i/o increases it); e.g.
PARALLEL_MAKE = "\
...
-l ${@int(os.sysconf(os.sysconf_names['SC_NPROCESSORS_ONLN'])) * 150/100} \
"
Enrico
Quoted 12 lines · Enrico Scholz
Koen Kooi <[email protected]>
Quoted 8 lines · writes:
meta/conf/local.conf.sample line 23
Quoted 2 lines
meta/conf/local.conf.sample +2
-0+# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"I've noticed that after 4 threads IO becomes a big bottleneck when you have things like webkit, qt, asio etc in the buildqueue. Combine that with issues like every make -j thread taking >2GB ram with asio and webkit this default seems a bit high. I'd use 0.5*numcpu with a lower bound of 2.
Quoted 6 lines
limitting the load mitigates this (high i/o increases it); e.g.
PARALLEL_MAKE = "\ ... -l ${@int(os.sysconf(os.sysconf_names['SC_NPROCESSORS_ONLN'])) * 150/100} \"
FYI, I think this points out the variability in system performance, between CPU, RAM and I/o.
As it stands the patch gives my machine the best performance. So I like it as it is. But my machine (dual 8-core w/ HT, 64 GB of RAM, and hardware raid). But on hardware with less RAM, slower disk, it may not perform optimally.
So the catch is what is the proper optimal setting? As I see it, assuming that the system has enough ram and I/O to fill the CPUs is the best approach (what was implemented.) And then in the comments document that this may not be the best setting for all systems, and the user should adjust it as necessary. Even suggesting some of the alternative systems such as the 150/100 above.
No setting is going to make everyone happy, but something has to be better then defaulting to '1'.
--Mark
Quoted 6 lines
Enrico
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 13 lines · Mark Hatle
Quoted 6 lines
Quoted 5 lines
I've noticed that after 4 threads IO becomes a big bottleneck when you have things like webkit, qt, asio etc in the buildqueue. Combine that with issues like every make -j thread taking >2GB ram with asio and webkit this default seems a bit high. I'd use 0.5*numcpu with a lower bound of 2.
Quoted 6 lines
limitting the load mitigates this (high i/o increases it); e.g.
PARALLEL_MAKE = "\ ... -l ${@int(os.sysconf(os.sysconf_names['SC_NPROCESSORS_ONLN'])) * 150/100} \"
Quoted 5 lines
FYI, I think this points out the variability in system performance, between CPU, RAM and I/o.
As it stands the patch gives my machine the best performance. So I like it as it is.
Point of my posting was not the exact scale (I just copied a piece of my ~/.bitbake.conf), but using '-l' in addition to '-j'. Doing heavy i/o or being low on RAM (which causes swapping) increases the load and 'make' will throttle then.
Quoted 2 lines
No setting is going to make everyone happy but something has to be better then defaulting to '1'.
That's true; but not due to performance gain but because it will help to detect races in the build process.
Enrico
Quoted 21 lines · Richard Purdie
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +3
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" #
meta/conf/local.conf.sample line 34
Quoted 10 lines
meta/conf/local.conf.sample +3
-0# The second option controls how many processes make should run in parallel when # running compile tasks: # #PARALLEL_MAKE ?= "-j 4" # +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example.
On our Fedora-18 build host this change leads to the following exception. :(
----------------------> snip <------------------------- Loading cache...done.
Loaded 2167 entries from dependency cache.
Parsing recipes...done.
Parsing of 1729 .bb files complete (1727 cached, 2 parsed). 2168 targets, 85 skipped, 1 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.20.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "Fedora-18" TARGET_SYS = "arm-angstrom-linux-gnueabi" MACHINE = "hikirk" DISTRO = "angstrom" DISTRO_VERSION = "v2013.12" TUNE_FEATURES = "armv5 thumb dsp" TARGET_FPU = "soft" meta-hipos = "(nobranch):6f2ad6754c9c9ddd6f1c5505e584e0eb62cfcf83"
meta-hipos-kirkwood = "(nobranch):1a43f75e217fea3c6b4493536b3b66945ce53921"
meta-hipos-fsl = "(nobranch):ad3beec4157a96d0c506969a29d14f6354e7406f"
meta-fsl-arm = "(nobranch):af392c22bf6b563525ede4a81b6755ff1dd2c1c6"
meta-fsl-arm-extra = "(nobranch):07ad83db0fb67c5023bd627a61efb7f474c52622"
meta-java = "(nobranch):9dfb7ecb87e67052e94641d6e02907c582264217"
meta-angstrom = "(nobranch):8fcd781b777a7a94f2b97514c9691a2b5c74eaf2"
meta-oe
meta-systemd
meta-multimedia
meta-networking
meta-efl
meta-gnome = "(nobranch):eb4563b83be0a57ede4269ab19688af6baa62cd2"
meta = "(nobranch):e86622a932bbd0acdea67ecfb15c8b06c27353d8"
NOTE: Preparing runqueue
ERROR: An uncaught exception occured in runqueue, please see the failure below:
ERROR: Running idle function
Traceback (most recent call last):
File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/runqueue.py", line 1111, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue instance at 0x1534b2d8>):
meta/conf/local.conf.sample line 35
Quoted 1 line
meta/conf/local.conf.sample +0
-0self.number_tasks = int(self.cfgData.getVar("BB_NUMBER_THREADS", True) or 1)
self.scheduler = self.cfgData.getVar("BB_SCHEDULER", True) or "speed"
File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 516, in DataSmart.getVar(var='BB_NUMBER_THREADS', expand=True, noweakdefault=False):
def getVar(self, var, expand=False, noweakdefault=False):
> return self.getVarFlag(var, "_content", expand, noweakdefault)
File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 606, in DataSmart.getVarFlag(var='BB_NUMBER_THREADS', flag='_content', expand=True, noweakdefault=False):
cachename = var + "[" + flag + "]"
> value = self.expand(value, cachename)
if value is not None and flag == "_content" and local_var is not None and "_removeactive" in local_var: File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 344, in DataSmart.expand(s='${@oe.utils.cpu_count()}', varname='BB_NUMBER_THREADS'):
def expand(self, s, varname = None):
> return self.expandWithRefs(s, varname).value
File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 334, in DataSmart.expandWithRefs(s='${@oe.utils.cpu_count()}', varname='BB_NUMBER_THREADS'):
except Exception as exc:
> raise ExpansionError(varname, s, exc)
ExpansionError: Failure expanding variable BB_NUMBER_THREADS, expression was ${@oe.utils.cpu_count()} which triggered exception AttributeError: 'module' object has no attribute 'cpu_count' ----------------------> snip <-------------------------
Any ideas? A missing python package at the build host? ...?
Signature · 7 lines
DResearch Fahrzeugelektronik GmbH Otto-Schmirgal-Str. 3, 10319 Berlin, Germany Tel: +49 30 515932-237 mailto:[email protected] Fax: +49 30 515932-299 Geschäftsführer: Dr. Michael Weber, Werner Mögle; Amtsgericht Berlin Charlottenburg; HRB 130120 B; Ust.-IDNr. DE273952058
Quoted 76 lines · On Wednesday 29 January 2014 11:14:47 Steffen Sledz wrote:
Quoted 27 lines · Richard Purdie
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +3
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" # # The second option controls how many processes make should run in parallel when # running compile tasks:meta/conf/local.conf.sample
Quoted 10 lines
meta/conf/local.conf.sample +3
-0# #PARALLEL_MAKE ?= "-j 4" # +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example.On our Fedora-18 build host this change leads to the following exception. :(
----------------------> snip <------------------------- Loading cache...done.
Loaded 2167 entries from dependency cache.Parsing recipes...done.
Parsing of 1729 .bb files complete (1727 cached, 2 parsed). 2168 targets, 85 skipped, 1 masked, 0 errors.NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.20.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "Fedora-18" TARGET_SYS = "arm-angstrom-linux-gnueabi" MACHINE = "hikirk" DISTRO = "angstrom" DISTRO_VERSION = "v2013.12" TUNE_FEATURES = "armv5 thumb dsp" TARGET_FPU = "soft" meta-hipos = "(nobranch):6f2ad6754c9c9ddd6f1c5505e584e0eb62cfcf83"meta-hipos-kirkwood = "(nobranch):1a43f75e217fea3c6b4493536b3b66945ce53921"
meta-hipos-fsl = "(nobranch):ad3beec4157a96d0c506969a29d14f6354e7406f"
meta-fsl-arm = "(nobranch):af392c22bf6b563525ede4a81b6755ff1dd2c1c6"
meta-fsl-arm-extra = "(nobranch):07ad83db0fb67c5023bd627a61efb7f474c52622"
meta-java = "(nobranch):9dfb7ecb87e67052e94641d6e02907c582264217"
meta-angstrom = "(nobranch):8fcd781b777a7a94f2b97514c9691a2b5c74eaf2"
meta-oe
meta-systemd
meta-multimedia
meta-networking
meta-efl
meta-gnome = "(nobranch):eb4563b83be0a57ede4269ab19688af6baa62cd2"
meta = "(nobranch):e86622a932bbd0acdea67ecfb15c8b06c27353d8"
Quoted 6 lines
NOTE: Preparing runqueue
ERROR: An uncaught exception occured in runqueue, please see the failure
below: ERROR: Running idle function
Traceback (most recent call last):File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/runqueue.py", line
1111, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue instance at 0x1534b2d8>):
Quoted 29 lines
Quoted 2 lines
self.number_tasks =
int(self.cfgData.getVar("BB_NUMBER_THREADS", True) or 1)self.scheduler =
self.cfgData.getVar("BB_SCHEDULER", True)or
"speed" File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 516, inDataSmart.getVar(var='BB_NUMBER_THREADS', expand=True, noweakdefault=False):
defgetVar(self, var, expand=False, noweakdefault=False):
> returnself.getVarFlag(var, "_content", expand, noweakdefault)File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line
606, inDataSmart.getVarFlag(var='BB_NUMBER_THREADS', flag='_content', expand=True, noweakdefault=False): cachename = var + "[" + flag + "]"Quoted 1 line
value =
self.expand(value, cachename)if value is not None and flag == "_content" and local_var isnot None and "
_removeactive" inlocal_var: File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 344, inDataSmart.expand(s='${@oe.utils.cpu_count()}', varname='BB_NUMBER_THREADS'):
defexpand(self, s, varname = None):
> returnself.expandWithRefs(s, varname).valueFile "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line
334, inDataSmart.expandWithRefs(s='${@oe.utils.cpu_count()}', varname='BB_NUMBER_THREADS'):
except Exception as exc:
> raiseExpansionError(varname, s, exc)
Quoted 6 lines
ExpansionError: Failure expanding variable
BB_NUMBER_THREADS, expression was ${@oe.utils.cpu_count()} which triggered exception AttributeError: 'module' object has no attribute 'cpu_count' ----------------------> snip <-------------------------Any ideas? A missing python package at the build host? ...?
Looks like you're missing Richard's other posted patch that adds this function.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 119 lines · Paul Eggleton
Quoted 76 lines · On Wednesday 29 January 2014 11:14:47 Steffen Sledz wrote:
Quoted 27 lines · Richard Purdie
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +3
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" # # The second option controls how many processes make should run in parallel when # running compile tasks:meta/conf/local.conf.sample
Quoted 10 lines
meta/conf/local.conf.sample +3
-0# #PARALLEL_MAKE ?= "-j 4" # +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example.On our Fedora-18 build host this change leads to the following exception. :(
----------------------> snip <------------------------- Loading cache...done.
Loaded 2167 entries from dependency cache.Parsing recipes...done.
Parsing of 1729 .bb files complete (1727 cached, 2 parsed). 2168 targets, 85 skipped, 1 masked, 0 errors.NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.20.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "Fedora-18" TARGET_SYS = "arm-angstrom-linux-gnueabi" MACHINE = "hikirk" DISTRO = "angstrom" DISTRO_VERSION = "v2013.12" TUNE_FEATURES = "armv5 thumb dsp" TARGET_FPU = "soft" meta-hipos = "(nobranch):6f2ad6754c9c9ddd6f1c5505e584e0eb62cfcf83"meta-hipos-kirkwood = "(nobranch):1a43f75e217fea3c6b4493536b3b66945ce53921"
meta-hipos-fsl = "(nobranch):ad3beec4157a96d0c506969a29d14f6354e7406f"
meta-fsl-arm = "(nobranch):af392c22bf6b563525ede4a81b6755ff1dd2c1c6"
meta-fsl-arm-extra = "(nobranch):07ad83db0fb67c5023bd627a61efb7f474c52622"
meta-java = "(nobranch):9dfb7ecb87e67052e94641d6e02907c582264217"
meta-angstrom = "(nobranch):8fcd781b777a7a94f2b97514c9691a2b5c74eaf2"
meta-oe
meta-systemd
meta-multimedia
meta-networking
meta-efl
meta-gnome = "(nobranch):eb4563b83be0a57ede4269ab19688af6baa62cd2"
meta = "(nobranch):e86622a932bbd0acdea67ecfb15c8b06c27353d8"Quoted 6 lines
NOTE: Preparing runqueue
ERROR: An uncaught exception occured in runqueue, please see the failure
below: ERROR: Running idle function
Traceback (most recent call last):File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/runqueue.py", line
1111, in
RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue instance at 0x1534b2d8>):Quoted 34 lines
Quoted 2 lines
self.number_tasks =
int(self.cfgData.getVar("BB_NUMBER_THREADS", True) or 1)self.scheduler =
self.cfgData.getVar("BB_SCHEDULER", True)or
"speed" File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 516, inDataSmart.getVar(var='BB_NUMBER_THREADS', expand=True, noweakdefault=False):
defgetVar(self, var, expand=False, noweakdefault=False):
> returnself.getVarFlag(var, "_content", expand, noweakdefault)File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line
606, inDataSmart.getVarFlag(var='BB_NUMBER_THREADS', flag='_content', expand=True, noweakdefault=False): cachename = var + "[" + flag + "]"Quoted 1 line
value =
self.expand(value, cachename)if value is not None and flag == "_content" and local_var isnot None and "
_removeactive" inlocal_var: File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 344, inDataSmart.expand(s='${@oe.utils.cpu_count()}', varname='BB_NUMBER_THREADS'):
defexpand(self, s, varname = None):
> returnself.expandWithRefs(s, varname).valueFile "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line
334, inDataSmart.expandWithRefs(s='${@oe.utils.cpu_count()}', varname='BB_NUMBER_THREADS'):
except Exception as exc:
> raiseExpansionError(varname, s, exc)ExpansionError: Failure expanding variable
BB_NUMBER_THREADS, expression was ${@oe.utils.cpu_count()} which triggered exception AttributeError: 'module' object has no attribute 'cpu_count' ----------------------> snip <-------------------------
Quoted 4 lines
Quoted 1 line
Any ideas? A missing python package at the build host? ...?
Looks like you're missing Richard's other posted patch that adds this function.
Hmmm? Exactly the same revisions work well on another (openSUSE-13.1) build host.
Signature · 7 lines
DResearch Fahrzeugelektronik GmbH Otto-Schmirgal-Str. 3, 10319 Berlin, Germany Tel: +49 30 515932-237 mailto:[email protected] Fax: +49 30 515932-299 Geschäftsführer: Dr. Michael Weber, Werner Mögle; Amtsgericht Berlin Charlottenburg; HRB 130120 B; Ust.-IDNr. DE273952058
Quoted 26 lines · Steffen Sledz
Quoted 25 lines · Richard Purdie
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +3
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" # # The second option controls how many processes make should run in parallel when # running compile tasks: # #PARALLEL_MAKE ?= "-j 4"
Quoted 8 lines
meta/conf/local.conf.sample line 34
Quoted 6 lines
meta/conf/local.conf.sample +3
-0# +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example.On our Fedora-18 build host this change leads to the following exception. :(
Do you have:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=721773072da08cf0f5e1206961ada3083b6722b4
?
Cheers,
Richard
Quoted 36 lines · Richard Purdie
Quoted 26 lines · Steffen Sledz
Quoted 25 lines · Richard Purdie
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +3
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" # # The second option controls how many processes make should run in parallel when # running compile tasks: # #PARALLEL_MAKE ?= "-j 4"Quoted 8 lines
meta/conf/local.conf.sample line 34
Quoted 6 lines
meta/conf/local.conf.sample +3
-0# +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example.On our Fedora-18 build host this change leads to the following exception. :(
Quoted 3 lines
Do you have:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=721773072da08cf0f5e1206961ada3083b6722b4
No.
But now it becomes really interesting. As mentioned before it works well on our openSUSE-13.1 build host (without this commit)???
Signature · 7 lines
DResearch Fahrzeugelektronik GmbH Otto-Schmirgal-Str. 3, 10319 Berlin, Germany Tel: +49 30 515932-237 mailto:[email protected] Fax: +49 30 515932-299 Geschäftsführer: Dr. Michael Weber, Werner Mögle; Amtsgericht Berlin Charlottenburg; HRB 130120 B; Ust.-IDNr. DE273952058
Quoted 43 lines · On Wednesday 29 January 2014 12:47:54 Steffen Sledz wrote:
Quoted 42 lines · Richard Purdie
Quoted 41 lines · Steffen Sledz
Quoted 27 lines · Richard Purdie
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +3
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" # # The second option controls how many processes make should run in parallel when # running compile tasks:meta/conf/local.conf.sample
Quoted 10 lines
meta/conf/local.conf.sample +3
-0# #PARALLEL_MAKE ?= "-j 4" # +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example.On our Fedora-18 build host this change leads to the following exception. :(>
Quoted 8 lines
Quoted 4 lines
Do you have:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=721773072da08cf0f5e12
06961ada3083b6722b4No.
But now it becomes really interesting. As mentioned before it works well on our openSUSE-13.1 build host (without this commit)???
Are you sure you're not overriding BB_NUMBER_THREADS / PARALLEL_MAKE on the machine where it's "working"? You can use bitbake -e | less (variable history) to be sure.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 48 lines · Paul Eggleton
Quoted 47 lines · On Wednesday 29 January 2014 12:47:54 Steffen Sledz wrote:
Quoted 46 lines · Richard Purdie
Quoted 36 lines · Steffen Sledz
Quoted 35 lines · Richard Purdie
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +6
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" # # The second option controls how many processes make should run in parallel when # running compile tasks: # #PARALLEL_MAKE ?= "-j 4" # +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +#Quoted 5 lines
Quoted 2 lines
# For a quad-core machine,
BB_NUMBER_THREADS= "4",PARALLEL_MAKE= "-j 4" would # be appropriate for example.On our Fedora-18 build host this change leads to the following exception. :(>
Do you have:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=721773072da08cf0f5e12
06961ada3083b6722b4
Quoted 8 lines
Quoted 4 lines
No.
But now it becomes really interesting. As mentioned before it works well on our openSUSE-13.1 build host (without this commit)???
Are you sure you're not overriding
BB_NUMBER_THREADS/PARALLEL_MAKEon the machine where it's "working"? You can use bitbake -e | less (variable history) to be sure.
Yes, this was a good hint.
Sorry for disturbance.
Signature · 7 lines
DResearch Fahrzeugelektronik GmbH Otto-Schmirgal-Str. 3, 10319 Berlin, Germany Tel: +49 30 515932-237 mailto:[email protected] Fax: +49 30 515932-299 Geschäftsführer: Dr. Michael Weber, Werner Mögle; Amtsgericht Berlin Charlottenburg; HRB 130120 B; Ust.-IDNr. DE273952058
Thank you for this contribution.
As a side note, please consider the documentation bits in the future as well. :-) I have just opened a separate ticket for that here:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5770
Quoted 38 lines · Richard Purdie
Its rather sad that people don't appear to read local.conf and then complain about slow builds when they're just using a single thread. Most systems have more than one core now so we might as well use a more automatic default for these values. This may lead to better experiences for new users.
[YOCTO #2528]
Signed-off-by Richard Purdie <[email protected]>---
meta/conf/local.conf.sample +6
-0@@ -18,12 +18,18 @@ # option determines how many tasks bitbake should run in parallel: # #BB_NUMBER_THREADS ?= "4" +# +# Default to setting automatically based on cpu count +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" # # The second option controls how many processes make should run in parallel when # running compile tasks: # #PARALLEL_MAKE ?= "-j 4" # +# Default to setting automatically based on cpu count +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example._______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core