build speedup possible?

Hi guys,

First of all kudos again for this very good tool! I'm using it almsot daily and am stoked about the functionality but simple usage at the same time. Now one issue with build systems is always build time. It seems the speed is not impacted much by just spending more horsepower (cpu cores). When looking at the load I see very frequently that just one core is busy with one package, especially during the configure step. Some packages themselves are build with just one core, I guess we cant do very much about that. However, why are independent packages not build in parallel? Are the dependencies too complex to make this work? Thanks for any insight!

cheers,
Andreas

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Quoted 1 line

Quoted 1 line

Andreas Naumann <[email protected]> 07.02.14 9.52 Uhr >>>

Hi Andreas,

This is not a solution more my experience:

I'm using a 4 Core 3.2 GHz on a Fedora 20 system and for a

During build I'm using GKrellM System Monitor and and shows me that all

Regards,

Frank

Quoted 1 line

Quoted 1 line

Quoted 1 line

Andreas Naumann <[email protected]> 07.02.14 9.52 Uhr >>>

Hi guys,

First of all kudos again for this very good tool! I'm using it almsot daily and am stoked about the functionality but simple usage at the same time.
Now one issue with build systems is always build time. It seems the speed is not impacted much by just spending more horsepower (cpu cores). When looking at the load I see very frequently that just one core is busy with one package, especially during the configure step. Some packages themselves are build with just one core, I guess we cant do very much about that.
However, why are independent packages not build in parallel? Are the dependencies too complex to make this work? Thanks for any insight!

Andreas, Maxime, Frank, all,

Quoted 6 lines · On Fri, Feb 7, 2014 at 10:00 AM, Frank Ihle <[email protected]…

Hi Andreas,

This is not a solution more my experience:

I'm using a 4 Core 3.2 GHz on a Fedora 20 system and for a

at91sam9x5ek_defconfig it takes about 7 min from 0 (without an downloaded DLs) to a bootable Image.

Quoted 2 lines

During build I'm using GKrellM System Monitor and and shows me that

all cores are in use. ATM I'm ok with these results, since i started with a compile time about 45 min.

Quoted 6 lines

Regards,

Frank

Quoted 1 line

Quoted 1 line

Quoted 1 line

Andreas Naumann <dev at andin.de <mailto:[email protected]>> 07.02.14

9.52 Uhr >>>

Quoted 14 lines

Hi guys,

First of all kudos again for this very good tool! I'm using it almsot daily and am stoked about the functionality but simple usage at the same time.
Now one issue with build systems is always build time. It seems the speed is not impacted much by just spending more horsepower (cpu cores). When looking at the load I see very frequently that just one core is busy with one package, especially during the configure step. Some packages themselves are build with just one core, I guess we cant do very much about that.
However, why are independent packages not build in parallel? Are the dependencies too complex to make this work? Thanks for any insight!

Well, this is a tricky thing. I see 3 points on this topic:
- building a package in parallel:
We already do this, unless the package itself (more precisely its
build-system) does not support it.
In such cases, it is explicitly stated in the *.mk file by setting
<PKG>_MAKE = $(MAKE1).
- the package configuration step:
Unfortunately the package configuration is highly serialized and
depends on the build-system used
by the package.
- building packages in parallel:
This point tends to hit some Buildroot's principles:
- simplicity;
- and build reproducibility (don't know if this word really exists in
English...).
Being able to reproduce build (mainly build failures) implies that
the sysroot should not change during

the configuration/build steps of a package, so the sysroot should be
copied for each package built in
parallel (iow, if 4 packages are built in parallel, there will be 4
sysroots at the same time, each of them
with a content that may be slightly different one from the others).
This may introduce some complexity
in Buildroot and it may also make build time worse due to the copies
of the sysroot for each package.

The patch series pointed by Maxime introduces support for building packages in parallel, but does not
implement the sysroot copy thing.
During the Buildroot Developer Days in last November, it has been decided that we won't enable building
packages in parallel (see:
http://elinux.org/Buildroot:DeveloperDaysELCE2013#Parallel_top-level_make ).

@others developers: feel free to correct me, if I speak with my ass ;)

BTW, this could certainly be added in the FAQ.

Regards,

--
Samuel

Thanks for all your input!

I understand the issues with the implicit/optional dependencies which may vary because of unpredictable execution order during the config step. Since determinism is more important to me than speed I wont remove the .NOTPARALLEL directive for now.
However...

Am 07.02.2014 11:31, schrieb Samuel Martin:

Quoted 60 lines

Andreas, Maxime, Frank, all,

Quoted 6 lines · On Fri, Feb 7, 2014 at 10:00 AM, Frank Ihle <[email protected]…

Hi Andreas,

This is not a solution more my experience:

I'm using a 4 Core 3.2 GHz on a Fedora 20 system and for a

at91sam9x5ek_defconfig it takes about 7 min from 0 (without an downloaded DLs) to a bootable Image.

Quoted 2 lines

During build I'm using GKrellM System Monitor and and shows me that

all cores are in use. ATM I'm ok with these results, since i started with a compile time about 45 min.

Quoted 6 lines

Regards,

Frank

Quoted 1 line

Quoted 1 line

Quoted 1 line

Andreas Naumann <dev at andin.de <mailto:[email protected]>> 07.02.14

9.52 Uhr >>>

Quoted 5 lines

Hi guys,

First of all kudos again for this very good tool! I'm using it almsot daily and am stoked about the functionality but simple usage at the

same

Quoted 3 lines

time.
Now one issue with build systems is always build time. It seems the speed is not impacted much by just spending more horsepower (cpu

cores).

Quoted 6 lines

When looking at the load I see very frequently that just one core is busy with one package, especially during the configure step. Some packages themselves are build with just one core, I guess we cant do very much about that.
However, why are independent packages not build in parallel? Are the dependencies too complex to make this work? Thanks for any insight!

Well, this is a tricky thing. I see 3 points on this topic:
- building a package in parallel:
We already do this, unless the package itself (more precisely its
build-system) does not support it.
In such cases, it is explicitly stated in the *.mk file by setting
<PKG>_MAKE = $(MAKE1).
- the package configuration step:
Unfortunately the package configuration is highly serialized and
depends on the build-system used
by the package.
- building packages in parallel:
This point tends to hit some Buildroot's principles:
- simplicity;
- and build reproducibility (don't know if this word really exists in
English...).
Being able to reproduce build (mainly build failures) implies that
the sysroot should not change during

Quoted 8 lines

the configuration/build steps of a package, so the sysroot should be
copied for each package built in
parallel (iow, if 4 packages are built in parallel, there will be 4
sysroots at the same time, each of them
with a content that may be slightly different one from the others).
This may introduce some complexity
in Buildroot and it may also make build time worse due to the copies
of the sysroot for each package.

... we have a hack around that sort of does separate sysroots. That is, before going to global staging we create a per packet staging (in order to create ipkgs).
Would it be possible to switch off parallelization just for the configure and staging steps?

Quoted 10 lines

The patch series pointed by Maxime introduces support for building packages in parallel, but does not
implement the sysroot copy thing.
During the Buildroot Developer Days in last November, it has been decided that we won't enable building
packages in parallel (see:
http://elinux.org/Buildroot:DeveloperDaysELCE2013#Parallel_top-level_make
).

Quoted 3 lines

@others developers: feel free to correct me, if I speak with my ass ;)

BTW, this could certainly be added in the FAQ.

Andreas, Maxime, Frank, all,

Quoted 21 lines · Frank Ihle

Hi Andreas,

This is not a solution more my experience:

I'm using a 4 Core 3.2 GHz on a Fedora 20 system and for a at91sam9x5ek_defconfig it takes about 7 min from 0 (without an downloaded DLs) to a bootable Image.

During build I'm using GKrellM System Monitor and and shows me that all cores are in use. ATM I'm ok with these results, since i started with a compile time about 45 min.

Regards,

Frank

Quoted 1 line

Quoted 1 line

Quoted 1 line

Andreas Naumann <[email protected]> 07.02.14 9.52 Uhr >>>

Hi guys,

Quoted 11 lines

First of all kudos again for this very good tool! I'm using it almsot daily and am stoked about the functionality but simple usage at the same time.
Now one issue with build systems is always build time. It seems the speed is not impacted much by just spending more horsepower (cpu cores). When looking at the load I see very frequently that just one core is busy with one package, especially during the configure step. Some packages themselves are build with just one core, I guess we cant do very much about that.
However, why are independent packages not build in parallel? Are the dependencies too complex to make this work? Thanks for any insight!

Well, this is a tricky thing. I see 3 points on this topic:
- building a package in parallel:
We already do this, unless the package itself (more precisely its
build-system) does not support it.
In such cases, it is explicitly stated in the *.mk file by setting
<PKG>_MAKE = $(MAKE1).
- the package configuration step:
Unfortunately the package configuration is highly serialized and depends
on the build-system used
by the package.
- building packages in parallel:
This point tends to hit some Buildroot's principles:
- simplicity;
- and build reproducibility (don't know if this word really exists in
English...).
Being able to reproduce build (mainly build failures) implies that the
sysroot should not change during

the configuration/build steps of a package, so the sysroot should be
copied for each package built in
parallel (iow, if 4 packages are built in parallel, there will be 4
sysroots at the same time, each of them
with a content that may be slightly different one from the others). This
may introduce some complexity
in Buildroot and it may also make build time worse due to the copies of
the sysroot for each package.

The patch series pointed by Maxime introduces support for building packages in parallel, but does not
implement the sysroot copy thing.
During the Buildroot Developer Days in last November, it has been decided that we won't enable building
packages in parallel (see:
http://elinux.org/Buildroot:DeveloperDaysELCE2013#Parallel_top-level_make ).

@others developers: feel free to correct me, if I speak with my ass ;)

BTW, this could certainly be added in the FAQ.

Quoted 56 lines · Samuel Martin

Andreas, Maxime, Frank, all,

Quoted 21 lines · Frank Ihle

Hi Andreas,

This is not a solution more my experience:

I'm using a 4 Core 3.2 GHz on a Fedora 20 system and for a at91sam9x5ek_defconfig it takes about 7 min from 0 (without an downloaded DLs) to a bootable Image.

During build I'm using GKrellM System Monitor and and shows me that all cores are in use. ATM I'm ok with these results, since i started with a compile time about 45 min.

Regards,

Frank

Quoted 1 line

Quoted 1 line

Quoted 1 line

Andreas Naumann <[email protected]> 07.02.14 9.52 Uhr >>>

Hi guys,

Quoted 11 lines

First of all kudos again for this very good tool! I'm using it almsot daily and am stoked about the functionality but simple usage at the same time.
Now one issue with build systems is always build time. It seems the speed is not impacted much by just spending more horsepower (cpu cores). When looking at the load I see very frequently that just one core is busy with one package, especially during the configure step. Some packages themselves are build with just one core, I guess we cant do very much about that.
However, why are independent packages not build in parallel? Are the dependencies too complex to make this work? Thanks for any insight!

Well, this is a tricky thing. I see 3 points on this topic:
- building a package in parallel:
We already do this, unless the package itself (more precisely its
build-system) does not support it.
In such cases, it is explicitly stated in the *.mk file by setting
<PKG>_MAKE = $(MAKE1).
- the package configuration step:
Unfortunately the package configuration is highly serialized and depends
on the build-system used
by the package.
- building packages in parallel:
This point tends to hit some Buildroot's principles:
- simplicity;
- and build reproducibility (don't know if this word really exists in
English...).
Being able to reproduce build (mainly build failures) implies that the
sysroot should not change during

Quoted 8 lines

the configuration/build steps of a package, so the sysroot should be
copied for each package built in
parallel (iow, if 4 packages are built in parallel, there will be 4
sysroots at the same time, each of them
with a content that may be slightly different one from the others). This
may introduce some complexity
in Buildroot and it may also make build time worse due to the copies of
the sysroot for each package.

The build can be reproduced even when building packages in parallel, we just need to record the order of the steps and replay that order if needed.

The sysroot copy is needed only to be sure that even if some package has an unspecified optional dependency it generates every time the same binary.

Quoted 12 lines

The patch series pointed by Maxime introduces support for building packages in parallel, but does not
implement the sysroot copy thing.
During the Buildroot Developer Days in last November, it has been decided that we won't enable building
packages in parallel (see:
http://elinux.org/Buildroot:DeveloperDaysELCE2013#Parallel_top-level_make ).

@others developers: feel free to correct me, if I speak with my ass ;)

Quoted 1 line

BTW, this could certainly be added in the FAQ.

I think it's a good idea. Maybe after merging the patch set.

I think that at least when doing development it will be useful to build packages in parallel.
Maybe it will be best to avoid building packages in parallel just for releases.

The build can be reproduced even when building packages in parallel, we just need to record the order of the steps and replay that order if needed.

The sysroot copy is needed only to be sure that even if some package has an unspecified optional dependency it generates every time the same binary.

I think that at least when doing development it will be useful to build packages in parallel.
Maybe it will be best to avoid building packages in parallel just for releases.

Dear Fabio Porcedda,

Quoted 3 lines · Fabio Porcedda

The build can be reproduced even when building packages in parallel, we just need to record the order of the steps and replay that order if needed.

And in practice, how do you achieve this? When building in parallel, how do you record the fact that the test for libfoo in package A configure script was run before or after the installation of libfoo in the global sysroot? It's not only about recording the order of the build of the package: it's knowing whether each specific test of a configure script was run before or after the corresponding library was installed in the global sysroot.

Quoted 3 lines

The sysroot copy is needed only to be sure that even if some package has an unspecified optional dependency it generates every time the same binary.

Which is *the* very important thing. So saying it's needed "only to be sure" is quite minimizing the very important consequence of parallel build with a global sysroot, IMO.

Quoted 3 lines

I think that at least when doing development it will be useful to build packages in parallel.
Maybe it will be best to avoid building packages in parallel just for releases.

I don't agree.

I do a lot of support on #buildroot, and I would *hate* to have people coming up with bug reports that are not reproducible because they depend on the specific scheduling of the build. I really don't want new comers to use this feature and then come back with bug reports that are basically unfixable because they cannot be reproduced.

I think we should really not minimize the support problem that parallelization without per-package sysroot is going to cause.

Quoted 7 lines

And in practice, how do you achieve this? When building in parallel, how do you record the fact that the test for libfoo in package A configure script was run before or after the installation of libfoo in the global sysroot? It's not only about recording the order of the build of the package: it's knowing whether each specific test of a configure script was run before or after the corresponding library was installed in the global sysroot.

Well i admit i haven't thought about that corner case, I don't know an easy solution for that.

Quoted 2 lines

I think we should really not minimize the support problem that parallelization without per-package sysroot is going to cause.

My suggestion was based on the assumption that there is a way to reproduce the exact build order.
But if that is not true i agree with you.

Dear Fabio Porcedda,

Quoted 8 lines · Fabio Porcedda

Quoted 7 lines

And in practice, how do you achieve this? When building in parallel, how do you record the fact that the test for libfoo in package A configure script was run before or after the installation of libfoo in the global sysroot? It's not only about recording the order of the build of the package: it's knowing whether each specific test of a configure script was run before or after the corresponding library was installed in the global sysroot.

Quoted 2 lines

Well i admit i haven't thought about that corner case, I don't know an easy solution for that.

IMO this is not a corner case, this is precisely the problem we have been pointing out since the beginning when mixing a global sysroot with top-level parallelization.

Quoted 6 lines

Quoted 2 lines

I think we should really not minimize the support problem that parallelization without per-package sysroot is going to cause.

My suggestion was based on the assumption that there is a way to reproduce the exact build order.
But if that is not true i agree with you.

I don't really see how to reproduce the exact build order, so the support problem will be here. That's really why I have been a bit skeptical and resistant to this patch series from the very beginning. I am obviously very interested in ways to reduce the build time. But doing that at the expense of reliability is not a good idea in my opinion, as users will face spurious failures that we will not be able to reproduce and fix.

If we want to get this to the next level, someone need to experiment with per-package sysroot. Of course, this would be a long experiment, without necessarily at the end the guarantee that we will accept it, because it may be too complex to match Buildroot simplicity principle.

Quoted 7 lines

I don't really see how to reproduce the exact build order, so the support problem will be here. That's really why I have been a bit skeptical and resistant to this patch series from the very beginning. I am obviously very interested in ways to reduce the build time. But doing that at the expense of reliability is not a good idea in my opinion, as users will face spurious failures that we will not be able to reproduce and fix.

About the spurious failures when building packages in parallel IMHO they depends only on the building order of the finished steps. If a build fails IMHO it depends on something that is not available so those failures can be reproduced considering only the finished steps and ignoring the overlapping unfinished steps.
What do you think about that?

If we want to get this to the next level, someone need to experiment with per-package sysroot. Of course, this would be a long experiment, without necessarily at the end the guarantee that we will accept it, because it may be too complex to match Buildroot simplicity principle.

Best regards,

Thomas

Quoted 4 lines

If we want to get this to the next level, someone need to experiment with per-package sysroot. Of course, this would be a long experiment, without necessarily at the end the guarantee that we will accept it, because it may be too complex to match Buildroot simplicity principle.

as I said, we have hacked br to a per-package sysroot in the past. This was in order to create isolated ipkgs and did its job. Even if we dont solve any of the dependency issues with that, it could be a starting point. If this is of interest I can try to go further with it.

And for the depency issue, how does OE solve it?