Allow for parallel builds and saved output

The MAKEALL script cleverly runs make with the appropriate options to use all of the cores on the system, but your average U-Boot build can't make much use of more than a few cores. If you happen to have a many-core server, your builds will leave most of the system idle.

In order to make full use of such a system, we need to build multiple targets in parallel, and this requires directing make output into multiple directories. We add a BUILD_NBUILDS variable, which allows users to specify how many builds to run in parallel.
When BUILD_NBUILDS is set greater than 1, we redefine BUILD_DIR for each build to be ${BUILD_DIR}/${target}. Also, we make "./build" the default BUILD_DIR when BUILD_NBUILDS is greater than 1.

MAKEALL now tracks which builds are still running, and when one finishes, it starts a new build.

Once each build finishes, we run "make tidy" on its directory, to reduce the footprint.

As a result, we are left with a build directory with all of the built targets still there for use, which means anyone who wanted to use MAKEALL as part of a test harness can now do so.

Signed-off-by Andy Fleming<[email protected]>

---
v2: - Update to keep BUILD_NBUILDS builds in flight, rather than batching
- Clean up style things
- Defer error output until build completion to make output *slightly* more readable

v3: - Add BUILD_MANY variable to clarify logic
- Added comment for done/skip prefixes, and renamed for clarity
- Changed CURRENT_COUNT to CURRENT_CNT to match TOTAL_CNT
- Put build wait logic into its own manage_build function
- Used wildcards to delete build management files
- Fixed a bug where error-less builds printed an error

Hi Andy,

Le 25/04/2012 07:33, Andy Fleming a ?crit :

*However*, two boards build without and do not build clean with: scpu and pndb3.

Also, the final report is not displayed as the same with and without the patch -- with it, unclean boards are listed one per line, without it, they are listed in a single line.

Last: breaking during a parallel build then trying a git clean -xfd causes errors such as files missing.

The MAKEALL script cleverly runs make with the appropriate options to use all of the cores on the system, but your average U-Boot build can't make much use of more than a few cores. If you happen to have a many-core server, your builds will leave most of the system idle.

In order to make full use of such a system, we need to build multiple targets in parallel, and this requires directing make output into multiple directories. We add a BUILD_NBUILDS variable, which allows users to specify how many builds to run in parallel.
When BUILD_NBUILDS is set greater than 1, we redefine BUILD_DIR for each build to be ${BUILD_DIR}/${target}. Also, we make "./build" the default BUILD_DIR when BUILD_NBUILDS is greater than 1.

MAKEALL now tracks which builds are still running, and when one finishes, it starts a new build.

Once each build finishes, we run "make tidy" on its directory, to reduce the footprint.

As a result, we are left with a build directory with all of the built targets still there for use, which means anyone who wanted to use MAKEALL as part of a test harness can now do so.

Signed-off-by Andy Fleming <[email protected]>

---
v2: - Update to keep BUILD_NBUILDS builds in flight, rather than batching
- Clean up style things
- Defer error output until build completion to make output *slightly* more readable

v3: - Add BUILD_MANY variable to clarify logic
- Added comment for done/skip prefixes, and renamed for clarity
- Changed CURRENT_COUNT to CURRENT_CNT to match TOTAL_CNT
- Put build wait logic into its own manage_build function
- Used wildcards to delete build management files
- Fixed a bug where error-less builds printed an error

Once again, my unscientific tests on my 24-thread box:

time ./MAKEALL 83xx

real    4m2.951s
user    26m6.534s
sys     4m0.070s

time ./MAKEALL 86xx

real    0m27.271s
user    3m10.706s
sys     0m27.123s

time BUILD_NBUILDS=50 BUILD_NCPUS=1 ./MAKEALL 86xx

real    0m38.430s
user    2m11.601s
sys     0m18.610s

time BUILD_NBUILDS=50 BUILD_NCPUS=1 ./MAKEALL 83xx

real    1m15.642s
user    15m2.125s
sys     2m25.818s

time BUILD_NBUILDS=50 BUILD_NCPUS=1 ./MAKEALL 85xx

real    4m29.840s
user    80m30.641s
sys     10m49.643s

Looks like 86xx was affected by having only 5 boards:

time BUILD_NBUILDS=10 BUILD_NCPUS=4 ./MAKEALL 86xx

real    0m11.887s
user    2m22.299s
sys     0m20.894s

 MAKEALL |  137 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 files changed, 122 insertions(+), 15 deletions(-)

Dear Andy Fleming,

In message <[email protected]> you

Quoted 4 lines

The MAKEALL script cleverly runs make with the appropriate options to use all of the cores on the system, but your average U-Boot build can't make much use of more than a few cores. If you happen to have a many-core server, your builds will leave most of the system idle.

Quoted 3 lines

 MAKEALL |  137

++++++++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 files changed, 122 insertions(+), 15 deletions(-)

Applied, thanks.

Best regards,

Wolfgang Denk

Dear Wolfgang Denk,

Quoted 3 lines

Dear Andy Fleming,

In message<[email protected]> you

Quoted 4 lines · wrote:

Quoted 4 lines

The MAKEALL script cleverly runs make with the appropriate options to use all of the cores on the system, but your average U-Boot build can't make much use of more than a few cores. If you happen to have a many-core server, your builds will leave most of the system idle.

[...]

Quoted 4 lines

Quoted 3 lines

  MAKEALL |  137

++++++++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 files changed, 122 insertions(+), 15 deletions(-)

Quoted 1 line

Applied, thanks.

What do you get if you run:

./MAKEALL -a arm -l

? :-)

Le 06/05/2012 03:48, Marek Vasut a ?crit :

Quoted 21 lines

Dear Wolfgang Denk,

Quoted 3 lines

Dear Andy Fleming,

In message<[email protected]> you

Quoted 4 lines · wrote:

Quoted 4 lines

The MAKEALL script cleverly runs make with the appropriate options to use all of the cores on the system, but your average U-Boot build can't make much use of more than a few cores. If you happen to have a many-core server, your builds will leave most of the system idle.

[...]

Quoted 5 lines

Quoted 3 lines

 MAKEALL |  137

++++++++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 files changed, 122 insertions(+), 15 deletions(-)

Applied, thanks.

What do you get if you run:

./MAKEALL -a arm -l

Quoted 1 line

? :-)

I am being hit by a spurious build failure in parallel build: pdnb3 succeeds when built alone but fails in a MAKEALL arm with the following .ERR:

mv: cannot stat `/home/uboot/src/u-boot-arm/build/pdnb3/include/autoconf.mk.tmp': No such file or directory

Looks like several builds competed for this temp file name :)...

Quoted 36 lines

Le 06/05/2012 03:48, Marek Vasut a ?crit :

Quoted 19 lines

Dear Wolfgang Denk,

Quoted 4 lines

Dear Andy Fleming,

In message<[email protected]> you

Quoted 4 lines · wrote:

Quoted 4 lines

The MAKEALL script cleverly runs make with the appropriate options to use all of the cores on the system, but your average U-Boot build can't make much use of more than a few cores. If you happen to have a many-core server, your builds will leave most of the system idle.

[...]

Quoted 4 lines

Quoted 3 lines

 MAKEALL |  137

++++++++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 files changed, 122 insertions(+), 15 deletions(-)

Quoted 7 lines

Quoted 1 line

Applied, thanks.

What do you get if you run:

./MAKEALL -a arm -l

? :-)

I am being hit by a spurious build failure in parallel build: pdnb3 succeeds when built alone but fails in a MAKEALL arm with the following .ERR:

mv: cannot stat `/home/uboot/src/u-boot-arm/build/pdnb3/include/autoconf.mk.tmp': No such file or directory

Quoted 1 line

Looks like several builds competed for this temp file name :)...

I already submitted the patch for this. It's because pdnb3 was being built twice (and therefore twice concurrently). Wolfgang just has to apply that patch, too. :)

Andy

Dear Andy Fleming,

Quoted 37 lines · Albert ARIBAUD

Quoted 36 lines

Le 06/05/2012 03:48, Marek Vasut a ?crit :

Quoted 19 lines

Dear Wolfgang Denk,

Quoted 4 lines

Dear Andy Fleming,

In message<[email protected]> you

Quoted 4 lines · wrote:

Quoted 4 lines

The MAKEALL script cleverly runs make with the appropriate options to use all of the cores on the system, but your average U-Boot build can't make much use of more than a few cores. If you happen to have a many-core server, your builds will leave most of the system idle.

[...]

Quoted 4 lines

Quoted 3 lines

MAKEALL | 137 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 files changed, 122 insertions(+), 15 deletions(-)

Quoted 7 lines

Quoted 1 line

Applied, thanks.

What do you get if you run:

./MAKEALL -a arm -l

? :-)

I am being hit by a spurious build failure in parallel build: pdnb3 succeeds when built alone but fails in a MAKEALL arm with the following .ERR:

mv: cannot stat `/home/uboot/src/u-boot-arm/build/pdnb3/include/autoconf.mk.tmp': No such file or directory

Quoted 5 lines

Quoted 1 line

Looks like several builds competed for this temp file name :)...

I already submitted the patch for this. It's because pdnb3 was being built twice (and therefore twice concurrently). Wolfgang just has to apply that patch, too. :)

Does it also fix ./MAKEALL -a arm -l ?

Quoted 4 lines

I already submitted the patch for this. It's because pdnb3 was being built twice (and therefore twice concurrently). Wolfgang just has to apply that patch, too. :)

Does it also fix ./MAKEALL -a arm -l ?