MAKEALL
Wolfgang,
I have some troubles to run MAKEALL with BUILD_NBUILDS. If I set BUILD_NBUILDS to 2 or greater, there is a good chance the total number of targets is not an integral multiple of BUILD_NBUILDS. It has two undesired results.
1. The status report has wrong number of passed builds.
2. This script throws out SIGTERM.
The second one is troubling me. I am using Jenkins to monitor and build automatically. I can trap the SIGTERM on some hosts but not all of them.
Can you shed some light on this?
York
Dear York,
Quoted 5 lines · In message <[email protected]> you wrote:
I have some troubles to run MAKEALL with
BUILD_NBUILDS. If I setBUILD_NBUILDSto 2 or greater, there is a good chance the total number of targets is not an integral multiple ofBUILD_NBUILDS. It has two undesired results.
Quoted 7 lines
1. The status report has wrong number of passed builds.
2. This script throws out SIGTERM.The second one is troubling me. I am using Jenkins to monitor and build automatically. I can trap the SIGTERM on some hosts but not all of them.
Can you shed some light on this?
I'm sorry, but I am not familiar with this BUILD_NBUILDS code at all. It was added by Andy Fleming, so maybe he can help.
Andy?
Best regards,
Wolfgang Denk
Signature · 3 lines
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Hi York,
Quoted 16 lines · Wolfgang Denk
Dear York,
Quoted 2 lines · In message <[email protected]> you wrote:
I have some troubles to run MAKEALL with
BUILD_NBUILDS. If I set
BUILD_NBUILDSQuoted 1 line
to 2 or greater, there is a good chance the total number of targets is
not an
Quoted 8 lines
integral multiple of
BUILD_NBUILDS. It has two undesired results.1. The status report has wrong number of passed builds.
2. This script throws out SIGTERM.The second one is troubling me. I am using Jenkins to monitor and build automatically. I can trap the SIGTERM on some hosts but not all of them.
Quoted 7 lines
Quoted 1 line
Can you shed some light on this?
I'm sorry, but I am not familiar with this
BUILD_NBUILDScode at all. It was added by Andy Fleming, so maybe he can help.Andy?
It might also be worth looking at tools/buildman, which automatically allocates one build thread per CPU.
Regards,
Simon
Hi Simon,
Quoted 23 lines · Simon Glass
Hi York,
Quoted 18 lines · Wolfgang Denk
Dear York,
Quoted 2 lines · In message <[email protected]> you wrote:
I have some troubles to run MAKEALL with
BUILD_NBUILDS. If I set
BUILD_NBUILDSQuoted 1 line
to 2 or greater, there is a good chance the total number of targets is
not an
Quoted 9 lines
integral multiple of
BUILD_NBUILDS. It has two undesired results.1. The status report has wrong number of passed builds.
2. This script throws out SIGTERM.The second one is troubling me. I am using Jenkins to monitor and build automatically. I can trap the SIGTERM on some hosts but not all of them.
Can you shed some light on this?
Quoted 8 lines
Quoted 5 lines
I'm sorry, but I am not familiar with this
BUILD_NBUILDScode at all. It was added by Andy Fleming, so maybe he can help.Andy?
It might also be worth looking at
tools/buildman, which automatically allocates one build thread per CPU.
Jumping in late, but my question is incidental and not urgent anyway.
Would using buildman make the multiple build / multiple CPU code in MAKEALL useless? I'm wondering whether we could apply the Unix philosophy here (1), let buildman alone deal with handling parallel builds, and remove code from MAKEALL.
(1) not the "unix is user-friendly, it's just picky about who its friends are" one; the "do one thing well" one.
Quoted 2 lines
Regards,
Simon
Amicalement,
Signature · 1 line
Albert.
Hi Albert,
Quoted 9 lines
Quoted 2 lines
It might also be worth looking at
tools/buildman, which automatically allocates one build thread per CPU.Jumping in late, but my question is incidental and not urgent anyway.
Would using buildman make the multiple build / multiple CPU code in MAKEALL useless? I'm wondering whether we could apply the Unix philosophy here (1), let buildman alone deal with handling parallel builds, and remove code from MAKEALL.
I think parallel build feature is missing from buildman. So, buildman cannot check if -j option is working correctly.
Note:
Please do not be confused by the difference of what "parallel" means.
[1] MAKEALL runs single "make" thread by default. (You can change this with BUILD_NBUILDS variable) One "make" thread runs multiple jobs by giving -j option. (You can change this with BUILD_NCPUS variable)
[2] buildman runs multiple "make" threads. Each make thread runs one job (that is, always -j 1 ).
Besides parallel build , I notice some differences between MAKEALL and buildman.
- MAKEALL runs "make mrproper" everty time before "make", but buildman doesn't.
This means objects are remaining, that were generated by the previous commit.
So, even if some build rules in makefiles get broken at an intermediate commit, buildman possibly cannot detect the error.
- MAKEALL can select in-tree-build or out-of-tree build by BUILD_DIR option, but buildman always does out-of-tree build.
(Simon, please correct me if I am wrong.)
If you touch only C sources, buildman is enough (and faster).
But if you change make targets, it is highly recommended to check with MAKEALL.
At least, I need both. (for Kbuild and Kconfig work)
I guess we cannot replace MAKEALL with buildman for now.
Best Regards
Masahiro Yamada
Hi,
Quoted 13 lines · Masahiro Yamada
Hi Albert,
Quoted 9 lines
Quoted 2 lines
It might also be worth looking at
tools/buildman, which automatically allocates one build thread per CPU.Jumping in late, but my question is incidental and not urgent anyway.
Would using buildman make the multiple build / multiple CPU code in MAKEALL useless? I'm wondering whether we could apply the Unix philosophy here (1), let buildman alone deal with handling parallel builds, and remove code from MAKEALL.
This is a good summary thank you, but I have a few comments as well.
Quoted 2 lines
I think parallel build feature is missing from buildman. So, buildman cannot check if -j option is working correctly.
It is not used by default, but there is a -j option. If you use -j, first make sure you reduce the number of threads to compensate. So for example you can use:
buildman -j 8 -T 1
to use -j8 with only a single thread on a 8-core machine. I typically find that building with -j1 and increasing the thread count is much faster. Another tip if you are not using your machine for anything else is to use -T with a number 20-30% larger than the number of CPUs you have.
Quoted 5 lines
Note:
Please do not be confused by the difference of what "parallel" means.
Quoted 8 lines
[1] MAKEALL runs single "make" thread by default. (You can change this with
BUILD_NBUILDSvariable) One "make" thread runs multiple jobs by giving -j option. (You can change this withBUILD_NCPUSvariable)[2] buildman runs multiple "make" threads. Each make thread runs one job (that is, always -j 1 ).
Indeed, this is the default behaviour, but it can be changed with options.
Quoted 11 lines
Besides parallel build , I notice some differences between MAKEALL and buildman.
- MAKEALL runs "make mrproper" everty time before "make", but buildman doesn't.
This means objects are remaining, that were generated by the previous commit.
So, even if some build rules in makefiles get broken at an intermediate commit, buildman possibly cannot detect the error.
That is correct. To get the MAKEALL behaviour you need to give the -f option.
Quoted 3 lines
- MAKEALL can select in-tree-build or out-of-tree build by
BUILD_DIRoption, but buildman always does out-of-tree build.
That's right, in fact buildman does not support in-tree build at all unfortunately. However, given that every thread has its own git tree, it would actually be fairly easy to add this feature.
Quoted 10 lines
(Simon, please correct me if I am wrong.)
If you touch only C sources, buildman is enough (and faster).
But if you change make targets, it is highly recommended to check with MAKEALL.
At least, I need both. (forKbuildandKconfigwork)I guess we cannot replace MAKEALL with buildman for now.
The intent was that buildman supported everything that MAKEALL does and more. I believe that is true except for the in-tree build you mention above. However, I think there might also be some small difference between the way the -f option is implemented in buildman, and the way MAKEALL works. It might be worth splitting the -f option into two:
- one to force rebuild of a board
- another to force a reconfigure on every build
Then we could be sure. In summary, if we do that and support in-tree builds then I think buildman has all the features of MAKEALL.
BTW a few other options that buildman has, which people may not have noticed:
-S - shows increase/decrease of image sizes across commits
-B - same but for individual functions
-k - keep the build outputs
-d - show detailed board information
-S - allows building every second commit, or only first and last, etc.
Regards,
Simon
Hello Simon,
Quoted 5 lines
It is not used by default, but there is a -j option. If you use -j, first make sure you reduce the number of threads to compensate. So for example you can use:
buildman -j 8 -T 1
Ooh! I did not know buildman has such options.
Thanks!
Quoted 7 lines
The intent was that buildman supported everything that MAKEALL does and more. I believe that is true except for the in-tree build you mention above. However, I think there might also be some small difference between the way the -f option is implemented in buildman, and the way MAKEALL works. It might be worth splitting the -f option into two:
Quoted 5 lines
- one to force rebuild of a board
- another to force a reconfigure on every buildThen we could be sure. In summary, if we do that and support in-tree builds then I think buildman has all the features of MAKEALL.
That's good to know.
Best Regards
Masahiro Yamada