package/bearssl: disable parallel build
PATCH 1/1 package/bearssl: disable parallel build
The bearssl package is not compatible with parallel builds.
Fixes:
- http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773
--- package/bearssl/bearssl.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/bearssl/bearssl.mk +1 -1
@@ -24,7 +24,7 @@ BEARSSL_TARGETS += lib endif define BEARSSL_BUILD_CMDS - $(TARGET_MAKE_ENV) $(MAKE) $(BEARSSL_MAKE_OPTS) -C $(@D) \ + $(TARGET_MAKE_ENV) $(MAKE1) $(BEARSSL_MAKE_OPTS) -C $(@D) \ $(BEARSSL_TARGETS) endef
Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hi James,
Le sam. 15 oct. 2022 à 02:58, James Hilliard <[email protected]> a écrit :
Quoted 2 lines
The bearssl package is not compatible with parallel builds.
bearssl was building perfectly fine up until now and it seems there are a lot of packages that are affected by parallel build issues (e.g. dmraid, musepack, slang, etc).
Did we change something recently on the autobuilders?
Quoted 26 lines
Fixes:
-
http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773Signed-off-by James Hilliard <[email protected]>--- package/bearssl/bearssl.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/bearssl/bearssl.mk +1
-2@@ -24,7 +24,7 @@ BEARSSL_TARGETS += lib endif define BEARSSL_BUILD_CMDS - $(TARGET_MAKE_ENV) $(MAKE) $(BEARSSL_MAKE_OPTS) -C $(@D) \ + $(TARGET_MAKE_ENV) $(MAKE1) $(BEARSSL_MAKE_OPTS) -C $(@D) \ $(BEARSSL_TARGETS) endef --2.34.1
Quoted 1 line
Best Regards,
Fabrice
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 9 lines · Fabrice Fontaine
Hi James,
Quoted 2 lines · James Hilliard
The bearssl package is not compatible with parallel builds.
bearssl was building perfectly fine up until now and it seems there are a lot of packages that are affected by parallel build issues (e.g. dmraid, musepack, slang, etc).
Some parallel build bugs don't reproduce reliably due to timing, I made a change to my autobuilders which massively increases the chance of triggering build failures
in packages which have dependency graph bugs. This triggers build build failures in -j1 builds which would otherwise only appear when running parallel builds(my autobuilders are generally running with -j1 to make the logs more readable).
Quoted 1 line
Did we change something recently on the autobuilders?
Yes, all my autobuilders were recently changed to use make master(will be make version 4.4 once released) with top level --shuffle=random.
Details:
https://github.com/mirror/make/blob/3e20e376b7aa26fa43f81f23323c717c4cae1bfb/NEWS#L123-L128
https://trofi.github.io/posts/238-new-make-shuffle-mode.html
http://trofi.github.io/posts/249-an-update-on-make-shuffle.html
Note that MAKE1 is currently broken itself without this when using shuffle mode:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
Quoted 24 lines
Quoted 24 lines
Fixes:
- http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773Signed-off-by James Hilliard <[email protected]>--- package/bearssl/bearssl.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/bearssl/bearssl.mk +1
-1@@ -24,7 +24,7 @@ BEARSSL_TARGETS += lib endif define BEARSSL_BUILD_CMDS - $(TARGET_MAKE_ENV) $(MAKE) $(BEARSSL_MAKE_OPTS) -C $(@D) \ + $(TARGET_MAKE_ENV) $(MAKE1) $(BEARSSL_MAKE_OPTS) -C $(@D) \ $(BEARSSL_TARGETS) endef
Quoted 6 lines
Quoted 3 lines
--
2.34.1Best Regards,
Fabrice
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
James, All,
On 2022-10-15 19:21 -0400, James Hilliard spake thusly:
Quoted 15 lines
Quoted 5 lines · Fabrice Fontaine
Quoted 1 line · James Hilliard
The bearssl package is not compatible with parallel builds.
bearssl was building perfectly fine up until now and it seems there are a lot of packages that are affected by parallel build issues (e.g. dmraid, musepack, slang, etc).
Some parallel build bugs don't reproduce reliably due to timing, I made a change to my autobuilders which massively increases the chance of triggering build failures
in packages which have dependency graph bugs. This triggers build build failures in -j1 builds which would otherwise only appear when running parallel builds(my autobuilders are generally running with -j1 to make the logs more readable).
Quoted 4 lines
Quoted 1 line
Did we change something recently on the autobuilders?
Yes, all my autobuilders were recently changed to use make master(will be make version 4.4 once released) with top level --shuffle=random.
By disabling parallel build because you are using ana as yet unreleased version, we remove the parallel build for everyone, which is not very nice at all.
Instead, we should really try and fix the packages rather than papering over the issue.
Notes:
- it *is* interesting to find those bugs;
- it is sad that the default mode for --shuffle will be random, not none.
Regards,
Yann E. MORIN.
Quoted 42 lines
Details:
https://github.com/mirror/make/blob/3e20e376b7aa26fa43f81f23323c717c4cae1bfb/NEWS#L123-L128
https://trofi.github.io/posts/238-new-make-shuffle-mode.html
http://trofi.github.io/posts/249-an-update-on-make-shuffle.htmlNote that MAKE1 is currently broken itself without this when using shuffle mode:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/Quoted 24 lines
Quoted 24 lines
Fixes:
- http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773Signed-off-by James Hilliard <[email protected]>--- package/bearssl/bearssl.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/bearssl/bearssl.mk +1
-1@@ -24,7 +24,7 @@ BEARSSL_TARGETS += lib endif define BEARSSL_BUILD_CMDS - $(TARGET_MAKE_ENV) $(MAKE) $(BEARSSL_MAKE_OPTS) -C $(@D) \ + $(TARGET_MAKE_ENV) $(MAKE1) $(BEARSSL_MAKE_OPTS) -C $(@D) \ $(BEARSSL_TARGETS) endefQuoted 6 lines
Quoted 3 lines
--
2.34.1Best Regards,
Fabrice
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 19 lines · Yann E. MORIN
James, All,
On 2022-10-15 19:21 -0400, James Hilliard spake thusly:
Quoted 15 lines
Quoted 5 lines · Fabrice Fontaine
Quoted 1 line · James Hilliard
The bearssl package is not compatible with parallel builds.
bearssl was building perfectly fine up until now and it seems there are a lot of packages that are affected by parallel build issues (e.g. dmraid, musepack, slang, etc).
Some parallel build bugs don't reproduce reliably due to timing, I made a change to my autobuilders which massively increases the chance of triggering build failures
in packages which have dependency graph bugs. This triggers build build failures in -j1 builds which would otherwise only appear when running parallel builds(my autobuilders are generally running with -j1 to make the logs more readable).
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Did we change something recently on the autobuilders?
Yes, all my autobuilders were recently changed to use make master(will be make version 4.4 once released) with top level --shuffle=random.
By disabling parallel build because you are using ana as yet unreleased version, we remove the parallel build for everyone, which is not very nice at all.
Failures with shuffle mode indicate that the package is not parallel compatible on existing make releases as well, it just makes those bugs more visible.
I'm running shuffle with -j1 because that way it's obvious in the logs where the failure is getting hit, otherwise these bugs tend to be transient and difficult to
trace as they otherwise are timing dependent.
Quoted 8 lines
Instead, we should really try and fix the packages rather than papering over the issue.
Notes:
- it *is* interesting to find those bugs;
- it is sad that the default mode for --shuffle will be random, not none.
Default is equivalent to none, I modified my autobuilder runner to pass --shuffle=random.
Quoted 47 lines
Regards,
Yann E. MORIN.Quoted 37 lines
Details:
https://github.com/mirror/make/blob/3e20e376b7aa26fa43f81f23323c717c4cae1bfb/NEWS#L123-L128
https://trofi.github.io/posts/238-new-make-shuffle-mode.html
http://trofi.github.io/posts/249-an-update-on-make-shuffle.htmlNote that MAKE1 is currently broken itself without this when using shuffle mode:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/Quoted 29 lines
Quoted 25 lines
Fixes:
- http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773Signed-off-by James Hilliard <[email protected]>--- package/bearssl/bearssl.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/bearssl/bearssl.mk +1
-2@@ -24,7 +24,7 @@ BEARSSL_TARGETS += lib endif define BEARSSL_BUILD_CMDS - $(TARGET_MAKE_ENV) $(MAKE) $(BEARSSL_MAKE_OPTS) -C $(@D) \ + $(TARGET_MAKE_ENV) $(MAKE1) $(BEARSSL_MAKE_OPTS) -C $(@D) \ $(BEARSSL_TARGETS) endef --Quoted 2 lines
2.34.1
Best Regards,
Quoted 5 lines
Quoted 1 line
Fabrice
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 7 lines
--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
James, All,
+Thomas, Peter, Arnout
On 2022-10-16 11:44 -0400, James Hilliard spake thusly:
Quoted 2 lines · Yann E. MORIN
Quoted 1 line
On 2022-10-15 19:21 -0400, James Hilliard spake thusly:
[--SNIP--]
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Yes, all my autobuilders were recently changed to use make master(will be make version 4.4 once released) with top level --shuffle=random.
By disabling parallel build because you are using ana as yet unreleased version, we remove the parallel build for everyone, which is not very nice at all.
Failures with shuffle mode indicate that the package is not parallel compatible on existing make releases as well, it just makes those bugs more visible.
I never said they were not, and I even said (further on) that it *is* interesting to find such bugs.
However, the behaviour so far has been ordering of prerequisites (and goals!), and for some packages, that ordering was enough to work in a parallel make.
For example, the bearsll parallel issue is caused by the ordering so the mkdir is very early. Yes, it is incorrect, but it happens to always work in practice. And yes it is easy to fix, here's a patch:
diff -durN a/mk/mkrules.sh b/mk/mkrules.sh
--- a/mk/mkrules.sh 2018-08-14 22:41:54.000000000 +0200
+++ b/mk/mkrules.sh 2022-10-16 17:54:22.282069851 +0200
@@ -531,23 +531,23 @@
(for f in $coresrc ; do
b="$(basename "$f" .c)\$O"
g="$(escsep "$f")"
- printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
+ printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
done
for f in $toolssrc ; do
b="$(basename "$f" .c)\$O"
g="$(escsep "$f")"
- printf '\n$(OBJDIR)$P%s: %s $(HEADERSTOOLS)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
+ printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSTOOLS)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
done
for f in $testcryptosrc $testspeedsrc ; do
b="$(basename "$f" .c)\$O"
g="$(escsep "$f")"
- printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
+ printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
done
for f in $testx509src ; do
b="$(basename "$f" .c)\$O"
g="$(escsep "$f")"
- printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) -DSRCDIRNAME=".." $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
+ printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) -DSRCDIRNAME=".." $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
done) >> Rules.mk
Binary files a/mk/.mkrules.sh.swp and b/mk/.mkrules.sh.swp differAnd then adding a CONFIGURE_CMDS that runs mk/mkrules.sh to regenerate the Makefile.
Quoted 4 lines
I'm running shuffle with -j1 because that way it's obvious in the logs where the failure is getting hit, otherwise these bugs tend to be transient and difficult to
trace as they otherwise are timing dependent.
Yes, this is very interesting to find such bugs, but disabling parallel build in packages is not nice when they can be (easily) fixed.
Quoted 5 lines
Quoted 4 lines
Notes:
- it *is* interesting to find those bugs;
- it is sad that the default mode for --shuffle will be random, not none.Default is equivalent to none,
Yeah, I noticed the hard way after having my patch rejected in makem as you may have noticed. :-/
Quoted 2 lines
I modified my autobuilder runner to pass
--shuffle=random.
Again, that is very interesting, but please, please, conordinate with the rest of us when you do such changes, because that is very weird, as Fabrice noted, to suddenly get build failures out of the blue when nothing seemed to have otherwise changed.
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 20 lines · Yann E. MORIN
James, All,
+Thomas, Peter, Arnout
On 2022-10-16 11:44 -0400, James Hilliard spake thusly:
Quoted 2 lines · Yann E. MORIN
Quoted 1 line
On 2022-10-15 19:21 -0400, James Hilliard spake thusly:
[--SNIP--]
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Yes, all my autobuilders were recently changed to use make master(will be make version 4.4 once released) with top level --shuffle=random.
By disabling parallel build because you are using ana as yet unreleased version, we remove the parallel build for everyone, which is not very nice at all.
Failures with shuffle mode indicate that the package is not parallel compatible on existing make releases as well, it just makes those bugs more visible.
I never said they were not, and I even said (further on) that it *is* interesting to find such bugs.
Quoted 7 lines
However, the behaviour so far has been ordering of prerequisites (and goals!), and for some packages, that ordering was enough to work in a parallel make.
For example, the bearsll parallel issue is caused by the ordering so the mkdir is very early. Yes, it is incorrect, but it happens to always work in practice. And yes it is easy to fix, here's a patch:
Yeah, I think a lot of these are bugs that usually don't appear most of the time but may randomly pop up under high load highly parallel conditions.
Randomly failing parallel builds has been an issue I've seen pop up a good bit for my project builds so I wanted to try and find a way to make them more visible.
Quoted 37 lines
diff -durN
a/mk/mkrules.shb/mk/mkrules.sh
---a/mk/mkrules.sh2018-08-14 22:41:54.000000000 +0200 +++b/mk/mkrules.sh2022-10-16 17:54:22.282069851 +0200 @@ -531,23 +531,23 @@
(for f in $coresrc ; do
b="$(basename "$f" .c)\$O"
g="$(escsep "$f")"
- printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
+ printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
donefor f in $toolssrc ; do
b="$(basename "$f" .c)\$O"
g="$(escsep "$f")"
- printf '\n$(OBJDIR)$P%s: %s $(HEADERSTOOLS)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
+ printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSTOOLS)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
donefor f in $testcryptosrc $testspeedsrc ; do
b="$(basename "$f" .c)\$O"
g="$(escsep "$f")"
- printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
+ printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
donefor f in $testx509src ; do
b="$(basename "$f" .c)\$O"
g="$(escsep "$f")"
- printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) -DSRCDIRNAME=".." $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
+ printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) -DSRCDIRNAME=".." $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
done) >> Rules.mk
Binary files a/mk/.mkrules.sh.swp and b/mk/.mkrules.sh.swp differAnd then adding a
CONFIGURE_CMDSthat runsmk/mkrules.shto regenerate theMakefile.
Quoted 7 lines
Quoted 4 lines
I'm running shuffle with -j1 because that way it's obvious in the logs where the failure is getting hit, otherwise these bugs tend to be transient and difficult to
trace as they otherwise are timing dependent.Yes, this is very interesting to find such bugs, but disabling parallel build in packages is not nice when they can be (easily) fixed.
Yeah, I hadn't investigated the root cause in detail as bearssl didn't appear to be very widely used. At least with per-package directories disabling parallel build for an individual package shouldn't meaningfully slow down the build in most cases AFAIU.
I agree fixing the root cause is ideal, I thought our typical approach was to just mark parallel incompatible stuff with MAKE1 and then fix things later on as needed. If a lot of packages have dependency ordering issues I assume we would want to prioritise fixing popular packages that have long build times first.
What do you think the best strategy there would be?
Quoted 10 lines
Quoted 6 lines
Quoted 5 lines
Notes:
- it *is* interesting to find those bugs;
- it is sad that the default mode for --shuffle will be random, not none.Default is equivalent to none,
Yeah, I noticed the hard way after having my patch rejected in makem as you may have noticed. :-/
Quoted 7 lines
Quoted 2 lines
I modified my autobuilder runner to pass
--shuffle=random.Again, that is very interesting, but please, please, conordinate with the rest of us when you do such changes, because that is very weird, as Fabrice noted, to suddenly get build failures out of the blue when nothing seemed to have otherwise changed.
I've been experimenting a bit in general with autobuilder settings to try and fix some issues with poor coverage.
One other reason for testing with shuffle=random is to make higher kconfig probability values more useful(as otherwise build failures will be concentrated in packages early in the dependency ordering) :
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
Quoted 11 lines
Regards,
Yann E. MORIN.--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Hello James,
Quoted 5 lines · On Sun, 16 Oct 2022 12:45:06 -0400 James Hilliard <[email protected]>…
Quoted 4 lines
Again, that is very interesting, but please, please, conordinate with the rest of us when you do such changes, because that is very weird, as Fabrice noted, to suddenly get build failures out of the blue when nothing seemed to have otherwise changed.
Quoted 7 lines
I've been experimenting a bit in general with autobuilder settings to try and fix some issues with poor coverage.
One other reason for testing with shuffle=random is to make higher kconfig probability values more useful(as otherwise build failures will be concentrated in packages early in the dependency ordering) :
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
Thanks a lot for all this work, which I find really useful. However, like Yann said I believe we need to coordinate better. I think we would prefer to see build results submitted on autobuild.buildroot.org to only be generated by the official/upstream autobuild-run script. Otherwise, it's super confusing as we get results that corresponding to variations of build configurations/build logic that we are not even aware they exist.
Another concern is that the autobuilder results are already extremely bad these days, and therefore the priority is more on addressing/fixing the existing issues rather than adding more. It's a lot easier/practical to fix new problems when they are not mixed together with zillions of old problems.
Again, thanks a lot for your work on this!
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 19 lines · Thomas Petazzoni
Hello James,
Quoted 5 lines · On Sun, 16 Oct 2022 12:45:06 -0400 James Hilliard <[email protected]>…
Quoted 4 lines
Again, that is very interesting, but please, please, conordinate with the rest of us when you do such changes, because that is very weird, as Fabrice noted, to suddenly get build failures out of the blue when nothing seemed to have otherwise changed.
Quoted 7 lines
I've been experimenting a bit in general with autobuilder settings to try and fix some issues with poor coverage.
One other reason for testing with shuffle=random is to make higher kconfig probability values more useful(as otherwise build failures will be concentrated in packages early in the dependency ordering) :
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
Quoted 7 lines
Thanks a lot for all this work, which I find really useful. However, like Yann said I believe we need to coordinate better. I think we would prefer to see build results submitted on autobuild.buildroot.org to only be generated by the official/upstream autobuild-run script. Otherwise, it's super confusing as we get results that corresponding to variations of build configurations/build logic that we are not even aware they exist.
The shuffle=random changes don't require modifying autobuild-run, I'm just tweaking the path to make in my autobuild service environment and then passing an additional config flag to autobuild-run like this:
--make-opts="--shuffle=random"
Quoted 6 lines
Another concern is that the autobuilder results are already extremely bad these days, and therefore the priority is more on addressing/fixing the existing issues rather than adding more. It's a lot easier/practical to fix new problems when they are not mixed together with zillions of old problems.
Well builds were mostly failing in the same place due to deterministic build ordering, I was trying to make it so that we would see failures in different places as that's more useful compared with always seeing the same errors.
To me seeing the same errors most of the time indicates there's some major coverage issues with the autobuilders which need fixing.
Quoted 8 lines
Again, thanks a lot for your work on this!
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 4 lines · On Mon, 17 Oct 2022 07:29:52 -0400 James Hilliard <[email protected]>…
The shuffle=random changes don't require modifying autobuild-run, I'm just tweaking the path to make in my autobuild service environment and then passing an additional config flag to autobuild-run like this:
--make-opts="--shuffle=random"
Right, but my point stands: it's custom usage/configuration of autobuild-run that we would prefer to discuss on the mailing list first before deploying/enabling.
Quoted 7 lines
Well builds were mostly failing in the same place due to deterministic build ordering, I was trying to make it so that we would see failures in different places as that's more useful compared with always seeing the same errors.
To me seeing the same errors most of the time indicates there's some major coverage issues with the autobuilders which need fixing.
I think you're missing a point here: many of the builds today are failing due to the glibc 2.36 bump. So pretty much every single build that is glibc based will indeed fail with some error related to this glibc bump.
Your additional randomization will not change anything to this. It's just that we now have the glibc bump issues + the newly discovered issues related to the extra randomization + the usual amount of build issues.
Best regards,
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 4 lines · Thomas Petazzoni
On Mon, 17 Oct 2022 07:29:52 -0400
James Hilliard <[email protected]> wrote:
Quoted 8 lines
Quoted 4 lines
The shuffle=random changes don't require modifying autobuild-run, I'm just tweaking the path to make in my autobuild service environment and then passing an additional config flag to autobuild-run like this:
--make-opts="--shuffle=random"Right, but my point stands: it's custom usage/configuration of autobuild-run that we would prefer to discuss on the mailing list first before deploying/enabling.
I've generally been deploying autobuild-run tweaks after initial local testing.
Typically I send them to the mailing list after running them deployed for a little to clean them up and make sure they work as expected and verify the builds that are failing are not due to bugs in the autobuild-run tweaks themselves.
They don't usually get looked at for a while in patchwork so I haven't really been bringing them up on the mailing list until after I've tested/deployed them myself for a little, I have a couple pending in patchwork at the moment for autobuild-run for example:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
Sometimes I override genrandconfig as well to test patches such as:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
I have a local patch to autobuild-run that redirects to an out of tree genrandconfig so that I can test tweaks there. I also have it validating that the in tree version of genrandconfig hasn't changed by comparing against a known sha256 hash before using the out of tree tweaked version to ensure that autobuild-run automatically falls back to using the in-tree genrandconfig if genrandconfig gets changed/fixes in upstream buildroot.
For autobuild-run changes that increase error rates significantly due to higher rates of bugs which need fixes in buildroot I sometimes have held off on sending them as patches until the rates of errors triggered get low enough that it would make sense for others to use those changes as well.
In this case I was also holding off on a patch automatically enabling shuffle mode to autobuild-run as using that feature currently requires running a yet to be released version of make and was triggering a few issues in buildroot that needed fixing on the buildroot side.
I also figured that by testing master make using the autobuilders we can get make issues fixed upstream before the next make release, such as this shuffle mode bug so far:
https://savannah.gnu.org/bugs/index.php?63215
What sort of autobuilder changes would you like me to discuss on the mailing list before deploying? I tend to experiment a bit but am usually pretty careful to make sure that my autobuilder tweaks only cause build failures due to real bugs that need fixing in buildroot/buildroot packages.
Quoted 6 lines
Quoted 6 lines
Well builds were mostly failing in the same place due to deterministic build ordering, I was trying to make it so that we would see failures in different places as that's more useful compared with always seeing the same errors.
Quoted 7 lines
Quoted 2 lines
To me seeing the same errors most of the time indicates there's some major coverage issues with the autobuilders which need fixing.
I think you're missing a point here: many of the builds today are failing due to the glibc 2.36 bump. So pretty much every single build that is glibc based will indeed fail with some error related to this glibc bump.
Probably a lot were failing for a couple days due to this glibc bug:
https://github.com/buildroot/buildroot/commit/b749d6dd7b5fc6cc8b85a316878721d42654759c
There's probably some failures still due to this issue but most of the others seem to be dependency bugs in package makefiles:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
However I'm actually referring to a separate issue with test coverage that has always been present.
Quoted 5 lines
Your additional randomization will not change anything to this. It's just that we now have the glibc bump issues + the newly discovered issues related to the extra randomization + the usual amount of build issues.
When I brought patch up this on irc:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
arnout mentioned:
2022-07-25.log-[03:28:02] <arnout> Lightsword: about the build time: I think the issue is that if you increase the number of selected packages, you also reduce the number of successful builds (because it fails as soon as one failing package is selected) - and that way, packages that get built late (i.e. that come late in the alphabet and have no reverse dependencies) don't get built.
This is indeed the case when I tested increasing the probability further, however by randomizing the dependency ordering using make 4.4's new shuffle mode we should be able to mitigate the alphabetical ordering bias issue and thus get better test coverage of packages with larger dependency trees by increasing the number of selected packages while using shuffle mode.
Looking at the current autobuilder failure logs I'm seeing that most autobuilders
other than my own have much lower build failure rates and not much variety in terms of failure reasons due to poor coverage of packages with larger dependency trees.
For my own autobuilders running with shuffle mode I'm seeing much higher failure rates for different reasons so these changes do appear to be increasing out test coverage overall, there's a lot of issues with makefile dependency ordering that are new and I suspect once those are fixed we'll be able to get much better overall coverage of more common issues, especially issues hit when running parallel builds.
Quoted 8 lines
Best regards,
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Hello James,
Quoted 9 lines · On Fri, 14 Oct 2022 18:58:13 -0600 James Hilliard <[email protected]>…
The bearssl package is not compatible with parallel builds.
Fixes:
- http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773Signed-off-by James Hilliard <[email protected]>--- package/bearssl/bearssl.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Based on the feedback which said that fixing bearssl was preferred over using $(MAKE1), I have marked this patch as Changes Requested. Could you see at fixing the bearssl build system, submit the fix upstream, and provide the fix for Buildroot?
Thanks a lot!
Thomas
Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 16 lines · Thomas Petazzoni
Hello James,
Quoted 9 lines · On Fri, 14 Oct 2022 18:58:13 -0600 James Hilliard <[email protected]>…
The bearssl package is not compatible with parallel builds.
Fixes:
- http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773Signed-off-by James Hilliard <[email protected]>--- package/bearssl/bearssl.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 4 lines
Based on the feedback which said that fixing bearssl was preferred over using $(MAKE1), I have marked this patch as Changes Requested. Could you see at fixing the bearssl build system, submit the fix upstream, and provide the fix for Buildroot?
I contacted the maintainer as described here https://bearssl.org/contrib.html
with the suggested change, so far I haven't received a response back.
Quoted 8 lines
Thanks a lot!
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot