New: Too many directories in BR2_EXTERNAL causes hang during make
https://bugs.busybox.net/show_bug.cgi?id=14996
Bug ID: 14996
Summary: Too many directories in BR2_EXTERNAL causes hang
during make
Product: buildroot
Version: unspecified
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P5
Component: Other
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected]
Target Milestone: ---
The top level Makefile in buildroot has a recursive rule which causes the appearance of a hang as the number of directories in BR2_EXTERNAL increases. When the number of directories in BR2_EXTERNAL is small, the recursion occurs, but make detects the recursion and determines the target does not have to be remade. This allows make to progress.
This is the failing rule:
define percent_defconfig
# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig %_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \
$$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN)endef
$(eval $(foreach d,$(call reverse,$(TOPDIR) $(BR2_EXTERNAL_DIRS)),$(call percent_defconfig,$(d))$(sep)))
The rule for %defconfig is created for each directory in BR2_EXTERNAL. When the rule is matched, the stem is 'defconfig_name'. The second prerequisite is expanded to $(1)/configs/defconfig_name_defconfig. The rule, and all of the other rules defined by this macro, are invoked again, but the stem is now $(1)/configs/defconfig_name_defconfig. The second prerequisite is now expanded to $(1)/configs/($1)/configs/defconfig_name_defconfig. This expansion continues until make detects the infinite recursion.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
https://bugs.busybox.net/show_bug.cgi?id=14996
Yann E. MORIN <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Assignee|[email protected] |[email protected] |.org | CC| |[email protected] Status|NEW |ASSIGNED
--- Comment #1 from Yann E. MORIN <[email protected]> --- David, All,
Thanks for the report, and the detailed explanations (which I did not yet entirely
groked, and will need a bit more time before I see the issue).
How many br2-external trees are you using? I've done a bit of testing with 0 to 1000 br2-external trees, and here are the results:
$ time make BR2_EXTERNAL='' list-defconfigs real 0m0.129s
$ time make BR2_EXTERNAL='(1 item)' list-defconfigs real 0m0.138s
$ time make BR2_EXTERNAL='(10 item)' list-defconfigs real 0m0.262s
$ time make BR2_EXTERNAL='(100 item)' list-defconfigs real 0m1.912s
$ time make BR2_EXTERNAL='(1000 item)' list-defconfigs real 0m19.344s
So, what we can see, is that it is roughly linear to the number of br2-external trees. Even the 1000-tree case is far from appearing stuck (granted, it is a bit
long, but after seeing 2s for 100 items, it did not feel excruciating to wait).
Also, I do not have any message from make 4.2.1 that reports any infinite recursion.
Can you provide a bit more details on your configuration?
Regards,
Yann E. MORIN.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #2 from David Lawson <[email protected]> ---
I am sorry, I am working on a github project to show the problem. Do you have a defconfig file in any of the external projects? It should only take one. I am concerned also about the name of the defconfig file.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #3 from Yann E. MORIN <[email protected]> --- David, All,
Here's how I prepared the test br2-external trees:
$ for i in $(seq 1 1000); do
mkdir -p br2-external-${i}/configs
touch br2-external-${i}/{Config.in,external.mk}
echo "name: BR_TEST_${i}" >br2-external-${i}/external.desc
touch br2-external-${i}/configs/foo{,_${i}}_defconfig
doneSo, I get minimal br2-external trees, each with two defconfigs:
one that is present in all br2external trees, and one that is specific to each tree.
And here is exactly how I tested that:
$ make --version GNU Make 4.2.1
$ for ext in 1 10 100 1000; do time make BR2_EXTERNAL="$(for i in $(seq 1
${ext}); do echo $(pwd)/br2-external-${i}; done)" list-defconfigs >/dev/null;
done 2>&1 |grep real
real 0m0.179s
real 0m0.245s
real 0m1.299s
real 0m18.277sWhat version of make are you using? How many br2-external trees do you have?
However, please do note that indeed, it was never envisioned that more than a few
br2-external trees be used simultaneously. But that it feels like it got stuck is
still not a nice behaviour...
Regards,
Yann E. MORIN.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #4 from Yann E. MORIN <[email protected]> --- David, All,
To be noted: on my machine, in the directory I was using, I was limited to using just 2446 br2-external trees, otherwise, make would whine with:
- with 2447:
make: execvp: /bin/sh: Argument list too long
- with 2248 and above:
-bash: /usr/bin/make: Argument list too long
Yes, note how 2446 succeeds, 2447 has a special error, and 2448 and above have another error... Meh...
But still, 2446 br2-external trees is clearly way more than was ever expected when the feature was added... back in 2016!
Regards,
Yann E. MORIN.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #5 from David Lawson <[email protected]> --- Here is a simple project that shows the problem:
https://github.com/dclawso/buldroot-recursive
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #6 from David Lawson <[email protected]> --- This shows the recursion occurs for only one external package, but it is largely benign as make detects the recursion and determines the target does not have to be remade.
However, with as few as 6 external packages, the number of dependency checks that are performed causes make to hang for several hours.
We currently are at 8 external packages and I left the compile running over the weekend with no progress.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
David Lawson <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Host| |ubuntu 22.04
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #7 from Yann E. MORIN <[email protected]> --- David, All,
OK, I could reproduce the issue with a much less convoluted example than your script.
The issue does only lanifests itself when actually trying to configure Buildroot.
Listing the defconfig like I was doing was not enough to trigger the issue.
What's weird, is that using 5 external trees does not exhibit the issue; it is very fast. But using 6 trees triggers the issue, and make is insanely long (I did
not even wait for it to finish, in fact).
Note however that your comments are a bit confusing. You are talking about external
packages, but that is incorrect; it is about _br2-external trees_. Also, your script
is very confusing, because it does very convoluted things (instead of providing a git
tree that was already fully prepared). But since we don't need it to reproduce the
issue (and thus to test a fix), I'm not going to comment further. ;-)
I'll lok into devising a fix...
Regards,
Yann E. MORIN.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #8 from Yann E. MORIN <[email protected]> --- David, All,
Could you please have a go with, and test that patch, please:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
Regards,
Yann E. MORIN.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #9 from David Lawson <[email protected]> --- The comments for $1 and $2 appear to be reversed. I am still attempting to incorporate this patch into the build tree I am using.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #10 from David Lawson <[email protected]> --- This change is working in my environment.
Thank you!
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #11 from Yann E. MORIN <[email protected]> --- David, All,
(In reply to David Lawson from comment #10)
Quoted 1 line
This change is working in my environment.
Thanks for the feedback. I've just resent a v3 with trivial changes. If you can validate it still works, could you please reply to the mail with your review tag, ple
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[email protected]
Tested-by Your NAME[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
--- Comment #12 from Yann E. MORIN <[email protected]> --- David, All,
(In reply to David Lawson from comment #10)
Quoted 1 line
This change is working in my environment.
Thanks for the feedback. I've just resent a v3 with trivial changes. If you can validate it still works, could you please reply to the mail with your tested tag, please:
Regards,
Yann E. MORIN.
PS. Sorry for the previous spurious message, PEBCAK.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14996] Too many directories in BR2_EXTERNAL causes hang during make
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14996
Yann E. MORIN <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |RESOLVED
Resolution|--- |FIXED--- Comment #13 from Yann E. MORIN <[email protected]> --- David, All,
Thank you for the report.
We believe this is now fixed with commit e6195c53041f (Makefile: fix use of many br2-external trees).
If you still encounter the issue, please re-open.
Regards,
Yann E. MORIN.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot