system/Config.in: introduce pre-build script
cover system/Config.in: introduce pre-build script
Hi all,
I am writing a separate "cover letter" for this change, even though it is just a single patch, and a small one at that. But I think an explanation and some background is helpful, which would be too long to include in the patch itself.
Very recently, we discovered the need to run some prepatory steps before launching a Buildroot build.
Specifically, we are building ARM64 images that also contain a 32-bit run-time environment from an Aarch32 toolchain (just the shared libraries and ld.so from sys-root, nothing fancier than that). -- No, we aren't doing this for fun, but because there are some binaries, which we don't have the sources for, that only come as 32-bit variants and they need to be able to execute in an ARM64 environment, as well.
We used to use /lib64 and /lib to store the shared libraries in the target file system. However, we have come to realize that using /lib and /lib32 makes a lot of things much easier (/lib64 merely becomes a symlink to /lib in this setup) and began the transition.
However, introducing the change leads to a problem for developers they don't
expect: they simply update their GIT repo as they are used to doing, and suddenly their already existing build is inconsistent with what is now expected, whereas in the past it has always been possible to do an incremental build even after a "git pull."
Enter a pre-build script that checks if /lib64 is a directory and not a symlink. If so, it removes $TARGET_DIR and all .stamp_target_installed files from the build/ directory. Problem solved: the $TARGET_DIR will be re-populated with the proper directory layout, even if only an incremental build is being performed. The developers (who aren't Buildroot experts and don't really care what mechanims we use to build our file system as long as it works) don't have to remember to run any additional steps manually nor do they have to rebuild everything from scratch or end up running into inexplicable build errors. Instead, the build system knows all on its own to do this additional step only once and only when actually required.
While the reason we want a pre-build script at this point in time is very specific to our setup, I am sure others have come across reasons of their own why such a feature might be helpful. Also, the feature seems to be pretty non-intrusive and adds hardly any overhead if not used.
Only after I came up with the attached patch did I do a quick Google search and discovered that there was an attempt almost seven years ago of adding the same feature. The patch here is almost identical to my proposal:
https://patchwork.ozlabs.org/patch/220466/
It looks like it was denied at the time. I am not exactly sure why. Maybe because the author didn't answer the questions regarding the use case?
In the hopes of a different outcome, I wrote this detailed description above. And, of course, I am more than willing to tweak and adapt this patch based community input and proposals.
Regards,
-Markus
Markus Mayer (1):
system/Config.in: introduce pre-build script
Makefile | 3 +++ system/Config.in | 10 ++++++++++ 2 files changed, 13 insertions(+)
Signature · 1 line
2.17.1
PATCH 1/1 system/Config.in: introduce pre-build script
We introduce the concept of a pre-build script that works similar to the already existing post-build and post-image scripts.
The pre-build script(s) are executed before the build commences. This allows a user to run some preperatory tasks prior to the build.
--- Makefile | 3 +++ system/Config.in | 10 ++++++++++ 2 files changed, 13 insertions(+)
Makefile +3 -0
@@ -579,6 +579,9 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG) .PHONY: prepare prepare: $(BUILD_DIR)/buildroot-config/auto.conf + @$(foreach s, $(call qstrip,$(BR2_ROOTFS_PRE_BUILD_SCRIPT)), \ + $(call MESSAGE,"Executing pre-build script $(s)"); \ + $(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep)) .PHONY: world world: target-post-image
system/Config.in +10 -0
@@ -518,6 +518,16 @@ config BR2_ROOTFS_OVERLAY They are copied as-is into the rootfs, excluding files ending with ~ and .git, .svn and .hg directories. +config BR2_ROOTFS_PRE_BUILD_SCRIPT + string "Custom scripts to run before commencing the build" + default "" + help + Specify a space-separated list of scripts to be run before the + build commences. + + This gives users the opportunity to do board-specific + preparations before starting the build. + config BR2_ROOTFS_POST_BUILD_SCRIPT string "Custom scripts to run before creating filesystem images" default ""
Signature · 1 line
2.17.1
Hi Markus,
Quoted 7 lines · Markus Mayer
We introduce the concept of a pre-build script that works similar to the already existing post-build and post-image scripts.
The pre-build script(s) are executed before the build commences. This allows a user to run some preperatory tasks prior to the build.
Signed-off-by Markus Mayer <[email protected]>
Although the use case you presented in the cover letter is probably not a
"good" one, as explained by Thomas, I'm pretty sure there are valid cases for having a pre-build script.
I think in general it's actually better to have a wrapper script or Makefile
around Buildroot, because that gives a lot more flexibility. Still, this proposal is symmetrical with the post-image script. And the overhead is negligible. So I applied to master after all.
Regards,
Arnout
Quoted 31 lines
--- Makefile | 3 +++ system/Config.in | 10 ++++++++++ 2 files changed, 13 insertions(+)Makefile +3
-0@@ -579,6 +579,9 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG) .PHONY: prepare prepare: $(BUILD_DIR)/buildroot-config/auto.conf + @$(foreach s, $(call qstrip,$(BR2_ROOTFS_PRE_BUILD_SCRIPT)), \ + $(call MESSAGE,"Executing pre-build script $(s)"); \ + $(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep)) .PHONY: world world: target-post-imagesystem/Config.in +3
-0@@ -518,6 +518,16 @@ config BR2_ROOTFS_OVERLAY They are copied as-is into the rootfs, excluding files ending with ~ and .git, .svn and .hg directories. +config BR2_ROOTFS_PRE_BUILD_SCRIPT + string "Custom scripts to run before commencing the build" + default ""
system/Config.in config BR2_ROOTFS_OVERLAY · line 533
Quoted 10 lines
system/Config.in +7
-0+ help + Specify a space-separated list of scripts to be run before the + build commences. + + This gives users the opportunity to do board-specific + preparations before starting the build. + config BR2_ROOTFS_POST_BUILD_SCRIPT string "Custom scripts to run before creating filesystem images" default ""
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
cover system/Config.in: introduce pre-build script
In reply to Markus MayerHello Markus,
Quoted 12 lines · On Tue, 21 Jan 2020 15:09:55 -0800 Markus Mayer <[email protected]> wrote:
Specifically, we are building ARM64 images that also contain a 32-bit run-time environment from an Aarch32 toolchain (just the shared libraries and ld.so from sys-root, nothing fancier than that). -- No, we aren't doing this for fun, but because there are some binaries, which we don't have the sources for, that only come as 32-bit variants and they need to be able to execute in an ARM64 environment, as well.
We used to use /lib64 and /lib to store the shared libraries in the target file system. However, we have come to realize that using /lib and /lib32 makes a lot of things much easier (/lib64 merely becomes a symlink to /lib in this setup) and began the transition.
Quoted 5 lines
However, introducing the change leads to a problem for developers they don't
expect: they simply update their GIT repo as they are used to doing, and suddenly their already existing build is inconsistent with what is now expected, whereas in the past it has always been possible to do an incremental build even after a "git pull."
We simply don't support incremental builds like you describe, i.e there is no way for Buildroot to guarantee after a "git pull" and a partial rebuild with just "make" that changes will be taken into account. A simple thing that adding a new --enable-foo option in a package that has already been built will not see this package being rebuilt when running "make".
So, while there may possibly be other situations where a pre-build script could be useful, your particular use-case is not a very convincing argument at all, because it clearly falls outside of what Buildroot supports.
So what would be nice to see is if there are other use-cases for a pre-build script that make sense.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com