Skip to contentSkip to conversation

system/Config.in: introduce pre-build script

PATCH In review Markus Mayer 3 participants 4 messages Jan 21, 2020 – Jan 6, 2022 Kbuild

lore
Share & tools
lore
#0 lore raw

cover system/Config.in: introduce pre-build script

Hi all,

I am writing a separate "cover letter" for this change, even though it is just a single patch, and a small one at that. But I think an explanation and some background is helpful, which would be too long to include in the patch itself.

Very recently, we discovered the need to run some prepatory steps before launching a Buildroot build.

Specifically, we are building ARM64 images that also contain a 32-bit run-time environment from an Aarch32 toolchain (just the shared libraries and ld.so from sys-root, nothing fancier than that). -- No, we aren't doing this for fun, but because there are some binaries, which we don't have the sources for, that only come as 32-bit variants and they need to be able to execute in an ARM64 environment, as well.

We used to use /lib64 and /lib to store the shared libraries in the target file system. However, we have come to realize that using /lib and /lib32 makes a lot of things much easier (/lib64 merely becomes a symlink to /lib in this setup) and began the transition.

However, introducing the change leads to a problem for developers they don't
expect: they simply update their GIT repo as they are used to doing, and suddenly their already existing build is inconsistent with what is now expected, whereas in the past it has always been possible to do an incremental build even after a "git pull."

Enter a pre-build script that checks if /lib64 is a directory and not a symlink. If so, it removes $TARGET_DIR and all .stamp_target_installed files from the build/ directory. Problem solved: the $TARGET_DIR will be re-populated with the proper directory layout, even if only an incremental build is being performed. The developers (who aren't Buildroot experts and don't really care what mechanims we use to build our file system as long as it works) don't have to remember to run any additional steps manually nor do they have to rebuild everything from scratch or end up running into inexplicable build errors. Instead, the build system knows all on its own to do this additional step only once and only when actually required.

While the reason we want a pre-build script at this point in time is very specific to our setup, I am sure others have come across reasons of their own why such a feature might be helpful. Also, the feature seems to be pretty non-intrusive and adds hardly any overhead if not used.

Only after I came up with the attached patch did I do a quick Google search and discovered that there was an attempt almost seven years ago of adding the same feature. The patch here is almost identical to my proposal:

https://patchwork.ozlabs.org/patch/220466/

It looks like it was denied at the time. I am not exactly sure why. Maybe because the author didn't answer the questions regarding the use case?

In the hopes of a different outcome, I wrote this detailed description above. And, of course, I am more than willing to tweak and adapt this patch based community input and proposals.

Regards,
-Markus

Markus Mayer (1):
system/Config.in: introduce pre-build script

 Makefile         |  3 +++
 system/Config.in | 10 ++++++++++
 2 files changed, 13 insertions(+)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/1 system/Config.in: introduce pre-build script

We introduce the concept of a pre-build script that works similar to the already existing post-build and post-image scripts.

The pre-build script(s) are executed before the build commences. This allows a user to run some preperatory tasks prior to the build.

---
 Makefile         |  3 +++
 system/Config.in | 10 ++++++++++
 2 files changed, 13 insertions(+)
Makefile +3 -0
index a58ca7821d2d..9bf9f1aa05ee 100644
@@ -579,6 +579,9 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 
 .PHONY: prepare
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
+	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_PRE_BUILD_SCRIPT)), \
+		$(call MESSAGE,"Executing pre-build script $(s)"); \
+		$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
 .PHONY: world
 world: target-post-image
system/Config.in +10 -0
index c8c5be40e096..7b890aed1abb 100644
@@ -518,6 +518,16 @@ config BR2_ROOTFS_OVERLAY
 	  They are copied as-is into the rootfs, excluding files ending
 	  with ~ and .git, .svn and .hg directories.
 
+config BR2_ROOTFS_PRE_BUILD_SCRIPT
+	string "Custom scripts to run before commencing the build"
+	default ""
+	help
+	  Specify a space-separated list of scripts to be run before the
+	  build commences.
+
+	  This gives users the opportunity to do board-specific
+	  preparations before starting the build.
+
 config BR2_ROOTFS_POST_BUILD_SCRIPT
 	string "Custom scripts to run before creating filesystem images"
 	default ""
Signature · 1 line
2.17.1
#2 lore raw

Hi Markus,

Quoted 7 lines · Markus Mayer

We introduce the concept of a pre-build script that works similar to the already existing post-build and post-image scripts.

The pre-build script(s) are executed before the build commences. This allows a user to run some preperatory tasks prior to the build.

Although the use case you presented in the cover letter is probably not a
"good" one, as explained by Thomas, I'm pretty sure there are valid cases for having a pre-build script.

I think in general it's actually better to have a wrapper script or Makefile
around Buildroot, because that gives a lot more flexibility. Still, this proposal is symmetrical with the post-image script. And the overhead is negligible. So I applied to master after all.

Regards,
Arnout

Quoted 31 lines
---
  Makefile         |  3 +++
  system/Config.in | 10 ++++++++++
  2 files changed, 13 insertions(+)
Makefile +3 -0
index a58ca7821d2d..9bf9f1aa05ee 100644
@@ -579,6 +579,9 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 
  .PHONY: prepare
  prepare: $(BUILD_DIR)/buildroot-config/auto.conf
+	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_PRE_BUILD_SCRIPT)), \
+		$(call MESSAGE,"Executing pre-build script $(s)"); \
+		$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
  .PHONY: world
  world: target-post-image
system/Config.in +3 -0
index c8c5be40e096..7b890aed1abb 100644
@@ -518,6 +518,16 @@ config BR2_ROOTFS_OVERLAY
  	  They are copied as-is into the rootfs, excluding files ending
  	  with ~ and .git, .svn and .hg directories.
 
+config BR2_ROOTFS_PRE_BUILD_SCRIPT
+	string "Custom scripts to run before commencing the build"
+	default ""

system/Config.in config BR2_ROOTFS_OVERLAY · line 533

Quoted 10 lines
system/Config.in +7 -0
+	help
+	  Specify a space-separated list of scripts to be run before the
+	  build commences.
+
+	  This gives users the opportunity to do board-specific
+	  preparations before starting the build.
+
  config BR2_ROOTFS_POST_BUILD_SCRIPT
  	string "Custom scripts to run before creating filesystem images"
  	default ""

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#3 lore raw

cover system/Config.in: introduce pre-build script

In reply to Markus Mayer

Hello Markus,

Quoted 12 lines · On Tue, 21 Jan 2020 15:09:55 -0800 Markus Mayer <[email protected]> wrote:

Specifically, we are building ARM64 images that also contain a 32-bit run-time environment from an Aarch32 toolchain (just the shared libraries and ld.so from sys-root, nothing fancier than that). -- No, we aren't doing this for fun, but because there are some binaries, which we don't have the sources for, that only come as 32-bit variants and they need to be able to execute in an ARM64 environment, as well.

We used to use /lib64 and /lib to store the shared libraries in the target file system. However, we have come to realize that using /lib and /lib32 makes a lot of things much easier (/lib64 merely becomes a symlink to /lib in this setup) and began the transition.

Quoted 5 lines

However, introducing the change leads to a problem for developers they don't
expect: they simply update their GIT repo as they are used to doing, and suddenly their already existing build is inconsistent with what is now expected, whereas in the past it has always been possible to do an incremental build even after a "git pull."

We simply don't support incremental builds like you describe, i.e there is no way for Buildroot to guarantee after a "git pull" and a partial rebuild with just "make" that changes will be taken into account. A simple thing that adding a new --enable-foo option in a package that has already been built will not see this package being rebuilt when running "make".

So, while there may possibly be other situations where a pre-build script could be useful, your particular use-case is not a very convincing argument at all, because it clearly falls outside of what Buildroot supports.

So what would be nice to see is if there are other use-cases for a pre-build script that make sense.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com