Skip to contentSkip to conversation

Pull request for branch yem/instrument-build

GIT PULL New Yann E. MORIN 4 participants 15 messages Oct 27, 2013 – Oct 29, 2013 Kbuild

lore
Share & tools
lore
#0 lore raw

From: "Yann E. MORIN" <[email protected]>

Hello All!

This series is an RFC to add support for instrumenting the build process, reworked after discussion both on the ML and during the developpers' day.

It works as thus;
- we register a set of hooks
- hooks are called before and after each steps

There are currently two hooks defined:
- log steps with timings (as an example of hos to do a hook)
- call a user-supplied script

This is just a RFC for now, and if it is accepted, new hooks can be defined later on.

I'm planning on adding at least one other hook, that runs all scripts present in support/step-hooks/ (or whatever its name).

Open for comments! ;-)

Changes v1 -> v2:
- removed the intra-steps instrumentation (Thomas, Peter)
- accept a list of user-supplied hooks, not only one hook (Thomas)

Regards,
Yann E. MORIN.

The following changes since commit 200bf74e8e4686c3f132a6600ae6803eda8b588a:

Update config.guess and config.sub scripts to latest versions. (2013-10-27 15:29:38 +0100)

are available in the git repository at:

git://gitorious.org/buildroot/buildroot.git yem/instrument-build

for you to fetch changes up to a4b9a0c71f89f16b00a75ba0b37f19c71c0300c3:

pkg-infra: add user-supplied step-hooks (2013-10-27 16:51:34 +0000)

----------------------------------------------------------------
Yann E. MORIN (4):
post-{build,images} hooks: export BUILD_DIR too
pkg-infra: introduce pre/post-step hooks
pkg-infra: add hook to log timing of steps
pkg-infra: add user-supplied step-hooks

 Makefile                            |  4 +--
 docs/manual/customize-rootfs.txt    |  5 ++--
 docs/manual/debugging-buildroot.txt | 33 +++++++++++++++++++++
 docs/manual/developer-guide.txt     |  2 ++
 package/Makefile.in                 |  5 ++++
 package/pkg-generic.mk              | 59 +++++++++++++++++++++++++++++++++++++
 6 files changed, 104 insertions(+), 4 deletions(-)
 create mode 100644 docs/manual/debugging-buildroot.txt
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH 1/4 post-{build, images} hooks: export BUILD_DIR too

From: "Yann E. MORIN" <[email protected]>

Also export BUILD_DIR for post-{build,images} hooks, so they do have a place to store generated files.

Note: this will be more einteresting for the instrumentation of steps, to come in a later patch.

---
 Makefile                         | 4 ++--
 docs/manual/customize-rootfs.txt | 5 +++--
 package/Makefile.in              | 5 +++++
 3 files changed, 10 insertions(+), 4 deletions(-)
Makefile +2 -2
index f266e2d..0113f75 100644
@@ -511,7 +511,7 @@ endif
 
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_BUILD_SCRIPT)), \
 		$(call MESSAGE,"Executing post-build script $(s)"); \
-		$(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+		$(USER_HOOKS_EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
 LOCALE_WHITELIST=$(BUILD_DIR)/locales.nopurge
@@ -557,7 +557,7 @@ endif
 target-post-image:
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \
-		$(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+		$(USER_HOOKS_EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
 toolchain-eclipse-register:
 	./support/scripts/eclipse-register-toolchain `readlink -f $(O)` $(notdir $(TARGET_CROSS)) $(BR2_ARCH)
docs/manual/customize-rootfs.txt +3 -2
index 49a6b4b..450b3d5 100644
@@ -41,6 +41,7 @@ there are a few ways to customize the resulting target filesystem.
     - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
     - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see
       xref:generic-package-reference[]
+    - +BUILD_DIR+: the directory where packages are extracted and built
     - +BINARIES_DIR+: the place where all binary files (aka images) are
       stored
     - +BASE_DIR+: the base output directory
@@ -78,8 +79,8 @@ in one of these _post-image scripts_ will require special handling
 
 Just like for the _post-build scripts_ mentioned above, you also have
 access to the following environment variables from your _post-image
-scripts_: +BUILDROOT_CONFIG+, +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+,
-+BINARIES_DIR+ and +BASE_DIR+.
+scripts_: +BUILDROOT_CONFIG+, +BUILD_DIR+, +HOST_DIR+, +STAGING_DIR+,
++TARGET_DIR+, +BINARIES_DIR+ and +BASE_DIR+.
 
 Additionally, each of the +BR2_ROOTFS_POST_BUILD_SCRIPT+ and
 +BR2_ROOTFS_POST_IMAGE_SCRIPT+ scripts will be passed the arguments
package/Makefile.in +5 -0
index 612f3c7..2224f27 100644
@@ -275,6 +275,11 @@ HOST_MAKE_ENV=PATH=$(HOST_PATH) \
 		PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig" \
 		PERLLIB="$(HOST_DIR)/usr/lib/perl"
 
+# This extra environment we can not export ourselves, so we have
+# to expicitly pass it to user-supplied external hooks (eg.
+# post-build, post-images)
+USER_HOOKS_EXTRA_ENV=\
+	BUILD_DIR=$(BUILD_DIR)
 
 ################################################################################
 # settings we need to pass to configure
Signature · 1 line
1.8.1.2
#2 lore raw

PATCH 2/4 pkg-infra: introduce pre/post-step hooks

From: "Yann E. MORIN" <[email protected]>

This hooks will let us instrument the build process in many ways:
- log current step to see what broke
- time each step to see what is worth optimising
- sanity-check installed files (rpath, overwritten files...)
- call user-provided script
- ...

The steps are coarse-grain, and all have a 'start' and a 'end' hooks. Here is the list of available steps (8 total):
- extract
- patch
- configure
- build
- install-host
- install-staging
- install-image
- install-target

The download, clean and uninstall steps are not instrumented on purpose.

---
 package/pkg-generic.mk | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
package/pkg-generic.mk +39 -0
index 4bba4b5..799555c 100644
@@ -21,6 +21,29 @@
 ################################################################################
 
 ################################################################################
+# Helper functions to catch start/end of each steps
+################################################################################
+
+# Those two functions are called by each steps below.
+# They are responsible for calling all hooks defined in $(STEP_HOOKS)
+# and pass each of them two arguments:
+#   $1: either 'start' or 'end'
+#   $2: the name of the step
+#   $3: the name of the package
+
+# Start step
+# $1: step name
+define step_start
+	$(foreach hook,$(STEP_HOOKS),$(call $(hook),start,$(1),$($(PKG)_NAME))$(sep))
+endef
+
+# End step
+# $1: step name
+define step_end
+	$(foreach hook,$(STEP_HOOKS),$(call $(hook),end,$(1),$($(PKG)_NAME))$(sep))
+endef
+
+################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
 
@@ -55,6 +78,7 @@ endif
 
 # Unpack the archive
 $(BUILD_DIR)/%/.stamp_extracted:
+	@$(call step_start,extract)
 	@$(call MESSAGE,"Extracting")
 	$(Q)mkdir -p $(@D)
 	$($(PKG)_EXTRACT_CMDS)
@@ -62,6 +86,7 @@ $(BUILD_DIR)/%/.stamp_extracted:
 	$(Q)chmod -R +rw $(@D)
 	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,extract)
 
 # Rsync the source directory if the <pkg>_OVERRIDE_SRCDIR feature is
 # used.
@@ -90,6 +115,7 @@ endif
 $(BUILD_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
 $(BUILD_DIR)/%/.stamp_patched: PATCH_BASE_DIRS = $($(PKG)_DIR_PREFIX)/$(RAWNAME) $(call qstrip,$(BR2_GLOBAL_PATCH_DIR))/$(RAWNAME)
 $(BUILD_DIR)/%/.stamp_patched:
+	@$(call step_start,patch)
 	@$(call MESSAGE,"Patching $($(PKG)_DIR_PREFIX)/$(RAWNAME)")
 	$(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(foreach p,$($(PKG)_PATCH),support/scripts/apply-patches.sh $(@D) $(DL_DIR) $(notdir $(p))$(sep))
@@ -106,31 +132,39 @@ $(BUILD_DIR)/%/.stamp_patched:
 	)
 	$(foreach hook,$($(PKG)_POST_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,patch)
 
 # Configure
 $(BUILD_DIR)/%/.stamp_configured:
+	@$(call step_start,configure)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	@$(call MESSAGE,"Configuring")
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,configure)
 
 # Build
 $(BUILD_DIR)/%/.stamp_built::
+	@$(call step_start,build)
 	@$(call MESSAGE,"Building")
 	$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,build)
 
 # Install to host dir
 $(BUILD_DIR)/%/.stamp_host_installed:
+	@$(call step_start,install-host)
 	@$(call MESSAGE,"Installing to host directory")
 	$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,install-host)
 
 # Install to staging dir
 $(BUILD_DIR)/%/.stamp_staging_installed:
+	@$(call step_start,install-staging)
 	@$(call MESSAGE,"Installing to staging directory")
 	$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
@@ -142,16 +176,20 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 				$(addprefix $(STAGING_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ;\
 	fi
 	$(Q)touch $@
+	@$(call step_end,install-staging)
 
 # Install to images dir
 $(BUILD_DIR)/%/.stamp_images_installed:
+	@$(call step_start,install-image)
 	@$(call MESSAGE,"Installing to images directory")
 	$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,install-image)
 
 # Install to target dir
 $(BUILD_DIR)/%/.stamp_target_installed:
+	@$(call step_start,install-target)
 	@$(call MESSAGE,"Installing to target")
 	$(if $(BR2_INIT_SYSTEMD),\
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
@@ -163,6 +201,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 		$(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
 	fi
 	$(Q)touch $@
+	@$(call step_end,install-target)
 
 # Clean package
 $(BUILD_DIR)/%/.stamp_cleaned:
Signature · 1 line
1.8.1.2
#3 lore raw

PATCH 2/4 pkg-infra: introduce pre/post-step hooks

Quoted 32 lines · Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

This hooks will let us instrument the build process in many ways:
- log current step to see what broke
- time each step to see what is worth optimising
- sanity-check installed files (rpath, overwritten files...)
- call user-provided script
- ...

The steps are coarse-grain, and all have a 'start' and a 'end' hooks. Here is the list of available steps (8 total):
- extract
- patch
- configure
- build
- install-host
- install-staging
- install-image
- install-target

The download, clean and uninstall steps are not instrumented on purpose.

---
package/pkg-generic.mk | 39 +++++++++++++++++++++++++++++++++++++++

1 file changed, 39 insertions(+)
package/pkg-generic.mk +0 -0
index 4bba4b5..799555c 100644

package/pkg-generic.mk line 24

Quoted 4 lines
package/pkg-generic.mk +1 -0
@@ -21,6 +21,29 @@
 ################################################################################
 
 ################################################################################
+# Helper functions to catch start/end of each steps

each step

package/pkg-generic.mk line 27

Quoted 3 lines
package/pkg-generic.mk +3 -0
+################################################################################
+
+# Those two functions are called by each steps below.

each step

package/pkg-generic.mk line 28

Quoted 1 line
package/pkg-generic.mk +1 -0
+# They are responsible for calling all hooks defined in $(STEP_HOOKS)

I personally find the name STEP_HOOKS pretty vague, if you don't know the context you have no clue what it could be. The other hooks POST_EXTRACT etc are clear from their name. Maybe, GLOBAL_INSTRUMENTATION_HOOKS?

Best regards,
Thomas

#4 lore raw

PATCH 2/4 pkg-infra: introduce pre/post-step hooks

Thomas D, Yann,

Thomas De Schampheleire <[email protected]> wrote on 10/27/2013 04:40:09 PM:

Quoted 1 line
Quoted 1 line

################################################################################

Quoted 2 lines
Quoted 2 lines

################################################################################

Quoted 6 lines
Quoted 1 line

+# Helper functions to catch start/end of each steps

each step

+################################################################################

Quoted 8 lines

package/pkg-generic.mk line 27

Quoted 2 lines
package/pkg-generic.mk +2 -0
+
+# Those two functions are called by each steps below.

each step

Quoted 1 line

+# They are responsible for calling all hooks defined in $(STEP_HOOKS)

I personally find the name STEP_HOOKS pretty vague, if you don't know

the

Quoted 1 line

context you have no clue what it could be. The other hooks POST_EXTRACT

etc

Quoted 3 lines

are clear from their name.

Maybe, GLOBAL_INSTRUMENTATION_HOOKS?

But that is so long to type :)

Thanks,
-Ryan

#5 lore raw

PATCH 2/4 pkg-infra: introduce pre/post-step hooks

In reply to Thomas De Schampheleire

On 2013-10-27 22:40 +0100, Thomas De Schampheleire spake thusly:

Quoted 3 lines
Quoted 2 lines · Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>
This hooks will let us instrument the build process in many ways:

[--SNIP--]

Quoted 12 lines

package/pkg-generic.mk line 24

Quoted 9 lines
package/pkg-generic.mk +1 -0
index 4bba4b5..799555c 100644
@@ -21,6 +21,29 @@
 ################################################################################
 
 ################################################################################
+# Helper functions to catch start/end of each steps

each step

Quoted 5 lines

package/pkg-generic.mk line 27

Quoted 3 lines
package/pkg-generic.mk +3 -0
+################################################################################
+
+# Those two functions are called by each steps below.

each step

Both: OK.

Quoted 6 lines

package/pkg-generic.mk line 28

Quoted 1 line
package/pkg-generic.mk +1 -0
+# They are responsible for calling all hooks defined in $(STEP_HOOKS)

I personally find the name STEP_HOOKS pretty vague, if you don't know the context you have no clue what it could be. The other hooks POST_EXTRACT etc are clear from their name.
Maybe, GLOBAL_INSTRUMENTATION_HOOKS?

I'm on the side of Ryan here: it is way too long. :-p I'll try to see if I can come with a better name, though.

Thanks for the review!

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#6 lore raw

PATCH 2/4 pkg-infra: introduce pre/post-step hooks

Hi Yann, Ryan,

Quoted 10 lines · Yann E. MORIN

On 2013-10-27 22:40 +0100, Thomas De Schampheleire spake thusly:

Quoted 4 lines

I personally find the name STEP_HOOKS pretty vague, if you don't know the context you have no clue what it could be. The other hooks POST_EXTRACT etc are clear from their name.
Maybe, GLOBAL_INSTRUMENTATION_HOOKS?

I'm on the side of Ryan here: it is way too long. :-p I'll try to see if I can come with a better name, though.

I'm fine with looking at other names, of course, but I don't see how GLOBAL_INSTRUMENTATION_HOOKS
is longer than
FOO_POST_INSTALL_STAGING_HOOKS FOO_POST_INSTALL_TARGET_HOOKS
or
CONNTRACK_TOOLS_PRE_CONFIGURE_HOOKS

Besides, it's not that you're writing this so regularly that the length is really so important, IMHO.

Best regards,
Thomas

#7 lore raw

PATCH 2/4 pkg-infra: introduce pre/post-step hooks

Thomas DS, All,

On 2013-10-29 21:03 +0100, Thomas De Schampheleire spake thusly:

Quoted 22 lines

Hi Yann, Ryan,

Quoted 10 lines · Yann E. MORIN

On 2013-10-27 22:40 +0100, Thomas De Schampheleire spake thusly:

Quoted 4 lines

I personally find the name STEP_HOOKS pretty vague, if you don't know the context you have no clue what it could be. The other hooks POST_EXTRACT etc are clear from their name.
Maybe, GLOBAL_INSTRUMENTATION_HOOKS?

I'm on the side of Ryan here: it is way too long. :-p I'll try to see if I can come with a better name, though.

I'm fine with looking at other names, of course, but I don't see how GLOBAL_INSTRUMENTATION_HOOKS
is longer than
FOO_POST_INSTALL_STAGING_HOOKS FOO_POST_INSTALL_TARGET_HOOKS
or
CONNTRACK_TOOLS_PRE_CONFIGURE_HOOKS

Quoted 2 lines

Besides, it's not that you're writing this so regularly that the length is really so important, IMHO.

Yeah, yeah, right. :-)

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#8 lore raw

PATCH 3/4 pkg-infra: add hook to log timing of steps

In reply to Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

The timing information is stored in the file $(O)/build-time.log

---
 package/pkg-generic.mk | 11 +++++++++++
 1 file changed, 11 insertions(+)
package/pkg-generic.mk +11 -0
index 799555c..72e0e82 100644
@@ -43,6 +43,17 @@ define step_end
 	$(foreach hook,$(STEP_HOOKS),$(call $(hook),end,$(1),$($(PKG)_NAME))$(sep))
 endef
 
+#######################################
+# Actual steps hooks
+
+# Time steps
+define step_time
+	printf "%s:%-5.5s:%-20.20s: %s\n"           \
+	       "$$(date +%s)" "$(1)" "$(2)" "$(3)"  \
+	       >>"$(BUILD_DIR)/build-time.log"
+endef
+STEP_HOOKS += step_time
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
Signature · 1 line
1.8.1.2
#9 lore raw

PATCH 4/4 pkg-infra: add user-supplied step-hooks

In reply to Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

Allow user to supply their own step-hooks by passing a variable on the make command-line:
make BR2_STEP_USER_HOOK=/path/to/my/script

This can be usefull to run site-specific actions at each step of the build process, such as logging installed, removed or modified files, do sanity checks on installed files...

It is possible to call more than one script, by passing a space-separated lists of scripts to call.

---
 docs/manual/debugging-buildroot.txt | 33 +++++++++++++++++++++++++++++++++
 docs/manual/developer-guide.txt     |  2 ++
 package/pkg-generic.mk              |  9 +++++++++
 3 files changed, 44 insertions(+)
 create mode 100644 docs/manual/debugging-buildroot.txt
docs/manual/debugging-buildroot.txt new file +33 -0
new file mode 100644
index 0000000..7e66208
@@ -0,0 +1,33 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[debugging-buildroot]]
+
+Debugging Buildroot
+-------------------
+
+It is possible to instrument the steps +Buildroot+ does when building
+packages. Define the variable +BR2_STEPS_USER_HOOKS+ to contain the path
+of one or more scripts (or other executables), in a space-separated list,
+you want called before and after each step. The script is called with
+three parameters:
+
+  - +start+ or +end+ to denote the start (resp. the end) of a step;
+  - the name of the step about to be started, or which just ended.
+  - the name of the package
+
+For example :
+
+----
+make BR2_STEPS_USER_HOOKS="/path/to/my/script1 /path/to/my/script2"
+----
+
+That script has access to the following variables:
+
+  - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
+  - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see
+    xref:generic-package-reference[]
+  - +BUILD_DIR+: the directory where packages are extracted and built
+  - +BINARIES_DIR+: the place where all binary files (aka images) are
+    stored
+  - +BASE_DIR+: the base output directory
docs/manual/developer-guide.txt +2 -0
index 43272f5..2e6b7aa 100644
@@ -13,3 +13,5 @@ include::patch-policy.txt[]
 include::download-infra.txt[]
 
 include::board-support.txt[]
+
+include::debugging-buildroot.txt[]
package/pkg-generic.mk +9 -0
index 72e0e82..625dabc 100644
@@ -54,6 +54,15 @@ define step_time
 endef
 STEP_HOOKS += step_time
 
+# User-supplied script
+define step_user
+	@$(foreach user_hook, $(BR2_STEPS_USER_HOOKS), \
+		$(USER_HOOKS_EXTRA_ENV) $(user_hook) "$(1)" "$(2)" "$(3)"$(sep))
+endef
+ifneq ($(BR2_STEPS_USER_HOOKS),)
+STEP_HOOKS += step_user
+endif
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
Signature · 1 line
1.8.1.2
#10 lore raw

PATCH 1/4 post-{build, images} hooks: export BUILD_DIR too

In reply to Yann E. MORIN
Quoted 65 lines · Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

Also export BUILD_DIR for post-{build,images} hooks, so they do have a place to store generated files.

Note: this will be more einteresting for the instrumentation of steps, to come in a later patch.

---

Makefile                         | 4 ++--
docs/manual/customize-rootfs.txt | 5 +++--
package/Makefile.in              | 5 +++++
3 files changed, 10 insertions(+), 4 deletions(-)
Makefile +1 -1
index f266e2d..0113f75 100644
@@ -511,7 +511,7 @@ endif
 
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_BUILD_SCRIPT)), \
 		$(call MESSAGE,"Executing post-build script $(s)"); \
-		$(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+		$(USER_HOOKS_EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)

LOCALE_WHITELIST=$(BUILD_DIR)/locales.nopurge

Makefile +0 -0
@@ -557,7 +557,7 @@ endif

target-post-image:

	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
		$(call MESSAGE,"Executing post-image script $(s)"); \
Makefile +1 -1
-		$(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+		$(USER_HOOKS_EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))

toolchain-eclipse-register:

	./support/scripts/eclipse-register-toolchain `readlink -f $(O)` $(notdir $(TARGET_CROSS)) $(BR2_ARCH)
docs/manual/customize-rootfs.txt +1 -0
index 49a6b4b..450b3d5 100644
@@ -41,6 +41,7 @@ there are a few ways to customize the resulting target filesystem.
    - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
    - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see
      xref:generic-package-reference[]
+    - +BUILD_DIR+: the directory where packages are extracted and built
    - +BINARIES_DIR+: the place where all binary files (aka images) are
      stored
    - +BASE_DIR+: the base output directory
@@ -78,8 +79,8 @@ in one of these _post-image scripts_ will require special handling
 

Just like for the _post-build scripts_ mentioned above, you also have access to the following environment variables from your _post-image

docs/manual/customize-rootfs.txt +2 -2
-scripts_: +BUILDROOT_CONFIG+, +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+,
-+BINARIES_DIR+ and +BASE_DIR+.
+scripts_: +BUILDROOT_CONFIG+, +BUILD_DIR+, +HOST_DIR+, +STAGING_DIR+,
++TARGET_DIR+, +BINARIES_DIR+ and +BASE_DIR+.

Additionally, each of the +BR2_ROOTFS_POST_BUILD_SCRIPT+ and

docs/manual/customize-rootfs.txt +1 -0
+BR2_ROOTFS_POST_IMAGE_SCRIPT+ scripts will be passed the arguments
package/Makefile.in +0 -0
index 612f3c7..2224f27 100644

package/Makefile.in HOST_MAKE_ENV=PATH=$(HOST_PATH) \ · line 282

Quoted 8 lines
package/Makefile.in +5 -0
@@ -275,6 +275,11 @@ HOST_MAKE_ENV=PATH=$(HOST_PATH) \
 		PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig" \
 		PERLLIB="$(HOST_DIR)/usr/lib/perl"
 
+# This extra environment we can not export ourselves, so we have
+# to expicitly pass it to user-supplied external hooks (eg.
+# post-build, post-images)
+USER_HOOKS_EXTRA_ENV=\
+	BUILD_DIR=$(BUILD_DIR)

I'd add a bit more info here: we cannot export BUILD_DIR globally because some packages, like uboot, also use this same variable for another purpose.

Best regards,
Thomas

#11 lore raw

PATCH 1/4 post-{build, images} hooks: export BUILD_DIR too

Dear Thomas De Schampheleire,

Quoted 7 lines · Thomas De Schampheleire

package/Makefile.in HOST_MAKE_ENV=PATH=$(HOST_PATH) \ · line 282

Quoted 5 lines
package/Makefile.in +5 -0
+# This extra environment we can not export ourselves, so we have
+# to expicitly pass it to user-supplied external hooks (eg.
+# post-build, post-images)
+USER_HOOKS_EXTRA_ENV=\
+	BUILD_DIR=$(BUILD_DIR)

I'd add a bit more info here: we cannot export BUILD_DIR globally because some packages, like uboot, also use this same variable for another purpose.

(Unfortunately, we've asked you to not wrapped patches you reply to, but now, it has the consequence that you no longer wrap what you write. Well, I guess with K9 it's hard to get both).

Regarding BUILD_DIR and other variables being exported, is it really safe to export other variables with a relatively generic name such as TARGET_DIR, BASE_DIR, HOST_DIR, BINARIES_DIR, etc. ? Should we instead expose them to post-build/post-image scripts as BR2_<something> ? But well, of course, it means breaking the existing API, which isn't nice.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#12 lore raw

PATCH 1/4 post-{build, images} hooks: export BUILD_DIR too

Thomas P, All,

On 2013-10-28 06:17 +0100, Thomas Petazzoni spake thusly:

Quoted 9 lines

Dear Thomas De Schampheleire,

Quoted 5 lines · Thomas De Schampheleire

package/Makefile.in HOST_MAKE_ENV=PATH=$(HOST_PATH) \ · line 282

Quoted 5 lines
package/Makefile.in +5 -0
+# This extra environment we can not export ourselves, so we have
+# to expicitly pass it to user-supplied external hooks (eg.
+# post-build, post-images)
+USER_HOOKS_EXTRA_ENV=\
+	BUILD_DIR=$(BUILD_DIR)

[--SNIP--]

Quoted 5 lines

Regarding BUILD_DIR and other variables being exported, is it really safe to export other variables with a relatively generic name such as TARGET_DIR, BASE_DIR, HOST_DIR, BINARIES_DIR, etc. ? Should we instead expose them to post-build/post-image scripts as BR2_<something> ? But well, of course, it means breaking the existing API, which isn't nice.

If we export all of those variables with this mechanism rathe than with the make's 'export', then we again dwell on the sagfe side, and we need not rename them.

We should have named them BR2_something from the beginning, even for our internal use, to avoid any name-clashing with the calling environment. For example:

    $ export BUILD_DIR=/path/to/my/project/build.dir
    $ mkdir -p "${BUILD_DIR}/buildroot.output"
    $ make -C "${BUILD_DIR}//buildroot.output"

Suppose the user has post-{build,image} or other rsource files (eg. kernel .config) in its ${BUILD_DIR}, then his/her configuration is broken.

But that's the way story goes, we fscked up at the beginning, we have to live with them, now... :-(

Anyway, I'll keep this change as-is, and make a new cset that export all these variables to using this new mechanism, so they only get exported for post-{build,image} scripts (and the new steps-user-hook).

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#13 lore raw

PATCH 1/4 post-{build, images} hooks: export BUILD_DIR too

In reply to Thomas Petazzoni

Hi Thomas,

On Mon, Oct 28, 2013 at 6:17 AM, Thomas Petazzoni
<[email protected]> wrote:
[..]

Quoted 5 lines

(Unfortunately, we've asked you to not wrapped patches you reply to, but now, it has the consequence that you no longer wrap what you write. Well, I guess with K9 it's hard to get both).

Hmm :-o
I'm not sure if I can easily fix this with K9. My earlier bug report and even partial patch on the k9 mailing list didn't get any response. I'll check the source but I'll try not to write long sentences in the mean time ;-)

Best regards,
Thomas

#14 lore raw

PATCH 1/4 post-{build, images} hooks: export BUILD_DIR too

In reply to Thomas De Schampheleire

Thomas DS, All,

On 2013-10-27 22:12 +0100, Thomas De Schampheleire spake thusly:

Quoted 5 lines
Quoted 4 lines · Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

Also export BUILD_DIR for post-{build,images} hooks, so they do have a place to store generated files.

[--SNIP--]

Quoted 9 lines

package/Makefile.in HOST_MAKE_ENV=PATH=$(HOST_PATH) \ · line 282

Quoted 5 lines
package/Makefile.in +5 -0
+# This extra environment we can not export ourselves, so we have
+# to expicitly pass it to user-supplied external hooks (eg.
+# post-build, post-images)
+USER_HOOKS_EXTRA_ENV=\
+	BUILD_DIR=$(BUILD_DIR)

I'd add a bit more info here: we cannot export BUILD_DIR globally because some packages, like uboot, also use this same variable for another purpose.

Will do, thanks.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'