Prototype implementation of per-package out of tree build

Hello,

There has recently been some interest in improving the support for overriden packages by doing out of tree build of all packages. Since I had the beginning of a prototype working, I cleaned it up, made numerous fixes to make it work apparently properly, and share it with the community.

The following patches should be considered as a prototype, I haven't tested all packages (of course), and there are quite certainly some corner case issues that haven't been solved. However, I think it's a good time to get more testing in order to find, and solve those issues.

It should also not be considered yet that we want this feature in Buildroot. It is again only a prototype, which aims at solving the remaining problems, and see if we think it is a good idea to have this in Buildroot or not.

For those who want to test, the 'out-of-tree' branch is available at:

git://git.free-electrons.com/users/thomas-petazzoni/buildroot.git

Here is a quick summary of what this patch set does:

* The tarballs are now extracted in $(O)/src/<pkg>-<version>

* Each tarball is extracted and patched only once, even if there is both a target build and a host build. Once a package has been extracted and patched, its source code is changed to not have write permissions, in order to catch misbehaving packages in terms of out of tree build.

* The builds continue to be done in $(O)/build/<pkg>-<version> for target packages and $(O)/build/host-<pkg>-<version> for host packages.

* A <pkg>_SUPPORTS_OUT_OF_TREE variable defines whether the package supports out of tree build or not. By default, it's set to NO, in which case the package source code is rsynced from its source directory to the build directory at the beginning of the configure step. It is set to YES by both the autotools and cmake package infrastructures (which have been modified to do out of tree build), and by some packages that use the generic infrastructure but nonetheless support out of tree build.

* The override source directory mechanism is updated to take advantage of this out of tree support. Now, if you override a package that supports out of tree build, the source code is no longer rsynced to the build directory. This makes the override source directory mechanism a lot more usable for big packages such as the Linux kernel.

Amongst the known remaining issues to solve are:

* The semantic of the 'make <pkg>-dirclean' command. Should it remove the build directory and the source directory? Only the build directory? If it removes both (which the proposed prototype currently does), it is strange because you remove one build variant (host or target) and the source directory, but the other build variant (target or host) remains.

* I am not sure on how we want to handle <pkg>-reconfigure and <pkg>-rebuild. I guess we want to tell the user to do his changes in the package source directory, output/src/<pkg>-<version>. But all files in this source directory are marked read-only, so it's not very practical.

In addition to this, for packages that don't support out of tree build, we do a rsync at the beginning of the configure step. So if the user does "make <pkg>-reconfigure", then the source code gets rsynced again from the source directory to the build directory, but not if the user does "make <pkg>-rebuild".

Comments, ideas, suggestions are welcome.

Best regards,

Thomas

This new directory will be used to extract the source directory of the different packages, as part of the out of tree support.

Note that we need to explicitly re-add write permissions on the contents of this directory, because write permissions are removed from the source code of packages in order to ensure that the packages are not incorrectly modifying their source directory during an out of tree build. We also re-add write permissions to the build directory since some packages copy files from their source directory to the build directory, causing rm to fail due to lack of permissions.

---
 Makefile |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

index 6f8ed0e..15ce4e4 100644
@@ -260,6 +260,7 @@ GENERATE_LOCALE=$(call qstrip,$(BR2_GENERATE_LOCALE))
 STAMP_DIR:=$(BASE_DIR)/stamps
 
 BINARIES_DIR:=$(BASE_DIR)/images
+SRC_DIR:=$(BASE_DIR)/src
 TARGET_DIR:=$(BASE_DIR)/target
 TOOLCHAIN_DIR=$(BASE_DIR)/toolchain
 TARGET_SKELETON=$(TOPDIR)/system/skeleton

@@ -383,7 +384,7 @@ TARGETS_LEGAL_INFO:=$(patsubst %,%-legal-info,\
 $(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) %
 
 dirs: $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
-	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
+	$(HOST_DIR) $(BINARIES_DIR) $(SRC_DIR) $(STAMP_DIR)
 
 $(BASE_TARGETS): dirs $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
 

@@ -409,7 +410,7 @@ world: toolchain $(TARGETS_ALL)
 # dependencies anywhere else
 #
 #############################################################
-$(TOOLCHAIN_DIR) $(BUILD_DIR) $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR):
+$(TOOLCHAIN_DIR) $(BUILD_DIR) $(HOST_DIR) $(BINARIES_DIR) $(SRC_DIR) $(STAMP_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR):
 	@mkdir -p $@
 
 $(STAGING_DIR):

@@ -704,9 +705,10 @@ ifeq ($(NEED_WRAPPER),y)
 endif
 
 clean:
+	-chmod u+w -R $(SRC_DIR)/* $(BUILD_DIR)/*
 	rm -rf $(STAGING_DIR) $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
 		$(STAMP_DIR) $(BUILD_DIR) $(TOOLCHAIN_DIR) $(BASE_DIR)/staging \
-		$(LEGAL_INFO_DIR)
+		$(LEGAL_INFO_DIR) $(SRC_DIR)
 
 distclean: clean
 ifeq ($(DL_DIR),$(TOPDIR)/dl)

1.7.9.5

I agree that that would be a good idea.

However, what if the patches are different for two configurations? They
can be different because:

- there is a configuration option for custom patches, like BR2_LINUX_KERNEL_PATCH

- there are patches applied depending on configuration options, like BR2_LINUX_KERNEL_EXT_XENOMAI_ADEOS_PATCH

- there are manual modifications applied depending on some configuration option, like BZIP2_NOLARGEFILE_FIX_MAKEFILE

- override.mk appends to the FOO_PATCH variable

- BR2_EXTRA_PACKAGE_PATCH_DIRS (not committed yet,
http://patchwork.ozlabs.org/patch/204216/ )

So in practice, I'm afraid it's not feasible.

Regards,
Arnout

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

The <pkg>_SUBDIR variable was currently somewhat handled at the generic package infrastructure level, with the $(PKG)_SRCDIR variable taking into account this SUBDIR. But in fact, this <pkg>_SUBDIR variable only makes sense in the context of the autotools infrastructure, to indicate that the configure script is not located at the root of the source tree, but instead in a sub-directory. In the case of generic packages, the package .mk file already has enough freedom to handle bizarre situations where the Makefile or other parts of the build system may not be located at the root of the package source tree.

In order to simplify the introduction of out of tree support, we therefore move the SUBDIR support as an autotools infrastructure specific feature.

---
 package/pkg-autotools.mk |   30 ++++++++++++++++++++----------
 package/pkg-generic.mk   |   11 +----------
 2 files changed, 21 insertions(+), 20 deletions(-)

index 890506b..843239d 100644
@@ -70,6 +70,16 @@ ifndef $(2)_MAKE
  endif
 endif
 
+ifndef $(2)_SUBDIR
+ ifdef $(3)_SUBDIR
+  $(2)_SUBDIR = $$($(3)_SUBDIR)
+ else
+  $(2)_SUBDIR ?=
+ endif
+endif
+
+$(2)_SRCSUBDIR = $$($(2)_SRCDIR)/$($(2)_SUBDIR)
+
 $(2)_CONF_ENV			?=
 $(2)_CONF_OPT			?=
 $(2)_MAKE_ENV			?=

@@ -93,7 +103,7 @@ ifeq ($(5),target)
 
 # Configure package for target
 define $(2)_CONFIGURE_CMDS
-	(cd $$($$(PKG)_SRCDIR) && rm -rf config.cache && \
+	(cd $$($$(PKG)_SRCSUBDIR) && rm -rf config.cache && \
 	$$(TARGET_CONFIGURE_OPTS) \
 	$$(TARGET_CONFIGURE_ARGS) \
 	$$($$(PKG)_CONF_ENV) \

@@ -120,7 +130,7 @@ else
 # because it often relies on host tools which may or may not be
 # installed.
 define $(2)_CONFIGURE_CMDS
-	(cd $$($$(PKG)_SRCDIR) && rm -rf config.cache; \
+	(cd $$($$(PKG)_SRCSUBDIR) && rm -rf config.cache; \
 	        $$(HOST_CONFIGURE_OPTS) \
 		CFLAGS="$$(HOST_CFLAGS)" \
 		LDFLAGS="$$(HOST_LDFLAGS)" \

@@ -216,11 +226,11 @@ endif
 ifndef $(2)_BUILD_CMDS
 ifeq ($(5),target)
 define $(2)_BUILD_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 else
 define $(2)_BUILD_CMDS
-	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 endif

@@ -231,7 +241,7 @@ endif
 #
 ifndef $(2)_INSTALL_CMDS
 define $(2)_INSTALL_CMDS
-	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -C $$($$(PKG)_SRCDIR) install
+	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -C $$($$(PKG)_SRCSUBDIR) install
 endef
 endif
 

@@ -241,7 +251,7 @@ endif
 #
 ifndef $(2)_INSTALL_STAGING_CMDS
 define $(2)_INSTALL_STAGING_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_STAGING_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_STAGING_OPT) -C $$($$(PKG)_SRCSUBDIR)
 	for i in $$$$(find $(STAGING_DIR)/usr/lib* -name "*.la"); do \
 		cp -f $$$$i $$$$i~; \
 		$$(SED) "s:\(['= ]\)/usr:\\1$(STAGING_DIR)/usr:g" $$$$i; \

@@ -255,7 +265,7 @@ endif
 #
 ifndef $(2)_INSTALL_TARGET_CMDS
 define $(2)_INSTALL_TARGET_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_TARGET_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_TARGET_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 

@@ -265,7 +275,7 @@ endif
 #
 ifndef $(2)_CLEAN_CMDS
 define $(2)_CLEAN_CMDS
-	-$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE)  $$($$(PKG)_CLEAN_OPT) -C $$($$(PKG)_SRCDIR)
+	-$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE)  $$($$(PKG)_CLEAN_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 

@@ -275,7 +285,7 @@ endif
 #
 ifndef $(2)_UNINSTALL_STAGING_CMDS
 define $(2)_UNINSTALL_STAGING_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_UNINSTALL_STAGING_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_UNINSTALL_STAGING_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 

@@ -288,7 +298,7 @@ endif
 #
 ifndef $(2)_UNINSTALL_TARGET_CMDS
 define $(2)_UNINSTALL_TARGET_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -k $$($$(PKG)_UNINSTALL_TARGET_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -k $$($$(PKG)_UNINSTALL_TARGET_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 

index 59de0f0..eeaa4fe 100644
@@ -218,16 +218,7 @@ endif
 $(2)_BASE_NAME	=  $(1)-$$($(2)_VERSION)
 $(2)_DL_DIR	=  $$(DL_DIR)/$$($(2)_BASE_NAME)
 $(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
-
-ifndef $(2)_SUBDIR
- ifdef $(3)_SUBDIR
-  $(2)_SUBDIR = $$($(3)_SUBDIR)
- else
-  $(2)_SUBDIR ?=
- endif
-endif
-
-$(2)_SRCDIR		       = $$($(2)_DIR)/$$($(2)_SUBDIR)
+$(2)_SRCDIR		       = $$($(2)_DIR)
 $(2)_BUILDDIR		       ?= $$($(2)_SRCDIR)
 
 ifneq ($$($(2)_OVERRIDE_SRCDIR),)

1.7.9.5

I'm a little nervous about the out-of-tree stuff in general: my experience is that out-of-tree builds don't work with quite a lot of packages, so while it would work with the key stuff (particularly the kernel and the toolchain, probably) it would always end up a little flakey.

On the other hand, I've been thinking about a totally different approach that people might find interesting. What if we made a local git repo out of every tree? Basically:

- extract the tarball
- git init
- git add .
- git commit -m 'initial tarball $filename'
- git am /path/to/relevant/patches # apply the patches and commit them in git

Then if you want to share files between, say, host-* and target-*, you can use the git_new_worktree script to make a second checkout from the same git repo.

If someone is working in one of the trees and makes a change, it's then easy to 'git checkout .' in the other tree and get the same changes. So your builds can be identical, but there's no need for the hackery around read-only files, etc. And there's a 'git clean' to guaranteed wipe out all the non-source files (to rebuild without having to erase and re-extract/re-patch the directory, so it's faster), 'git diff' to tell you if any of your source files have been changed, etc.

Using git for the patch queue also makes it easy to update or add patches using 'git rebase'. For example, sometimes I've patched a particular package and notice a bug in my patch, so I would like to patch the patch. The way I do this is to 'git am' the patch sequence, then modify the code, then 'git rebase -i' to merge and rearrange the patches, then 'git format-patch' to re-extract the set of patches and drop them back into buildroot.

If buildroot did all this explicitly and automatically as part of the build process, it would grow a dependency on git. But on the other hand, it could automate a lot of neat stuff, like extracting patches. And "out of tree" builds could just be "in tree" builds with a different worktree pointing at the same local git repo.

This is just an idea, but it has a lot of potential I think. I already do it manually when working on packages inside buildroot, so I know it works in essence. And git is so fast that there's basically no overhead.

Have fun,

Avery