Override source directories

Hello,

Here is a set of RFC patches that implement the basic mechanism I have been referring to since a couple of weeks/months about allowing users to override the source directory for packages.

From an user perspective, it allows ones to create a "local.mk" file in the Buildroot main source directory, with variable definitions such as :

ZLIB_OVERRIDE_SRCDIR = /tmp/zlib MYAPP_OVERRIDE_SRCDIR = /home/foobar/project/myapp LIBGLIB2_OVERRIDE_SRCDIR = /home/foobar/git/libglib2

At build time, Buildroot would use those directories at the source for the packages, instead of downloading, extracing and patching them in the usual way.

Internally, this is implemented inside our package infrastructure, so this mechanism should work for all packages that rely on the infrastructure. The GENTARGETS macro makes sure that the package build process bypasses the download, extract and patch step when <pkg>_OVERRIDE_SRCDIR is defined, and instead a symlinking step is added. It's very simple.

This series of patches also add the <pkg>-reconfigure and <pkg>-rebuild targets, which makes it easy for users to restart the package build process at the compilation step or at the configuration step, without having to mess up with stamp file details.

There are still a few issues to solve and discuss, though :

* Is local.mk inside the main source directory the good solution? Should we instead provide a configuration option to set the path to the override file? Or another solution?

* As the mechanism is implemented today, "make clean" removes the symbolic link $(O)/build/<pkg>-<version>, but not the stamp files created inside this source directory. So next time the build is started, things will go wrong. To solve this, the only solution I see is to separate the directory with the stamp files from the source directory, at least in the case of overriden source directories.

* Other things?

Feedback, comments and ideas welcome.

Regards,

Thomas

The following changes since commit 7234c23967fa70b760c64700e56bb1778019fbeb:

CHANGES: update with recent changes (2011-05-18 21:02:30 +0200)

are available in the git repository at:
http://free-electrons.com/~thomas/buildroot.git for-2011.08/sourcedir

Thomas Petazzoni (4):
package: enhance infrastructure to support source dir override
package: remove stamp files in <pkg>-dirclean
package: add <pkg>-rebuild and <pkg>-reconfigure Include a local.mk for package overrides

 Makefile                    |    1 +
 package/Makefile.package.in |   30 ++++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 0 deletions(-)

Thanks,

Thomas Petazzoni

When a variable <pkg>_OVERRIDE_SRCDIR is defined, then Buildroot will no longer try to download, extract and patch the package. It will simply use the value of this variable as the source directory for the package. A symbolic link to this directory will be created, so that for Buildroot, the package sources are still in $(O)/build/pkg-version/.

This can be used to tell Buildroot that the sources for a given package are inside some directory that you control, and which can be versioned in Git/SVN, or handled in whichever way you want.

Note that Buildroot will still create its .stamp_* files and do the build inside this source directory. Will out-of-tree builds would have been possible with AUTOTARGETS, it wouldn't work well in a generic way for GENTARGETS packages. That's the reason why we've choosen to keep an in-tree build solution.

---
 package/Makefile.package.in |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

index bf4e1b4..1afc18d 100644
@@ -250,6 +250,13 @@ $(BUILD_DIR)/%/.stamp_extracted:
 	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
+# Set up symbolic link to the source directory if the
+# <pkg>_OVERRIDE_SRCDIR feature is used.
+$(BUILD_DIR)/%/.stamp_symlinked:
+	@$(call MESSAGE,"Symlinking to source dir")
+	ln -s $(SRCDIR) $(@D)
+	$(Q)touch $@
+
 # Patch
 #
 # The NOHOSTPKG variable is the uppercased package name, without the

@@ -418,6 +425,7 @@ $(2)_TARGET_INSTALL_STAGING =	$$($(2)_DIR)/.stamp_staging_installed
 $(2)_TARGET_INSTALL_HOST =      $$($(2)_DIR)/.stamp_host_installed
 $(2)_TARGET_BUILD =		$$($(2)_DIR)/.stamp_built
 $(2)_TARGET_CONFIGURE =		$$($(2)_DIR)/.stamp_configured
+$(2)_TARGET_SYMLINK =		$$($(2)_DIR)/.stamp_symlinked
 $(2)_TARGET_PATCH =		$$($(2)_DIR)/.stamp_patched
 $(2)_TARGET_EXTRACT =		$$($(2)_DIR)/.stamp_extracted
 $(2)_TARGET_SOURCE =		$$($(2)_DIR)/.stamp_downloaded

@@ -463,8 +471,15 @@ $(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
 $(1)-build:		$(1)-configure \
 			$$($(2)_TARGET_BUILD)
 
+ifeq ($($(2)_OVERRIDE_SRCDIR),)
 $(1)-configure:		$(1)-patch \
 			$$($(2)_TARGET_CONFIGURE)
+else
+$(1)-configure:		$(1)-symlink \
+			$$($(2)_TARGET_CONFIGURE)
+endif
+
+$(1)-symlink:		$$($(2)_TARGET_SYMLINK)
 
 $(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
 

@@ -492,6 +507,7 @@ $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
 $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
 $$($(2)_TARGET_BUILD):			PKG=$(2)
 $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
+$$($(2)_TARGET_SYMLINK):		SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
 $$($(2)_TARGET_PATCH):			PKG=$(2)
 $$($(2)_TARGET_PATCH):			NOHOSTPKG=$(3)
 $$($(2)_TARGET_EXTRACT):		PKG=$(2)

1.7.1

Hello Thomas,

We started using your "source override" patches, but we encountered a problem.
Package dependencies are not build when a package source is overriden. To overcome this problem we made the following change in Makefile.package.in

ifeq ($($(2)_OVERRIDE_SRCDIR),)

$(1)-configure:		$(1)-patch \
			$$($(2)_TARGET_CONFIGURE)
else
$(1)-configure:		$(1)-symlink \
+                      $(1)-depends \
			$$($(2)_TARGET_CONFIGURE)

endif

$(1)-symlink: $$($(2)_TARGET_SYMLINK)

Cheers,
Alper

View this message in context: http://old.nabble.com/-RFC--Override-source-directories-tp31651180p31919307.html
Sent from the Buildroot (busybox) mailing list archive at Nabble.com.

Hello Alper,

Le Fri, 24 Jun 2011 05:04:56 -0700 (PDT), Alper YILDIRIM <[email protected]> a ?crit :

Yes, this is correct. I also found the problem a few days ago, and fixed it locally, but I haven't had the time to post a newer version of the patch set, because I'm fixing other issues with this new feature.

Regards,

Thomas

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

When the package build directory was a real directory, then removing the directory was enough to remove all its stamp files. However, now that the build directory can in fact be a symbolic link to the real source directory, we need to explicitly remove the stamp files before removing the symbolic link. Otherwise, next time the symbolic is re-created, some old stamp files would be found.

Unfortunately, there is still a bug: if one does a global "make clean", the symbolic links are removed since $(BUILD_DIR) is removed, but not the stamp files. Should we create the stamp files outside of the package build directory ?

---
 package/Makefile.package.in |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

index 1afc18d..702395a 100644
@@ -335,6 +335,7 @@ $(BUILD_DIR)/%/.stamp_uninstalled:
 
 # Remove package sources
 $(BUILD_DIR)/%/.stamp_dircleaned:
+	rm -f $(@D)/.stamp_*
 	rm -Rf $(@D)
 
 ################################################################################

1.7.1

We are often asked "how can I restart the build of a package ?" or "how can I restart the build of package from the configure part ?". Obviously, tweaking with stamp files is possible, but not very user friendly.

Therefore this patch adds two new per-package targets: <pkg>-rebuild and <pkg>-reconfigure. They will respectively restart the build/install or the configure/build/install of the package <pkg>.

---
 package/Makefile.package.in |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)

index 702395a..8abfd8b 100644
@@ -501,6 +501,19 @@ $(1)-clean:		$(1)-uninstall \
 
 $(1)-dirclean:		$$($(2)_TARGET_DIRCLEAN)
 
+$(1)-clean-for-rebuild:
+			rm -f $$($(2)_TARGET_BUILD)
+			rm -f $$($(2)_TARGET_INSTALL_STAGING)
+			rm -f $$($(2)_TARGET_INSTALL_TARGET)
+			rm -f $$($(2)_TARGET_INSTALL_HOST)
+
+$(1)-rebuild:		$(1)-clean-for-rebuild $(1)
+
+$(1)-clean-for-reconfigure: $(1)-clean-for-rebuild
+			rm -f $$($(2)_TARGET_CONFIGURE)
+
+$(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
+
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
 $$($(2)_TARGET_INSTALL_TARGET):		PKG=$(2)

1.7.1

The user can now create a local.mk file at the top of the Buildroot source tree to override the source directory for various packages.

We could probably turn this into a configuration option, so that a different override file can be used for different projects using the same Buildroot source tree.

An example local.mk file:

ZLIB_OVERRIDE_SRCDIR = /tmp/zlib

---
 Makefile |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

index 5f61b93..07a51d8 100644
@@ -318,6 +318,7 @@ else ifeq ($(BR2_TOOLCHAIN_CTNG),y)
 include toolchain/toolchain-crosstool-ng.mk
 endif
 
+-include local.mk
 include package/*/*.mk
 
 include boot/common.mk

1.7.1

Hi Thomas,

I do a similar thing but with a python wrapper, so i think it would be great to have this feature directly inside Buildroot.

Very useful for a user point of view :)

The advantage to have this file on top of Buildroot directory is that we can easily find it and made local changes.
But a configuration option could be more comprehensive for end user, and add flexibility.

Maybe i am wrong, but it can be easy to parse the previous local.mk file and call the updated <pkg>-dirclean rules just for local package before make clean call 'rm' ?

I will be happy to see this feature inside Buildroot :)

Regards,

Benoit Mauduit

Hi Thomas,

It is definitely what i want from buildroot. This feature is a must when working with local application repositories.

I prefer having a configuration option for this, it will be more flexible.

I guess, it won't hurt to have an extra stamp directory for locally overridden source directories.

When do you plan to release this feature.
Please don't wait too much.

Cheers,
Alper

Dnia 2011-05-18, ?ro o godzinie 23:48 +0200, Thomas Petazzoni pisze:

Hi Thomas,
[cut]

Any chance you could implement <pkg>-remove target, parallel to <pkg>-rebuild target? Because this is what I really miss in Buildroot. I often find some packages useless, and I have to build from scratch to remove them. I suppose others would also appreciate "remove package" function.

Patryk "LeadMan" Benderz
Linux Registered User #377521
()  ascii ribbon campaign - against html e-mail 
/\  www.asciiribbon.org   - against proprietary attachments

Hello Patryk,

Le Thu, 09 Jun 2011 08:38:46 +0200,
Patryk Benderz <[email protected]> a ?crit :

This is also on the TODO-list, and some experiments have been made to implement this, but it is a very different work than the one I'm currently proposing through this patch set. As Buildroot is designed today, it does not know which package installed which files in TARGET_DIR and STAGING_DIR, which makes it impossible for Buildroot to remove a particular package. The package infrastructure needs to be extended to maintain the list of the files installed by each package (this is relatively simple). But then, when removing a package, you need to ensure that all reverse-dependencies of this package are also removed, or at least recompiled. This is not completely trivial to implement.

Regards,

Thomas

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com