Skip to contentSkip to conversation

Add POST_RSYNC hook

PATCH New Thomas De Schampheleire 3 participants 7 messages Nov 7, 2013 – Nov 10, 2013 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 Add POST_RSYNC hook

This patch series adopts the POST_RSYNC hook patch by Tzu-Jung Lee and changes the default rsync exclusion list, as discussed on the Buildroot developer days.
The other rsync-related patch submitted by Tzu-Jung Lee (custom rsync command) should not be needed anymore as its use case is covered by the second patch in this series.

---
 Makefile                                  |  14 ++++++++------
 docs/manual/adding-packages-autotools.txt |   3 +--
 docs/manual/adding-packages-cmake.txt     |   3 +--
 docs/manual/adding-packages-generic.txt   |  39 +--------------------------------------
 docs/manual/adding-packages-hooks.txt     |  61 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 docs/manual/adding-packages.txt           |   2 ++
 package/pkg-generic.mk                    |   4 +++-
 7 files changed, 77 insertions(+), 49 deletions(-)
#1 lore raw

PATCH v2 1/1 infra: Add POST_RSYNC_HOOKS support

One of the use cases is for the 'local packages' to restore the SCM info. Some packages use this information to generate version info during build time. In this case, the local package can have this hook to restore it by symbolic link for example.

---
v2: no changes

 docs/manual/adding-packages-generic.txt | 1 +
 package/pkg-generic.mk                  | 2 ++
 2 files changed, 3 insertions(+)
docs/manual/adding-packages-generic.txt +1 -0
@@ -467,6 +467,7 @@ The following hook points are available:
 
 * +LIBFOO_POST_DOWNLOAD_HOOKS+
 * +LIBFOO_POST_EXTRACT_HOOKS+
+* +LIBFOO_POST_RSYNC_HOOKS+
 * +LIBFOO_PRE_PATCH_HOOKS+
 * +LIBFOO_POST_PATCH_HOOKS+
 * +LIBFOO_PRE_CONFIGURE_HOOKS+
package/pkg-generic.mk +2 -0
@@ -69,6 +69,7 @@ endif
 	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
 	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
 	rsync -au --cvs-exclude --include core $(SRCDIR)/ $(@D)
+	$(foreach hook,$($(PKG)_POST_RSYNC_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Handle the SOURCE_CHECK and SHOW_EXTERNAL_DEPS cases for rsynced
@@ -339,6 +340,7 @@ endif
 # post-steps hooks
 $(2)_POST_DOWNLOAD_HOOKS        ?=
 $(2)_POST_EXTRACT_HOOKS         ?=
+$(2)_POST_RSYNC_HOOKS           ?=
 $(2)_PRE_PATCH_HOOKS            ?=
 $(2)_POST_PATCH_HOOKS           ?=
 $(2)_PRE_CONFIGURE_HOOKS        ?=
#2 lore raw

PATCH v2 1/1 infra: centralize rsync exclude list for VCS files

In reply to Thomas De Schampheleire

Buildroot has three places where rsync is used:
1. to copy the target skeleton
2. to copy the rootfs overlay(s)
3. to copy overridden package sources

In all of these cases, we want to exclude version control files by default. Place 1 and 2 used an identical set of explicit --exclude options, while place 3 used the option --cvs-exclude. This last option, however, not only excludes version control files, but also binary files (.o, .so) and any file or directory named 'core' (a problem for the linux kernel that has several directories with this name). Moreover, the exact list of excluded files when using --cvs-exclude depends on the version of rsync.

This patch creates one global variable RSYNC_VCS_EXCLUSIONS that can be used by the various rsync commands. It excludes the version control files of svn, git, hg, cvs and bzr.

---
v2: no changes

 Makefile               |  14 ++++++++------
 package/pkg-generic.mk |   2 +-
 2 files changed, 9 insertions(+), 7 deletions(-)
Makefile +8 -6
@@ -426,11 +426,14 @@ ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
 TARGET_SKELETON=$(BR2_ROOTFS_SKELETON_CUSTOM_PATH)
 endif
 
+RSYNC_VCS_EXCLUSIONS = \
+	--exclude .svn --exclude .git --exclude .hg --exclude .bzr \
+	--exclude CVS
+
 $(BUILD_DIR)/.root:
 	mkdir -p $(TARGET_DIR)
-	rsync -a \
-		--exclude .empty --exclude .svn --exclude .git \
-		--exclude .hg --exclude=CVS --exclude '*~' \
+	rsync -a $(RSYNC_VCS_EXCLUSIONS) \
+		--exclude .empty --exclude '*~' \
 		$(TARGET_SKELETON)/ $(TARGET_DIR)/
 	cp support/misc/target-dir-warning.txt $(TARGET_DIR_WARNING_FILE)
 	@ln -snf lib $(TARGET_DIR)/$(LIB_SYMLINK)
@@ -504,9 +507,8 @@ endif
 
 	@$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
 		$(call MESSAGE,"Copying overlay $(d)"); \
-		rsync -a \
-			--exclude .empty --exclude .svn --exclude .git \
-			--exclude .hg --exclude=CVS --exclude '*~' \
+		rsync -a $(RSYNC_VCS_EXCLUSIONS) \
+			--exclude .empty --exclude '*~' \
 			$(d)/ $(TARGET_DIR)$(sep))
 
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_BUILD_SCRIPT)), \
package/pkg-generic.mk +1 -1
@@ -68,7 +68,7 @@ endif
 $(BUILD_DIR)/%/.stamp_rsynced:
 	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
 	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
-	rsync -au --cvs-exclude --include core $(SRCDIR)/ $(@D)
+	rsync -au $(RSYNC_VCS_EXCLUSIONS) $(SRCDIR)/ $(@D)
 	$(foreach hook,$($(PKG)_POST_RSYNC_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
#3 lore raw

PATCH v2 1/1 manual: split info on hooks to a separate section/file

In reply to Thomas De Schampheleire

Split out the information on hooks to a separate section (and source file). Not only because the hooks are useful for all infrastructures (and thus don't really fit specifically in the generic infrastructure section), but also for clarity when the info on hooks will be expanded in later patches.

---
v2: no changes

 docs/manual/adding-packages-autotools.txt |   3 +-
 docs/manual/adding-packages-cmake.txt     |   3 +-
 docs/manual/adding-packages-generic.txt   |  40 +-------------------------
 docs/manual/adding-packages-hooks.txt     |  41 +++++++++++++++++++++++++++
 docs/manual/adding-packages.txt           |   2 +
 5 files changed, 46 insertions(+), 43 deletions(-)
docs/manual/adding-packages-autotools.txt +1 -2
@@ -162,8 +162,7 @@ well for most autotools-based packages. 
 still possible to customize what is done in any particular step:
 
 * By adding a post-operation hook (after extract, patch, configure,
-  build or install). See the reference documentation of the generic
-  infrastructure for details.
+  build or install). See xref:hooks[] for details.
 
 * By overriding one of the steps. For example, even if the autotools
   infrastructure is used, if the package +.mk+ file defines its
docs/manual/adding-packages-cmake.txt +1 -2
@@ -135,8 +135,7 @@ for most CMake-based packages. However, 
 possible to customize what is done in any particular step:
 
 * By adding a post-operation hook (after extract, patch, configure,
-  build or install). See the reference documentation of the generic
-  infrastructure for details.
+  build or install). See xref:hooks[] for details.
 
 * By overriding one of the steps. For example, even if the CMake
   infrastructure is used, if the package +.mk+ file defines its own
docs/manual/adding-packages-generic.txt +1 -39
@@ -451,42 +451,4 @@ In the action definitions, you can use t
 * Of course the +$(HOST_DIR)+, +$(STAGING_DIR)+ and +$(TARGET_DIR)+
   variables to install the packages properly.
 
-The last feature of the generic infrastructure is the ability to add
-hooks. These define further actions to perform after existing steps.
-Most hooks aren't really useful for generic packages, since the +.mk+
-file already has full control over the actions performed in each step
-of the package construction. The hooks are more useful for packages
-using the autotools infrastructure described below.  However, since
-they are provided by the generic infrastructure, they are documented
-here. The exception is +LIBFOO_POST_PATCH_HOOKS+.  Patching the
-package and producing legal info are not user definable, so
-+LIBFOO_POST_PATCH_HOOKS+ and +LIBFOO_POST_LEGAL_INFO_HOOKS+ are
-useful for generic packages.
-
-The following hook points are available:
-
-* +LIBFOO_POST_DOWNLOAD_HOOKS+
-* +LIBFOO_POST_EXTRACT_HOOKS+
-* +LIBFOO_POST_RSYNC_HOOKS+
-* +LIBFOO_PRE_PATCH_HOOKS+
-* +LIBFOO_POST_PATCH_HOOKS+
-* +LIBFOO_PRE_CONFIGURE_HOOKS+
-* +LIBFOO_POST_CONFIGURE_HOOKS+
-* +LIBFOO_POST_BUILD_HOOKS+
-* +LIBFOO_POST_INSTALL_HOOKS+ (for host packages only)
-* +LIBFOO_POST_INSTALL_STAGING_HOOKS+ (for target packages only)
-* +LIBFOO_POST_INSTALL_TARGET_HOOKS+ (for target packages only)
-* +LIBFOO_POST_LEGAL_INFO_HOOKS+
-
-These variables are 'lists' of variable names containing actions to be
-performed at this hook point. This allows several hooks to be
-registered at a given hook point. Here is an example:
-
-----------------------
-define LIBFOO_POST_PATCH_FIXUP
-	action1
-	action2
-endef
-
-LIBFOO_POST_PATCH_HOOKS += LIBFOO_POST_PATCH_FIXUP
-----------------------
+Finally, you can also use hooks. See xref:hooks[] for more information.
docs/manual/adding-packages-hooks.txt new file +41 -0

41 lines not on this page · raw message

docs/manual/adding-packages.txt +2 -0
@@ -18,6 +18,8 @@ include::adding-packages-autotools.txt[]
 
 include::adding-packages-cmake.txt[]
 
+include::adding-packages-hooks.txt[]
+
 include::adding-packages-gettext.txt[]
 
 include::adding-packages-tips.txt[]
#4 lore raw

PATCH v2 1/1 manual: add some info on the POST_RSYNC hook

In reply to Thomas De Schampheleire

---
v2: rsync is also used for site method 'local' (comment ThomasP on IRC)

 docs/manual/adding-packages-hooks.txt |  20 ++++++++++++++++++++
 1 files changed, 20 insertions(+), 0 deletions(-)
docs/manual/adding-packages-hooks.txt +20 -0
@@ -39,3 +39,23 @@ endef
 
 LIBFOO_POST_PATCH_HOOKS += LIBFOO_POST_PATCH_FIXUP
 ----------------------
+
+Using the +POST_RSYNC+ hook
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The +POST_RSYNC+ hook is run only for packages that use a local source,
+either through the +local+ site method or the +OVERRIDE_SRCDIR+
+mechanism. In this case, package sources are copied using +rsync+ from
+the local location into the buildroot build directory. The +rsync+
+command does not copy all files from the source directory, though.
+Files belonging to a version control system, like the directories
++.git+, +.hg+, etc. are not copied. For most packages this is
+sufficient, but a given package can perform additional actions using
+the +POST_RSYNC+ hook.
+
+In principle, the hook can contain any command you want. One specific
+use case, though, is the intentional copying of the version control
+directory using +rsync+. The +rsync+ command you use in the hook can, among
+others, use the following variables:
+
+* +$(SRCDIR)+: the path to the overridden source directory
+* +$(@D)+: the path to the build directory
#5 lore raw

PATCH v2 1/1 manual: add some info on the POST_RSYNC hook

2013/11/7 Thomas De Schampheleire <[email protected]>

Quoted 2 lines
Quoted 41 lines

---
v2: rsync is also used for site method 'local' (comment ThomasP on IRC)

 docs/manual/adding-packages-hooks.txt |  20 ++++++++++++++++++++
 1 files changed, 20 insertions(+), 0 deletions(-)
docs/manual/adding-packages-hooks.txt +16 -0
@@ -39,3 +39,23 @@ endef
 
 LIBFOO_POST_PATCH_HOOKS += LIBFOO_POST_PATCH_FIXUP
 ----------------------
+
+Using the +POST_RSYNC+ hook
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The +POST_RSYNC+ hook is run only for packages that use a local source,
+either through the +local+ site method or the +OVERRIDE_SRCDIR+
+mechanism. In this case, package sources are copied using +rsync+ from
+the local location into the buildroot build directory. The +rsync+
+command does not copy all files from the source directory, though.
+Files belonging to a version control system, like the directories
++.git+, +.hg+, etc. are not copied. For most packages this is
+sufficient, but a given package can perform additional actions using
+the +POST_RSYNC+ hook.
+
+In principle, the hook can contain any command you want. One specific
+use case, though, is the intentional copying of the version control
+directory using +rsync+. The +rsync+ command you use in the hook can,

among

docs/manual/adding-packages-hooks.txt +4 -0
+others, use the following variables:
+
+* +$(SRCDIR)+: the path to the overridden source directory
+* +$(@D)+: the path to the build directory

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 4 lines
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131107/ed00a074/attachment.html>
#6 lore raw

PATCH v2 1/1 Add POST_RSYNC hook

In reply to Thomas De Schampheleire
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas De Schampheleire <[email protected]> writes:

This patch series adopts the POST_RSYNC hook patch by Tzu-Jung Lee and changes the default rsync exclusion list, as discussed on the Buildroot developer days.
The other rsync-related patch submitted by Tzu-Jung Lee (custom rsync command) should not be needed anymore as its use case is covered by the second patch in this series.

Committed series, thanks.

Signature · 1 line
Bye, Peter Korsgaard