Skip to contentSkip to conversation

Add POST_RSYNC hook

PATCH Superseded Thomas De Schampheleire 2 participants 6 messages Nov 1, 2013 Kbuild

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Add POST_RSYNC hook

This patch series adopts the POST_RSYNC hook patch by Tzu-Jung Lee and changes the default rsync exclusion list, as discussed on the Buildroot developer days.
The other rsync-related patch submitted by Tzu-Jung Lee (custom rsync command) should not be needed anymore as its use case is covered by the second patch in this series.

---
 Makefile                                  |  14 +++--
 docs/manual/adding-packages-autotools.txt |   3 +-
 docs/manual/adding-packages-cmake.txt     |   3 +-
 docs/manual/adding-packages-generic.txt   |  39 +-----------------
 docs/manual/adding-packages-hooks.txt     |  60 +++++++++++++++++++++++++++
 docs/manual/adding-packages.txt           |   2 +
 package/pkg-generic.mk                    |   4 +-
 7 files changed, 76 insertions(+), 49 deletions(-)
#1 lore raw

PATCH 1/1 infra: Add POST_RSYNC_HOOKS support

One of the use cases is for the 'local packages' to restore the SCM info. Some packages use this information to generate version info during build time. In this case, the local package can have this hook to restore it by symbolic link for example.

---
Include the comments against v1 from Thomas De Schampheleire:

Update manual for the new hook.
Set the default hook to empty

 docs/manual/adding-packages-generic.txt | 1 +
 package/pkg-generic.mk                  | 2 ++
 2 files changed, 3 insertions(+)
docs/manual/adding-packages-generic.txt +1 -0
@@ -467,6 +467,7 @@ The following hook points are available:
 
 * +LIBFOO_POST_DOWNLOAD_HOOKS+
 * +LIBFOO_POST_EXTRACT_HOOKS+
+* +LIBFOO_POST_RSYNC_HOOKS+
 * +LIBFOO_PRE_PATCH_HOOKS+
 * +LIBFOO_POST_PATCH_HOOKS+
 * +LIBFOO_PRE_CONFIGURE_HOOKS+
package/pkg-generic.mk +2 -0
@@ -69,6 +69,7 @@ endif
 	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
 	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
 	rsync -au --cvs-exclude --include core $(SRCDIR)/ $(@D)
+	$(foreach hook,$($(PKG)_POST_RSYNC_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Handle the SOURCE_CHECK and SHOW_EXTERNAL_DEPS cases for rsynced
@@ -339,6 +340,7 @@ endif
 # post-steps hooks
 $(2)_POST_DOWNLOAD_HOOKS        ?=
 $(2)_POST_EXTRACT_HOOKS         ?=
+$(2)_POST_RSYNC_HOOKS           ?=
 $(2)_PRE_PATCH_HOOKS            ?=
 $(2)_POST_PATCH_HOOKS           ?=
 $(2)_PRE_CONFIGURE_HOOKS        ?=
#2 lore raw

PATCH 1/1 infra: centralize rsync exclude list for VCS files

In reply to Thomas De Schampheleire

Buildroot has three places where rsync is used:
1. to copy the target skeleton
2. to copy the rootfs overlay(s)
3. to copy overridden package sources

In all of these cases, we want to exclude version control files by default. Place 1 and 2 used an identical set of explicit --exclude options, while place 3 used the option --cvs-exclude. This last option, however, not only excludes version control files, but also binary files (.o, .so) and any file or directory named 'core' (a problem for the linux kernel that has several directories with this name). Moreover, the exact list of excluded files when using --cvs-exclude depends on the version of rsync.

This patch creates one global variable RSYNC_VCS_EXCLUSIONS that can be used by the various rsync commands. It excludes the version control files of svn, git, hg, cvs and bzr.

---
 Makefile               |  14 ++++++++------
 package/pkg-generic.mk |   2 +-
 2 files changed, 9 insertions(+), 7 deletions(-)
Makefile +8 -6
@@ -426,11 +426,14 @@ ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
 TARGET_SKELETON=$(BR2_ROOTFS_SKELETON_CUSTOM_PATH)
 endif
 
+RSYNC_VCS_EXCLUSIONS = \
+	--exclude .svn --exclude .git --exclude .hg --exclude .bzr \
+	--exclude CVS
+
 $(BUILD_DIR)/.root:
 	mkdir -p $(TARGET_DIR)
-	rsync -a \
-		--exclude .empty --exclude .svn --exclude .git \
-		--exclude .hg --exclude=CVS --exclude '*~' \
+	rsync -a $(RSYNC_VCS_EXCLUSIONS) \
+		--exclude .empty --exclude '*~' \
 		$(TARGET_SKELETON)/ $(TARGET_DIR)/
 	cp support/misc/target-dir-warning.txt $(TARGET_DIR_WARNING_FILE)
 	@ln -snf lib $(TARGET_DIR)/$(LIB_SYMLINK)
@@ -504,9 +507,8 @@ endif
 
 	@$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
 		$(call MESSAGE,"Copying overlay $(d)"); \
-		rsync -a \
-			--exclude .empty --exclude .svn --exclude .git \
-			--exclude .hg --exclude=CVS --exclude '*~' \
+		rsync -a $(RSYNC_VCS_EXCLUSIONS) \
+			--exclude .empty --exclude '*~' \
 			$(d)/ $(TARGET_DIR)$(sep))
 
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_BUILD_SCRIPT)), \
package/pkg-generic.mk +1 -1
@@ -68,7 +68,7 @@ endif
 $(BUILD_DIR)/%/.stamp_rsynced:
 	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
 	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
-	rsync -au --cvs-exclude --include core $(SRCDIR)/ $(@D)
+	rsync -au $(RSYNC_VCS_EXCLUSIONS) $(SRCDIR)/ $(@D)
 	$(foreach hook,$($(PKG)_POST_RSYNC_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
#3 lore raw

PATCH 1/1 manual: split info on hooks to a separate section/file

In reply to Thomas De Schampheleire

Split out the information on hooks to a separate section (and source file). Not only because the hooks are useful for all infrastructures (and thus don't really fit specifically in the generic infrastructure section), but also for clarity when the info on hooks will be expanded in later patches.

---
 docs/manual/adding-packages-autotools.txt |   3 +-
 docs/manual/adding-packages-cmake.txt     |   3 +-
 docs/manual/adding-packages-generic.txt   |  40 +-------------------------
 docs/manual/adding-packages-hooks.txt     |  41 +++++++++++++++++++++++++++
 docs/manual/adding-packages.txt           |   2 +
 5 files changed, 46 insertions(+), 43 deletions(-)
docs/manual/adding-packages-autotools.txt +1 -2
@@ -162,8 +162,7 @@ well for most autotools-based packages. 
 still possible to customize what is done in any particular step:
 
 * By adding a post-operation hook (after extract, patch, configure,
-  build or install). See the reference documentation of the generic
-  infrastructure for details.
+  build or install). See xref:hooks[] for details.
 
 * By overriding one of the steps. For example, even if the autotools
   infrastructure is used, if the package +.mk+ file defines its
docs/manual/adding-packages-cmake.txt +1 -2
@@ -135,8 +135,7 @@ for most CMake-based packages. However, 
 possible to customize what is done in any particular step:
 
 * By adding a post-operation hook (after extract, patch, configure,
-  build or install). See the reference documentation of the generic
-  infrastructure for details.
+  build or install). See xref:hooks[] for details.
 
 * By overriding one of the steps. For example, even if the CMake
   infrastructure is used, if the package +.mk+ file defines its own
docs/manual/adding-packages-generic.txt +1 -39
@@ -451,42 +451,4 @@ In the action definitions, you can use t
 * Of course the +$(HOST_DIR)+, +$(STAGING_DIR)+ and +$(TARGET_DIR)+
   variables to install the packages properly.
 
-The last feature of the generic infrastructure is the ability to add
-hooks. These define further actions to perform after existing steps.
-Most hooks aren't really useful for generic packages, since the +.mk+
-file already has full control over the actions performed in each step
-of the package construction. The hooks are more useful for packages
-using the autotools infrastructure described below.  However, since
-they are provided by the generic infrastructure, they are documented
-here. The exception is +LIBFOO_POST_PATCH_HOOKS+.  Patching the
-package and producing legal info are not user definable, so
-+LIBFOO_POST_PATCH_HOOKS+ and +LIBFOO_POST_LEGAL_INFO_HOOKS+ are
-useful for generic packages.
-
-The following hook points are available:
-
-* +LIBFOO_POST_DOWNLOAD_HOOKS+
-* +LIBFOO_POST_EXTRACT_HOOKS+
-* +LIBFOO_POST_RSYNC_HOOKS+
-* +LIBFOO_PRE_PATCH_HOOKS+
-* +LIBFOO_POST_PATCH_HOOKS+
-* +LIBFOO_PRE_CONFIGURE_HOOKS+
-* +LIBFOO_POST_CONFIGURE_HOOKS+
-* +LIBFOO_POST_BUILD_HOOKS+
-* +LIBFOO_POST_INSTALL_HOOKS+ (for host packages only)
-* +LIBFOO_POST_INSTALL_STAGING_HOOKS+ (for target packages only)
-* +LIBFOO_POST_INSTALL_TARGET_HOOKS+ (for target packages only)
-* +LIBFOO_POST_LEGAL_INFO_HOOKS+
-
-These variables are 'lists' of variable names containing actions to be
-performed at this hook point. This allows several hooks to be
-registered at a given hook point. Here is an example:
-
-----------------------
-define LIBFOO_POST_PATCH_FIXUP
-	action1
-	action2
-endef
-
-LIBFOO_POST_PATCH_HOOKS += LIBFOO_POST_PATCH_FIXUP
-----------------------
+Finally, you can also use hooks. See xref:hooks[] for more information.
docs/manual/adding-packages-hooks.txt new file +41 -0

41 lines not on this page · raw message

docs/manual/adding-packages.txt +2 -0
@@ -18,6 +18,8 @@ include::adding-packages-autotools.txt[]
 
 include::adding-packages-cmake.txt[]
 
+include::adding-packages-hooks.txt[]
+
 include::adding-packages-gettext.txt[]
 
 include::adding-packages-tips.txt[]
#4 lore raw

PATCH 1/1 manual: add some info on the POST_RSYNC hook

In reply to Thomas De Schampheleire
---
 docs/manual/adding-packages-hooks.txt |  19 +++++++++++++++++++
 1 files changed, 19 insertions(+), 0 deletions(-)
docs/manual/adding-packages-hooks.txt +19 -0
@@ -39,3 +39,22 @@ endef
 
 LIBFOO_POST_PATCH_HOOKS += LIBFOO_POST_PATCH_FIXUP
 ----------------------
+
+Using the +POST_RSYNC+ hook
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The +POST_RSYNC+ hook is run only for packages that have a local source
+override, using the +OVERRIDE_SRCDIR+ mechanism. In that case, the
+package sources are copied using +rsync+ from a local location into the
+buildroot build directory. The +rsync+ command does not copy all files
+from the source directory, though. Files belonging to a version control
+system, like the directories +.git+, +.hg+, etc. are not copied. For
+most packages this is sufficient, but a given package can perform
+additional actions using the +POST_RSYNC+ hook.
+
+In principle, the hook can contain any command you want. One specific
+use case, though, is the intentional copying of the version control
+directory using +rsync+. The +rsync+ command you use in the hook can, among
+others, use the following variables:
+
+* +$(SRCDIR)+: the path to the overridden source directory
+* +$(@D)+: the path to the build directory
#5 lore raw

PATCH 1/1 Add POST_RSYNC hook

In reply to Thomas De Schampheleire

Hi Thomas, all,

2013/11/1 Thomas De Schampheleire <[email protected]>

Quoted 20 lines

This patch series adopts the POST_RSYNC hook patch by Tzu-Jung Lee and changes the default rsync exclusion list, as discussed on the Buildroot developer days.
The other rsync-related patch submitted by Tzu-Jung Lee (custom rsync command) should not be needed anymore as its use case is covered by the second patch in this series.

---
 Makefile                                  |  14 +++--
 docs/manual/adding-packages-autotools.txt |   3 +-
 docs/manual/adding-packages-cmake.txt     |   3 +-
 docs/manual/adding-packages-generic.txt   |  39 +-----------------
 docs/manual/adding-packages-hooks.txt     |  60
+++++++++++++++++++++++++++
 docs/manual/adding-packages.txt           |   2 +
 package/pkg-generic.mk                    |   4 +-
 7 files changed, 76 insertions(+), 49 deletions(-)
Quoted 5 lines

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

For the whole series:

Regards,

Signature · 4 lines
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131101/ef0689dd/attachment.html>