Allow adding per-package overrlice rsync exclusions

PATCH Superseded Adrian Perez de Castro 3 participants 5 messages Nov 7, 2017 – Nov 8, 2017

Share & tools

Adrian Perez de Castro

author

2017-11-07 01:57 UTC #0 lore raw

PATCH 1/1 Allow adding per-package overrlice rsync exclusions

This allows using <PKG>_SRCDIR_OVERRIDE_RSYNC_EXCLUSIONS in local.mk to skip copying parts of source trees unneeded for building. For example, when developing WebKitGTK+, it's handy to skip copying all the tests and other build directories, which are huge:

    WEBKITGTK_OVERRIDE_SRCDIR = /home/aperez/WebKit
    WEBKITGTK_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS = \
        --exclude JSTests --exclude ManualTests \
	--exclude PerformanceTests --exclude WebDriverTests \
	--exclude WebKitBuild --exclude WebKitLibraries \
	--exclude WebKit.xcworkspace --exclude Websites \
	--exclude Examples

This saves a good chunk of time when rsync is used for the first time to copy the source tree over before building.

Signed-off-by Adrian Perez de Castro <[email protected]>

---
 package/pkg-generic.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

package/pkg-generic.mk +1 -1

index 0e28675fbe..c895afc498 100644
@@ -181,7 +181,7 @@ $(BUILD_DIR)/%/.stamp_rsynced:
 	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
 	$(foreach hook,$($(PKG)_PRE_RSYNC_HOOKS),$(call $(hook))$(sep))
 	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
-	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)
+	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)
 	$(foreach hook,$($(PKG)_POST_RSYNC_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@

Signature · 1 line

2.15.0

Arnout Vandecappelle

2017-11-07 20:57 UTC #1 lore raw

PATCH 1/1 Allow adding per-package overrlice rsync exclusions

Hi Adrien,

Quoted 13 lines · Adrian Perez de Castro

This allows using <PKG>_SRCDIR_OVERRIDE_RSYNC_EXCLUSIONS in local.mk to skip copying parts of source trees unneeded for building. For example, when developing WebKitGTK+, it's handy to skip copying all the tests and other build directories, which are huge:
    WEBKITGTK_OVERRIDE_SRCDIR = /home/aperez/WebKit
    WEBKITGTK_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS = \
        --exclude JSTests --exclude ManualTests \
	--exclude PerformanceTests --exclude WebDriverTests \
	--exclude WebKitBuild --exclude WebKitLibraries \
	--exclude WebKit.xcworkspace --exclude Websites \
	--exclude Examples

Quoted 2 lines

This saves a good chunk of time when rsync is used for the first time to copy the source tree over before building.

Looks like an interesting feature. Although, if the out-of-tree build feature
gets merged, it's probably less useful. That is, assuming that webkitgtk supports out-of-tree build, which is not a given...

I could bikeshed a little on the name, but it's actually OK.

Quoted 11 lines

Signed-off-by Adrian Perez de Castro <[email protected]>
---
 package/pkg-generic.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/pkg-generic.mk +0 -0
index 0e28675fbe..c895afc498 100644

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_rsynced: · line 184

Quoted 5 lines

package/pkg-generic.mk +1 -1

@@ -181,7 +181,7 @@ $(BUILD_DIR)/%/.stamp_rsynced:
 	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
 	$(foreach hook,$($(PKG)_PRE_RSYNC_HOOKS),$(call $(hook))$(sep))
 	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
-	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)
+	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)

Would have been nice to split the long line. But even without split:

Reviewed-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

Could you also add this feature to the documentation of _OVERRIDE_SRCDIR in the
manual?

Regards,
Arnout

Quoted 4 lines

package/pkg-generic.mk +0 -0

 	$(foreach hook,$($(PKG)_POST_RSYNC_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Thomas Petazzoni

2017-11-07 21:05 UTC #2 lore raw

PATCH 1/1 Allow adding per-package overrlice rsync exclusions

Hello,

Quoted 8 lines · Arnout Vandecappelle

Quoted 8 lines

    WEBKITGTK_OVERRIDE_SRCDIR = /home/aperez/WebKit
    WEBKITGTK_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS = \
        --exclude JSTests --exclude ManualTests \
	--exclude PerformanceTests --exclude WebDriverTests \
	--exclude WebKitBuild --exclude WebKitLibraries \
	--exclude WebKit.xcworkspace --exclude Websites \
	--exclude Examples

Quoted 6 lines

Quoted 2 lines
This saves a good chunk of time when rsync is used for the first time to copy the source tree over before building.
Looks like an interesting feature. Although, if the out-of-tree build feature
gets merged, it's probably less useful. That is, assuming that webkitgtk supports out-of-tree build, which is not a given...

Also my thinking: per-package out of tree build is going to make this new feature a lot less relevant. That being said, per-package out of tree build is not there yet, and this new feature is just a very simple change, so I believe it's OK to merge it.

Quoted 8 lines

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_rsynced: · line 184
Quoted 1 line
package/pkg-generic.mk +1 -0
+	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)  
Would have been nice to split the long line. But even without split:
Reviewed-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Could you also add this feature to the documentation of _OVERRIDE_SRCDIR in the
manual?

I think we really want the documentation update as part of the patch series :)

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Adrian Perez de Castro

author

2017-11-08 12:46 UTC #3 lore raw

PATCH 1/1 Allow adding per-package overrlice rsync exclusions

Hello,

I have just noticed that there's a typo in the patch subject line (s/overrlice/override/), which I have to fix O:-)

Quoted 18 lines · Thomas Petazzoni

Hello,
Quoted 14 lines · Arnout Vandecappelle
Quoted 10 lines
    WEBKITGTK_OVERRIDE_SRCDIR = /home/aperez/WebKit
    WEBKITGTK_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS = \
        --exclude JSTests --exclude ManualTests \
	--exclude PerformanceTests --exclude WebDriverTests \
	--exclude WebKitBuild --exclude WebKitLibraries \
	--exclude WebKit.xcworkspace --exclude Websites \
	--exclude Examples
This saves a good chunk of time when rsync is used for the first time to copy the source tree over before building.
Looks like an interesting feature. Although, if the out-of-tree build feature
gets merged, it's probably less useful. That is, assuming that webkitgtk supports out-of-tree build, which is not a given...

WebKit supports out-of-tree builds (thanks to CMake, and that our build definitions do not do anything weird in this regard). I know that at least we the WebKitGTK+ developers make out-of-tree builds regularly so it is not likely to break in the future ? and if it did, it would be a bug on our side.

Quoted 4 lines

Also my thinking: per-package out of tree build is going to make this new feature a lot less relevant. That being said, per-package out of tree build is not there yet, and this new feature is just a very simple change, so I believe it's OK to merge it.

How about packages which do not support out-of-tree builds? Wouldn't it still be useful to be to define additional rsync flags for them?

Quoted 11 lines

Quoted 8 lines
package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_rsynced: · line 184
Quoted 1 line
package/pkg-generic.mk +1 -0
+	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)  
Would have been nice to split the long line. But even without split:
Reviewed-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Could you also add this feature to the documentation of _OVERRIDE_SRCDIR in the
manual?
I think we really want the documentation update as part of the patch series :)

Good point, I'll add add the edit for the manual as well and submit a new version of the patch.

Cheers,

--
Adri?n ?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20171108/08ca02ae/attachment.asc>

Thomas Petazzoni

2017-11-08 17:06 UTC #4 lore raw

PATCH 1/1 Allow adding per-package overrlice rsync exclusions

Hello,

Quoted 7 lines · Adrian Perez de Castro

Quoted 4 lines
Also my thinking: per-package out of tree build is going to make this new feature a lot less relevant. That being said, per-package out of tree build is not there yet, and this new feature is just a very simple change, so I believe it's OK to merge it.
How about packages which do not support out-of-tree builds? Wouldn't it still be useful to be to define additional rsync flags for them?

Indeed, packages that don't support out of tree build will continue using rsync, and therefore it makes sense to have this exclusion thing.

Quoted 5 lines

Quoted 2 lines
I think we really want the documentation update as part of the patch series :)
Good point, I'll add add the edit for the manual as well and submit a new version of the patch.

Thanks :)

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com