<pkg>-rsync: support user custom cmds
PATCH 1/1 <pkg>-rsync: support user custom cmds
--- package/pkg-generic.mk | 4 ++++ 1 file changed, 4 insertions(+)
package/pkg-generic.mk +4 -0
@@ -309,6 +309,10 @@ $(2)_INSTALL_IMAGES ?= NO $(2)_INSTALL_TARGET ?= YES $(2)_DIR_PREFIX = $(if $(4),$(4),$(TOP_SRCDIR)/package) +# default rsync command +$(2)_RSYNC_CMDS ?= \ + rsync -au --cvs-exclude --include core $(SRCDIR)/ $(@D) + # define sub-target stamps $(2)_TARGET_INSTALL_TARGET = $$($(2)_DIR)/.stamp_target_installed $(2)_TARGET_INSTALL_STAGING = $$($(2)_DIR)/.stamp_staging_installed
Signature · 1 line
1.8.3.2
PATCH 1/1 <pkg>-rsync: support user custom cmds
Hi Tzu-Jung Lee,
Quoted 1 line · Tzu-Jung Lee
Signed-off-by Tzu-Jung Lee <[email protected]>
Could you explain more about the use case of this patch? Why do you want to override the rsync command?
Quoted 9 lines
--- package/pkg-generic.mk | 4 ++++ 1 file changed, 4 insertions(+)package/pkg-generic.mk +0
-0
package/pkg-generic.mk $(2)_INSTALL_IMAGES ?= NO · line 318
Quoted 10 lines
package/pkg-generic.mk +4
-0@@ -309,6 +309,10 @@ $(2)_INSTALL_IMAGES ?= NO $(2)_INSTALL_TARGET ?= YES $(2)_DIR_PREFIX = $(if $(4),$(4),$(TOP_SRCDIR)/package) +# default rsync command +$(2)_RSYNC_CMDS ?= \ + rsync -au --cvs-exclude --include core $(SRCDIR)/ $(@D) + # define sub-target stamps $(2)_TARGET_INSTALL_TARGET = $$($(2)_DIR)/.stamp_target_installed $(2)_TARGET_INSTALL_STAGING = $$($(2)_DIR)/.stamp_staging_installed
I fail to understand how this will work without further changes. Even if a package defines FOO_RSYNC_CMDS, where is it used? I guess the original rsync commands should be replaced by the new variable.
Also here, documentation should be updated to reflect new features.
Best regards,
Thomas
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
This patch allows users to override the options or entire command line of rsync. The default options filters out object or libraries during rsync. For local packages which come with pre-built binaries, the copy in build/ directory will not have complete files as src/
---
Include the comments aginst v1 from Thomas De Schampheleire:
Elaborate the use case of this feature.
Update the document to relect the change.
Restore a missing replacement for the original rsync command line.
docs/manual/adding-packages-generic.txt | 9 +++++++++ package/pkg-generic.mk | 6 +++++- 2 files changed, 14 insertions(+), 1 deletion(-)
docs/manual/adding-packages-generic.txt +9 -0
@@ -358,6 +358,15 @@ LIBFOO_VERSION = 2.32 Now, the variables that define what should be performed at the different steps of the build process. +* +LIBFOO_RSYNC_CMDS+ lists the actions to be performed at the + fetch step for a package to be built from local source. By default, + the local source directory is copied to the build directory by rsync + with default options '-au --cvs-exclude --include core'. This works + fine and is efficient for general cases since it doesn't copy SCM + metadata as well as object files, and libraries. For packages that + come with pre-built binaries, user can override the default options + or entire command line to copy them at this fetch step. + * +LIBFOO_CONFIGURE_CMDS+ lists the actions to be performed to configure the package before its compilation.
package/pkg-generic.mk +5 -1
@@ -62,7 +62,7 @@ $(BUILD_DIR)/%/.stamp_extracted: $(BUILD_DIR)/%/.stamp_rsynced: @$(call MESSAGE,"Syncing from source dir $(SRCDIR)") @test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1) - rsync -au --cvs-exclude --include core $(SRCDIR)/ $(@D) + $($(PKG)_RSYNC_CMDS) $(foreach hook,$($(PKG)_POST_RSYNC_HOOKS),$(call $(hook))$(sep)) $(Q)touch $@
@@ -310,6 +310,10 @@ $(2)_INSTALL_IMAGES ?= NO $(2)_INSTALL_TARGET ?= YES $(2)_DIR_PREFIX = $(if $(4),$(4),$(TOP_SRCDIR)/package) +# default rsync command +$(2)_RSYNC_CMDS ?= \ + rsync -au --cvs-exclude --include core $(SRCDIR)/ $(@D) + # define sub-target stamps $(2)_TARGET_INSTALL_TARGET = $$($(2)_DIR)/.stamp_target_installed $(2)_TARGET_INSTALL_STAGING = $$($(2)_DIR)/.stamp_staging_installed
Signature · 1 line
1.8.3.2
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
Dear Tzu-Jung Lee,
Quoted 4 lines · Tzu-Jung Lee
This patch allows users to override the options or entire command line of rsync. The default options filters out object or libraries during rsync. For local packages which come with pre-built binaries, the copy in build/ directory will not have complete files as src/
Hum, I am not sure this is the right way to fix this. When it was introduced, I thought the --cvs-exclude option was only ignoring things like CVS/ directories, .svn/ directories or .git/ directories. But now that I read the rsync man page, I see that it excludes a huge number of other file name patterns, which I think isn't desirable.
So rather than providing a way for the user to override the rsync command, I'd prefer the rsync command to behave appropriately. I believe it should simply copy /everything/, including version control directories. Yes that's longer, but there's no reasonable generic way to determine which files should be copied and which files should not.
Best regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
Quoted 14 lines · Thomas Petazzoni
Dear Tzu-Jung Lee,
Quoted 4 lines · Tzu-Jung Lee
This patch allows users to override the options or entire command line of rsync. The default options filters out object or libraries during rsync. For local packages which come with pre-built binaries, the copy in build/ directory will not have complete files as src/
Hum, I am not sure this is the right way to fix this. When it was introduced, I thought the --cvs-exclude option was only ignoring things like CVS/ directories, .svn/ directories or .git/ directories. But now that I read the rsync man page, I see that it excludes a huge number of other file name patterns, which I think isn't desirable.
Quoted 5 lines
So rather than providing a way for the user to override the rsync command, I'd prefer the rsync command to behave appropriately. I believe it should simply copy /everything/, including version control directories. Yes that's longer, but there's no reasonable generic way to determine which files should be copied and which files should not.
In my version of 'man rsync', here is the list of excluded patterns when using cvs-exclude:
RCS SCCS CVS CVS.adm RCSLOG cvslog.* tags TAGS .make.state .nse_depinfo *~ #* .#* ,* _$* *$ *.old *.bak *.BAK *.orig *.rej .del-* *.a *.olb *.o *.obj *.so *.exe *.Z *.elc *.ln core .svn/ .git/ .hg/ .bzr/
I can see how excluding *.a *.o *.obj *.so *.exe can be debatable. I would certainly like to keep excluding .svn, .git, .hg, and .bzr. These directories can become very big, and in the general use case, aren't needed.
Frankly, I don't care a lot about the other patterns.
Here are three suggestions:
- follow the proposed patch, keeping the default as it is and providing a way to overrule it per package.
- simplifying the default exclusion list to .svn, .git, .hg and .bzr (but this wouldn't cover the use case of Tzu-Jung Lee)
- make the exclusion list a global config option, defaulting to e.g. .svn, .git, .hg, .bzr and maybe others, but changeable by the user.
Best regards,
Thomas
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
Op 28-jul.-2013 15:15 schreef "Thomas De Schampheleire" < [email protected]> het volgende:
Quoted 38 lines
Quoted 14 lines · Thomas Petazzoni
Dear Tzu-Jung Lee,
Quoted 4 lines · Tzu-Jung Lee
This patch allows users to override the options or entire command line of rsync. The default options filters out object or libraries during rsync. For local packages which come with pre-built binaries, the copy in build/ directory will not have complete files as src/
Hum, I am not sure this is the right way to fix this. When it was introduced, I thought the --cvs-exclude option was only ignoring things like CVS/ directories, .svn/ directories or .git/ directories. But now that I read the rsync man page, I see that it excludes a huge number of other file name patterns, which I think isn't desirable.
Quoted 5 lines
So rather than providing a way for the user to override the rsync command, I'd prefer the rsync command to behave appropriately. I believe it should simply copy /everything/, including version control directories. Yes that's longer, but there's no reasonable generic way to determine which files should be copied and which files should not.
In my version of 'man rsync', here is the list of excluded patterns when using cvs-exclude:
RCS SCCS CVS CVS.adm RCSLOG cvslog.* tags TAGS .make.state .nse_depinfo *~ #* .#* ,* _$* *$ *.old *.bak *.BAK *.orig *.rej .del-* *.a *.olb *.o *.obj *.so *.exe *.Z *.elc *.ln core .svn/ .git/ .hg/ .bzr/
I can see how excluding *.a *.o *.obj *.so *.exe can be debatable. I would certainly like to keep excluding .svn, .git, .hg, and .bzr. These directories can become very big, and in the general use case, aren't needed.
Frankly, I don't care a lot about the other patterns.
Quoted 7 lines
Here are three suggestions:
- follow the proposed patch, keeping the default as it is and providing a way to overrule it per package.
- simplifying the default exclusion list to .svn, .git, .hg and .bzr (but this wouldn't cover the use case of Tzu-Jung Lee)
- make the exclusion list a global config option, defaulting to e.g. .svn, .git, .hg, .bzr and maybe others, but changeable by the user.
Other suggestion:
- add a config option 'don't exclude version control files when rsyncing'
and don't provide further flexibility.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130728/d9f467a1/attachment.html>
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
Quoted 43 lines · Thomas De Schampheleire
Op 28-jul.-2013 15:15 schreef "Thomas De Schampheleire" <[email protected]> het volgende:
Quoted 38 lines
Quoted 14 lines · Thomas Petazzoni
Dear Tzu-Jung Lee,
Quoted 4 lines · Tzu-Jung Lee
This patch allows users to override the options or entire command line of rsync. The default options filters out object or libraries during rsync. For local packages which come with pre-built binaries, the copy in build/ directory will not have complete files as src/
Hum, I am not sure this is the right way to fix this. When it was introduced, I thought the --cvs-exclude option was only ignoring things like CVS/ directories, .svn/ directories or .git/ directories. But now that I read the rsync man page, I see that it excludes a huge number of other file name patterns, which I think isn't desirable.
Quoted 5 lines
So rather than providing a way for the user to override the rsync command, I'd prefer the rsync command to behave appropriately. I believe it should simply copy /everything/, including version control directories. Yes that's longer, but there's no reasonable generic way to determine which files should be copied and which files should not.
In my version of 'man rsync', here is the list of excluded patterns when using cvs-exclude:
RCS SCCS CVS CVS.adm RCSLOG cvslog.* tags TAGS .make.state .nse_depinfo *~ #* .#* ,* _$* *$ *.old *.bak *.BAK *.orig *.rej .del-* *.a *.olb *.o *.obj *.so *.exe *.Z *.elc *.ln core .svn/ .git/ .hg/ .bzr/
I can see how excluding *.a *.o *.obj *.so *.exe can be debatable. I would certainly like to keep excluding .svn, .git, .hg, and .bzr. These directories can become very big, and in the general use case, aren't needed.
Frankly, I don't care a lot about the other patterns.
It actually benefit me a lot since most of my 'local package' are manged by git. And the sync to larger project such as Linux kernel does takes a lot of time.
Indeed some projects need the SCM metadata to generate the version info. For git or recent version of svn, this can be easily solved by providing a POST_RSYNC_HOOKS, which links to the metadata directory.
For those packages that require special treat, they can still override the default
rsync command.
Quoted 7 lines
Quoted 7 lines
Here are three suggestions:
- follow the proposed patch, keeping the default as it is and providing a way to overrule it per package.
- simplifying the default exclusion list to .svn, .git, .hg and .bzr (but this wouldn't cover the use case of Tzu-Jung Lee)
- make the exclusion list a global config option, defaulting to e.g. .svn, .git, .hg, .bzr and maybe others, but changeable by the user.
I'm okay with the default setting being either exclusive or conservative. As long as the HOOKS will be provided, and the default can be OVERRIDEN. For example, if the default fetch step is reverted to 'cp -r', I can still override it for
the large local projects to save the sync time.
Quoted 4 lines
Other suggestion:
- add a config option 'don't exclude version control files when rsyncing' and don't provide further flexibility.
Thanks.
Roy
PATCH v3 1/1 <pkg>-rsync: support user custom cmds
This patch allows users to override the options or entire command line of rsync. The default options filters out object or libraries during rsync. For local packages which come with pre-built binaries, the copy in build/ directory will not have complete files as src/
---
Fix the rsync default command line.
docs/manual/adding-packages-generic.txt | 9 +++++++++ package/pkg-generic.mk | 6 +++++- 2 files changed, 14 insertions(+), 1 deletion(-)
docs/manual/adding-packages-generic.txt +9 -0
@@ -358,6 +358,15 @@ LIBFOO_VERSION = 2.32 Now, the variables that define what should be performed at the different steps of the build process. +* +LIBFOO_RSYNC_CMDS+ lists the actions to be performed at the + fetch step for a package to be built from local source. By default, + the local source directory is copied to the build directory by rsync + with default options '-au --cvs-exclude --include core'. This works + fine and is efficient for general cases since it doesn't copy SCM + metadata as well as object files, and libraries. For packages that + come with pre-built binaries, user can override the default options + or entire command line to copy them at this fetch step. + * +LIBFOO_CONFIGURE_CMDS+ lists the actions to be performed to configure the package before its compilation.
package/pkg-generic.mk +5 -1
@@ -62,7 +62,7 @@ $(BUILD_DIR)/%/.stamp_extracted: $(BUILD_DIR)/%/.stamp_rsynced: @$(call MESSAGE,"Syncing from source dir $(SRCDIR)") @test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1) - rsync -au --cvs-exclude --include core $(SRCDIR)/ $(@D) + $($(PKG)_RSYNC_CMDS) $(foreach hook,$($(PKG)_POST_RSYNC_HOOKS),$(call $(hook))$(sep)) $(Q)touch $@
@@ -310,6 +310,10 @@ $(2)_INSTALL_IMAGES ?= NO $(2)_INSTALL_TARGET ?= YES $(2)_DIR_PREFIX = $(if $(4),$(4),$(TOP_SRCDIR)/package) +# default rsync command +$(2)_RSYNC_CMDS ?= \ + rsync -au --cvs-exclude --include core $$($(2)_OVERRIDE_SRCDIR)/ $$($(2)_DIR) + # define sub-target stamps $(2)_TARGET_INSTALL_TARGET = $$($(2)_DIR)/.stamp_target_installed $(2)_TARGET_INSTALL_STAGING = $$($(2)_DIR)/.stamp_staging_installed
Signature · 1 line
1.8.3.2
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
In reply to Tzu-Jung LeeHi all,
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Here are three suggestions:
- follow the proposed patch, keeping the default as it is and providing a way to overrule it per package.
- simplifying the default exclusion list to .svn, .git, .hg and .bzr (but this wouldn't cover the use case of Tzu-Jung Lee)
- make the exclusion list a global config option, defaulting to e.g. .svn, .git, .hg, .bzr and maybe others, but changeable by the user.
- add a config option 'don't exclude version control files when rsyncing' and don't provide further flexibility.
Quoted 5 lines
I'm okay with the default setting being either exclusive or conservative. As long as the HOOKS will be provided, and the default can be OVERRIDEN. For example, if the default fetch step is reverted to 'cp -r', I can still override it for
the large local projects to save the sync time.
I don't think we should now consider the remote possibility of changing the default fetch step to something else. When ever that happens, we can discuss exceptions or hooks for that.
Rather I think we should focus on the current situation which has some limitations. In my eyes, there are two independent selections:
1. does the user want rsync to copy binary files (like .o, .so etc.)
2. does the user want rsync to copy version control files (like .git, .svn, etc.)
We could make things simple and allow no choice for 1 (always copy binary files).
The second item could be a global config option, which is relatively simple and would cover most use cases IMO. It would not cover the case where package A does want version control files, and package B does not, I'm not sure if we need to consider this...
What are the opinions of other buildroot contributors?
Thanks,
Thomas
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
Quoted 29 lines · Thomas De Schampheleire
Hi all,
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Here are three suggestions:
- follow the proposed patch, keeping the default as it is and providing a way to overrule it per package.
- simplifying the default exclusion list to .svn, .git, .hg and .bzr (but this wouldn't cover the use case of Tzu-Jung Lee)
- make the exclusion list a global config option, defaulting to e.g. .svn, .git, .hg, .bzr and maybe others, but changeable by the user.
- add a config option 'don't exclude version control files when rsyncing' and don't provide further flexibility.Quoted 5 lines
I'm okay with the default setting being either exclusive or conservative. As long as the HOOKS will be provided, and the default can be OVERRIDEN. For example, if the default fetch step is reverted to 'cp -r', I can still override it for
the large local projects to save the sync time.I don't think we should now consider the remote possibility of changing the default fetch step to something else. When ever that happens, we can discuss exceptions or hooks for that.
Rather I think we should focus on the current situation which has some limitations. In my eyes, there are two independent selections:
1. does the user want rsync to copy binary files (like .o, .so etc.)
2. does the user want rsync to copy version control files (like .git, .svn, etc.)
Quoted 8 lines
We could make things simple and allow no choice for 1 (always copy binary files).
The second item could be a global config option, which is relatively simple and would cover most use cases IMO. It would not cover the case where package A does want version control files, and package B does not, I'm not sure if we need to consider this...What are the opinions of other buildroot contributors?
ping?
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
Quoted 38 lines · Thomas De Schampheleire
Quoted 36 lines · Thomas De Schampheleire
Hi all,
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Here are three suggestions:
- follow the proposed patch, keeping the default as it is and providing a way to overrule it per package.
- simplifying the default exclusion list to .svn, .git, .hg and .bzr (but this wouldn't cover the use case of Tzu-Jung Lee)
- make the exclusion list a global config option, defaulting to e.g. .svn, .git, .hg, .bzr and maybe others, but changeable by the user.
- add a config option 'don't exclude version control files when rsyncing' and don't provide further flexibility.Quoted 5 lines
I'm okay with the default setting being either exclusive or conservative. As long as the HOOKS will be provided, and the default can be OVERRIDEN. For example, if the default fetch step is reverted to 'cp -r', I can still override it for
the large local projects to save the sync time.I don't think we should now consider the remote possibility of changing the default fetch step to something else. When ever that happens, we can discuss exceptions or hooks for that.
Rather I think we should focus on the current situation which has some limitations. In my eyes, there are two independent selections:
1. does the user want rsync to copy binary files (like .o, .so etc.)
2. does the user want rsync to copy version control files (like .git, .svn, etc.)We could make things simple and allow no choice for 1 (always copy binary files).
The second item could be a global config option, which is relatively simple and would cover most use cases IMO. It would not cover the case where package A does want version control files, and package B does not, I'm not sure if we need to consider this...
Quoted 4 lines
Quoted 1 line
What are the opinions of other buildroot contributors?
ping?
If we don't allow any configurability, the default behavior must copy everything,
since we don't know which files user would like to include or exclude,
If we're considering efficiency for some cases, especially those have giant VCS database such as linux kernel, I think the combination of the two patches proposed
already simple enough. They don't really add that much of complexity, I think.
1. Support customized RSYNC (this patch)
2. Allow each package add their post-rsync hook if they need.
We can change the default RSYNC command back to plain rsync. or 'cp'. If an user is in favor of current behavior, which exclude VCS database, he can customize his RSYNC command to the one we are using now.
And for those packages that do require VCS to be presented in the build directory,
The 2nd patch allows it to add a post-rsync-hook, which 'link' the VCS directory from the source dir to the build dir.
Roy
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
Quoted 40 lines · Tzu-Jung Lee
Quoted 38 lines · Thomas De Schampheleire
Quoted 36 lines · Thomas De Schampheleire
Hi all,
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Here are three suggestions:
- follow the proposed patch, keeping the default as it is and providing a way to overrule it per package.
- simplifying the default exclusion list to .svn, .git, .hg and .bzr (but this wouldn't cover the use case of Tzu-Jung Lee)
- make the exclusion list a global config option, defaulting to e.g. .svn, .git, .hg, .bzr and maybe others, but changeable by the user.
- add a config option 'don't exclude version control files when rsyncing' and don't provide further flexibility.Quoted 5 lines
I'm okay with the default setting being either exclusive or conservative. As long as the HOOKS will be provided, and the default can be OVERRIDEN. For example, if the default fetch step is reverted to 'cp -r', I can still override it for
the large local projects to save the sync time.I don't think we should now consider the remote possibility of changing the default fetch step to something else. When ever that happens, we can discuss exceptions or hooks for that.
Rather I think we should focus on the current situation which has some limitations. In my eyes, there are two independent selections:
1. does the user want rsync to copy binary files (like .o, .so etc.)
2. does the user want rsync to copy version control files (like .git, .svn, etc.)We could make things simple and allow no choice for 1 (always copy binary files).
The second item could be a global config option, which is relatively simple and would cover most use cases IMO. It would not cover the case where package A does want version control files, and package B does not, I'm not sure if we need to consider this...
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
What are the opinions of other buildroot contributors?
ping?
If we don't allow any configurability, the default behavior must copy everything,
since we don't know which files user would like to include or exclude,
Agreed - *if* we don't allow configurability.
Quoted 6 lines
If we're considering efficiency for some cases, especially those have giant VCS database such as linux kernel, I think the combination of the two patches proposed
already simple enough. They don't really add that much of complexity, I think.1. Support customized RSYNC (this patch)
Like Thomas, I really don't see the point of being able to choose cp
instead of rsync.
Quoted 1 line
2. Allow each package add their post-rsync hook if they need.
Adding the VCS stuff in a post-rsync-hook is really hackish.
Instead, I think the rsync command should use $($(PKG)_RSYNC_OPTS),
which allows the user to specify additional --include options in the override.mk file. The documentation of the override.mk file should also be updated to explain this possibility.
Regards,
Arnout
Quoted 18 lines
We can change the default RSYNC command back to plain rsync. or 'cp'.
If an user is in favor of current behavior, which exclude VCS database, he can customize his RSYNC command to the one we are using now.
And for those packages that do require VCS to be presented in the build directory,
The 2nd patch allows it to add a post-rsync-hook, which 'link' the VCS directory from the source dir to the build dir.Roy
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
Quoted 78 lines · Arnout Vandecappelle
Quoted 52 lines · Tzu-Jung Lee
Quoted 49 lines · Thomas De Schampheleire
Quoted 43 lines · Thomas De Schampheleire
Hi all,
Quoted 13 lines
Quoted 11 lines
Quoted 11 lines
Here are three suggestions:
- follow the proposed patch, keeping the default as it is and providing a way to overrule it per package.
- simplifying the default exclusion list to .svn, .git, .hg and .bzr (but this wouldn't cover the use case of Tzu-Jung Lee)
- make the exclusion list a global config option, defaulting to e.g. .svn, .git, .hg, .bzr and maybe others, but changeable by the user.
- add a config option 'don't exclude version control files when rsyncing'
and don't provide further flexibility.Quoted 7 lines
I'm okay with the default setting being either exclusive or conservative.
As long as the HOOKS will be provided, and the default can be OVERRIDEN.
For example, if the default fetch step is reverted to 'cp -r', I can still override it for
the large local projects to save the sync time.I don't think we should now consider the remote possibility of changing the default fetch step to something else. When ever that happens, we can discuss exceptions or hooks for that.
Rather I think we should focus on the current situation which has some limitations. In my eyes, there are two independent selections:
1. does the user want rsync to copy binary files (like .o, .so etc.)
2. does the user want rsync to copy version control files (like .git, .svn, etc.)We could make things simple and allow no choice for 1 (always copy binary files).
The second item could be a global config option, which is relatively simple and would cover most use cases IMO. It would not cover the case where package A does want version control files, and package B does not, I'm not sure if we need to consider this...What are the opinions of other buildroot contributors?
Quoted 6 lines
Quoted 1 line
ping?
If we don't allow any configurability, the default behavior must copy everything,
since we don't know which files user would like to include or exclude,Agreed - *if* we don't allow configurability.
Quoted 8 lines
If we're considering efficiency for some cases, especially those have giant VCS
database such as linux kernel, I think the combination of the two patches proposed
already simple enough. They don't really add that much of complexity, I think.1. Support customized RSYNC (this patch)
Like Thomas, I really don't see the point of being able to choose cp
instead of rsync.
Quoted 8 lines
Quoted 1 line
2. Allow each package add their post-rsync hook if they need.
Adding the VCS stuff in a post-rsync-hook is really hackish.
Instead, I think the rsync command should use $($(PKG)_RSYNC_OPTS), which
allows the user to specify additional --include options in the override.mk file. The documentation of the override.mk file should also be updated to explain this possibility.
$($(PKG)_RSYNC_OPTS) alone only gives the choice to sync / cp them or not.
In some cases we actually prefer to simply link the VCS stuff instead of copying it
to the build directory, especially if the VCS of the package is very huge.
Not only the time taken by rsyncing them from a possibly networked mounted file system, but also the additional storage space taken by the build. If we are building multiple rootfs images that are built with Linux kernel from local
source. It takes a few GB additional space in the build directories.
So $($(PKG)_RSYNC_OPTS) can be replace the 1) custom RSYNC cmd patch. But we still need rsync-post-hook or other less hacky alternatives.
Thanks.
Roy
PATCH v2 1/1 <pkg>-rsync: support user custom cmds
Quoted 3 lines · Tzu-Jung Lee
Quoted 2 lines · Arnout Vandecappelle
Quoted 1 line · Tzu-Jung Lee
[snip]
Quoted 35 lines
Quoted 13 lines
Quoted 8 lines
If we're considering efficiency for some cases, especially those have giant VCS
database such as linux kernel, I think the combination of the two patches proposed
already simple enough. They don't really add that much of complexity, I think.1. Support customized RSYNC (this patch)
Like Thomas, I really don't see the point of being able to choose cp
instead of rsync.Quoted 8 lines
Quoted 1 line
2. Allow each package add their post-rsync hook if they need.
Adding the VCS stuff in a post-rsync-hook is really hackish.
Instead, I think the rsync command should use $($(PKG)_RSYNC_OPTS), which
allows the user to specify additional --include options in the override.mk file. The documentation of the override.mk file should also be updated to explain this possibility.$($(PKG)_RSYNC_OPTS) alone only gives the choice to sync / cp them or not.
In some cases we actually prefer to simply link the VCS stuff instead of copying it
to the build directory, especially if the VCS of the package is very huge.Not only the time taken by rsyncing them from a possibly networked mounted file system, but also the additional storage space taken by the build. If we are building multiple rootfs images that are built with Linux kernel from local
source. It takes a few GB additional space in the build directories.
Quoted 2 lines
So $($(PKG)_RSYNC_OPTS) can be replace the 1) custom RSYNC cmd patch. But we still need rsync-post-hook or other less hacky alternatives.
Yes, that sounds reasonable.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F