Add support for local package downloads (file://)
PATCH 1/1 Add support for local package downloads (file://)
This patch series add support for file:// URLs for both the package infrastructure, as for the BR2_PRIMARY_SITE and BR2_BACKUP_SITE options. URLs starting with file:// will be handled by a plain 'cp' instead of wget. A new SITE_METHOD 'file' is added. Documentation is updated as well.
Note that this is mainly useful in situations where developers working on a project are sharing the same filesystem (e.g. disk, NFS, ...), as typically is the case in corporate environments. In that case, one may add one or more local project-specific packages, and even use a central storage location as 'proxy' for package downloads (using BR2_PRIMARY_SITE with a file:// URL).
--- docs/buildroot.html | 29 +++++++++++++++++------------ package/Makefile.package.in | 14 +++++++++++--- 2 files changed, 28 insertions(+), 15 deletions(-)
PATCH 1/1 Add file:// helpers to package download infrastructure
--- package/Makefile.package.in | 22 +++++++++++++++++++--- 1 files changed, 19 insertions(+), 3 deletions(-)
package/Makefile.package.in +19 -3
@@ -89,14 +89,15 @@ # working copy of the source repository for their corresponding SCM, # checking out the requested version / commit / tag, and create an # archive out of it. DOWNLOAD_WGET is the normal wget-based download -# mechanism. +# mechanism. DOWNLOAD_FILE is for packages residing on standard +# filesystem paths (local, nfs, ...). # -# The SOURCE_CHECK_{GIT,SVN,BZR,WGET} helpers are in charge of simply +# The SOURCE_CHECK_{GIT,SVN,BZR,WGET,FILE} helpers are in charge of simply # checking that the source is available for download. This can be used # to make sure one will be able to get all the sources needed for # one's build configuration. # -# The SHOW_EXTERNAL_DEPS_{GIT,SVN,BZR,WGET} helpers simply output to +# The SHOW_EXTERNAL_DEPS_{GIT,SVN,BZR,WGET,FILE} helpers simply output to # the console the names of the files that will be downloaded, or path # and revision of the source repositories, producing a list of all the # "external dependencies" of a given build configuration.
@@ -170,6 +171,20 @@ echo $(2) endef + +define DOWNLOAD_FILE + test -e $(DL_DIR)/$(2) || \ + cp `echo "$(call qstrip,$(1))/$(2)" | sed 's_^.*://__'` $(DL_DIR) +endef + +define SOURCE_CHECK_FILE + test -e `echo "$(call qstrip,$(1))/$(2)" | sed 's_^.*://__'` +endef + +define SHOW_EXTERNAL_DEPS_FILE + echo $(2) +endef + ################################################################################ # DOWNLOAD -- Download helper. Will try to download source from: # 1) BR2_PRIMARY_SITE if enabled
@@ -192,6 +207,7 @@ git) $($(DL_MODE)_GIT) && exit ;; \ svn) $($(DL_MODE)_SVN) && exit ;; \ bzr) $($(DL_MODE)_BZR) && exit ;; \ + file) $($(DL_MODE)_FILE) && exit ;; \ *) $(call $(DL_MODE)_WGET,$(1),$(2)) && exit ;; \ esac ; \ fi ; \
PATCH 1/1 Allow BR2_PRIMARY_SITE and BR2_BACKUP_SITE to be a file:// URL
In reply to Thomas De Schampheleire--- package/Makefile.package.in | 14 +++++++++++--- 1 files changed, 11 insertions(+), 3 deletions(-)
package/Makefile.package.in +11 -3
@@ -198,9 +198,14 @@ # $(call DOWNLOAD,$(FOO_SITE),$(FOO_SOURCE)) ################################################################################ +geturischeme=$(firstword $(subst ://, ,$(call qstrip,$(1)))) + define DOWNLOAD $(Q)if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \ - $(call $(DL_MODE)_WGET,$(BR2_PRIMARY_SITE),$(2)) && exit ; \ + case "$(call geturischeme,$(BR2_PRIMARY_SITE))" in \ + file) $(call $(DL_MODE)_FILE,$(BR2_PRIMARY_SITE),$(2)) && exit ;; \ + *) $(call $(DL_MODE)_WGET,$(BR2_PRIMARY_SITE),$(2)) && exit ;; \ + esac ; \ fi ; \ if test -n "$(1)" ; then \ case "$($(PKG)_SITE_METHOD)" in \
@@ -212,7 +217,10 @@ esac ; \ fi ; \ if test -n "$(call qstrip,$(BR2_BACKUP_SITE))" ; then \ - $(call $(DL_MODE)_WGET,$(BR2_BACKUP_SITE),$(2)) && exit ; \ + case "$(call geturischeme,$(BR2_BACKUP_SITE))" in \ + file) $(call $(DL_MODE)_FILE,$(BR2_BACKUP_SITE),$(2)) && exit ;; \ + *) $(call $(DL_MODE)_WGET,$(BR2_BACKUP_SITE),$(2)) && exit ;; \ + esac ; \ fi ; \ exit 1 endef
@@ -419,7 +427,7 @@ $(2)_SITE_METHOD = $($(3)_SITE_METHOD) else # Try automatic detection using the scheme part of the URI - $(2)_SITE_METHOD = $(firstword $(subst ://, ,$(call qstrip,$($(2)_SITE)))) + $(2)_SITE_METHOD = $(call geturischeme,$($(2)_SITE)) endif endif
PATCH 1/1 Allow BR2_PRIMARY_SITE and BR2_BACKUP_SITE to be a file:// URL
Hello,
Le Tue, 15 Mar 2011 08:27:07 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :
package/Makefile.package.in line 208
Quoted 7 lines
define DOWNLOAD
package/Makefile.package.in +4
-1$(Q)if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \ - $(call $(DL_MODE)_WGET,$(BR2_PRIMARY_SITE),$(2)) && exit ; \ + case "$(call geturischeme,$(BR2_PRIMARY_SITE))" in \ + file) $(call $(DL_MODE)_FILE,$(BR2_PRIMARY_SITE),$(2)) && exit ;; \ + *) $(call $(DL_MODE)_WGET,$(BR2_PRIMARY_SITE),$(2)) && exit ;; \ + esac ; \
Does this bring any additional feature that using the existing BR2_DL_DIR option ?
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 1/1 Allow BR2_PRIMARY_SITE and BR2_BACKUP_SITE to be a file:// URL
Quoted 13 lines · Thomas Petazzoni
Hello,
Le Tue, 15 Mar 2011 08:27:07 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :Quoted 7 lines
?define DOWNLOAD
? ? ? $(Q)if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
- ? ? ? ? ? ? $(call $(DL_MODE)_WGET,$(BR2_PRIMARY_SITE),$(2)) && exit ; \
+ ? ? ? ? ? ? case "$(call geturischeme,$(BR2_PRIMARY_SITE))" in \
+ ? ? ? ? ? ? ? ? ? ? file) $(call $(DL_MODE)_FILE,$(BR2_PRIMARY_SITE),$(2)) && exit ;; \
+ ? ? ? ? ? ? ? ? ? ? *) $(call $(DL_MODE)_WGET,$(BR2_PRIMARY_SITE),$(2)) && exit ;; \
+ ? ? ? ? ? ? esac ; \
Quoted 2 lines
Does this bring any additional feature that using the existing
BR2_DL_DIRoption ?
I don't think that BR2_DL_DIR and BR2_PRIMARY_SITE serve the same purpose. The former specifies where packages should be stored, which implies write permissions. The latter specifies where packages come from, which only implies read permissions. In a project with multiple developers, one wouldn't want the central 'DL_DIR' to become cluttered with packages from each individual developer's experiments.
Moreover, the current uses of DL_DIR expect it to be a locally accessible directory. In order to support scp, we'd need to modify all the existing download rules.
PATCH 1/1 Allow BR2_PRIMARY_SITE and BR2_BACKUP_SITE to be a file:// URL
Le Sun, 24 Jul 2011 13:54:15 +0200,
Thomas De Schampheleire <[email protected]> a ?crit :
Quoted 6 lines
I don't think that
BR2_DL_DIRandBR2_PRIMARY_SITEserve the same purpose. The former specifies where packages should be stored, which implies write permissions. The latter specifies where packages come from, which only implies read permissions. In a project with multiple developers, one wouldn't want the central 'DL_DIR' to become cluttered with packages from each individual developer's experiments.
Sounds like a good and convincing argumentation. I'll try to look at integrating the feature.
Regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 1/1 Update documentation to mention file:// support
In reply to Thomas De Schampheleire--- docs/buildroot.html | 29 +++++++++++++++++------------ 1 files changed, 17 insertions(+), 12 deletions(-)
docs/buildroot.html +17 -12
@@ -1045,8 +1045,8 @@ in the package directory inside Buildroot will be applied to the package after extraction.</li> - <li><code>LIBFOO_SITE</code> may contain the Internet location - of the package. It can either be the HTTP or FTP location of a + <li><code>LIBFOO_SITE</code> may contain the location + of the package. It can either be the HTTP, FTP or filesystem location of a tarball, or the URL of a Git or Subversion repository (see <code>LIBFOO_SITE_METHOD</code> below). If <code>HOST_LIBFOO_SITE</code> is not specified, it
@@ -1055,17 +1055,20 @@ <code>http://$$(BR2_SOURCEFORGE_MIRROR).dl.sourceforge.net/sourceforge/packagename</code>. <br/>Examples:<br/> <code>LIBFOO_SITE=http://www.libfoosoftware.org/libfoo</code><br/> - <code>LIBFOO_SITE=http://svn.xiph.org/trunk/Tremor/</code></li> + <code>LIBFOO_SITE=http://svn.xiph.org/trunk/Tremor/</code><br/> + <code>LIBFOO_SITE=file://project/store/</code></li> <li><code>LIBFOO_SITE_METHOD</code> may contain the method to fetch the package source code. It can either be <code>wget</code> (for normal FTP/HTTP downloads of - tarballs), <code>svn</code>, <code>git</code> or <code>bzr</code>. + tarballs), <code>file</code> (for local filesystem copy), + <code>svn</code>, <code>git</code> or <code>bzr</code>. When not specified, it is guessed from the URL given - in <code>LIBFOO_SITE</code>: <code>svn://</code>, <code>git://</code> - and <code>bzr://</code> URLs will use the <code>svn</code>, - <code>git</code> and <code>bzr</code> methods respectively. All other - URL-types will use the <code>wget</code> method. So for example, in the + in <code>LIBFOO_SITE</code>: <code>svn://</code>, <code>git://</code>, + <code>bzr://</code> and <code>file://</code> URLs will use the + <code>svn</code>, <code>git</code>, <code>bzr</code> and + <code>file</code> methods respectively. All other URL-types will + use the <code>wget</code> method. So for example, in the case of a package whose source code is available through Subversion repository on HTTP, one <i>must</i> specifiy <code>LIBFOO_SITE_METHOD=svn</code>. For <code>svn</code>
@@ -1073,10 +1076,12 @@ checkout/clone of the repository which is then tarballed and stored into the download cache. Next builds will not checkout/clone again, but will use the tarball - directly. When <code>HOST_LIBFOO_SITE_METHOD</code> is not - specified, it defaults to the value - of <code>LIBFOO_SITE_METHOD</code>. See <code>package/multimedia/tremor/</code> - for an example.</li> + directly. Note that the <code>file</code> method is only useful in + projects where all developers have access to the same filesystem + (which can be an NFS share, of course). When + <code>HOST_LIBFOO_SITE_METHOD</code> is not specified, it defaults + to the value of <code>LIBFOO_SITE_METHOD</code>. + See <code>package/multimedia/tremor/</code> for an example.</li> <li><code>LIBFOO_DEPENDENCIES</code> lists the dependencies (in terms of package name) that are required for the current target package to
PATCH 1/1 Add support for local package downloads (file://)
In reply to Thomas De SchampheleireQuoted 4 lines · Thomas De Schampheleire
This patch series add support for file:// URLs for both the package infrastructure, as for the
BR2_PRIMARY_SITEandBR2_BACKUP_SITEoptions. URLs starting with file:// will be handled by a plain 'cp' instead of wget. A newSITE_METHOD'file' is added. Documentation is updated as well.
Hi Thomas,
This looks like a more thorough implementation of what I started and attached to bug 3343.
I'd recommend taking Thomas' patches over mine!
Mine also had starting points for scp and rsync as well.
Quoted 5 lines
Note that this is mainly useful in situations where developers working on a project are sharing the same filesystem (e.g. disk, NFS, ...), as typically is the case in corporate environments. In that case, one may add one or more local project-specific packages, and even use a central storage location as 'proxy' for package downloads (using
BR2_PRIMARY_SITEwith a file:// URL).
This was my motivation too. It also was found to be helpful for quicker prototyping and debugging.
Quoted 6 lines
Signed-off-by Thomas De Schampheleire <[email protected]>--- docs/buildroot.html | 29 +++++++++++++++++------------ package/Makefile.package.in | 14 +++++++++++--- 2 files changed, 28 insertions(+), 15 deletions(-)
slainte mhath, RGB
--
Richard Guy Briggs -- ~\ -- ~\ <hpv.tricolour.net>
<www.TriColour.net> -- \___ o \@ @ Ride yer bike!
Ottawa, ON, CANADA -- Lo_>__M__\\/\%__\\/\%
Vote! -- <greenparty.ca>_____GTVS6#790__(*)__(*)________(*)(*)_________________
PATCH 1/1 Add support for local package downloads (file://)
Hi Richard,
On Tue, Mar 15, 2011 at 3:26 PM, Richard Guy Briggs <[email protected]> wrote:
[...]
Quoted 2 lines
This looks like a more thorough implementation of what I started and attached to bug 3343.
I wasn't aware of this bug before, but I'm glad that I'm not the only one with this feature request.
Quoted 4 lines
I'd recommend taking Thomas' patches over mine!
Mine also had starting points for scp and rsync as well.
Thanks. I like the fact that you envisioned support for extra protocols like scp and rsync. Preferably we'd merge these changes with mine, don't you think? (but we should fix your laziness ;-) )
Best regards,
Thomas
PATCH 1/1 Add support for local package downloads (file://)
Quoted 9 lines · Thomas De Schampheleire
Hi Richard,
On Tue, Mar 15, 2011 at 3:26 PM, Richard Guy Briggs <[email protected]> wrote:
[...]Quoted 2 lines
This looks like a more thorough implementation of what I started and attached to bug 3343.
I wasn't aware of this bug before, but I'm glad that I'm not the only one with this feature request.
Maybe it makes sense to accept yours first just for the file:// protocol and then work on a seperate patch at least for scp and rsync, if not a new bug (enhancement) number altogether.
Quoted 7 lines
Quoted 3 lines
I'd recommend taking Thomas' patches over mine!
Mine also had starting points for scp and rsync as well.
Thanks. I like the fact that you envisioned support for extra protocols like scp and rsync. Preferably we'd merge these changes with mine, don't you think? (but we should fix your laziness ;-) )
I threw my patch out there mainly for discussion purposes.
The file/cp part worked fine for our needs, but the documentation hadn't been done yet.
The scp and rsync bits I had a couple of issues, one being how to verify the existence of the object sought, and how to deal with the multiple potential types of objects those two protocols can manage. It would probably be reasonable to restrict the types of objects they could fetch based on overlap with other protocols such as file:// above. Which is to say, no point in using scp or rsync to get a local file when cp will do so more efficiently with a less complicated procedure.
Quoted 2 lines
Best regards,
Thomas
slainte mhath, RGB
--
Richard Guy Briggs -- ~\ -- ~\ <hpv.tricolour.net>
<www.TriColour.net> -- \___ o \@ @ Ride yer bike!
Ottawa, ON, CANADA -- Lo_>__M__\\/\%__\\/\%
Vote! -- <greenparty.ca>_____GTVS6#790__(*)__(*)________(*)(*)_________________
PATCH 1/1 Add support for local package downloads (file://)
In reply to Thomas De SchampheleireQuoted 13 lines · On Tue, 15 Mar 2011 08:27:05 +0100 Thomas De Schampheleire…
This patch series add support for file:// URLs for both the package infrastructure, as for the
BR2_PRIMARY_SITEandBR2_BACKUP_SITEoptions. URLs starting with file:// will be handled by a plain 'cp' instead of wget. A newSITE_METHOD'file' is added. Documentation is updated as well.Note that this is mainly useful in situations where developers working on a project are sharing the same filesystem (e.g. disk, NFS, ...), as typically is the case in corporate environments. In that case, one may add one or more local project-specific packages, and even use a central storage location as 'proxy' for package downloads (using
BR2_PRIMARY_SITEwith a file:// URL).
Quoted 2 lines
Signed-off-by: Thomas De Schampheleire
<[email protected]>
I agree with the principle, I need to do a bit of testing before acking the patches.
Thanks for this work!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 1/1 Add support for local package downloads (file://)
On Tue, Mar 15, 2011 at 11:54 PM, Thomas Petazzoni <[email protected]> wrote:
[...]
Quoted 3 lines
I agree with the principle, I need to do a bit of testing before acking the patches.
Ok. Would you prefer we merge the rsync and scp changes of Richard first, and resubmit a patch?
Quoted 2 lines
Thanks for this work!
Always glad to contribute!
Best regards,
Thomas
PATCH 1/1 Add support for local package downloads (file://)
In reply to Thomas PetazzoniHi Thomas,
Quoted 16 lines · Thomas Petazzoni
Quoted 13 lines · On Tue, 15 Mar 2011 08:27:05 +0100 Thomas De Schampheleire…
This patch series add support for file:// URLs for both the package infrastructure, as for the
BR2_PRIMARY_SITEandBR2_BACKUP_SITEoptions. URLs starting with file:// will be handled by a plain 'cp' instead of wget. A newSITE_METHOD'file' is added. Documentation is updated as well.Note that this is mainly useful in situations where developers working on a project are sharing the same filesystem (e.g. disk, NFS, ...), as typically is the case in corporate environments. In that case, one may add one or more local project-specific packages, and even use a central storage location as 'proxy' for package downloads (using
BR2_PRIMARY_SITEwith a file:// URL).
Quoted 7 lines
Quoted 2 lines
Signed-off-by: Thomas De Schampheleire
<[email protected]>I agree with the principle, I need to do a bit of testing before acking the patches.
Thanks for this work!
Did you have some time to have a better look at these patches?
Thanks,
Thomas