Implement basic non-wget download methods

Packages can now be sourced from Git and Subversion repositories by setting a _SITE_METHOD variable to 'git' or 'svn', respectively and without the quotes.

The package's _VERSION variable defines which commit, revision, tag or branch should be checked out. For Git, it can be HEAD, a commit ID, a tag name or branch name (anything that can be checked out with git checkout). For Subversion, it must be a revision number, or HEAD.

Signed-off-by Maxime Petazzoni<[email protected]>

---
  Config.in                   |    4 ++
  Makefile                    |    1 +
  package/Makefile.package.in |   70 ++++++++++++++++++++++++++++++++++++++++--
  3 files changed, 71 insertions(+), 4 deletions(-)

Config.in +4 -0

index 6adfe49..0fe4cd1 100644
@@ -41,6 +41,10 @@ config BR2_GIT
  	string "Git command to download source tree"
  	default "git clone"
 
+config BR2_GIT_CO
+	string "Git command to checkout a given commit or tag"
+	default "git checkout"
+
  config BR2_ZCAT
  	string "zcat command"
  	default "gzip -d -c"

Makefile +0 -0

index 2e49a6b..65b43f1 100644

package/Makefile.package.in +18 -0

+# version / commit / tag, and create an archive out of it.
+################################################################################
+
+define VCS_PACK_SOURCE
+	$(TAR) czf $($(PKG)_SOURCE) --exclude-vcs $($(PKG)_BASE_NAME)/&&  \
+	rm -rf $($(PKG)_DL_DIR)
+endef
+
+define DOWNLOAD_GIT
+	pushd $(DL_DIR)>  /dev/null&&  \
+	$(GIT) $($(PKG)_SITE) $($(PKG)_DL_DIR)&&  \
+	pushd $($(PKG)_DL_DIR)>  /dev/null&&  \
+	$(GIT_CO) $($(PKG)_DL_VERSION)&&  \
+	popd>  /dev/null&&  \
+	$(VCS_PACK_SOURCE)&&  \
+	popd>  /dev/null
+endef
+

package/Makefile.package.in +16 -3

+
+################################################################################
  # DOWNLOAD -- Download helper. Will try to download source from:
  # 1) BR2_PRIMARY_SITE if enabled
  # 2) Download site

@@ -83,8 +115,20 @@ TERM_RESET := $(shell tput rmso)
 
  define DOWNLOAD
  	$(Q)test -e $(DL_DIR)/$(2) || \
-	for site in $(call qstrip,$(BR2_PRIMARY_SITE)) $(1) $(call qstrip,$(BR2_BACKUP_SITE)); \
-	do $(WGET) -P $(DL_DIR) $$site/$(2)&&  exit; done
+	(if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
+		$(call DOWNLOAD_WGET,$(BR2_PRIMARY_SITE),$(2))&&  exit ; \
+	fi ; \
+	if test -n "$(1)" ; then \
+		case "$($(PKG)_SITE_METHOD)" in \
+			git) $(DOWNLOAD_GIT)&&  exit ;; \
+			svn) $(DOWNLOAD_SVN)&&  exit ;; \
+			*) $(call DOWNLOAD_WGET,$(1),$(2))&&  exit ;; \
+		esac ; \
+	fi ; \
+	if test -n "$(call qstrip,$(BR2_BACKUP_SITE))" ; then \
+		$(call DOWNLOAD_WGET,$(BR2_BACKUP_SITE),$(2))&&  exit ; \
+	fi ; \
+	exit 1)
  endef
 
  # Utility programs used to build packages

@@ -244,13 +288,23 @@ ifndef $(2)_VERSION
   endif
  endif
 
-$(2)_DIR			=  $$(BUILD_DIR)/$(1)-$$($(2)_VERSION)

package/Makefile.package.in +10 -1

+$(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
 
  ifndef $(2)_SOURCE
   ifdef $(3)_SOURCE
    $(2)_SOURCE = $($(3)_SOURCE)
   else
-  $(2)_SOURCE			?= $(1)-$$($(2)_VERSION).tar.gz
+  $(2)_SOURCE			?= $$($(2)_BASE_NAME).tar.gz
   endif
  endif
 

@@ -269,6 +323,14 @@ ifndef $(2)_SITE
   endif
  endif
 
+ifndef $(2)_SITE_METHOD
+ ifdef $(3)_SITE_METHOD
+  $(2)_SITE_METHOD = $($(3)_SITE_METHOD)
+ else
+	$(2)_SITE_METHOD = wget
+ endif
+endif
+
  $(2)_DEPENDENCIES		?=
  $(2)_INSTALL_STAGING		?= NO
  $(2)_INSTALL_TARGET		?= YES

Makefile SVN_UP:=$(call qstrip,$(BR2_SVN_UP)) $(QUIET) · line 252

Quoted 9 lines

Makefile +1 -0

index 2e49a6b..65b43f1 100644
@@ -249,6 +249,7 @@ SVN_UP:=$(call qstrip,$(BR2_SVN_UP)) $(QUIET)
  BZR_CO:=$(call qstrip,$(BR2_BZR_CO)) $(QUIET)
  BZR_UP:=$(call qstrip,$(BR2_BZR_UP)) $(QUIET)
  GIT:=$(call qstrip,$(BR2_GIT)) $(QUIET)
+GIT_CO:=$(call qstrip,$(BR2_GIT_CO)) $(QUIET)

Now I would rename GIT to GIT_CLONE to make the difference clear.

Makefile SVN_UP:=$(call qstrip,$(BR2_SVN_UP)) $(QUIET) · line 255

Quoted 3 lines

Makefile +2 -0

  ################################################################################
+# The DOWNLOAD_{GIT,SVN,BZR} helpers are in charge of getting a working copy of
+# the source repository for their corresponding SCM, checkouting the requested

checkouting -> checking out

Your overall approach is very clean and easy to extend to other VCSs.

The flip side of this generality is that for each new version that one wants to download it requires a new git clone or svn checkout. This takes much more bandwidth (and disk space) than git pull or svn update. Think about big repos such as Linux.

Quoted 2 lines

+$(2)_BASE_NAME = $(1)-$$($(2)_VERSION)
+$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_BASE_NAME)

I don't like saving in $(DL_DIR). That is a directory that I keep with care so I don't have to re-download everything if I wipe buildroot and start off again, and I also appreciate the chance of having it on a shared storage so that different developers or different hosts save bandwidth and time. So I wouldn't like it to be polluted with repository clones.

IMHO in your proposed infrastructure the clones should go in a place where make clean removes them. After all you never reuse them (you reuse the tarball instead).

Hi,

Quoted 2 lines · Luca Ceresoli

Quoted 1 line

+GIT_CO:=$(call qstrip,$(BR2_GIT_CO)) $(QUIET)

Now I would rename GIT to GIT_CLONE to make the difference clear.

I aimed for the least modifications here. But if the consensus is on having GIT_CLONE and GIT_CO, I'll change it.

Indeed. But as I understand it, versions for packages don't change that often. I agree that when they do, you end up downloading more stuff to clone the full repository again instead of just updating.

But support for git fetch and svn update can easily be added later down the road if this becomes a real and frequent issue.

Quoted 9 lines

Quoted 2 lines

+$(2)_BASE_NAME = $(1)-$$($(2)_VERSION)
+$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_BASE_NAME)

I don't like saving in $(DL_DIR). That is a directory that I keep with care so I don't have to re-download everything if I wipe buildroot and start off again, and I also appreciate the chance of having it on a shared storage so that different developers or different hosts save bandwidth and time. So I wouldn't like it to be polluted with repository clones.

Quoted 3 lines

IMHO in your proposed infrastructure the clones should go in a place where make clean removes them. After all you never reuse them (you reuse the tarball instead).

I think you missed the 'rm' line in VCS_PACK_SOURCE. Working copies are not kept around. I archive them with tar --exclude-vcs, and delete the working copy directory so only the archive remains for the extract phase to use.

So you can still have a shared download directories with the archives you need in there, even for VCS sourced packages, and Buildroot will know where to find them and will use them. For example for Tremor, a tremor-16259.tar.gz archive will be created in DL_DIR, and subsequent builds will use this directly (unless TREMOR_VERSION changed of course).

Quoted 5 lines

Quoted 2 lines

Quoted 1 line

+GIT_CO:=$(call qstrip,$(BR2_GIT_CO)) $(QUIET)

Now I would rename GIT to GIT_CLONE to make the difference clear.

I aimed for the least modifications here. But if the consensus is on having GIT_CLONE and GIT_CO, I'll change it.

In a broader sense what are these configuration options for? Is it to allow adding switches (e.g. git checkout -f) to the commands or is it to allow specifying the path to the executable
(e.g. /usr/bin/svn, /opt/git4.0-pre/bin/git)?
If it's the first case, I'd side with GIT_CLONE and GIT_CO. If it's the second case, I'd leave it as GIT but get rid of the GIT_CO and just put the actual command in Makefile.package.in. Is is likely that they will ever differ from "clone" or "checkout" anyway?

I have to agree with Maxime here, $(DL_DIR) seems the right place for these directories and the creation of the tarball in $(DL_DIR) integrates nicely with the existing infrastructure. I would prefer the use of "git archive"/"svn export" instead of plain "tar" here though. I feel it's better to use the VCS native facility than relying on the installed version of tar to know about the particulars of the VCS metainformation.

In addition, when tar is not used, the name of the cloned directory need not be tied to the version; a prefix can be specified in the archive command or exporting done to a temporary directory. $($(PKG)_BASE_NAME) can just be $($(PKG)_NAME).$($(PKG)_SITE_METHOD) or similar.

The result is that the working copy can be left in $(DL_DIR) so it's available for updating at a later time; I would leave out the "rm" line so re-use is possible. It could be regarded as a kind of hook to help the possible future feature (discussed in the other thread) of using BR as a development environment.

Another feature not present in this is the guessing of the download scheme/protocol from the URI (in Luca's patch). It seems to fit in nicely with this patch without significant modification and adds that extra bit of flexibility while falling back to sensible defaults, in my opinion.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Maxime" == Maxime Petazzoni <[email protected]> writes:

Quoted 8 lines

In a broader sense what are these configuration options for? Is it to allow adding switches (e.g. git checkout -f) to the commands or is it to allow specifying the path to the executable
(e.g. /usr/bin/svn, /opt/git4.0-pre/bin/git)?
If it's the first case, I'd side with GIT_CLONE and GIT_CO. If it's the second case, I'd leave it as GIT but get rid of the GIT_CO and just put the actual command in Makefile.package.in. Is is likely that they will ever differ from "clone" or "checkout" anyway?

Yes, they are really old and not used (besides SVN_CO) - Feel free to clean this up while making these changes.