Pull request for branch for-2010.11/download-rework
Subsystems (2)
Hello,
Here is a pull request for Maxime's work on cleaning up the download code and adding support for Git and SVN download. After review/testing, I've added my Signed-off-by to those patches.
Some notes :
* 'make source-check' doesn't work, but it didn't work previously either. Doing 'make source-check' does an allyesconfig, which enables all packages. But some packages (e.g Barebox), need a configuration or their Makefile produce an error. This is not a regression introduced by this patchset, and it's something that should be fixed separatly.
* 'make external-deps' used to show only the basename of all the files that are downloaded and not the full URL. This patchset keeps this behaviour for http/ftp downloads, but for SVN/Git downloads it doesn't really make sense, so it shows the full URL. In the future, 'make external-deps' should probably show the full URL for http/ftp downloads as well, but that can be handled as a separate patch.
* 'make external-deps' does not work with external toolchains, but it's not a regression introduced by this patchset, and can therefore be fixed separatly. There are already patches floating around to fix this.
Regards,
Thomas
The following changes since commit 33070528796dafe90ebab836356fea03f2dc24f9:
Marcus Osdoba (1):
usbutils: bump to 0.90
are available in the git repository at:
git://git.busybox.net/~tpetazzoni/git/buildroot for-2010.11/download-rework
Maxime Petazzoni (8):
Remove the spider feature
Cleanup the VCS commands configuration
Update defconfigs with new VCS config options
Implement basic non-wget download methods
Use the normal download method for Tremor
Use the normal download method for libsvgtiny
Re-implement the source-check and external-deps targets Revert "Makefile.package.in: allow packages to override download step"
Config.in | 24 ++---- Makefile | 16 +--- configs/arm_toolchain_defconfig | 5 +- configs/at91rm9200df_defconfig | 5 +- configs/at91rm9200df_ext_bare_defconfig | 5 +- configs/at91rm9200df_ext_defconfig | 5 +- configs/at91sam9260dfc_defconfig | 5 +- configs/at91sam9260dfc_ext_bare_defconfig | 5 +- configs/at91sam9260dfc_ext_defconfig | 5 +- configs/at91sam9261ek_defconfig | 5 +- configs/at91sam9261ek_ext_bare_defconfig | 5 +- configs/at91sam9261ek_ext_defconfig | 5 +- configs/at91sam9263ek_defconfig | 5 +- configs/at91sam9263ek_ext_bare_defconfig | 5 +- configs/at91sam9263ek_ext_defconfig | 5 +- configs/at91sam9g20dfc_defconfig | 5 +- configs/at91sam9g20dfc_ext_bare_defconfig | 5 +- configs/at91sam9g20dfc_ext_defconfig | 5 +- configs/atngw100-base_defconfig | 5 +- configs/atngw100_defconfig | 5 +- configs/atstk1005_defconfig | 5 +- configs/atstk100x_defconfig | 5 +- configs/i386_defconfig | 5 +- configs/i686_defconfig | 5 +- configs/integrator926_defconfig | 5 +- configs/integrator926_huge_defconfig | 5 +- configs/kb9202_defconfig | 5 +- configs/v100sc2_defconfig | 5 +- package/Makefile.package.in | 138 ++++++++++++++++++++++++++--- package/libsvgtiny/libsvgtiny.mk | 11 +-- package/mpfr/mpfr.mk | 2 +- package/multimedia/tremor/tremor.mk | 12 +-- toolchain/wget-show-external-deps.sh | 6 -- 33 files changed, 190 insertions(+), 149 deletions(-) delete mode 100755 toolchain/wget-show-external-deps.sh
Thanks,
Signature · 1 line
Thomas Petazzoni
PATCH 1/8 Remove the spider feature
From: Maxime Petazzoni <[email protected]>
In preparation for the re-work of the DOWNLOAD helper to support multiple download methods and protocols, the spider feature used with wget is removed for now until it is re-implemented on top of the new download methods.
--- Makefile | 7 +++---- package/Makefile.package.in | 10 ---------- package/mpfr/mpfr.mk | 2 -- 3 files changed, 3 insertions(+), 16 deletions(-)
Makefile +3 -4
@@ -244,7 +244,7 @@ ifeq ($(ARCH),xtensa) ARCH:=$(ARCH)_$(call qstrip,$(BR2_xtensa_core_name)) endif -WGET:=$(call qstrip,$(BR2_WGET)) $(SPIDER) $(QUIET) +WGET:=$(call qstrip,$(BR2_WGET)) $(QUIET) SVN_CO:=$(call qstrip,$(BR2_SVN_CO)) $(QUIET) SVN_UP:=$(call qstrip,$(BR2_SVN_UP)) $(QUIET) BZR_CO:=$(call qstrip,$(BR2_BZR_CO)) $(QUIET)
@@ -437,11 +437,10 @@ endif source: $(TARGETS_SOURCE) $(HOST_SOURCE) _source-check: - $(MAKE) $(EXTRAMAKEARGS) SPIDER=--spider source + @echo "TODO $@" external-deps: - @$(MAKE) -Bs BR2_WGET=$(TOPDIR)/toolchain/wget-show-external-deps.sh \ - $(EXTRAMAKEARGS) SPIDER=--spider source + @echo "TODO $@" show-targets: @echo $(TARGETS)
package/Makefile.package.in +0 -10
@@ -81,16 +81,11 @@ TERM_RESET := $(shell tput rmso) # $(call DOWNLOAD,$(FOO_SITE),$(FOO_SOURCE)) ################################################################################ -# support make source-check/external-deps -ifneq ($(SPIDER),) -DOWNLOAD=$(WGET) -P $(DL_DIR) $(1)/$(2) -else define DOWNLOAD $(Q)test -e $(DL_DIR)/$(2) || \ for site in $(call qstrip,$(BR2_PRIMARY_SITE)) $(1) $(call qstrip,$(BR2_BACKUP_SITE)); \ do $(WGET) -P $(DL_DIR) $$site/$(2) && exit; done endef -endif # Utility programs used to build packages TAR ?= tar
@@ -117,19 +112,14 @@ endef # Retrieve the archive $(BUILD_DIR)/%/.stamp_downloaded: -# support make source-check/external-deps -ifeq ($(SPIDER),) # Only show the download message if it isn't already downloaded $(Q)(test -e $(DL_DIR)/$($(PKG)_SOURCE) && \ (test -z $($(PKG)_PATCH) || test -e $(DL_DIR)$($(PKG)_PATCH))) || \ $(call MESSAGE,"Downloading") -endif $(call DOWNLOAD,$($(PKG)_SITE),$($(PKG)_SOURCE)) $(if $($(PKG)_PATCH),$(call DOWNLOAD,$($(PKG)_SITE),$($(PKG)_PATCH))) -ifeq ($(SPIDER),) $(Q)mkdir -p $(@D) $(Q)touch $@ -endif # Unpack the archive $(BUILD_DIR)/%/.stamp_extracted:
package/mpfr/mpfr.mk +0 -2
@@ -21,10 +21,8 @@ MPFR_PATCH_SOURCE:=$(DL_DIR)/$(MPFR_PATCH_FILE) $(MPFR_PATCH_SOURCE): $(call DOWNLOAD,$(MPFR_SITE),$(MPFR_PATCH)) -ifeq ($(SPIDER),) # this breaks source-check/external-deps mv $(DL_DIR)/$(MPFR_PATCH) $@ endif -endif $(DL_DIR)/$(MPFR_SOURCE): $(call DOWNLOAD,$(MPFR_SITE),$(MPFR_SOURCE))
Signature · 1 line
1.7.0.4
PATCH 2/8 Cleanup the VCS commands configuration
In reply to Thomas PetazzoniFrom: Maxime Petazzoni <[email protected]>
It's not really necessary to differenciate the commands for checking out or updating a repository. Only the path to the binary and eventual top-level flags are useful to configure.
--- Config.in | 24 ++++++++---------------- Makefile | 6 ++---- 2 files changed, 10 insertions(+), 20 deletions(-)
Config.in +8 -16
@@ -21,25 +21,17 @@ config BR2_WGET string "Wget command" default "wget --passive-ftp -nd" -config BR2_SVN_CO - string "Subversion (svn) command to download source tree" - default "svn co" +config BR2_SVN + string "Subversion (svn) command" + default "svn" -config BR2_SVN_UP - string "Subversion (svn) command to update source tree" - default "svn up" - -config BR2_BZR_CO - string "Bazaar (bzr) command to download source tree" - default "bzr co" - -config BR2_BZR_UP - string "Bazaar (bzr) command to update source tree" - default "bzr up" +config BR2_BZR + string "Bazaar (bzr) command" + default "bzr" config BR2_GIT - string "Git command to download source tree" - default "git clone" + string "Git command" + default "git" config BR2_ZCAT string "zcat command"
Makefile +2 -4
@@ -245,10 +245,8 @@ ARCH:=$(ARCH)_$(call qstrip,$(BR2_xtensa_core_name)) endif WGET:=$(call qstrip,$(BR2_WGET)) $(QUIET) -SVN_CO:=$(call qstrip,$(BR2_SVN_CO)) $(QUIET) -SVN_UP:=$(call qstrip,$(BR2_SVN_UP)) $(QUIET) -BZR_CO:=$(call qstrip,$(BR2_BZR_CO)) $(QUIET) -BZR_UP:=$(call qstrip,$(BR2_BZR_UP)) $(QUIET) +SVN:=$(call qstrip,$(BR2_SVN)) $(QUIET) +BZR:=$(call qstrip,$(BR2_BZR)) $(QUIET) GIT:=$(call qstrip,$(BR2_GIT)) $(QUIET) ZCAT:=$(call qstrip,$(BR2_ZCAT)) BZCAT:=$(call qstrip,$(BR2_BZCAT))
Signature · 1 line
1.7.0.4
PATCH 3/8 Update defconfigs with new VCS config options
In reply to Thomas PetazzoniFrom: Maxime Petazzoni <[email protected]>
--- configs/arm_toolchain_defconfig | 5 ++--- configs/at91rm9200df_defconfig | 5 ++--- configs/at91rm9200df_ext_bare_defconfig | 5 ++--- configs/at91rm9200df_ext_defconfig | 5 ++--- configs/at91sam9260dfc_defconfig | 5 ++--- configs/at91sam9260dfc_ext_bare_defconfig | 5 ++--- configs/at91sam9260dfc_ext_defconfig | 5 ++--- configs/at91sam9261ek_defconfig | 5 ++--- configs/at91sam9261ek_ext_bare_defconfig | 5 ++--- configs/at91sam9261ek_ext_defconfig | 5 ++--- configs/at91sam9263ek_defconfig | 5 ++--- configs/at91sam9263ek_ext_bare_defconfig | 5 ++--- configs/at91sam9263ek_ext_defconfig | 5 ++--- configs/at91sam9g20dfc_defconfig | 5 ++--- configs/at91sam9g20dfc_ext_bare_defconfig | 5 ++--- configs/at91sam9g20dfc_ext_defconfig | 5 ++--- configs/atngw100-base_defconfig | 5 ++--- configs/atngw100_defconfig | 5 ++--- configs/atstk1005_defconfig | 5 ++--- configs/atstk100x_defconfig | 5 ++--- configs/i386_defconfig | 5 ++--- configs/i686_defconfig | 5 ++--- configs/integrator926_defconfig | 5 ++--- configs/integrator926_huge_defconfig | 5 ++--- configs/kb9202_defconfig | 5 ++--- configs/v100sc2_defconfig | 5 ++--- 26 files changed, 52 insertions(+), 78 deletions(-)
configs/arm_toolchain_defconfig +2 -3
@@ -131,9 +131,8 @@ BR2_AT91_LINUXPATCH_SITE="http://maxim.org.za/AT91RM9200/2.6" # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91rm9200df_defconfig +2 -3
@@ -136,9 +136,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-arm/kernel-patches-$(BR2_KER # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91rm9200df_ext_bare_defconfig +2 -3
@@ -131,9 +131,8 @@ BR2_AT91_LINUXPATCH_SITE="http://maxim.org.za/AT91RM9200/2.6" # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91rm9200df_ext_defconfig +2 -3
@@ -142,9 +142,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-arm/kernel-patches-$(BR2_KER # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9260dfc_defconfig +2 -3
@@ -149,9 +149,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-arm/kernel-patches-$(BR2_KER # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9260dfc_ext_bare_defconfig +2 -3
@@ -131,9 +131,8 @@ BR2_AT91_LINUXPATCH_SITE="http://maxim.org.za/AT91RM9200/2.6" # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9260dfc_ext_defconfig +2 -3
@@ -150,9 +150,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-arm/kernel-patches-$(BR2_KER # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9261ek_defconfig +2 -3
@@ -152,9 +152,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-arm/kernel-patches-$(BR2_KER # Build options # BR2_WGET="wget --passive-ftp" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9261ek_ext_bare_defconfig +2 -3
@@ -134,9 +134,8 @@ BR2_AT91_LINUXPATCH_SITE="http://maxim.org.za/AT91RM9200/2.6" # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9261ek_ext_defconfig +2 -3
@@ -153,9 +153,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-arm/kernel-patches-$(BR2_KER # Build options # BR2_WGET="wget --passive-ftp" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9263ek_defconfig +2 -3
@@ -150,9 +150,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-arm/kernel-patches-$(BR2_KER # Build options # BR2_WGET="wget --passive-ftp" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9263ek_ext_bare_defconfig +2 -3
@@ -133,9 +133,8 @@ BR2_AT91_LINUXPATCH_SITE="http://maxim.org.za/AT91RM9200/2.6" # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9263ek_ext_defconfig +2 -3
@@ -152,9 +152,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-arm/kernel-patches-$(BR2_KER # Build options # BR2_WGET="wget --passive-ftp" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9g20dfc_defconfig +2 -3
@@ -150,9 +150,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-arm/kernel-patches-$(BR2_KER # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9g20dfc_ext_bare_defconfig +2 -3
@@ -131,9 +131,8 @@ BR2_AT91_LINUXPATCH_SITE="http://maxim.org.za/AT91RM9200/2.6" # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/at91sam9g20dfc_ext_defconfig +2 -3
@@ -150,9 +150,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-arm/kernel-patches-$(BR2_KER # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/atngw100-base_defconfig +2 -3
@@ -84,9 +84,8 @@ BR2_TARGET_AVR32_ATNGW100_BASE=y # Build options # BR2_WGET="wget --passive-ftp --retry-connrefused --waitretry=10" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="zcat" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/atngw100_defconfig +2 -3
@@ -69,11 +69,10 @@ BR2_TARGET_AVR32_ATNGW100=y # Commands # BR2_WGET="wget --passive-ftp --retry-connrefused --waitretry=10" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" +BR2_SVN="svn" BR2_BZR_CO="bzr co" BR2_BZR_UP="bzr up" -BR2_GIT="git clone" +BR2_GIT="git" BR2_ZCAT="zcat" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/atstk1005_defconfig +2 -3
@@ -82,9 +82,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-avr32/kernel-patches-$(BR2_K # Build options # BR2_WGET="wget --passive-ftp --retry-connrefused --waitretry=10" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="zcat" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/atstk100x_defconfig +2 -3
@@ -83,9 +83,8 @@ BR2_KERNEL_ARCH_PATCH_DIR="target/device/Atmel/arch-avr32/kernel-patches-$(BR2_K # Build options # BR2_WGET="wget --passive-ftp --retry-connrefused --waitretry=10" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="zcat" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/i386_defconfig +2 -3
@@ -85,9 +85,8 @@ BR2_TARGET_I686=y # Build options # BR2_WGET="wget --passive-ftp" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/i686_defconfig +2 -3
@@ -85,9 +85,8 @@ BR2_TARGET_I686=y # Build options # BR2_WGET="wget --passive-ftp" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/integrator926_defconfig +2 -3
@@ -81,9 +81,8 @@ BR2_BOARD_PATH="target/device/ARMLTD/$(BR2_BOARD_NAME)" # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/integrator926_huge_defconfig +2 -3
@@ -81,9 +81,8 @@ BR2_BOARD_PATH="target/device/ARMLTD/$(BR2_BOARD_NAME)" # Build options # BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/kb9202_defconfig +2 -3
@@ -92,9 +92,8 @@ BR2_TARGET_ARM=y # BR2_PRIMARY_SITE="" BR2_WGET="wget --passive-ftp -nd" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="gzip -d -c" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
configs/v100sc2_defconfig +2 -3
@@ -63,9 +63,8 @@ BR2_TARGET_VALKA_V100SC2=y # BR2_PRIMARY_SITE="" BR2_WGET="wget --passive-ftp --retry-connrefused --waitretry=10" -BR2_SVN_CO="svn co" -BR2_SVN_UP="svn up" -BR2_GIT="git clone" +BR2_SVN="svn" +BR2_GIT="git" BR2_ZCAT="zcat" BR2_BZCAT="bzcat" BR2_TAR_OPTIONS=""
Signature · 1 line
1.7.0.4
PATCH 4/8 Implement basic non-wget download methods
In reply to Thomas PetazzoniFrom: Maxime Petazzoni <[email protected]>
Packages can now be sourced from Git and Subversion repositories. The download method will be autodetected from the URI (git://, svn://, etc). If the repository is accessed through http(s), you can force the download method by setting a _SITE_METHOD variable to either 'git' or 'svn', respectively and without the quotes.
The package's _VERSION variable defines which commit, revision, tag or branch should be checked out. For Git, it can be HEAD, a commit ID, a tag name or branch name (anything that can be checked out with git checkout). For Subversion, it must be a revision number, or HEAD.
--- package/Makefile.package.in | 68 ++++++++++++++++++++++++++++++++++++++++-- 1 files changed, 64 insertions(+), 4 deletions(-)
package/Makefile.package.in +64 -4
@@ -69,6 +69,35 @@ TERM_BOLD := $(shell tput smso) TERM_RESET := $(shell tput rmso) ################################################################################ +# The DOWNLOAD_{GIT,SVN} helpers are in charge of getting a working copy of +# the source repository for their corresponding SCM, checking out the requested +# version / commit / tag, and create an archive out of it. +################################################################################ + +define DOWNLOAD_GIT + pushd $(DL_DIR) > /dev/null && \ + $(GIT) clone $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \ + pushd $($(PKG)_BASE_NAME) > /dev/null && \ + $(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \ + gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \ + popd > /dev/null && \ + rm -rf $($(PKG)_DL_DIR) && \ + popd > /dev/null +endef + +define DOWNLOAD_SVN + pushd $(DL_DIR) > /dev/null && \ + $(SVN) export -r $($(PKG)_DL_VERSION) $($(PKG)_SITE) $($(PKG)_DL_DIR) && \ + $(TAR) czf $($(PKG)_SOURCE) $($(PKG)_BASE_NAME)/ && \ + rm -rf $($(PKG)_DL_DIR) && \ + popd > /dev/null +endef + +define DOWNLOAD_WGET + $(WGET) -P $(DL_DIR) $(call qstrip,$(1))/$(2) +endef + +################################################################################ # DOWNLOAD -- Download helper. Will try to download source from: # 1) BR2_PRIMARY_SITE if enabled # 2) Download site
@@ -83,8 +112,20 @@ TERM_RESET := $(shell tput rmso) define DOWNLOAD $(Q)test -e $(DL_DIR)/$(2) || \ - for site in $(call qstrip,$(BR2_PRIMARY_SITE)) $(1) $(call qstrip,$(BR2_BACKUP_SITE)); \ - do $(WGET) -P $(DL_DIR) $$site/$(2) && exit; done + (if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \ + $(call DOWNLOAD_WGET,$(BR2_PRIMARY_SITE),$(2)) && exit ; \ + fi ; \ + if test -n "$(1)" ; then \ + case "$($(PKG)_SITE_METHOD)" in \ + git) $(DOWNLOAD_GIT) && exit ;; \ + svn) $(DOWNLOAD_SVN) && exit ;; \ + *) $(call DOWNLOAD_WGET,$(1),$(2)) && exit ;; \ + esac ; \ + fi ; \ + if test -n "$(call qstrip,$(BR2_BACKUP_SITE))" ; then \ + $(call DOWNLOAD_WGET,$(BR2_BACKUP_SITE),$(2)) && exit ; \ + fi ; \ + exit 1) endef # Utility programs used to build packages
@@ -244,13 +285,23 @@ ifndef $(2)_VERSION endif endif -$(2)_DIR = $$(BUILD_DIR)/$(1)-$$($(2)_VERSION) +# Keep the package version that may contain forward slashes in the _DL_VERSION +# variable, then replace all forward slashes ('/') by underscores ('_') to +# sanitize the package version that is used in paths, directory and file names. +# Forward slashes may appear in the package's version when pointing to a +# version control system branch or tag, for example remotes/origin/1_10_stable. +$(2)_DL_VERSION = $($(2)_VERSION) +$(2)_VERSION = $(subst /,_,$($(2)_VERSION)) + +$(2)_BASE_NAME = $(1)-$$($(2)_VERSION) +$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_BASE_NAME) +$(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASE_NAME) ifndef $(2)_SOURCE ifdef $(3)_SOURCE $(2)_SOURCE = $($(3)_SOURCE) else - $(2)_SOURCE ?= $(1)-$$($(2)_VERSION).tar.gz + $(2)_SOURCE ?= $$($(2)_BASE_NAME).tar.gz endif endif
@@ -269,6 +320,15 @@ ifndef $(2)_SITE endif endif +ifndef $(2)_SITE_METHOD + ifdef $(3)_SITE_METHOD + $(2)_SITE_METHOD = $($(3)_SITE_METHOD) + else + # Try automatic detection using the scheme part of the URI + $(2)_SITE_METHOD = $(firstword $(subst ://, ,$(call qstrip,$($(2)_SITE)))) + endif +endif + $(2)_DEPENDENCIES ?= $(2)_INSTALL_STAGING ?= NO $(2)_INSTALL_TARGET ?= YES
Signature · 1 line
1.7.0.4
PATCH 5/8 Use the normal download method for Tremor
In reply to Thomas PetazzoniFrom: Maxime Petazzoni <[email protected]>
Now that the generic download method supports Subversion, fall back to the normal usage of AUTOTARGETS for Tremor.
--- package/multimedia/tremor/tremor.mk | 12 ++---------- 1 files changed, 2 insertions(+), 10 deletions(-)
package/multimedia/tremor/tremor.mk +2 -10
@@ -5,19 +5,11 @@ ############################################################ TREMOR_SITE:=http://svn.xiph.org/trunk/Tremor/ +TREMOR_SITE_METHOD:=svn TREMOR_VERSION:=16259 -TREMOR_SVNDIR = Tremor-svn-r$(TREMOR_VERSION) -TREMOR_SOURCE:= $(TREMOR_SVNDIR).tar.bz2 + TREMOR_AUTORECONF = YES TREMOR_INSTALL_STAGING = YES TREMOR_INSTALL_TARGET = YES -$(DL_DIR)/$(TREMOR_SOURCE): - $(SVN_CO) -r $(TREMOR_VERSION) $(TREMOR_SITE) $(BUILD_DIR)/$(TREMOR_SVNDIR) - tar -cv -C $(BUILD_DIR) $(TREMOR_SVNDIR) | bzip2 - -c > $@ - rm -rf $(BUILD_DIR)/$(TREMOR_SVNDIR) - -# use custom download step -TREMOR_TARGET_SOURCE := $(DL_DIR)/$(TREMOR_SOURCE) - $(eval $(call AUTOTARGETS,package/multimedia,tremor))
Signature · 1 line
1.7.0.4
PATCH 6/8 Use the normal download method for libsvgtiny
In reply to Thomas PetazzoniFrom: Maxime Petazzoni <[email protected]>
Now that the generic download method supports Subversion, fall back to the normal usage of AUTOTARGETS for libsvgtiny.
--- package/libsvgtiny/libsvgtiny.mk | 11 +---------- 1 files changed, 1 insertions(+), 10 deletions(-)
package/libsvgtiny/libsvgtiny.mk +1 -10
@@ -6,20 +6,11 @@ LIBSVGTINY_SITE = svn://svn.netsurf-browser.org/trunk/libsvgtiny LIBSVGTINY_VERSION = 9800 -LIBSVGTINY_SVNDIR = libsvgtiny-svn-r$(LIBSVGTINY_VERSION) -LIBSVGTINY_SOURCE = $(LIBSVGTINY_SVNDIR).tar.bz2 + LIBSVGTINY_INSTALL_STAGING = YES LIBSVGTINY_INSTALL_TARGET = YES LIBSVGTINY_DEPENDENCIES = libxml2 host-gperf host-pkg-config -$(DL_DIR)/$(LIBSVGTINY_SOURCE): - $(SVN_CO) -r $(LIBSVGTINY_VERSION) $(LIBSVGTINY_SITE) $(BUILD_DIR)/$(LIBSVGTINY_SVNDIR) - tar -cv -C $(BUILD_DIR) $(LIBSVGTINY_SVNDIR) | bzip2 - -c > $@ - rm -rf $(BUILD_DIR)/$(LIBSVGTINY_SVNDIR) - -# use custom download step -LIBSVGTINY_TARGET_SOURCE := $(DL_DIR)/$(LIBSVGTINY_SOURCE) - define LIBSVGTINY_BUILD_CMDS $(TARGET_CONFIGURE_OPTS) $(MAKE) -C $(@D) PREFIX=/usr endef
Signature · 1 line
1.7.0.4
PATCH 7/8 Re-implement the source-check and external-deps targets
In reply to Thomas PetazzoniFrom: Maxime Petazzoni <[email protected]>
The new DL_MODE variable dispatches between the various download implementations of each method (Git, Subversion, Wget) to deal with the normal download (default mode, 'DOWNLOAD'), the source-check ('SOURCE_CHECK') and to show the external dependencies for external-deps ('SHOW_EXTERNAL_DEPS').
For the latter, the legacy script wget-show-external-deps.sh is no longer required as $(WGET) isn't called directly anymore but always through the DOWNLOAD helper.
--- Makefile | 13 +---- package/Makefile.package.in | 86 ++++++++++++++++++++++++++++----- package/mpfr/mpfr.mk | 2 + toolchain/wget-show-external-deps.sh | 6 -- 4 files changed, 77 insertions(+), 30 deletions(-) delete mode 100755 toolchain/wget-show-external-deps.sh
Makefile +2 -11
@@ -244,19 +244,10 @@ ifeq ($(ARCH),xtensa) ARCH:=$(ARCH)_$(call qstrip,$(BR2_xtensa_core_name)) endif -WGET:=$(call qstrip,$(BR2_WGET)) $(QUIET) -SVN:=$(call qstrip,$(BR2_SVN)) $(QUIET) -BZR:=$(call qstrip,$(BR2_BZR)) $(QUIET) -GIT:=$(call qstrip,$(BR2_GIT)) $(QUIET) ZCAT:=$(call qstrip,$(BR2_ZCAT)) BZCAT:=$(call qstrip,$(BR2_BZCAT)) TAR_OPTIONS=$(call qstrip,$(BR2_TAR_OPTIONS)) -xf -DL_DIR=$(call qstrip,$(BR2_DL_DIR)) -ifeq ($(DL_DIR),) -DL_DIR:=$(TOPDIR)/dl -endif - GNU_TARGET_SUFFIX:=-$(call qstrip,$(BR2_GNU_TARGET_SUFFIX)) STAGING_DIR:=$(call qstrip,$(BR2_STAGING_DIR))
@@ -435,10 +426,10 @@ endif source: $(TARGETS_SOURCE) $(HOST_SOURCE) _source-check: - @echo "TODO $@" + $(MAKE) DL_MODE=SOURCE_CHECK $(EXTRAMAKEARGS) source external-deps: - @echo "TODO $@" + @$(MAKE) -Bs DL_MODE=SHOW_EXTERNAL_DEPS $(EXTRAMAKEARGS) source show-targets: @echo $(TARGETS)
package/Makefile.package.in +73 -13
143 lines not on this page · raw message
package/mpfr/mpfr.mk +2 -0
@@ -21,8 +21,10 @@ MPFR_PATCH_SOURCE:=$(DL_DIR)/$(MPFR_PATCH_FILE) $(MPFR_PATCH_SOURCE): $(call DOWNLOAD,$(MPFR_SITE),$(MPFR_PATCH)) +ifeq ($(DL_MODE),DOWNLOAD) mv $(DL_DIR)/$(MPFR_PATCH) $@ endif +endif $(DL_DIR)/$(MPFR_SOURCE): $(call DOWNLOAD,$(MPFR_SITE),$(MPFR_SOURCE))
toolchain/wget-show-external-deps.sh deleted +0 -6
@@ -1,6 +0,0 @@ -#!/bin/sh -# -# replacement for wget (see BR2_WGET) which simply shows the file name to be -# downloaded. Used by the external-deps make target. - -exec basename ${!#}
Signature · 1 line
1.7.0.4
PATCH 8/8 Revert "Makefile.package.in: allow packages to override download step"
In reply to Thomas PetazzoniFrom: Maxime Petazzoni <[email protected]>
This reverts commit 48cf66f1a2e2c501abeee98b7a2268b0d6d2010d.
--- package/Makefile.package.in | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
package/Makefile.package.in +1 -1
@@ -402,7 +402,7 @@ $(2)_TARGET_BUILD = $$($(2)_DIR)/.stamp_built $(2)_TARGET_CONFIGURE = $$($(2)_DIR)/.stamp_configured $(2)_TARGET_PATCH = $$($(2)_DIR)/.stamp_patched $(2)_TARGET_EXTRACT = $$($(2)_DIR)/.stamp_extracted -$(2)_TARGET_SOURCE ?= $$($(2)_DIR)/.stamp_downloaded +$(2)_TARGET_SOURCE = $$($(2)_DIR)/.stamp_downloaded $(2)_TARGET_UNINSTALL = $$($(2)_DIR)/.stamp_uninstalled $(2)_TARGET_CLEAN = $$($(2)_DIR)/.stamp_cleaned $(2)_TARGET_DIRCLEAN = $$($(2)_DIR)/.stamp_dircleaned
Signature · 1 line
1.7.0.4
PATCH 5/8 Use the normal download method for Tremor
In reply to Thomas PetazzoniHi,
will this new code still try to access a tarball in the dl directory and then try to get a tarball from the primary mirror before it actually tries to get the snapshot from the SVN repository?
IMO it is very important that it continues to work that way. I don't want our nightly builds to pull snapshots from the SVN repository over and over again. This wastes bandwidth, slows down the builds and breaks the build every so often (if the SVN server is unreachable).
Sven
Signature · 5 lines
Sven Neumann Head of Software Development RAUMFELD GmbH | Reichenberger Str. 124 | 10999 Berlin | Germany Tel: +49.30.340.60.98.0 | Fax: +49.30.340.60.98.99 | s.neumann at raumfeld.com
PATCH 5/8 Use the normal download method for Tremor
Hello Sven,
Quoted 3 lines · On Mon, 13 Sep 2010 10:42:31 +0200 Sven Neumann <[email protected]> wrote:
will this new code still try to access a tarball in the dl directory and then try to get a tarball from the primary mirror before it actually tries to get the snapshot from the SVN repository?
Yes, this is exactly what it does. In patch 4/8 :
+define DOWNLOAD_GIT
+ pushd $(DL_DIR) > /dev/null && \
+ $(GIT) clone $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
+ pushd $($(PKG)_BASE_NAME) > /dev/null && \
+ $(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
+ gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
+ popd > /dev/null && \
+ rm -rf $($(PKG)_DL_DIR) && \
+ popd > /dev/null
+endef
Does the git clone and then creates a tarball stored in $(DL_DIR).
And then:
define DOWNLOAD
$(Q)test -e $(DL_DIR)/$(2) || \
- for site in $(call qstrip,$(BR2_PRIMARY_SITE)) $(1) $(call qstrip,$(BR2_BACKUP_SITE)); \
- do $(WGET) -P $(DL_DIR) $$site/$(2) && exit; done
+ (if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
+ $(call DOWNLOAD_WGET,$(BR2_PRIMARY_SITE),$(2)) && exit ; \
+ fi ; \
+ if test -n "$(1)" ; then \
+ case "$($(PKG)_SITE_METHOD)" in \
+ git) $(DOWNLOAD_GIT) && exit ;; \
+ svn) $(DOWNLOAD_SVN) && exit ;; \
+ *) $(call DOWNLOAD_WGET,$(1),$(2)) && exit ;; \
+ esac ; \
+ fi ; \
+ if test -n "$(call qstrip,$(BR2_BACKUP_SITE))" ; then \
+ $(call DOWNLOAD_WGET,$(BR2_BACKUP_SITE),$(2)) && exit ; \
+ fi ; \
+ exit 1)
endef
First checks if BR2_PRIMARY_SITE is set, and if yes, tries to fetch the tarball from here. Only if it's not available from the primary site, we go to actually fetching using git, svn or wget from the project official location.
Does this sounds ok for you ?
Quoted 4 lines
IMO it is very important that it continues to work that way. I don't want our nightly builds to pull snapshots from the SVN repository over and over again. This wastes bandwidth, slows down the builds and breaks the build every so often (if the SVN server is unreachable).
Absolutely, this is why SVN and Git fetch methods have been implemented to be as similar as the existing tarball download method.
Thanks for your feedback,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 5/8 Use the normal download method for Tremor
Ar Luan, 2010-09-13 ag 10:55 +0200, scr?obh Thomas Petazzoni:
Quoted 11 lines
(quoted diff) +10
-0+define DOWNLOAD_GIT + pushd $(DL_DIR) > /dev/null && \ + $(GIT) clone $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \ + pushd $($(PKG)_BASE_NAME) > /dev/null && \ + $(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \ + gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \ + popd > /dev/null && \ + rm -rf $($(PKG)_DL_DIR) && \ + popd > /dev/null +endef
Quoted 1 line
Does the git clone and then creates a tarball stored in $(DL_DIR).
Back a few months ago when Maxime (and Luca Ceresoli) first sent his patches to the list I took them on board and massaged them into something I could use on a daily basis and have had moderate success (I will share if anyone's interested). One of the problems with the above patch is that if the "git archive" is interrupted (e.g. Ctrl-C) a half-finished tarball is left in $(DL_DIR). Off you go again to "make" and DOWNLOAD sees that it has a "dl/package-version.tar.gz" -> I don't need to download that -> I'll extract that to "output/build/..." and then an error occurs when EOF is met unexpectedly.
Anyway, this fixes it: just archive to a tempfile and rename.
+define DOWNLOAD_GIT
+ pushd $(DL_DIR) > /dev/null && \
+ $(GIT) clone $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
+ pushd $($(PKG)_BASE_NAME) > /dev/null && \
+ $(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
+ gzip -c > $(DL_DIR)/$($(PKG)_SOURCE).tmp && \
+ mv $(DL_DIR)/$($(PKG)_SOURCE).tmp $(DL_DIR)/$($(PKG)_SOURCE) && \
+ popd > /dev/null && \
+ rm -rf $($(PKG)_DL_DIR) && \
+ popd > /dev/null
PATCH 5/8 Use the normal download method for Tremor
Hello Quotient,
[ I suppose Quotient isn't your real first name, but anyway ]
Quoted 11 lines · On Mon, 13 Sep 2010 16:45:32 +0100 Quotient Remainder…
Back a few months ago when Maxime (and Luca Ceresoli) first sent his patches to the list I took them on board and massaged them into something I could use on a daily basis and have had moderate success (I will share if anyone's interested). One of the problems with the above patch is that if the "git archive" is interrupted (e.g. Ctrl-C) a half-finished tarball is left in $(DL_DIR). Off you go again to "make" and DOWNLOAD sees that it has a "dl/package-version.tar.gz" -> I don't need to download that -> I'll extract that to "output/build/..." and then an error occurs when EOF is met unexpectedly.
Quoted 1 line
Anyway, this fixes it: just archive to a tempfile and rename.
I totally agree, but the current wget download method has exactly the same bug, so this problem is not a regression introduced by this patch set specifically.
Rather than trying to make this patch set perfect by trying to make it fix all bugs, I'd rather like to see it merged (it adds new features without removing any, and without introducing any regression, or at least, I hope).
The issue you're raising being perfectly valid, I'd like to see it fixed some day. FWIW, I even proposed a fix back in April for exactly this problem on interrupt wget downloads
(http://lists.busybox.net/pipermail/buildroot/2010-April/034014.html),
but that fix got rejected due to implementation concerns. I didn't had the time since then to look into it. Obviously, help is very welcome to propose further patches to fix that issues for all of wget, git and svn download methods.
Regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 5/8 Use the normal download method for Tremor
Quoted 8 lines · Thomas Petazzoni
I totally agree, but the current wget download method has exactly the same bug, so this problem is not a regression introduced by this patch set specifically.
Rather than trying to make this patch set perfect by trying to make it fix all bugs, I'd rather like to see it merged (it adds new features without removing any, and without introducing any regression, or at least, I hope).
I agree with Thomas, I think it's best to integrate this first patchset, and then work on the issues that remain but are not directly introduced by it.
Quoted 8 lines
The issue you're raising being perfectly valid, I'd like to see it fixed some day. FWIW, I even proposed a fix back in April for exactly this problem on interrupt wget downloads
(http://lists.busybox.net/pipermail/buildroot/2010-April/034014.html),
but that fix got rejected due to implementation concerns. I didn't had the time since then to look into it. Obviously, help is very welcome to propose further patches to fix that issues for all of wget, git and svn download methods.
Sure. I think using temporary files is a good solution, as using hashes doesn't play very well with Git/Svn downloads.
- Maxime
Signature · 10 lines
Maxime Petazzoni <http://www.bulix.org> ``One by one, the penguins took away my sanity.'' Linux kernel and software developer at MontaVista Software -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20100914/0f5dcd02/attachment.pgp>
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Thomas> Hello,
Thomas> Here is a pull request for Maxime's work on cleaning up the download Thomas> code and adding support for Git and SVN download. After Thomas> review/testing, I've added my Signed-off-by to those patches.
Pulled and pushed, thanks both of you.
Signature · 1 line
Bye, Peter Korsgaard