Skip to contentSkip to conversation

Some download-related changes (branch yem/check-downloads)

PATCH Superseded Yann E. MORIN 2 participants 21 messages Jul 2, 2014

v10 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v10 cover Some download-related changes (branch yem/check-downloads)

Hello All!

This series contains those download-related changes:

- Patches 1..8 : move download helpers to shell scripts, one such script for each download helper

- Patch 9 : don't use DL_DIR as scratchpad for temporary VCS
checkouts, since DL_DIR is a precious location

- Patches 10..11: check hashes of downloaded files, to ensure the files are genuine (with doc)

- Patch 12 : add hashes for ca-certificates

Hashes for more packages (eg. openssh, openssh, dropbear...) can be added at a later step.

Changes v9 -> v10:
- use /bin/bash instead of /bin/sh, since we use bashisms (Peter)
- drop patches already applied (Peter)

Changes v8 -> v9:
- include change by Ryan about svn peg-revision
- rebase on top of master after the crazy $$ springling ;-)

Changes v7 -> v8:
- convert scp and localfiles (Thomas DS, Peter)
- typoes, expand MITM (Thomas DS)
- use a cleaner and saner code path to detect errors in helpers

Changes v6 -> v7:
- use single-line title in manual (Samuel)
- use leftover cruft in bzr helper (Thomas)

Changes v5 -> v6:
- ensure we can run multiple parallel downloads from different build dirs with the same BR2_DL_DIR (Thomas)
- also convert bzr
- fix for stray failed downloads

Changes v4 -> v5:
- update ca-certificates hashes after version bump
- fix detection of comments and empty lines in .hash file
- rebase on top of master

Changes v3 -> v4:
- enhance the manual, typoes (Gustavo, Samuel, Thomas DS)
- remove spurious test bump on package/fis (Samuel)

Changes v2 -> v3:
- avoid partial downloads (Thomas DS)
- store hash-type in .hash file (Gustavo)
- typoes (Samuel, Gustavo)
- add possibility to treat missing hashes as an error

Changes v1 -> v2:
- don't do any behavioural change when switching to shell scripts, just reproduce exactly what was in the Makefile (Luca, Arnout)
- support more than one hash algorithm (Arnout, Gustavo)
- typoes (Luca, Baruch)

Regards,
Yann E. MORIN.

The following changes since commit 2d4fd5d358966b9959a0ea9f93daa3a08c6068e7:

pkg-infra: also set PKGDIR for the download step (2014-07-02 17:16:14 +0200)

are available in the git repository at:

git://gitorious.org/buildroot/buildroot.git yem/check-downloads

for you to fetch changes up to 6c119c59e1be7918d55ec414cec18f32ed7844b6:

package/ca-certificates: add tarball's hashes (2014-07-02 23:04:19 +0200)

----------------------------------------------------------------
Yann E. MORIN (12):
pkg-infra: move the git download helper to a script
pkg-infra: move the svn download helper to a script
pkg-infra: move the cvs download helper to a script
pkg-infra: move the hg download helper to a script
pkg-infra: move the wget download helper to a script
pkg-infra: move the bzr download helper to a script
pkg-infra: move the scp download helper to a script
pkg-infra: move the cp (aka localfiles) download helper to a script
pkg-infra: don't use DL_DIR as scratchpad for temporary downloads
pkg-infra: add possiblity to check downloaded files against known hashes
manual: add documentation about packages' hashes package/ca-certificates: add tarball's hashes

 docs/manual/adding-packages-directory.txt    | 66 +++++++++++++++++++++
 package/ca-certificates/ca-certificates.hash |  3 +
 package/pkg-download.mk                      | 89 ++++++++++++----------------
 support/download/bzr                         | 38 ++++++++++++
 support/download/check-hash                  | 77 ++++++++++++++++++++++++
 support/download/cp                          | 26 ++++++++
 support/download/cvs                         | 47 +++++++++++++++
 support/download/git                         | 56 +++++++++++++++++
 support/download/hg                          | 46 ++++++++++++++
 support/download/scp                         | 28 +++++++++
 support/download/svn                         | 44 ++++++++++++++
 support/download/wget                        | 35 +++++++++++
 12 files changed, 504 insertions(+), 51 deletions(-)
 create mode 100644 package/ca-certificates/ca-certificates.hash
 create mode 100755 support/download/bzr
 create mode 100755 support/download/check-hash
 create mode 100755 support/download/cp
 create mode 100755 support/download/cvs
 create mode 100755 support/download/git
 create mode 100755 support/download/hg
 create mode 100755 support/download/scp
 create mode 100755 support/download/svn
 create mode 100755 support/download/wget
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH v10 1/12 pkg-infra: move the git download helper to a script

The git download helper is getting a bit more complex. Fixing it in the Makefile when it breaks (like the recent breakage with a non-existing sha1-cset) proves to be challenging, to say the least.

Move it into a shell script in support/download/git, which will make it much easier to read, maintain, fix and enhance in the future.

---
Changes v9 -> v10:
- use bash as shell (Peter)

---
 package/pkg-download.mk | 17 +++--------------
 support/download/git    | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+), 14 deletions(-)
 create mode 100755 support/download/git
package/pkg-download.mk +3 -14
index e07fd1b..c848f6a 100644
@@ -12,7 +12,7 @@ WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
 SVN := $(call qstrip,$(BR2_SVN))
 CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
-GIT := $(call qstrip,$(BR2_GIT))
+export GIT := $(call qstrip,$(BR2_GIT))
 HG := $(call qstrip,$(BR2_HG)) $(QUIET)
 SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
 SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
@@ -84,19 +84,8 @@ github = https://github.com/$(1)/$(2)/archive/$(3)
 # problems
 define DOWNLOAD_GIT
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	 ((test "`git ls-remote $($(PKG)_SITE) $($(PKG)_DL_VERSION)`" && \
-	   echo "Doing shallow clone" && \
-	   $(GIT) clone --depth 1 -b $($(PKG)_DL_VERSION) --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME)) || \
-	  (echo "Doing full clone" && \
-	   $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME))) && \
-	pushd $($(PKG)_BASE_NAME) > /dev/null && \
-	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ -o $(DL_DIR)/.$($(PKG)_SOURCE).tmp $($(PKG)_DL_VERSION) && \
-	gzip -c $(DL_DIR)/.$($(PKG)_SOURCE).tmp > $(DL_DIR)/$($(PKG)_SOURCE) && \
-	rm -f $(DL_DIR)/.$($(PKG)_SOURCE).tmp && \
-	popd > /dev/null && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(EXTRA_ENV) support/download/git $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
+					  $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
 endef
 
 # TODO: improve to check that the given PKG_DL_VERSION exists on the remote
support/download/git new file +37 -0
new file mode 100755
index 0000000..4b5971e
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for git
+# Call it with:
+#   $1: git repo
+#   $2: git cset
+#   $3: package's basename (eg. foobar-1.2.3)
+#   $4: output file
+# And this environment:
+#   BR2_DL_DIR: path to Buildroot's download dir
+#   GIT       : the git command to call
+
+repo="${1}"
+cset="${2}"
+basename="${3}"
+output="${4}"
+
+repodir="${BR2_DL_DIR}/${basename}"
+
+if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
+    printf "Doing shallow clone\n"
+    ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${repodir}"
+else
+    printf "Doing full clone\n"
+    ${GIT} clone --bare "${repo}" "${repodir}"
+fi
+
+pushd "${repodir}"
+${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}"
+gzip -c "${output}.tmp" >"${output}"
+rm -f "${output}.tmp"
+popd
+
+rm -rf "${repodir}"
Signature · 1 line
1.9.1
#2 lore raw

PATCH 1/12 pkg-infra: move the git download helper to a script

Quoted 56 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

The git download helper is getting a bit more complex. Fixing it in the Makefile when it breaks (like the recent breakage with a non-existing sha1-cset) proves to be challenging, to say the least.

Move it into a shell script in support/download/git, which will make it much easier to read, maintain, fix and enhance in the future.

---
Changes v9 -> v10:
- use bash as shell (Peter)

---
 package/pkg-download.mk | 17 +++--------------
 support/download/git    | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+), 14 deletions(-)
 create mode 100755 support/download/git
package/pkg-download.mk +1 -14
index e07fd1b..c848f6a 100644
@@ -12,7 +12,7 @@ WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
 SVN := $(call qstrip,$(BR2_SVN))
 CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
-GIT := $(call qstrip,$(BR2_GIT))
+export GIT := $(call qstrip,$(BR2_GIT))
 HG := $(call qstrip,$(BR2_HG)) $(QUIET)
 SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
 SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
@@ -84,19 +84,8 @@ github = https://github.com/$(1)/$(2)/archive/$(3)
 # problems
 define DOWNLOAD_GIT
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	 ((test "`git ls-remote $($(PKG)_SITE) $($(PKG)_DL_VERSION)`" && \
-	   echo "Doing shallow clone" && \
-	   $(GIT) clone --depth 1 -b $($(PKG)_DL_VERSION) --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME)) || \
-	  (echo "Doing full clone" && \
-	   $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME))) && \
-	pushd $($(PKG)_BASE_NAME) > /dev/null && \
-	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ -o $(DL_DIR)/.$($(PKG)_SOURCE).tmp $($(PKG)_DL_VERSION) && \
-	gzip -c $(DL_DIR)/.$($(PKG)_SOURCE).tmp > $(DL_DIR)/$($(PKG)_SOURCE) && \
-	rm -f $(DL_DIR)/.$($(PKG)_SOURCE).tmp && \
-	popd > /dev/null && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)

[snip]

package/pkg-download.mk

Quoted 5 lines
package/pkg-download.mk +5 -0
+pushd "${repodir}"
+${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}"
+gzip -c "${output}.tmp" >"${output}"
+rm -f "${output}.tmp"
+popd

Purely cosmetical, but the pushd / popd calls are mising >/dev/null, so it is quite verbose:

make libuci-source

Quoted 1 line
Quoted 1 line
Quoted 1 line

libuci ed938cc8e423d4a33b8f31e6e6d1eb0805ae3d10 Downloading

Doing full clone
Cloning into bare repository '/var/lib/downloads/libuci-ed938cc8e423d4a33b8f31e6e6d1eb0805ae3d10'...
remote: Counting objects: 1826, done.
remote: Compressing objects: 100% (619/619), done.
remote: Total 1826 (delta 1205), reused 1798 (delta 1188)
Receiving objects: 100% (1826/1826), 302.48 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1205/1205), done.
Checking connectivity... done. /var/lib/downloads/libuci-ed938cc8e423d4a33b8f31e6e6d1eb0805ae3d10 ~/source/buildroot ~/source/buildroot

Committed with that fixed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#3 lore raw

PATCH v10 2/12 pkg-infra: move the svn download helper to a script

In reply to Yann E. MORIN

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

---
 package/pkg-download.mk |  9 +++------
 support/download/svn    | 25 +++++++++++++++++++++++++
 2 files changed, 28 insertions(+), 6 deletions(-)
 create mode 100755 support/download/svn
package/pkg-download.mk +3 -6
index c848f6a..e823f68 100644
@@ -9,7 +9,7 @@
 
 # Download method commands
 WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
-SVN := $(call qstrip,$(BR2_SVN))
+export SVN := $(call qstrip,$(BR2_SVN))
 CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
@@ -133,11 +133,8 @@ endef
 
 define DOWNLOAD_SVN
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	$(SVN) export $($(PKG)_SITE)@$($(PKG)_DL_VERSION) $($(PKG)_DL_DIR) && \
-	$(TAR) czf $($(PKG)_SOURCE) $($(PKG)_BASE_NAME)/ && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(EXTRA_ENV) support/download/svn $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
+					  $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
 endef
 
 define SOURCE_CHECK_SVN
support/download/svn new file +25 -0
new file mode 100755
index 0000000..0abcc86
@@ -0,0 +1,25 @@
+#!/bin/bash
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for svn
+# Call it with:
+#   $1: svn repo
+#   $2: svn revision
+#   $3: package's basename (eg. foobar-1.2.3)
+#   $4: output file
+# And this environment:
+#   SVN       : the svn command to call
+#   BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+rev="${2}"
+basename="${3}"
+output="${4}"
+
+pushd "${BR2_DL_DIR}"
+${SVN} export "${repo}@${rev}" "${basename}"
+tar czf "${output}" "${basename}"
+rm -rf "${basename}"
+popd
Signature · 1 line
1.9.1
#4 lore raw

PATCH 2/12 pkg-infra: move the svn download helper to a script

Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

Committed with >/dev/null added to pushd/popd and Ryans' reviewed-by added, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#5 lore raw

PATCH v10 3/12 pkg-infra: move the cvs download helper to a script

In reply to Yann E. MORIN

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

---
 package/pkg-download.mk | 11 ++++-------
 support/download/cvs    | 27 +++++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 7 deletions(-)
 create mode 100755 support/download/cvs
package/pkg-download.mk +4 -7
index e823f68..9f3afac 100644
@@ -10,7 +10,7 @@
 # Download method commands
 WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
 export SVN := $(call qstrip,$(BR2_SVN))
-CVS := $(call qstrip,$(BR2_CVS))
+export CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
 HG := $(call qstrip,$(BR2_HG)) $(QUIET)
@@ -114,12 +114,9 @@ endef
 
 define DOWNLOAD_CVS
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	$(CVS) -z3 -d:pserver:anonymous@$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
-		co -d $($(PKG)_BASE_NAME) -r :$($(PKG)_DL_VERSION) -P $($(PKG)_RAWNAME) && \
-	$(TAR) czf $($(PKG)_SOURCE) $($(PKG)_BASE_NAME)/ && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(EXTRA_ENV) support/download/cvs $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
+					  $($(PKG)_DL_VERSION) $($(PKG)_RAWNAME) \
+					  $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
 endef
 
 # Not all CVS servers support ls/rls, use login to see if we can connect
support/download/cvs new file +27 -0
new file mode 100755
index 0000000..60787a6
@@ -0,0 +1,27 @@
+#!/bin/bash
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for cvs
+# Call it with:
+#   $1: cvs repo
+#   $2: cvs revision
+#   $3: package's name (eg. foobar)
+#   $4: package's basename (eg. foobar-1.2.3)
+#   $5: output file
+# And this environment:
+#   CVS       : the cvs command to call
+#   BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+rev="${2}"
+rawname="${3}"
+basename="${4}"
+output="${5}"
+
+cd "${BR2_DL_DIR}"
+${CVS} -z3 -d":pserver:anonymous@${repo}" \
+       co -d "${basename}" -r ":${rev}" -P "${rawname}"
+tar czf "${output}" "${basename}"
+rm -rf "${basename}"
Signature · 1 line
1.9.1
#6 lore raw

PATCH 3/12 pkg-infra: move the cvs download helper to a script

Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#7 lore raw

PATCH v10 4/12 pkg-infra: move the hg download helper to a script

In reply to Yann E. MORIN

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

[tested a particular scenario that used to fail, when the 'hg archive' step is interrupted, now working fine]

---
Changes v9 -> v10:
- use bash as shell (Peter)

---
 package/pkg-download.mk | 11 +++--------
 support/download/hg     | 25 +++++++++++++++++++++++++
 2 files changed, 28 insertions(+), 8 deletions(-)
 create mode 100755 support/download/hg
package/pkg-download.mk +3 -8
index 9f3afac..3290b6b 100644
@@ -13,7 +13,7 @@ export SVN := $(call qstrip,$(BR2_SVN))
 export CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
-HG := $(call qstrip,$(BR2_HG)) $(QUIET)
+export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
 SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
 SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
 LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
@@ -161,13 +161,8 @@ endef
 
 define DOWNLOAD_HG
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	rm -rf $($(PKG)_BASE_NAME) && \
-	$(HG) clone --noupdate --rev $($(PKG)_DL_VERSION) $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
-	$(HG) archive --repository $($(PKG)_BASE_NAME) --type tgz --prefix $($(PKG)_BASE_NAME)/ \
-	              --rev $($(PKG)_DL_VERSION) $(DL_DIR)/$($(PKG)_SOURCE) && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(EXTRA_ENV) support/download/hg $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
+					  $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
 endef
 
 # TODO: improve to check that the given PKG_DL_VERSION exists on the remote
support/download/hg new file +25 -0
new file mode 100755
index 0000000..d2e69c3
@@ -0,0 +1,25 @@
+#!/bin/bash
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for hg
+# Call it with:
+#   $1: hg repo
+#   $2: hg cset
+#   $3: package's basename (eg. foobar-1.2.3)
+#   $4: output file
+# And this environment:
+#   HG        : the hg command to call
+#   BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+cset="${2}"
+basename="${3}"
+output="${4}"
+
+cd "${BR2_DL_DIR}"
+${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
+${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \
+              --rev "${cset}" "${output}"
+rm -rf "${basename}"
Signature · 1 line
1.9.1
#8 lore raw

PATCH 4/12 pkg-infra: move the hg download helper to a script

Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

Quoted 6 lines

[tested a particular scenario that used to fail, when the 'hg archive' step is interrupted, now working fine]

---
Changes v9 -> v10:
- use bash as shell (Peter)

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#9 lore raw

PATCH v10 5/12 pkg-infra: move the wget download helper to a script

In reply to Yann E. MORIN

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

---
 package/pkg-download.mk | 13 +++----------
 support/download/wget   | 21 +++++++++++++++++++++
 2 files changed, 24 insertions(+), 10 deletions(-)
 create mode 100755 support/download/wget
package/pkg-download.mk +3 -10
index 3290b6b..7554582 100644
@@ -8,7 +8,7 @@
 ################################################################################
 
 # Download method commands
-WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
+export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
 export SVN := $(call qstrip,$(BR2_SVN))
 export CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
@@ -175,17 +175,10 @@ define SHOW_EXTERNAL_DEPS_HG
   echo $($(PKG)_SOURCE)
 endef
 
-# Download a file using wget. Only download the file if it doesn't
-# already exist in the download directory. If the download fails,
-# remove the file (because wget -O creates a 0-byte file even if the
-# download fails).  To handle an interrupted download as well, download
-# to a temporary file first.  The temporary file will be overwritten
-# the next time the download is tried.
+
 define DOWNLOAD_WGET
 	test -e $(DL_DIR)/$(2) || \
-	($(WGET) -O $(DL_DIR)/$(2).tmp '$(call qstrip,$(1))' && \
-	 mv $(DL_DIR)/$(2).tmp $(DL_DIR)/$(2)) || \
-	(rm -f $(DL_DIR)/$(2).tmp ; exit 1)
+	$(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2)
 endef
 
 define SOURCE_CHECK_WGET
support/download/wget new file +21 -0
new file mode 100755
index 0000000..91ffd10
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for wget
+# Call it with:
+#   $1: URL
+#   $2: output file
+# And this environment:
+#   WGET      : the wget command to call
+
+url="${1}"
+output="${2}"
+
+if ${WGET} -O "${output}.tmp" "${url}"; then
+    mv "${output}.tmp" "${output}"
+else
+    rm -f "${output}.tmp"
+    exit 1
+fi
Signature · 1 line
1.9.1
#10 lore raw

PATCH 5/12 pkg-infra: move the wget download helper to a script

Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#11 lore raw

PATCH v10 6/12 pkg-infra: move the bzr download helper to a script

In reply to Yann E. MORIN

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

---
 package/pkg-download.mk |  4 ++--
 support/download/bzr    | 19 +++++++++++++++++++
 2 files changed, 21 insertions(+), 2 deletions(-)
 create mode 100755 support/download/bzr
package/pkg-download.mk +2 -2
index 7554582..e420b05 100644
@@ -11,7 +11,7 @@
 export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
 export SVN := $(call qstrip,$(BR2_SVN))
 export CVS := $(call qstrip,$(BR2_CVS))
-BZR := $(call qstrip,$(BR2_BZR))
+export BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
 export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
 SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
@@ -101,7 +101,7 @@ endef
 
 define DOWNLOAD_BZR
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	$(BZR) export $(DL_DIR)/$($(PKG)_SOURCE) $($(PKG)_SITE) -r $($(PKG)_DL_VERSION)
+	$(EXTRA_ENV) support/download/bzr $($(PKG)_SITE) $($(PKG)_DL_VERSION) $(DL_DIR)/$($(PKG)_SOURCE)
 endef
 
 define SOURCE_CHECK_BZR
support/download/bzr new file +19 -0
new file mode 100755
index 0000000..f07732e
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for bzr
+# Call it with:
+#   $1: bzr repo
+#   $2: bzr revision
+#   $3: output file
+# And this environment:
+#   BZR       : the bzr command to call
+#   BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+rev="${2}"
+output="${3}"
+
+${BZR} export "${output}" "${repo}" -r "${rev}"
Signature · 1 line
1.9.1
#12 lore raw

PATCH 6/12 pkg-infra: move the bzr download helper to a script

Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#13 lore raw

PATCH v10 7/12 pkg-infra: move the scp download helper to a script

In reply to Yann E. MORIN

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

---
 package/pkg-download.mk |  5 +++--
 support/download/scp    | 16 ++++++++++++++++
 2 files changed, 19 insertions(+), 2 deletions(-)
 create mode 100755 support/download/scp
package/pkg-download.mk +3 -2
index e420b05..eec91b8 100644
@@ -14,7 +14,7 @@ export CVS := $(call qstrip,$(BR2_CVS))
 export BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
 export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
-SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
+export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
 SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
 LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
@@ -147,7 +147,8 @@ endef
 # to prepend the path with a slash: scp://[user@]host:/absolutepath
 define DOWNLOAD_SCP
 	test -e $(DL_DIR)/$(2) || \
-	$(SCP) '$(call stripurischeme,$(call qstrip,$(1)))' $(DL_DIR)/$(2)
+	$(EXTRA_ENV) support/download/scp '$(call stripurischeme,$(call qstrip,$(1)))' \
+					  $(DL_DIR)/$(2)
 endef
 
 define SOURCE_CHECK_SCP
support/download/scp new file +16 -0
new file mode 100755
index 0000000..2997927
@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for scp
+# Call it with:
+#   $1: URL
+#   $2: output file
+# And this environment:
+#   SCP       : the scp command to call
+
+url="${1}"
+output="${2}"
+
+${SCP} "${url}" "${output}"
Signature · 1 line
1.9.1
#14 lore raw

PATCH 7/12 pkg-infra: move the scp download helper to a script

Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#15 lore raw

PATCH v10 8/12 pkg-infra: move the cp (aka localfiles) download helper to a script

In reply to Yann E. MORIN

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

---
 package/pkg-download.mk |  5 +++--
 support/download/cp     | 16 ++++++++++++++++
 2 files changed, 19 insertions(+), 2 deletions(-)
 create mode 100755 support/download/cp
package/pkg-download.mk +3 -2
index eec91b8..d3cd0c1 100644
@@ -16,7 +16,7 @@ export GIT := $(call qstrip,$(BR2_GIT))
 export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
 export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
 SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
-LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
+export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 # Default spider mode is 'DOWNLOAD'. Other possible values are 'SOURCE_CHECK'
 # used by the _source-check target and 'SHOW_EXTERNAL_DEPS', used by the
@@ -192,7 +192,8 @@ endef
 
 define DOWNLOAD_LOCALFILES
 	test -e $(DL_DIR)/$(2) || \
-		$(LOCALFILES) $(call stripurischeme,$(call qstrip,$(1))) $(DL_DIR)
+	$(EXTRA_ENV) support/download/cp $(call stripurischeme,$(call qstrip,$(1))) \
+					 $(DL_DIR)
 endef
 
 define SOURCE_CHECK_LOCALFILES
support/download/cp new file +16 -0
new file mode 100755
index 0000000..3e9f548
@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for cp
+# Call it with:
+#   $1: source file
+#   $2: output file
+# And this environment:
+#   LOCALFILES: the cp command to call
+
+source="${1}"
+output="${2}"
+
+${LOCALFILES} "${source}" "${output}"
Signature · 1 line
1.9.1
#16 lore raw

PATCH 8/12 pkg-infra: move the cp (aka localfiles) download helper to a script

Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.

---
Changes v9 -> v10:
- use bash as shell (Peter)

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#17 lore raw

PATCH v10 9/12 pkg-infra: don't use DL_DIR as scratchpad for temporary downloads

In reply to Yann E. MORIN

DL_DIR can be a very precious place for some users: they use it to store all the downloaded archives to share across all their Buildroot (and maybe non-Buildroot) builds.

We do not want to trash this location with our temporary downloads (e.g. git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs).

Turns out that we already have some kind of scratchpad, the BUILD_DIR. Although it is not really a disposable location, that's the best we have so far.

Also, we create the temporary tarballs with mktemp using the final tarrball, as template, since we want the temporary to be on the same filesystem as the final location, so the 'mv' is just a plain, atomic rename(2), and we are not left with a half-copied file as the final location.

Using mktemp ensures all temp file names are unique, so it allows for parallel downloads from different build dirs at the same time, without cloberring each downloads.

Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared locations, sometime with little place (eg. tmpfs), and some of the repositories we clone/checkout can be very big.

Cc Samuel Martin <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>

[tested a particular scenario that used to fail: two separate builds using a shared DL_DIR, ccache enabled, so that they run almost synchronously. These would download the same file at the same time, corrupting each other. With the patches in this series, all works fine.]

---
Change v7 -> v8:
- use a cleaner and saner code path to detect errors
- convert the cp and scp helpers, too (Thomas DS)

Changes v6 -> v7:
- drop leftovers from a previous attempt in bzr (Thomas)

Changes v5 -> v6:
- ensure we can do parallel downloads from multiple build-dirs (Thomas)

Changes v4 -> v5
- explain why we create the temp files where we create them (Arnout)

Changes v3 -> v4:
- remove spurious bump in package/fis (Samuel)

---
 support/download/bzr  | 25 ++++++++++++++++++++++---
 support/download/cp   | 12 +++++++++++-
 support/download/cvs  | 34 +++++++++++++++++++++++++++-------
 support/download/git  | 33 ++++++++++++++++++++++++++-------
 support/download/hg   | 35 ++++++++++++++++++++++++++++-------
 support/download/scp  | 14 +++++++++++++-
 support/download/svn  | 33 ++++++++++++++++++++++++++-------
 support/download/wget | 26 ++++++++++++++++++++------
 8 files changed, 173 insertions(+), 39 deletions(-)
support/download/bzr +22 -3
index f07732e..3f52ee9 100755
@@ -9,11 +9,30 @@ set -e
 #   $2: bzr revision
 #   $3: output file
 # And this environment:
-#   BZR       : the bzr command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   BZR      : the bzr command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 rev="${2}"
 output="${3}"
 
-${BZR} export "${output}" "${repo}" -r "${rev}"
+tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - the we move to a temp file in the final location, so it is
+#   on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${BZR} export "${tmp_dl}" "${repo}" -r "${rev}"; then
+    if mv "${tmp_dl}" "${tmp_output}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
+
+# Cleanup
+rm -f "${tmp_dl}" "${tmp_output}"
+exit ${ret}
support/download/cp +11 -1
index 3e9f548..8f6bc06 100755
@@ -13,4 +13,14 @@ set -e
 source="${1}"
 output="${2}"
 
-${LOCALFILES} "${source}" "${output}"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+ret=1
+if ${LOCALFILES} "${source}" "${tmp_output}"; then
+    mv "${tmp_output}" "${output}"
+    ret=0
+fi
+
+# Cleanup
+rm -f "${tmp_output}"
+exit ${ret}
support/download/cvs +27 -7
index 60787a6..9aeed79 100755
@@ -11,8 +11,8 @@ set -e
 #   $4: package's basename (eg. foobar-1.2.3)
 #   $5: output file
 # And this environment:
-#   CVS       : the cvs command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   CVS      : the cvs command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 rev="${2}"
@@ -20,8 +20,28 @@ rawname="${3}"
 basename="${4}"
 output="${5}"
 
-cd "${BR2_DL_DIR}"
-${CVS} -z3 -d":pserver:anonymous@${repo}" \
-       co -d "${basename}" -r ":${rev}" -P "${rawname}"
-tar czf "${output}" "${basename}"
-rm -rf "${basename}"
+repodir="${basename}.tmp-cvs-checkout"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create a temporary tarball in the final location, so it is
+#   on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${CVS} -z3 -d":pserver:anonymous@${repo}" \
+           co -d "${repodir}" -r ":${rev}" -P "${rawname}"; then
+    if tar czf "${tmp_output}" "${repodir}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
+
+# Cleanup
+rm -rf "${repodir}" "${tmp_output}"
+exit ${ret}
support/download/git +26 -7
index 4b5971e..e553cf2 100755
@@ -10,15 +10,28 @@ set -e
 #   $3: package's basename (eg. foobar-1.2.3)
 #   $4: output file
 # And this environment:
-#   BR2_DL_DIR: path to Buildroot's download dir
-#   GIT       : the git command to call
+#   GIT      : the git command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 cset="${2}"
 basename="${3}"
 output="${4}"
 
-repodir="${BR2_DL_DIR}/${basename}"
+repodir="${basename}.tmp-git-checkout"
+tmp_tar="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create the uncomporessed tarball in tht same trashable location
+# - then we create a temporary compressed tarball in the final location, so
+#   it is on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
 
 if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
     printf "Doing shallow clone\n"
@@ -28,10 +41,16 @@ else
     ${GIT} clone --bare "${repo}" "${repodir}"
 fi
 
+ret=1
 pushd "${repodir}"
-${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}"
-gzip -c "${output}.tmp" >"${output}"
-rm -f "${output}.tmp"
+if ${GIT} archive --prefix="${basename}/" -o "${tmp_tar}" \
+                  --format=tar "${cset}"; then
+    if gzip -c "${tmp_tar}" >"${tmp_output}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
 popd
 
-rm -rf "${repodir}"
+rm -rf "${repodir}" "${tmp_tar}" "${tmp_output}"
+exit ${ret}
support/download/hg +28 -7
index d2e69c3..6e9e26b 100755
@@ -10,16 +10,37 @@ set -e
 #   $3: package's basename (eg. foobar-1.2.3)
 #   $4: output file
 # And this environment:
-#   HG        : the hg command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   HG       : the hg command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 cset="${2}"
 basename="${3}"
 output="${4}"
 
-cd "${BR2_DL_DIR}"
-${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
-${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \
-              --rev "${cset}" "${output}"
-rm -rf "${basename}"
+repodir="${basename}.tmp-hg-checkout"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create a temporary tarball in the final location, so it is
+#   on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${HG} clone --noupdate --rev "${cset}" "${repo}" "${repodir}"; then
+    if ${HG} archive --repository "${repodir}" --type tgz \
+                     --prefix "${basename}" --rev "${cset}" \
+                     "${tmp_output}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
+
+# Cleanup
+rm -rf "${repodir}" "${tmp_output}"
+exit ${ret}
support/download/scp +13 -1
index 2997927..d3aad43 100755
@@ -12,5 +12,17 @@ set -e
 
 url="${1}"
 output="${2}"
+tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
 
-${SCP} "${url}" "${output}"
+ret=1
+if ${SCP} "${url}" "${tmp_dl}"; then
+    if mv "${tmp_dl}" "${tmp_output}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
+
+# Cleanup
+rm -f "${tmp_dl}" "${tmp_output}"
+exit ${ret}
support/download/svn +26 -7
index 0abcc86..39cbbcb 100755
@@ -10,16 +10,35 @@ set -e
 #   $3: package's basename (eg. foobar-1.2.3)
 #   $4: output file
 # And this environment:
-#   SVN       : the svn command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   SVN      : the svn command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 rev="${2}"
 basename="${3}"
 output="${4}"
 
-pushd "${BR2_DL_DIR}"
-${SVN} export "${repo}@${rev}" "${basename}"
-tar czf "${output}" "${basename}"
-rm -rf "${basename}"
-popd
+repodir="${basename}.tmp-svn-checkout"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create a temporary tarball in the final location, so it is
+#   on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${SVN} export "${repo}@${rev}" "${repodir}"; then
+    if tar czf "${tmp_output}" "${repodir}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
+
+# Cleanup
+rm -rf "${repodir}" "${tmp_output}"
+exit ${ret}
support/download/wget +20 -6
index 91ffd10..cbeca3b 100755
@@ -8,14 +8,28 @@ set -e
 #   $1: URL
 #   $2: output file
 # And this environment:
-#   WGET      : the wget command to call
+#   WGET     : the wget command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 url="${1}"
 output="${2}"
 
-if ${WGET} -O "${output}.tmp" "${url}"; then
-    mv "${output}.tmp" "${output}"
-else
-    rm -f "${output}.tmp"
-    exit 1
+tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we copy to a temporary tarball in the final location, so it is
+#   on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${WGET} -O "${tmp_dl}" "${url}"; then
+    if mv "${tmp_dl}" "${tmp_output}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
 fi
+
+rm -f "${tmp_dl}" "${tmp_output}"
+exit ${ret}
Signature · 1 line
1.9.1
#18 lore raw

PATCH v10 10/12 pkg-infra: add possiblity to check downloaded files against known hashes

In reply to Yann E. MORIN

Some of the packages that Buildroot might build are sensitive packages, related to security: openssl, dropbear, ca-certificates...

Some of those packages are downloaded over plain http, because there is no way to get them over a secure channel, such as https.

In these dark times of pervasive surveillance, the potential for harm that a tampered-with package could generate, we may want to check the integrity of those sensitive packages.

So, each package may now provide a list of hashes for all files that needs to be downloaded, and Buildroot will just fail if any downloaded file does not match its known hash, in which case it is removed.

Hashes can be any of the md5, sha1 or sha2 variants, and will be checked even if the file was pre-downloaded.

Cc Baruch Siach <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Gustavo Zacarias <[email protected]>
Cc Thomas De Schampheleire <[email protected]>

---
Changes v9 -> v10:
- use bash as shell (Peter)

Changes v7 -> v8
- expand MITM to its full-length meaning (Thomas DS)
- typo (Thomas DS)

Changes v4 -> v5:
- fix detection of comments and empty lines

---
Note: this is not a bullet-proof solution, since Buildroot may itself be compromised. But since we do sign our releases, then we secure the list of hashes at the same time. Only random snapshots from the repository may be at risk of tampering, although this is highly doubtfull, given how git stores its data.

---
 package/pkg-download.mk     | 20 ++++++++++--
 support/download/check-hash | 76 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 93 insertions(+), 3 deletions(-)
 create mode 100755 support/download/check-hash
package/pkg-download.mk +17 -3
index d3cd0c1..7f208d5 100644
@@ -58,6 +58,17 @@ domainseparator=$(if $(1),$(1),/)
 # github(user,package,version): returns site of GitHub repository
 github = https://github.com/$(1)/$(2)/archive/$(3)
 
+# Helper for checking a tarball's checksum
+# If the hash does not match, remove the incorrect file
+# $(1): the path to the file with the hashes
+# $(2): the full path to the file to check
+define VERIFY_HASH
+	if ! support/download/check-hash $(1) $(2); then \
+		rm -f $(2); \
+		exit 1; \
+	fi
+endef
+
 ################################################################################
 # The DOWNLOAD_* helpers are in charge of getting a working copy
 # of the source repository for their corresponding SCM,
@@ -148,7 +159,8 @@ endef
 define DOWNLOAD_SCP
 	test -e $(DL_DIR)/$(2) || \
 	$(EXTRA_ENV) support/download/scp '$(call stripurischeme,$(call qstrip,$(1)))' \
-					  $(DL_DIR)/$(2)
+					  $(DL_DIR)/$(2) && \
+	$(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
 endef
 
 define SOURCE_CHECK_SCP
@@ -179,7 +191,8 @@ endef
 
 define DOWNLOAD_WGET
 	test -e $(DL_DIR)/$(2) || \
-	$(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2)
+	$(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2) && \
+	$(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
 endef
 
 define SOURCE_CHECK_WGET
@@ -193,7 +206,8 @@ endef
 define DOWNLOAD_LOCALFILES
 	test -e $(DL_DIR)/$(2) || \
 	$(EXTRA_ENV) support/download/cp $(call stripurischeme,$(call qstrip,$(1))) \
-					 $(DL_DIR)
+					 $(DL_DIR) && \
+	$(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
 endef
 
 define SOURCE_CHECK_LOCALFILES
support/download/check-hash new file +76 -0

76 lines not on this page · raw message

Signature · 1 line
1.9.1
#19 lore raw

PATCH v10 11/12 manual: add documentation about packages' hashes

In reply to Yann E. MORIN

Although md5 is, for legacy reasons, a supported hash type, it is not documented on purpose, since it is now known to be weak.

Cc Baruch Siach <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Samuel Martin <[email protected]>
Cc Thomas De Schampheleire <[email protected]>

---
Changes v6 -> v7:
- use single-line title (Samuel)

---
 docs/manual/adding-packages-directory.txt | 66 +++++++++++++++++++++++++++++++
 support/download/check-hash               |  1 +
 2 files changed, 67 insertions(+)
docs/manual/adding-packages-directory.txt +66 -0
index 69b012c..c2f0a1a 100644
@@ -357,3 +357,69 @@ different way, using different infrastructures:
 
 Further formatting details: see xref:writing-rules-mk[the writing
 rules].
+
+[[adding-packages-hash]]
+=== The +.hash+ file
+
+Optionally, you can add a third file, named +libfoo.hash+, that contains
+the hashes of the downloaded files for the +libfoo+ package.
+
+The hashes stored in that file are used to validate the integrity of the
+downloaded files.
+
+The format of this file is one line for each file for which to check the
+hash, each line being space-separated, with these three fields:
+
+* the type of hash, one of:
+** +sha1+, +sha224+, +sha256+, +sha384+, +sha512+
+* the hash of the file:
+** for +sha1+, 40 hexadecimal characters
+** for +sha224+, 56 hexadecimal characters
+** for +sha256+, 64 hexadecimal characters
+** for +sha384+, 96 hexadecimal characters
+** for +sha512+, 128 hexadecimal characters
+* the name of the file, without any directory component
+
+Lines starting with a +#+ sign are considered comments, and ignored. Empty
+lines are ignored.
+
+There can be more than one hash for a single file, each on its own line. In
+this case, all hashes must match.
+
+Ideally, the hashes stored in this file should match the hashes published by
+upstream, e.g. on their website, in the e-mail announcement... If upstream
+provides more than one type of hash (say, +sha1+ and +sha512+), then it is
+best to add all those hashes in the +.hash+ file. If upstream does not
+provide any hash, then compute at least one yourself, and mention this in a
+comment line above the hashes.
+
+*Note:* the number of spaces does not matter, so one can use spaces to
+properly align the different fields.
+
+The example below defines a +sha1+ and a +sha256+ published by upstream for
+the main +libfoo-1.2.3.tar.bz2+ tarball, plus two locally-computed hashes,
+a +sha256+ for a downloaded patch, and a +sha1+ for a downloaded binary blob:
+
+----
+# Hashes from: http://www.foosoftware.org/download/libfoo-1.2.3.tar.bz2.{sha1,sha256}:
+sha1   486fb55c3efa71148fe07895fd713ea3a5ae343a                         libfoo-1.2.3.tar.bz2
+sha256 efc8103cc3bcb06bda6a781532d12701eb081ad83e8f90004b39ab81b65d4369 libfoo-1.2.3.tar.bz2
+
+# No upstream hashes for the following:
+sha256 ff52101fb90bbfc3fe9475e425688c660f46216d7e751c4bbdb1dc85cdccacb9 libfoo-fix-blabla.patch
+sha1   2d608f3c318c6b7557d551a5a09314f03452f1a1                         libfoo-data.bin
+----
+
+If the +.hash+ file is present, and it contains one or more hashes for a
+downloaded file, the hash(es) computed by Buildroot (after download) must
+match the hash(es) stored in the +.hash+ file. If one or more hashes do
+not match, Buildroot considers this an error, deletes the downloaded file,
+and aborts.
+
+If the +.hash+ file is present, but it does not contain a hash for a
+downloaded file, no check is done for that file. If you set the
+environment variable +BR2_ENFORCE_CHECK_HASH+ to a non-empty value, and
+there is no hash for a downloaded file, Buildroot considers this an
+error, deletes the downloaded file, and aborts.
+
+If the +.hash+ file is missing, then no check is done at all.
support/download/check-hash +1 -0
index 9ea7c41..f1e0c1b 100755
@@ -22,6 +22,7 @@ check_one_hash() {
     _known="${2}"
     _file="${3}"
 
+    # Note: md5 is supported, but undocumented on purpose.
     # Note: sha3 is not supported, since there is currently no implementation
     #       (the NIST has yet to publish the parameters).
     case "${_h}" in
Signature · 1 line
1.9.1
#20 lore raw

PATCH v10 12/12 package/ca-certificates: add tarball's hashes

In reply to Yann E. MORIN

ca-certificates contains sensitive security-related information, and we want to ensure the archive that we download has not been compromised.

Add the sha1 and sha256 hashes from Debian's packaging.

---
Changes v4 -> v5:
- update hashes since we've bumped the version

---
 package/ca-certificates/ca-certificates.hash | 3 +++
 1 file changed, 3 insertions(+)
 create mode 100644 package/ca-certificates/ca-certificates.hash
package/ca-certificates/ca-certificates.hash new file +3 -0
new file mode 100644
index 0000000..bcd0723
@@ -0,0 +1,3 @@
+# hashes from: $(CA_CERTIFICATES_SITE)/ca-certificates_$(CA_CERTIFICATES_VERSION).dsc :
+sha1   ad57a45f0422fafd78a2e8191e5204f2306cc91b                         ca-certificates_20140223.tar.xz
+sha256 815b7cd97200b0d76450bb3e7d9b65997ac494ab6467b17369f65b2ef94bcb0c ca-certificates_20140223.tar.xz
Signature · 1 line
1.9.1