some download-related changes
v3 cover some download-related changes
From: "Yann E. MORIN" <[email protected]>
Hello All!
This series contains those download-related changes:
- Patches 1..2 : some cleanups and preparatory works in the download infra,
- Patches 3..7 : move download helpers to shell scripts, one such script for each download helper
- Patches 8..9 : don't use DL_DIR as scratchpad for temporary VCS checkouts, since DL_DIR is a precious location
- Patches 10..12: check hashes of downloaded files, to ensure the files are genuine
Samuel, I removed your reviewed-by tag on the patches that have changed quite a bit since last round: patches 8, 10 & 11.
Changes v2 -> v3:
- avoid partial downloads (Thomas DS)
- store hash-type in .hash file (Gustavo)
- typoes (Samuel, Gustavo)
- add possibility to treat missing hashes as an error
Changes v1 -> v2:
- don't do any behavioural change when switching to shell scripts, just reproduce exactly what was in the Makefile (Luca, Arnout)
- support more than one hash algorithm (Arnout, Gustavo)
- typoes (Luca, Baruch)
Regards,
Yann E. MORIN.
----------------------------------------------------------------
Yann E. MORIN (12):
Makefile: rename USER_HOOKS_EXTRA_ENV to EXTRA_ENV
Makefile: add BR2_DL_DIR to EXTRA_ENV
pkg-infra: move the git download helper to a script
pkg-infra: move the svn download helper to a script
pkg-infra: move the cvs download helper to a script
pkg-infra: move the hg download helper to a script
pkg-infra: move the wget download helper to a script
pkg-infra: don't use DL_DIR as scratchpad for temporary VCS checkouts
pkg-infra: also set PKGDIR for the download step
pkg-infra: add possiblity to check downloaded files against known hashes
manual: add documentation about packages' hashes package/ca-certificates: add tarball's hashes
Makefile | 4 +- docs/manual/adding-packages-directory.txt | 65 +++++++++++++++++++++++ package/Makefile.in | 5 +- package/ca-certificates/ca-certificates.hash | 3 ++ package/fis/fis.mk | 2 +- package/pkg-download.mk | 79 +++++++++++----------------- package/pkg-generic.mk | 3 +- support/download/check-hash | 77 +++++++++++++++++++++++++++ support/download/cvs | 33 ++++++++++++ support/download/git | 41 +++++++++++++++ support/download/hg | 31 +++++++++++ support/download/svn | 30 +++++++++++ support/download/wget | 24 +++++++++ 13 files changed, 344 insertions(+), 53 deletions(-) create mode 100644 package/ca-certificates/ca-certificates.hash create mode 100755 support/download/check-hash create mode 100755 support/download/cvs create mode 100755 support/download/git create mode 100755 support/download/hg create mode 100755 support/download/svn create mode 100755 support/download/wget
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 1/12 Makefile: rename USER_HOOKS_EXTRA_ENV to EXTRA_ENV
From: "Yann E. MORIN" <[email protected]>
This variable contains extra environment variables that we can not export since they are clashing with some build systems (eg. BUILD_DIR with u-boot).
So, we may need these variables for uses other than the user's hooks for instrumentation. For example, we'll use them later on to export BUILD_DIR to the download helper scripts.
Fix comment, too.
--- Makefile | 4 ++-- package/Makefile.in | 4 ++-- package/pkg-generic.mk | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-)
Makefile +2 -2
@@ -577,7 +577,7 @@ endif @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_BUILD_SCRIPT)), \ $(call MESSAGE,"Executing post-build script $(s)"); \ - $(USER_HOOKS_EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep)) + $(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep)) ifeq ($(BR2_ENABLE_LOCALE_PURGE),y) LOCALE_WHITELIST=$(BUILD_DIR)/locales.nopurge
@@ -623,7 +623,7 @@ endif target-post-image: $(TARGETS_ROOTFS) target-finalize @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \ $(call MESSAGE,"Executing post-image script $(s)"); \ - $(USER_HOOKS_EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep)) + $(EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep)) toolchain-eclipse-register: ./support/scripts/eclipse-register-toolchain `readlink -f $(O)` $(notdir $(TARGET_CROSS)) $(BR2_ARCH)
package/Makefile.in +2 -2
@@ -277,11 +277,11 @@ HOST_MAKE_ENV=PATH=$(HOST_PATH) \ PKG_CONFIG_SYSROOT_DIR="/" \ PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig" -# This extra environment we can not export ourselves (eg. because some +# This is extra environment we can not export ourselves (eg. because some # packages use that variable internally, eg. uboot), so we have to # explicitly pass it to user-supplied external hooks (eg. post-build, # post-images) -USER_HOOKS_EXTRA_ENV=\ +EXTRA_ENV=\ BUILD_DIR=$(BUILD_DIR) ################################################################################
package/pkg-generic.mk +1 -1
@@ -58,7 +58,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # User-supplied script define step_user @$(foreach user_hook, $(BR2_INSTRUMENTATION_SCRIPTS), \ - $(USER_HOOKS_EXTRA_ENV) $(user_hook) "$(1)" "$(2)" "$(3)"$(sep)) + $(EXTRA_ENV) $(user_hook) "$(1)" "$(2)" "$(3)"$(sep)) endef ifneq ($(BR2_INSTRUMENTATION_SCRIPTS),) GLOBAL_INSTRUMENTATION_HOOKS += step_user
Signature · 1 line
1.8.3.2
PATCH v3 2/12 Makefile: add BR2_DL_DIR to EXTRA_ENV
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Also export BR2_DL_DIR for incoming download helper scripts.
--- package/Makefile.in | 1 + 1 file changed, 1 insertion(+)
package/Makefile.in +1 -0
@@ -282,6 +282,7 @@ HOST_MAKE_ENV=PATH=$(HOST_PATH) \ # explicitly pass it to user-supplied external hooks (eg. post-build, # post-images) EXTRA_ENV=\ + BR2_DL_DIR=$(BR2_DL_DIR) \ BUILD_DIR=$(BUILD_DIR) ################################################################################
Signature · 1 line
1.8.3.2
PATCH v3 3/12 pkg-infra: move the git download helper to a script
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
The git download helper is getting a bit more complex. Fixing it in the Makefile when it breaks (like the recent breakage with a non-existing sha1-cset) proves to be challenging, to say the least.
Move it into a shell script in support/download/git, which will make it much easier to read, maintain, fix and enhance in the future.
--- package/pkg-download.mk | 17 +++-------------- support/download/git | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 14 deletions(-) create mode 100755 support/download/git
package/pkg-download.mk +3 -14
@@ -12,7 +12,7 @@ WGET := $(call qstrip,$(BR2_WGET)) $(QUIET) SVN := $(call qstrip,$(BR2_SVN)) CVS := $(call qstrip,$(BR2_CVS)) BZR := $(call qstrip,$(BR2_BZR)) -GIT := $(call qstrip,$(BR2_GIT)) +export GIT := $(call qstrip,$(BR2_GIT)) HG := $(call qstrip,$(BR2_HG)) $(QUIET) SCP := $(call qstrip,$(BR2_SCP)) $(QUIET) SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
@@ -84,19 +84,8 @@ github = https://github.com/$(1)/$(2)/tarball/$(3) # problems define DOWNLOAD_GIT test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - (pushd $(DL_DIR) > /dev/null && \ - ((test "`git ls-remote $($(PKG)_SITE) $($(PKG)_DL_VERSION)`" && \ - echo "Doing shallow clone" && \ - $(GIT) clone --depth 1 -b $($(PKG)_DL_VERSION) --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME)) || \ - (echo "Doing full clone" && \ - $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME))) && \ - pushd $($(PKG)_BASE_NAME) > /dev/null && \ - $(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ -o $(DL_DIR)/.$($(PKG)_SOURCE).tmp $($(PKG)_DL_VERSION) && \ - gzip -c $(DL_DIR)/.$($(PKG)_SOURCE).tmp > $(DL_DIR)/$($(PKG)_SOURCE) && \ - rm -f $(DL_DIR)/.$($(PKG)_SOURCE).tmp && \ - popd > /dev/null && \ - rm -rf $($(PKG)_DL_DIR) && \ - popd > /dev/null) + $(EXTRA_ENV) support/download/git $($(PKG)_SITE) $($(PKG)_DL_VERSION) \ + $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE) endef # TODO: improve to check that the given PKG_DL_VERSION exists on the remote
support/download/git new file +37 -0
@@ -0,0 +1,37 @@ +#!/bin/sh + +# We want to catch any command failure, and exit immediately +set -e + +# Download helper for git +# Call it with: +# $1: git repo +# $2: git cset +# $3: package's basename (eg. foobar-1.2.3) +# $4: output file +# And this environment: +# BR2_DL_DIR: path to Buildroot's download dir +# GIT : the git command to call + +repo="${1}" +cset="${2}" +basename="${3}" +output="${4}" + +repodir="${BR2_DL_DIR}/${basename}" + +if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then + printf "Doing shallow clone\n" + ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${repodir}" +else + printf "Doing full clone\n" + ${GIT} clone --bare "${repo}" "${repodir}" +fi + +pushd "${repodir}" +${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}" +gzip -c "${output}.tmp" >"${output}" +rm -f "${output}.tmp" +popd + +rm -rf "${repodir}"
Signature · 1 line
1.8.3.2
PATCH v3 4/12 pkg-infra: move the svn download helper to a script
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.
--- package/pkg-download.mk | 9 +++------ support/download/svn | 25 +++++++++++++++++++++++++ 2 files changed, 28 insertions(+), 6 deletions(-) create mode 100755 support/download/svn
package/pkg-download.mk +3 -6
@@ -9,7 +9,7 @@ # Download method commands WGET := $(call qstrip,$(BR2_WGET)) $(QUIET) -SVN := $(call qstrip,$(BR2_SVN)) +export SVN := $(call qstrip,$(BR2_SVN)) CVS := $(call qstrip,$(BR2_CVS)) BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT))
@@ -133,11 +133,8 @@ endef define DOWNLOAD_SVN test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - (pushd $(DL_DIR) > /dev/null && \ - $(SVN) export -r $($(PKG)_DL_VERSION) $($(PKG)_SITE) $($(PKG)_DL_DIR) && \ - $(TAR) czf $($(PKG)_SOURCE) $($(PKG)_BASE_NAME)/ && \ - rm -rf $($(PKG)_DL_DIR) && \ - popd > /dev/null) + $(EXTRA_ENV) support/download/svn $($(PKG)_SITE) $($(PKG)_DL_VERSION) \ + $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE) endef define SOURCE_CHECK_SVN
support/download/svn new file +25 -0
@@ -0,0 +1,25 @@ +#!/bin/sh + +# We want to catch any command failure, and exit immediately +set -e + +# Download helper for svn +# Call it with: +# $1: svn repo +# $2: svn revision +# $3: package's basename (eg. foobar-1.2.3) +# $4: output file +# And this environment: +# SVN : the svn command to call +# BR2_DL_DIR: path to Buildroot's download dir + +repo="${1}" +rev="${2}" +basename="${3}" +output="${4}" + +pushd "${BR2_DL_DIR}" +${SVN} export -r "${rev}" "${repo}" "${basename}" +tar czf "${output}" "${basename}" +rm -rf "${basename}" +popd
Signature · 1 line
1.8.3.2
PATCH v3 5/12 pkg-infra: move the cvs download helper to a script
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.
--- package/pkg-download.mk | 11 ++++------- support/download/cvs | 27 +++++++++++++++++++++++++++ 2 files changed, 31 insertions(+), 7 deletions(-) create mode 100755 support/download/cvs
package/pkg-download.mk +4 -7
@@ -10,7 +10,7 @@ # Download method commands WGET := $(call qstrip,$(BR2_WGET)) $(QUIET) export SVN := $(call qstrip,$(BR2_SVN)) -CVS := $(call qstrip,$(BR2_CVS)) +export CVS := $(call qstrip,$(BR2_CVS)) BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT)) HG := $(call qstrip,$(BR2_HG)) $(QUIET)
@@ -114,12 +114,9 @@ endef define DOWNLOAD_CVS test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - (pushd $(DL_DIR) > /dev/null && \ - $(CVS) -z3 -d:pserver:anonymous@$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \ - co -d $($(PKG)_BASE_NAME) -r :$($(PKG)_DL_VERSION) -P $($(PKG)_RAWNAME) && \ - $(TAR) czf $($(PKG)_SOURCE) $($(PKG)_BASE_NAME)/ && \ - rm -rf $($(PKG)_DL_DIR) && \ - popd > /dev/null) + $(EXTRA_ENV) support/download/cvs $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \ + $($(PKG)_DL_VERSION) $($(PKG)_RAWNAME) \ + $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE) endef # Not all CVS servers support ls/rls, use login to see if we can connect
support/download/cvs new file +27 -0
@@ -0,0 +1,27 @@ +#!/bin/sh + +# We want to catch any command failure, and exit immediately +set -e + +# Download helper for cvs +# Call it with: +# $1: cvs repo +# $2: cvs revision +# $3: package's name (eg. foobar) +# $4: package's basename (eg. foobar-1.2.3) +# $5: output file +# And this environment: +# CVS : the cvs command to call +# BR2_DL_DIR: path to Buildroot's download dir + +repo="${1}" +rev="${2}" +rawname="${3}" +basename="${4}" +output="${5}" + +cd "${BR2_DL_DIR}" +${CVS} -z3 -d":pserver:anonymous@${repo}" \ + co -d "${basename}" -r ":${rev}" -P "${rawname}" +tar czf "${output}" "${basename}" +rm -rf "${basename}"
Signature · 1 line
1.8.3.2
PATCH v3 6/12 pkg-infra: move the hg download helper to a script
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.
--- package/pkg-download.mk | 11 +++-------- support/download/hg | 25 +++++++++++++++++++++++++ 2 files changed, 28 insertions(+), 8 deletions(-) create mode 100755 support/download/hg
package/pkg-download.mk +3 -8
@@ -13,7 +13,7 @@ export SVN := $(call qstrip,$(BR2_SVN)) export CVS := $(call qstrip,$(BR2_CVS)) BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT)) -HG := $(call qstrip,$(BR2_HG)) $(QUIET) +export HG := $(call qstrip,$(BR2_HG)) $(QUIET) SCP := $(call qstrip,$(BR2_SCP)) $(QUIET) SSH := $(call qstrip,$(BR2_SSH)) $(QUIET) LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
@@ -161,13 +161,8 @@ endef define DOWNLOAD_HG test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - (pushd $(DL_DIR) > /dev/null && \ - rm -rf $($(PKG)_BASE_NAME) && \ - $(HG) clone --noupdate --rev $($(PKG)_DL_VERSION) $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \ - $(HG) archive --repository $($(PKG)_BASE_NAME) --type tgz --prefix $($(PKG)_BASE_NAME)/ \ - --rev $($(PKG)_DL_VERSION) $(DL_DIR)/$($(PKG)_SOURCE) && \ - rm -rf $($(PKG)_DL_DIR) && \ - popd > /dev/null) + $(EXTRA_ENV) support/download/hg $($(PKG)_SITE) $($(PKG)_DL_VERSION) \ + $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE) endef # TODO: improve to check that the given PKG_DL_VERSION exists on the remote
support/download/hg new file +25 -0
@@ -0,0 +1,25 @@ +#!/bin/sh + +# We want to catch any command failure, and exit immediately +set -e + +# Download helper for hg +# Call it with: +# $1: hg repo +# $2: hg cset +# $3: package's basename (eg. foobar-1.2.3) +# $4: output file +# And this environment: +# HG : the hg command to call +# BR2_DL_DIR: path to Buildroot's download dir + +repo="${1}" +cset="${2}" +basename="${3}" +output="${4}" + +cd "${BR2_DL_DIR}" +${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}" +${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \ + --rev "${cset}" "${output}" +rm -rf "${basename}"
Signature · 1 line
1.8.3.2
PATCH v3 7/12 pkg-infra: move the wget download helper to a script
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script.
--- package/pkg-download.mk | 6 ++---- support/download/wget | 21 +++++++++++++++++++++ 2 files changed, 23 insertions(+), 4 deletions(-) create mode 100755 support/download/wget
package/pkg-download.mk +2 -4
@@ -8,7 +8,7 @@ ################################################################################ # Download method commands -WGET := $(call qstrip,$(BR2_WGET)) $(QUIET) +export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET) export SVN := $(call qstrip,$(BR2_SVN)) export CVS := $(call qstrip,$(BR2_CVS)) BZR := $(call qstrip,$(BR2_BZR))
@@ -183,9 +183,7 @@ endef # the next time the download is tried. define DOWNLOAD_WGET test -e $(DL_DIR)/$(2) || \ - ($(WGET) -O $(DL_DIR)/$(2).tmp '$(call qstrip,$(1))' && \ - mv $(DL_DIR)/$(2).tmp $(DL_DIR)/$(2)) || \ - (rm -f $(DL_DIR)/$(2).tmp ; exit 1) + $(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2) endef define SOURCE_CHECK_WGET
support/download/wget new file +21 -0
@@ -0,0 +1,21 @@ +#!/bin/sh + +# We want to catch any command failure, and exit immediately +set -e + +# Download helper for wget +# Call it with: +# $1: URL +# $2: output file +# And this environment: +# WGET : the wget command to call + +url="${1}" +output="${2}" + +if ${WGET} -O "${output}.tmp" "${url}"; then + mv "${output}.tmp" "${output}" +else + rm -f "${output}.tmp" + exit 1 +fi
Signature · 1 line
1.8.3.2
PATCH v3 8/12 pkg-infra: don't use DL_DIR as scratchpad for temporary VCS checkouts
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
DL_DIR can be a very precious place for some users: they use it to store all the downloaded archives to share across all their Buildroot (and maybe non-Buildroot) builds.
We do not want to trash this location with our temporary VCS (git, Hg, svn, cvs) repository clones/checkouts.
Turns out that we already have some kind of scratchpad, the BUILD_DIR. Although it is not really a disposable location, that's the best we have so far.
Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared locations, sometime with little place (eg. tmpfs), and some of the repositories we clone/checkout can be very big.
--- package/fis/fis.mk | 2 +- support/download/cvs | 18 ++++++++++++------ support/download/git | 16 ++++++++++------ support/download/hg | 20 +++++++++++++------- support/download/svn | 19 ++++++++++++------- support/download/wget | 7 +++++-- 6 files changed, 53 insertions(+), 29 deletions(-)
package/fis/fis.mk +1 -1
@@ -6,7 +6,7 @@ FIS_SITE = http://svn.chezphil.org/utils/trunk FIS_SITE_METHOD = svn -FIS_VERSION = 2892 +FIS_VERSION = 9852 define FIS_BUILD_CMDS $(TARGET_CC) $(TARGET_CFLAGS) -std=c99 -o $(@D)/fis \
support/download/cvs +12 -6
@@ -11,8 +11,8 @@ set -e # $4: package's basename (eg. foobar-1.2.3) # $5: output file # And this environment: -# CVS : the cvs command to call -# BR2_DL_DIR: path to Buildroot's download dir +# CVS : the cvs command to call +# BUILD_DIR: path to Buildroot's build dir repo="${1}" rev="${2}"
@@ -20,8 +20,14 @@ rawname="${3}" basename="${4}" output="${5}" -cd "${BR2_DL_DIR}" +repodir="${basename}.tmp-cvs-checkout" + +cd "${BUILD_DIR}" +# Remove leftovers from a previous failed run +rm -rf "${repodir}" "${output}.tmp" + ${CVS} -z3 -d":pserver:anonymous@${repo}" \ - co -d "${basename}" -r ":${rev}" -P "${rawname}" -tar czf "${output}" "${basename}" -rm -rf "${basename}" + co -d "${repodir}" -r ":${rev}" -P "${rawname}" +tar czf "${output}.tmp" "${repodir}" +mv "${output}.tmp" "${output}" +rm -rf "${repodir}"
support/download/git +10 -6
@@ -10,15 +10,19 @@ set -e # $3: package's basename (eg. foobar-1.2.3) # $4: output file # And this environment: -# BR2_DL_DIR: path to Buildroot's download dir -# GIT : the git command to call +# GIT : the git command to call +# BUILD_DIR: path to Buildroot's build dir repo="${1}" cset="${2}" basename="${3}" output="${4}" -repodir="${BR2_DL_DIR}/${basename}" +repodir="${basename}.tmp-git-checkout" + +cd "${BUILD_DIR}" +# Remove leftovers from a previous failed run +rm -rf "${repodir}" "${output}.tmp" "${output}.tmp2" if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then printf "Doing shallow clone\n"
@@ -30,8 +34,8 @@ fi pushd "${repodir}" ${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}" -gzip -c "${output}.tmp" >"${output}" -rm -f "${output}.tmp" +gzip -c "${output}.tmp" >"${output}.tmp2" +mv "${output}.tmp2" "${output}" popd -rm -rf "${repodir}" +rm -rf "${repodir}" "${output}.tmp"
support/download/hg +13 -7
@@ -10,16 +10,22 @@ set -e # $3: package's basename (eg. foobar-1.2.3) # $4: output file # And this environment: -# HG : the hg command to call -# BR2_DL_DIR: path to Buildroot's download dir +# HG : the hg command to call +# BUILD_DIR: path to Buildroot's build dir repo="${1}" cset="${2}" basename="${3}" output="${4}" -cd "${BR2_DL_DIR}" -${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}" -${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \ - --rev "${cset}" "${output}" -rm -rf "${basename}" +repodir="${basename}.tmp-hg-checkout" + +cd "${BUILD_DIR}" +# Remove leftovers from a previous failed run +rm -rf "${repodir}" "${output}.tmp" + +${HG} clone --noupdate --rev "${cset}" "${repo}" "${repodir}" +${HG} archive --repository "${repodir}" --type tgz --prefix "${basename}" \ + --rev "${cset}" "${output}.tmp" +mv "${output}.tmp" "${output}" +rm -rf "${repodir}"
support/download/svn +12 -7
@@ -10,16 +10,21 @@ set -e # $3: package's basename (eg. foobar-1.2.3) # $4: output file # And this environment: -# SVN : the svn command to call -# BR2_DL_DIR: path to Buildroot's download dir +# SVN : the svn command to call +# BUILD_DIR: path to Buildroot's build dir repo="${1}" rev="${2}" basename="${3}" output="${4}" -pushd "${BR2_DL_DIR}" -${SVN} export -r "${rev}" "${repo}" "${basename}" -tar czf "${output}" "${basename}" -rm -rf "${basename}" -popd +repodir="${basename}.tmp-svn-checkout" + +cd "${BUILD_DIR}" +# Remove leftovers from a previous failed run +rm -rf "${repodir}" "${output}.tmp" + +${SVN} export -r "${rev}" "${repo}" "${repodir}" +tar czf "${output}.tmp" "${repodir}" +mv "${output}.tmp" "${output}" +rm -rf "${repodir}"
support/download/wget +5 -2
@@ -8,14 +8,17 @@ set -e # $1: URL # $2: output file # And this environment: -# WGET : the wget command to call +# WGET : the wget command to call url="${1}" output="${2}" +# Remove leftovers from a previous failed run +rm -rf "${output}.tmp" + if ${WGET} -O "${output}.tmp" "${url}"; then mv "${output}.tmp" "${output}" else - rm -f "${output}.tmp" + rm -f "${tmpfile}" exit 1 fi
Signature · 1 line
1.8.3.2
PATCH 8/12 pkg-infra: don't use DL_DIR as scratchpad for temporary VCS checkouts
Yann, all,
Quoted 32 lines · Yann E. MORIN
From: "Yann E. MORIN" <[email protected]>
DL_DIRcan be a very precious place for some users: they use it to store all the downloaded archives to share across all their Buildroot (and maybe non-Buildroot) builds.We do not want to trash this location with our temporary VCS (git, Hg, svn, cvs) repository clones/checkouts.
Turns out that we already have some kind of scratchpad, the
BUILD_DIR. Although it is not really a disposable location, that's the best we have so far.Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared locations, sometime with little place (eg. tmpfs), and some of the repositories we clone/checkout can be very big.
Signed-off-by "Yann E. MORIN" <[email protected]>--- package/fis/fis.mk | 2 +- support/download/cvs | 18 ++++++++++++------ support/download/git | 16 ++++++++++------ support/download/hg | 20 +++++++++++++------- support/download/svn | 19 ++++++++++++------- support/download/wget | 7 +++++-- 6 files changed, 53 insertions(+), 29 deletions(-)package/fis/fis.mk +0
-0
package/fis/fis.mk line 12
Quoted 8 lines
package/fis/fis.mk +1
-1@@ -6,7 +6,7 @@ FIS_SITE = http://svn.chezphil.org/utils/trunk FIS_SITE_METHOD = svn -FIS_VERSION = 2892 +FIS_VERSION = 9852 define FIS_BUILD_CMDS $(TARGET_CC) $(TARGET_CFLAGS) -std=c99 -o $(@D)/fis \
This change seems not related to the rest of the patch! left over form your test maybe? ;-)
Quoted 23 lines
support/download/cvs +5
-3@@ -11,8 +11,8 @@ set -e # $4: package's basename (eg. foobar-1.2.3) # $5: output file # And this environment: -# CVS : the cvs command to call -# BR2_DL_DIR: path to Buildroot's download dir +# CVS : the cvs command to call +# BUILD_DIR: path to Buildroot's build dir repo="${1}" rev="${2}"@@ -20,8 +20,14 @@ rawname="${3}" basename="${4}" output="${5}" -cd "${BR2_DL_DIR}" +repodir="${basename}.tmp-cvs-checkout" + +cd "${BUILD_DIR}"
support/download/cvs rawname="${3}" · line 32
Quoted 10 lines
support/download/cvs +6
-3+# Remove leftovers from a previous failed run +rm -rf "${repodir}" "${output}.tmp" + ${CVS} -z3 -d":pserver:anonymous@${repo}" \ - co -d "${basename}" -r ":${rev}" -P "${rawname}" -tar czf "${output}" "${basename}" -rm -rf "${basename}" + co -d "${repodir}" -r ":${rev}" -P "${rawname}" +tar czf "${output}.tmp" "${repodir}" +mv "${output}.tmp" "${output}"
You move the temporary repository in the package build_dir. Maybe the temporary output archive should be generated there as well, so that the mv command will only move something valid.
Regards,
Signature · 1 line
Samuel
PATCH v3 8/12 pkg-infra: don't use DL_DIR as scratchpad for temporary VCS checkouts
Samuel, All,
On 2014-03-06 11:25 +0100, Samuel Martin spake thusly:
Quoted 6 lines · Yann E. MORIN
Quoted 5 lines
From: "Yann E. MORIN" <[email protected]>
DL_DIRcan be a very precious place for some users: they use it to store all the downloaded archives to share across all their Buildroot (and maybe non-Buildroot) builds.
[--SNIP--]
Quoted 11 lines
Quoted 11 lines
package/fis/fis.mk +1
-1@@ -6,7 +6,7 @@ FIS_SITE = http://svn.chezphil.org/utils/trunk FIS_SITE_METHOD = svn -FIS_VERSION = 2892 +FIS_VERSION = 9852
Quoted 5 lines
package/fis/fis.mk line 12
Quoted 2 lines
package/fis/fis.mk +0
-0define FIS_BUILD_CMDS $(TARGET_CC) $(TARGET_CFLAGS) -std=c99 -o $(@D)/fis \This change seems not related to the rest of the patch! left over form your test maybe? ;-)
Yes, it's a leftover from some tests... Sigh...
Thanks for spotting this. :-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 8/12 pkg-infra: don't use DL_DIR as scratchpad for temporary VCS checkouts
In reply to Samuel MartinOn 06/03/14 11:25, Samuel Martin wrote:
[snip]
Quoted 16 lines
Quoted 12 lines
+
support/download/cvs rawname="${3}" · line 32
Quoted 11 lines
support/download/cvs +7
-3+cd "${BUILD_DIR}" +# Remove leftovers from a previous failed run +rm -rf "${repodir}" "${output}.tmp" + ${CVS} -z3 -d":pserver:anonymous@${repo}" \ - co -d "${basename}" -r ":${rev}" -P "${rawname}" -tar czf "${output}" "${basename}" -rm -rf "${basename}" + co -d "${repodir}" -r ":${rev}" -P "${rawname}" +tar czf "${output}.tmp" "${repodir}" +mv "${output}.tmp" "${output}"You move the temporary repository in the package
build_dir. Maybe the temporary output archive should be generated there as well, so that the mv command will only move something valid.
No! If the DL_DIR is on a different mount point than the build dir (e.g.
on a shared network drive), then the move will become a copy so non-atomic. We want to avoid that!
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH v3 9/12 pkg-infra: also set PKGDIR for the download step
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
This will be needed to get the hash file, to check the downloaded files.
--- package/pkg-generic.mk | 1 + 1 file changed, 1 insertion(+)
package/pkg-generic.mk +1 -0
@@ -534,6 +534,7 @@ $$($(2)_TARGET_PATCH): RAWNAME=$(patsubst host-%,%,$(1)) $$($(2)_TARGET_PATCH): PKGDIR=$(pkgdir) $$($(2)_TARGET_EXTRACT): PKG=$(2) $$($(2)_TARGET_SOURCE): PKG=$(2) +$$($(2)_TARGET_SOURCE): PKGDIR=$(pkgdir) $$($(2)_TARGET_DIRCLEAN): PKG=$(2) # Compute the name of the Kconfig option that correspond to the
Signature · 1 line
1.8.3.2
PATCH v3 10/12 pkg-infra: add possiblity to check downloaded files against known hashes
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Some of the packages that Buildroot might build are sensitive packages, related to security: openssl, dropbear, ca-certificates...
Some of those packages are downloaded over plain http, because there is no way to get them over a secure channel, such as https.
In these dark times of pervasive surveillance, the potential for harm that a tampered-with package could generate, we may want to check the integrity of those sensitive packages.
So, each package may now provide a list of hashes for all files that needs to be downloaded, and Buildroot will just fail if any downloaded file does not match its known hash, in which case it is removed.
Hashes can be any of the md5, sha1 or sha2 variants, and will be checked even if the file was pre-downloaded.
---
Note: this is not a bullet-proof solution, since Buildroot may itself be compromised. But since we do sign our releases, then we secure the list of hashes at the same time. Only random snapshots from the repository may be at risk of tampering, although this is highly doubtfull, given how git stores its data.
--- package/pkg-download.mk | 27 ++++++++++------ support/download/check-hash | 76 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 94 insertions(+), 9 deletions(-) create mode 100755 support/download/check-hash
package/pkg-download.mk +18 -9
@@ -58,6 +58,17 @@ domainseparator=$(if $(1),$(1),/) # github(user,package,version): returns site of github repository github = https://github.com/$(1)/$(2)/tarball/$(3) +# Helper for checking a tarball's checksum +# If the hash does not match, remove the incorrect file +# $(1): the path to the file with the hashes +# $(2): the full path to the file to check +define VERIFY_HASH + if ! support/download/check-hash $(1) $(2); then \ + rm -f $(2); \ + exit 1; \ + fi +endef + ################################################################################ # The DOWNLOAD_* helpers are in charge of getting a working copy # of the source repository for their corresponding SCM,
@@ -147,7 +158,8 @@ endef # to prepend the path with a slash: scp://[user@]host:/absolutepath define DOWNLOAD_SCP test -e $(DL_DIR)/$(2) || \ - $(SCP) '$(call stripurischeme,$(call qstrip,$(1)))' $(DL_DIR)/$(2) + $(SCP) '$(call stripurischeme,$(call qstrip,$(1)))' $(DL_DIR)/$(2) && \ + $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2)) endef define SOURCE_CHECK_SCP
@@ -175,15 +187,11 @@ define SHOW_EXTERNAL_DEPS_HG echo $($(PKG)_SOURCE) endef -# Download a file using wget. Only download the file if it doesn't -# already exist in the download directory. If the download fails, -# remove the file (because wget -O creates a 0-byte file even if the -# download fails). To handle an interrupted download as well, download -# to a temporary file first. The temporary file will be overwritten -# the next time the download is tried. + define DOWNLOAD_WGET test -e $(DL_DIR)/$(2) || \ - $(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2) + $(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2) && \ + $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2)) endef define SOURCE_CHECK_WGET
@@ -196,7 +204,8 @@ endef define DOWNLOAD_LOCALFILES test -e $(DL_DIR)/$(2) || \ - $(LOCALFILES) $(call stripurischeme,$(call qstrip,$(1))) $(DL_DIR) + $(LOCALFILES) $(call stripurischeme,$(call qstrip,$(1))) $(DL_DIR) && \ + $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2)) endef define SOURCE_CHECK_LOCALFILES
support/download/check-hash new file +76 -0
76 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH 10/12 pkg-infra: add possiblity to check downloaded files against known hashes
Quoted 16 lines · Yann E. MORIN
From: "Yann E. MORIN" <[email protected]>
Some of the packages that Buildroot might build are sensitive packages, related to security: openssl, dropbear, ca-certificates...
Some of those packages are downloaded over plain http, because there is no way to get them over a secure channel, such as https.
In these dark times of pervasive surveillance, the potential for harm that a tampered-with package could generate, we may want to check the integrity of those sensitive packages.
So, each package may now provide a list of hashes for all files that needs to be downloaded, and Buildroot will just fail if any downloaded file does not match its known hash, in which case it is removed.
Quoted 8 lines
Hashes can be any of the md5, sha1 or sha2 variants, and will be checked even if the file was pre-downloaded.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Baruch Siach <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc Gustavo Zacarias <[email protected]>
Regards,
Signature · 1 line
Samuel
PATCH v3 11/12 manual: add documentation about packages' hashes
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Although md5 is, for legacy resaons, a supported hash type, it is not documented on purpose, since it is now known to be weak.
--- docs/manual/adding-packages-directory.txt | 65 +++++++++++++++++++++++++++++++ support/download/check-hash | 1 + 2 files changed, 66 insertions(+)
docs/manual/adding-packages-directory.txt +65 -0
@@ -346,3 +346,68 @@ different way, using different infrastructures: Further formatting details: see xref:writing-rules-mk[the writing rules]. + +The +.hash+ file +~~~~~~~~~~~~~~~~ +[[adding-packages-hash]] + +Optionally, you can add a third file, named +libfoo.hash+, that contains +the hashes of the downloaded files for the +libfoo+ package. + +The hashes stored in that file are used to validate the integrity of the +downloaded files. + +The format for this file is one line for each file for which to check the +hash, each line being space-separated, with these three fields: + +* the type of hash, one of: +** +sha1+, +sha224+, +sha256+, +sha384+, +sha512+ +* the hash of the file: +** for +sha1+, 40 hexa-decimal characters +** for +sha224+, 56 hexa-decimal characters +** for +sha256+, 64 hexa-decimal characters +** for +sha384+, 96 hexa-decimal characters +** for +sha512+, 128 hexa-decimal characters +* the name of the file, without any directory component + +Lines starting with a +#+ sign are considered comments, and ignored. Empty +lines are ignored. + +There can be more than one hash for a single file, each of its own line. In +this case, all hashes must match. + +Ideally, the hashes stored in this file should match the hashes published by +upstream, eg, on their website, in the email announcement... If no such info +is available, then compute them yourself, and state so in a comment line +above the hashes. + +*Note:* the number of spaces does not matter, so one can use spaces to +properly align the different fields. + +The example below defines a +sha1+ and a +sha256+ published by upstream for +the main +libfoo-1.2.3.tar.bz2+ tarball, plus two locally-computed hashes, +a +sha256+ for a downloaded patch, and a +sha1+ for a downloaded binary blob: + +---- +# Hashes from: http://www.foosoftware.org/download/libfoo-1.2.3.tar.bz2.{sha1,sha256}: +sha1 486fb55c3efa71148fe07895fd713ea3a5ae343a libfoo-1.2.3.tar.bz2 +sha256 efc8103cc3bcb06bda6a781532d12701eb081ad83e8f90004b39ab81b65d4369 libfoo-1.2.3.tar.bz2 + +# No upstream hashes for the following: +sha256 ff52101fb90bbfc3fe9475e425688c660f46216d7e751c4bbdb1dc85cdccacb9 libfoo-fix-blabla.patch +sha1 2d608f3c318c6b7557d551a5a09314f03452f1a1 libfoo-data.bin +---- + +If the +.hash+ file is present, and there is one or more hashes for a +downloaded file, the hash(es) of the downloaded file must match the +hash(es) stored in the +.hash+ file. If one or more hashes do not match, +Buildroot considers this as an error, deletes the downloaded file, and +aborts. + +If the +.hash+ file is present, but there is no hash for a downloaded +file, no check is done for that downloaded file. If you set the +environment variable +BR2_ENFORCE_CHECK_HASH+ to a non-empty value, and +there is no hash for a downloaded file, Buildroot considers this as an +error, deletes the downloaded file, and aborts. + +If the +.hash+ file is missing, then no check is done at all.
support/download/check-hash +1 -0
@@ -22,6 +22,7 @@ check_one_hash() { _known="${2}" _file="${3}" + # Note: md5 is supported, but un-documented on-purpose. # Note: sha3 is not supported, since there is currently no implemetation # (the NIST has yet to publish the parameters). case "${_h}" in
Signature · 1 line
1.8.3.2
PATCH 11/12 manual: add documentation about packages' hashes
Yann, all,
Quoted 42 lines · Yann E. MORIN
From: "Yann E. MORIN" <[email protected]>
Although md5 is, for legacy resaons, a supported hash type, it is not documented on purpose, since it is now known to be weak.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Baruch Siach <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc Samuel Martin <[email protected]>--- docs/manual/adding-packages-directory.txt | 65 +++++++++++++++++++++++++++++++ support/download/check-hash | 1 + 2 files changed, 66 insertions(+)docs/manual/adding-packages-directory.txt +19
-0@@ -346,3 +346,68 @@ different way, using different infrastructures: Further formatting details: see xref:writing-rules-mk[the writing rules]. + +The +.hash+ file +~~~~~~~~~~~~~~~~ +[[adding-packages-hash]] + +Optionally, you can add a third file, named +libfoo.hash+, that contains +the hashes of the downloaded files for the +libfoo+ package. + +The hashes stored in that file are used to validate the integrity of the +downloaded files. + +The format for this file is one line for each file for which to check the +hash, each line being space-separated, with these three fields: + +* the type of hash, one of: +** +sha1+, +sha224+, +sha256+, +sha384+, +sha512+ +* the hash of the file: +** for +sha1+, 40 hexa-decimal characters +** for +sha224+, 56 hexa-decimal characters
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 377
Quoted 10 lines
docs/manual/adding-packages-directory.txt +10
-0+** for +sha256+, 64 hexa-decimal characters +** for +sha384+, 96 hexa-decimal characters +** for +sha512+, 128 hexa-decimal characters +* the name of the file, without any directory component + +Lines starting with a +#+ sign are considered comments, and ignored. Empty +lines are ignored. + +There can be more than one hash for a single file, each of its own line. In +this case, all hashes must match.
Maybe a note explaining why it's better to provide more than 1 hash for a file could be added.
Quoted 49 lines
docs/manual/adding-packages-directory.txt +36
-0+ +Ideally, the hashes stored in this file should match the hashes published by +upstream, eg, on their website, in the email announcement... If no such info +is available, then compute them yourself, and state so in a comment line +above the hashes. + +*Note:* the number of spaces does not matter, so one can use spaces to +properly align the different fields. + +The example below defines a +sha1+ and a +sha256+ published by upstream for +the main +libfoo-1.2.3.tar.bz2+ tarball, plus two locally-computed hashes, +a +sha256+ for a downloaded patch, and a +sha1+ for a downloaded binary blob: + +---- +# Hashes from: http://www.foosoftware.org/download/libfoo-1.2.3.tar.bz2.{sha1,sha256}: +sha1 486fb55c3efa71148fe07895fd713ea3a5ae343a libfoo-1.2.3.tar.bz2 +sha256 efc8103cc3bcb06bda6a781532d12701eb081ad83e8f90004b39ab81b65d4369 libfoo-1.2.3.tar.bz2 + +# No upstream hashes for the following: +sha256 ff52101fb90bbfc3fe9475e425688c660f46216d7e751c4bbdb1dc85cdccacb9 libfoo-fix-blabla.patch +sha1 2d608f3c318c6b7557d551a5a09314f03452f1a1 libfoo-data.bin +---- + +If the +.hash+ file is present, and there is one or more hashes for a +downloaded file, the hash(es) of the downloaded file must match the +hash(es) stored in the +.hash+ file. If one or more hashes do not match, +Buildroot considers this as an error, deletes the downloaded file, and +aborts. + +If the +.hash+ file is present, but there is no hash for a downloaded +file, no check is done for that downloaded file. If you set the +environment variable +BR2_ENFORCE_CHECK_HASH+ to a non-empty value, and +there is no hash for a downloaded file, Buildroot considers this as an +error, deletes the downloaded file, and aborts. + +If the +.hash+ file is missing, then no check is done at all.support/download/check-hash +1
-1@@ -22,6 +22,7 @@ check_one_hash() { _known="${2}" _file="${3}" + # Note: md5 is supported, but un-documented on-purpose. # Note: sha3 is not supported, since there is currently no implemetation # (the NIST has yet to publish the parameters). case "${_h}" in --
Quoted 2 lines
1.8.3.2
Regards,
Signature · 1 line
Samuel
PATCH v3 11/12 manual: add documentation about packages' hashes
Samuel, All,
On 2014-03-06 11:56 +0100, Samuel Martin spake thusly:
Quoted 1 line · Yann E. MORIN
[--SNIP--]
Quoted 32 lines
Quoted 32 lines
docs/manual/adding-packages-directory.txt +24
-0@@ -346,3 +346,68 @@ different way, using different infrastructures: Further formatting details: see xref:writing-rules-mk[the writing rules]. + +The +.hash+ file +~~~~~~~~~~~~~~~~ +[[adding-packages-hash]] + +Optionally, you can add a third file, named +libfoo.hash+, that contains +the hashes of the downloaded files for the +libfoo+ package. + +The hashes stored in that file are used to validate the integrity of the +downloaded files. + +The format for this file is one line for each file for which to check the +hash, each line being space-separated, with these three fields: + +* the type of hash, one of: +** +sha1+, +sha224+, +sha256+, +sha384+, +sha512+ +* the hash of the file: +** for +sha1+, 40 hexa-decimal characters +** for +sha224+, 56 hexa-decimal characters +** for +sha256+, 64 hexa-decimal characters +** for +sha384+, 96 hexa-decimal characters +** for +sha512+, 128 hexa-decimal characters +* the name of the file, without any directory component +
Quoted 8 lines
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 377
Quoted 5 lines
docs/manual/adding-packages-directory.txt +5
-0+Lines starting with a +#+ sign are considered comments, and ignored. Empty +lines are ignored. + +There can be more than one hash for a single file, each of its own line. In +this case, all hashes must match.Maybe a note explaining why it's better to provide more than 1 hash for a file could be added.
As I said to Gustavo on IRC, I'd prefer we only document the format of the .hash file in the manual, not define any policy. Ie. I don't think it is sensible to say something like:
For security considerations, adding more than one hash will ower the risk of collusions if more than one hash type is provided.
However, we can say, and I will add, something like:
If upstream provides more than one type of hash (say, sha1 and sha512), then it is best to add all those hashes in the .hash file.
This is more policy-neutral.
We have to keep in mind that this feature is a first-level stop-gap for security-conscious people, but in no way a security measure. Those security-conscious users are encouraged to check the downloaded files using a side-band channel (eg. manually checking signatures and so on...)
Buildroot itself can't check signatures: if the user does not have a chain-of-trust, from his own key and up to the signer's key, there is no point in checking the signature in the first place. We can't expect all users to have such a chain-of-trust, even less that all have a PGP key.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 11/12 manual: add documentation about packages' hashes
In reply to Yann E. MORINHi Yann,
Quoted 25 lines · Yann E. MORIN
From: "Yann E. MORIN" <[email protected]>
Although md5 is, for legacy resaons, a supported hash type, it is not documented on purpose, since it is now known to be weak.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Baruch Siach <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc Samuel Martin <[email protected]>--- docs/manual/adding-packages-directory.txt | 65 +++++++++++++++++++++++++++++++ support/download/check-hash | 1 + 2 files changed, 66 insertions(+)docs/manual/adding-packages-directory.txt +2
-0@@ -346,3 +346,68 @@ different way, using different infrastructures: Further formatting details: see xref:writing-rules-mk[the writing rules]. + +The +.hash+ file
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 360
Quoted 10 lines
docs/manual/adding-packages-directory.txt +10
-0+~~~~~~~~~~~~~~~~ +[[adding-packages-hash]] + +Optionally, you can add a third file, named +libfoo.hash+, that contains +the hashes of the downloaded files for the +libfoo+ package. + +The hashes stored in that file are used to validate the integrity of the +downloaded files. + +The format for this file is one line for each file for which to check the
I would say: The format _of_ this file
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 370
Quoted 10 lines
docs/manual/adding-packages-directory.txt +10
-0+hash, each line being space-separated, with these three fields: + +* the type of hash, one of: +** +sha1+, +sha224+, +sha256+, +sha384+, +sha512+ +* the hash of the file: +** for +sha1+, 40 hexa-decimal characters +** for +sha224+, 56 hexa-decimal characters +** for +sha256+, 64 hexa-decimal characters +** for +sha384+, 96 hexa-decimal characters +** for +sha512+, 128 hexa-decimal characters
hexadecimal is usually written in one word in English, for example:
http://en.wikipedia.org/wiki/Hexadecimal
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 376
Quoted 6 lines
docs/manual/adding-packages-directory.txt +6
-0+* the name of the file, without any directory component + +Lines starting with a +#+ sign are considered comments, and ignored. Empty +lines are ignored. + +There can be more than one hash for a single file, each of its own line. In
I think you can say:
..., each on its own line
..., each on a line of its own
but not 'each of its own line'
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 380
Quoted 4 lines
docs/manual/adding-packages-directory.txt +4
-0+this case, all hashes must match. + +Ideally, the hashes stored in this file should match the hashes published by +upstream, eg, on their website, in the email announcement... If no such info
'eg' should be written with dots (e.g. , see
http://en.wikipedia.org/wiki/Exempli_gratia#exempli_gratia ) e-mail (with dash)
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 382
Quoted 2 lines
docs/manual/adding-packages-directory.txt +2
-0+is available, then compute them yourself, and state so in a comment line +above the hashes.
I would write for this last sentence:
If upstream does not provide hashes, then compute them yourself, and mention this in a comment line above the hashes.
Quoted 9 lines
docs/manual/adding-packages-directory.txt +9
-0+ +*Note:* the number of spaces does not matter, so one can use spaces to +properly align the different fields. + +The example below defines a +sha1+ and a +sha256+ published by upstream for +the main +libfoo-1.2.3.tar.bz2+ tarball, plus two locally-computed hashes, +a +sha256+ for a downloaded patch, and a +sha1+ for a downloaded binary blob: + +----
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 401
Quoted 10 lines
docs/manual/adding-packages-directory.txt +10
-0+# Hashes from: http://www.foosoftware.org/download/libfoo-1.2.3.tar.bz2.{sha1,sha256}: +sha1 486fb55c3efa71148fe07895fd713ea3a5ae343a libfoo-1.2.3.tar.bz2 +sha256 efc8103cc3bcb06bda6a781532d12701eb081ad83e8f90004b39ab81b65d4369 libfoo-1.2.3.tar.bz2 + +# No upstream hashes for the following: +sha256 ff52101fb90bbfc3fe9475e425688c660f46216d7e751c4bbdb1dc85cdccacb9 libfoo-fix-blabla.patch +sha1 2d608f3c318c6b7557d551a5a09314f03452f1a1 libfoo-data.bin +---- + +If the +.hash+ file is present, and there is one or more hashes for a
'there are', but I would rather write:
..., and it contains one or more hashes for a downloaded file, ...
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 403
Quoted 2 lines
docs/manual/adding-packages-directory.txt +2
-0+downloaded file, the hash(es) of the downloaded file must match the +hash(es) stored in the +.hash+ file.
Maybe:
..., the hash(es) computed by Buildroot (after download) must match the hash(es) stored in the +.hash+ file.
Quoted 2 lines
If one or more hashes do not match,
+Buildroot considers this as an error, deletes the downloaded file, and
considers this an error ('as' is technically allowed but not commonly added, in my experience).
docs/manual/adding-packages-directory.txt different way, using different infrastructures:
Quoted 4 lines
docs/manual/adding-packages-directory.txt +4
-0+aborts. + +If the +.hash+ file is present, but there is no hash for a downloaded +file, no check is done for that downloaded file.
Similarly as above, I would write: "but it does not contain a hash for a downloaded file".
Also I think the second occurrence of 'downloaded' is redundant: 'for that file'.
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 410
Quoted 3 lines
If you set the
docs/manual/adding-packages-directory.txt +2
-0+environment variable +BR2_ENFORCE_CHECK_HASH+ to a non-empty value, and +there is no hash for a downloaded file, Buildroot considers this as an
without 'as'
support/download/check-hash check_one_hash() { · line 25
Quoted 12 lines
docs/manual/adding-packages-directory.txt +3
-0+error, deletes the downloaded file, and aborts. + +If the +.hash+ file is missing, then no check is done at all.support/download/check-hash +1
-0@@ -22,6 +22,7 @@ check_one_hash() { _known="${2}" _file="${3}" + # Note: md5 is supported, but un-documented on-purpose.
undocumented (see http://en.wikipedia.org/wiki/Undocumented_feature)
on purpose (see http://www.thefreedictionary.com/on+purpose)
Quoted 5 lines
support/download/check-hash +0
-1# Note: sha3 is not supported, since there is currently no implemetation # (the NIST has yet to publish the parameters). case "${_h}" in --1.8.3.2
Best regards,
Thomas
PATCH v3 11/12 manual: add documentation about packages' hashes
Thomas, All,
On 2014-03-06 18:14 +0100, Thomas De Schampheleire spake thusly:
Quoted 1 line · Yann E. MORIN
[--SNIP--]
Quoted 4 lines
docs/manual/adding-packages-directory.txt
Quoted 4 lines
docs/manual/adding-packages-directory.txt +0
-0
[--SNIP--]
Quoted 11 lines
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 376
Quoted 6 lines
docs/manual/adding-packages-directory.txt +6
-0+* the name of the file, without any directory component + +Lines starting with a +#+ sign are considered comments, and ignored. Empty +lines are ignored. + +There can be more than one hash for a single file, each of its own line. InI think you can say:
..., each on its own line
..., each on a line of its own
but not 'each of its own line'
Funny how one reads what one thinks he has written, and not what was actually written. Of course, I meant 'on', not 'of'.
Quoted 9 lines
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 380
Quoted 4 lines
docs/manual/adding-packages-directory.txt +4
-0+this case, all hashes must match. + +Ideally, the hashes stored in this file should match the hashes published by +upstream, eg, on their website, in the email announcement... If no such info'eg' should be written with dots (e.g. , see
http://en.wikipedia.org/wiki/Exempli_gratia#exempli_gratia ) e-mail (with dash)
Quoted 5 lines
docs/manual/adding-packages-directory.txt different way, using different infrastructures: · line 382
Quoted 2 lines
docs/manual/adding-packages-directory.txt +2
-0+is available, then compute them yourself, and state so in a comment line +above the hashes.I would write for this last sentence:
If upstream does not provide hashes, [...]
Furthermore, since there is no hash, it can be plural. I now wrote:
If upstream does not provide any hash, [...]
All applied. Thank you!
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 12/12 package/ca-certificates: add tarball's hashes
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
ca-certificates contains sensitive security-related information, and we want to ensure the archive that we download has not been compromised.
Add the sha1 and sha256 hashes from Debian's packaging.
--- package/ca-certificates/ca-certificates.hash | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 package/ca-certificates/ca-certificates.hash
package/ca-certificates/ca-certificates.hash new file +3 -0
@@ -0,0 +1,3 @@ +# hashes from: $(CA_CERTIFICATES_SITE)/ca-certificates_20130906.dsc : +sha1 7f197c1bf7c7fc82e9f8f2fec6d8cc65f6a6187b ca-certificates_20130906.tar.gz +sha256 dd10520091d469e95e11e5fafb7422d3be0a66071984d09009ed3e0232cb277d ca-certificates_20130906.tar.gz
Signature · 1 line
1.8.3.2