Enable hash checking for git downloads

Since our git downloads are reproducible since a while now, we can actually check the hash of git downloads. This does however still require a number of fixes first.

Patch 1 fixes the case when tar is compiled to generated Posix format tarballs.

Patch 2 is just a cleanup. This patch is optional.

Patch 3-5 fix incorrect hash files, which weren't noticed before because hashes weren't checked.

Patch 6-8 adds a hash for packages that actually have a hash file but set the hash type to 'none'. These patches are optional.

Patches 9-15 add exclusions for the different packages with CUSTOM_GIT configuration.

Patch 16, finally, enables the hash check.

Regards,
Arnout

[PATCH 01/16] download/git: create GNU format tar files [PATCH 02/16] aer-inject: remove redundant _SITE_METHOD [PATCH 03/16] fmc: correct hash file
[PATCH 04/16] linux-firmware: correct hash
[PATCH 05/16] squashfs: correct hash
[PATCH 06/16] ubus: add hash
[PATCH 07/16] uhttpd: add hash
[PATCH 08/16] vboot-utils: add hash
[PATCH 09/16] linux: exclude from hash check except for latest [PATCH 10/16] linux-headers: rework hash exclusion
[PATCH 11/16] uboot: exclude from hash check except for latest [PATCH 12/16] barebox: exclude from hash check except for latest [PATCH 13/16] at91bootstrap3: exclude from hash when downloading from [PATCH 14/16] mxs-bootlets: exclude from hash when downloading from [PATCH 15/16] arm-trusted-firmware: exclude from hash when [PATCH 16/16] pkg-download: enable hash check for git downloads

On most distros, the tar format defaults to GNU. However, at build time the default format may be changed to posix. Also, future versions of tar will default to posix.

Since we want the tarballs created by the git download method to be reproducible (so their hash can be checked), we should explicitly specify the format. Since existing tarballs on sources.buildroot.org use the GNU format, and also the existing hashes in the *.hash files are based on GNU format tarballs, we use the GNU format.

In addition, the Posix format encodes atime and ctime as well as mtime, but tar offers no option like --mtime to override them. In the GNU format, atime and ctime are only encoded if the --incremental option is given.

---
Peter, can you test if this solves the issue for you?

---
 support/download/git | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

index 7921411835..056057c700 100755
@@ -94,6 +94,8 @@ popd >/dev/null
 # Generate the archive, sort with the C locale so that it is reproducible
 find "${basename}" -not -type d >"${basename}.list"
 LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
-tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" \
+# Create GNU-format tarballs, since that's the format of the tarballs on
+# sources.buildroot.org and used in the *.hash files
+tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
          -T "${basename}.list.sorted" >"${output}.tar"
 gzip -n <"${output}.tar" >"${output}"

2.11.0

Hello,

Applied to master, thanks. We require tar 1.17 at least, and I checked that tar 1.17 supports --format=gnu.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Committed to 2017.02.x, thanks.

Bye, Peter Korsgaard

The _SITE has a git:// URL, so _SITE_METHOD is not needed.

---
 package/aer-inject/aer-inject.mk | 1 -
 1 file changed, 1 deletion(-)

index 69189b1668..e00570e272 100644
@@ -6,7 +6,6 @@
 
 AER_INJECT_VERSION = 9bd5e2c7886fca72f139cd8402488a2235957d41
 AER_INJECT_SITE = git://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git
-AER_INJECT_SITE_METHOD = git
 AER_INJECT_LICENSE = GPLv2
 AER_INJECT_LICENSE_FILES = README
 AER_INJECT_DEPENDENCIES = host-flex host-bison

2.11.0

It had sha256sum instead of sha256. This wasn't noticed before since hashes for git downloads are not checked.

---
 package/fmc/fmc.hash | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index 2402831855..13d9d77ee5 100644
@@ -1,2 +1,2 @@
-# Locally Computed
-sha256sum	a91e0c9b7c7f238634c64a755c05671f33f2acdb6ae2d09cad4d683b364ee8e4	fmc-fsl-sdk-v2.0.tar.gz
+# Locally calculated
+sha256	a91e0c9b7c7f238634c64a755c05671f33f2acdb6ae2d09cad4d683b364ee8e4	fmc-fsl-sdk-v2.0.tar.gz

2.11.0

Arnout,

Matthew L Weber / Pr Software Engineer
Airborne Information Systems / Security Systems and Software / Secure Platforms
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
www.rockwellcollins.com

Note: Any Export License Required Information and License Restricted
Third Party Intellectual Property (TPIP) content must be encrypted and
sent to matthew.weber at corp.rockwellcollins.com.

Committed to 2017.02.x, thanks.

Bye, Peter Korsgaard

When linux-firmware was bumped to 6d3bc8886 in commit 3ff5896ff, a hash file was added but the hash was wrong. This wasn't noticed before since hashes are not checked for git downloads.

---
 package/linux-firmware/linux-firmware.hash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index e8ee2b72a1..164603947e 100644
@@ -1,2 +1,2 @@
 # Locally calculated
-sha256 256378564e36b8f222adb8fbe746529e73413c6cbe2b0e5574f41dfd62d92577 linux-firmware-6d3bc8886517d171068fd1263176b8b5c51df204.tar.gz
+sha256 1cd14477a392fe8c7c44e62623b85466df90ed72a55ae213b637e1c4335739fd linux-firmware-6d3bc8886517d171068fd1263176b8b5c51df204.tar.gz

2.11.0

Committed to 2017.02.x, thanks.

Bye, Peter Korsgaard

When squashfs was bumped to 3de1687d in commit ee90313c64, the hash file was not updated. This wasn't noticed before since hashes are not checked for git downloads.

---
 package/squashfs/squashfs.hash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index eb456be364..74fdb44354 100644
@@ -1,2 +1,2 @@
 # Locally computed
-sha256	e807202ab4dcfeea214df49d3a728b9a85644a7e7262fc6fcd875a0def99d4ed	squashfs-9c1db6d13a51a2e009f0027ef336ce03624eac0d.tar.gz
+sha256	2c49392816615b73bf5d6cdce9776cb66a01f5a1494b1d9af580667e32bab001  squashfs-3de1687d7432ea9b302c2db9521996f506c140a3.tar.gz

2.11.0

Committed to 2017.02.x, thanks.

Bye, Peter Korsgaard

ubus had a hash file, but is says "none" because it is a git download. Since our git downloads are not reproducible, we can actually add a hash.

---
 package/ubus/ubus.hash | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index af9b652b0e..4e6a755c7f 100644
@@ -1,2 +1,2 @@
-# No hash for this git snapshot
-none  xxx  ubus-259450f414d8c9ee41896e8e6d6bc57ec00e2b63.tar.gz
+# Locally calculated
+sha256  0dbf75c2103f3f567b9bf1bb719ea9776d18401bbf2dfd8bed2c2ebbdd1021da  ubus-259450f414d8c9ee41896e8e6d6bc57ec00e2b63.tar.gz

2.11.0

Hello,

I guess you meant "are *now* reproducible", right? Ditto for the next commits. No need to resend just for that, it can be fixed up when applying.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Right. Looks like my fingers were quicker than my brains again.

Ah, the joys of 'git commit -c HEAD' :-)

Regards,
Arnout

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

uhttpd had a hash file, but is says "none" because it is a git download. Since our git downloads are not reproducible, we can actually add a hash.

---
 package/uhttpd/uhttpd.hash | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index 59e54ad49a..bfeb5571c5 100644
@@ -1,2 +1,2 @@
-# No hash for this git snapshot
-none	xxx	uhttpd-59e0c739634f46a164d939e54416287b91ff8a9b.tar.gz
+# Locally calculated
+sha256 b01aec584daaa367b85eddd950da27e3a41bcb265507cf42f79ff2ebcf0972d4  uhttpd-59e0c739634f46a164d939e54416287b91ff8a9b.tar.gz

2.11.0

vboot-utils had a hash file, but is says "none" because it is a git download. Since our git downloads are not reproducible, we can actually add a hash.

---
 package/vboot-utils/vboot-utils.hash | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index e16142389b..9a5b334324 100644
@@ -1,2 +1,2 @@
-# Git shapshot
-none	xxx	bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz
+# Locally computed
+sha256  e119782a374655117e3d9a4e667b0056c76961c4593ba907f860d1310f6fbc2a  vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz

2.11.0

We don't have a hash for any linux version. We currently also don't have a hash for the latest version, but if we ever add a hash, it will only be for the latest version.

This simplifies the code a little because soon we will add hash checks for git as well, so we also need an exclusion in that case. It's much easier to always exclude except in the single case where we do have a hash.

---
 linux/linux.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

index 7f4432e7b1..a52083f00a 100644
@@ -20,7 +20,6 @@ ifeq ($(BR2_LINUX_KERNEL_CUSTOM_TARBALL),y)
 LINUX_TARBALL = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION))
 LINUX_SITE = $(patsubst %/,%,$(dir $(LINUX_TARBALL)))
 LINUX_SOURCE = $(notdir $(LINUX_TARBALL))
-BR_NO_CHECK_HASH_FOR += $(LINUX_SOURCE)
 else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_GIT),y)
 LINUX_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
 LINUX_SITE_METHOD = git

@@ -32,9 +31,6 @@ LINUX_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
 LINUX_SITE_METHOD = svn
 else
 LINUX_SOURCE = linux-$(LINUX_VERSION).tar.xz
-ifeq ($(BR2_LINUX_KERNEL_CUSTOM_VERSION),y)
-BR_NO_CHECK_HASH_FOR += $(LINUX_SOURCE)
-endif
 # In X.Y.Z, get X and Y. We replace dots and dashes by spaces in order
 # to use the $(word) function. We support versions such as 4.0, 3.1,
 # 2.6.32, 2.6.32-rc1, 3.0-rc6, etc.

@@ -51,6 +47,10 @@ LINUX_SITE := $(LINUX_SITE)/testing
 endif # -rc
 endif
 
+ifeq ($(BR2_LINUX_KERNEL)$(BR2_LINUX_KERNEL_LATEST_VERSION),y)
+BR_NO_CHECK_HASH_FOR += $(LINUX_SOURCE)
+endif
+
 LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))
 
 # We rely on the generic package infrastructure to download and apply

2.11.0

Although we currently don't have a .hash file for linux-headers, there already are exclusions for the BR2_KERNEL_HEADERS_AS_KERNEL case (copied from linux.mk). However, there is no exclusion for the BR2_KERNEL_HEADERS_VERSION case.

For the BR2_KERNEL_HEADERS_AS_KERNEL case, the exclusion is actually not needed. Indeed, KERNEL_HEADERS_SOURCE is computed to be the same value as LINUX_SOURCE, and linux.mk already adds LINUX_SOURCE to BR_NO_CHECK_HASH_FOR.

For the other cases, we should exclude the BR2_KERNEL_HEADERS_VERSION case because there the user supplies the version so it can't be included in the .hash file.

---
 package/linux-headers/linux-headers.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

index 8f339cfec5..d83b9b0b02 100644
@@ -16,7 +16,6 @@ ifeq ($(BR2_LINUX_KERNEL_CUSTOM_TARBALL),y)
 LINUX_HEADERS_TARBALL = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION))
 LINUX_HEADERS_SITE = $(patsubst %/,%,$(dir $(LINUX_HEADERS_TARBALL)))
 LINUX_HEADERS_SOURCE = $(notdir $(LINUX_HEADERS_TARBALL))
-BR_NO_CHECK_HASH_FOR += $(LINUX_HEADERS_SOURCE)
 else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_GIT),y)
 LINUX_HEADERS_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
 LINUX_HEADERS_SITE_METHOD = git

@@ -29,9 +28,6 @@ LINUX_HEADERS_SITE_METHOD = hg
 LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.gz
 else
 LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
-ifeq ($(BR2_LINUX_KERNEL_CUSTOM_VERSION),y)
-BR_NO_CHECK_HASH_FOR += $(LINUX_HEADERS_SOURCE)
-endif
 # In X.Y.Z, get X and Y. We replace dots and dashes by spaces in order
 # to use the $(word) function. We support versions such as 4.0, 3.1,
 # 2.6.32, 2.6.32-rc1, 3.0-rc6, etc.

@@ -80,6 +76,10 @@ LINUX_HEADERS_SITE = $(BR2_KERNEL_MIRROR)/linux/kernel/v4.x
 endif
 LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
 
+ifeq ($(BR2_KERNEL_HEADERS_VERSION),y)
+BR_NO_CHECK_HASH_FOR += $(LINUX_HEADERS_SOURCE)
+endif
+
 endif # ! BR2_KERNEL_HEADERS_AS_KERNEL
 
 LINUX_HEADERS_LICENSE = GPLv2

2.11.0

Instead of excluding the hash specifically for the custom tarball and custom version cases, exclude it always except in the one case where we do have a hash: the latest version.

This simplifies the code a little because soon we will add hash checks for git as well, so we also need an exclusion in that case.

---
 boot/uboot/uboot.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

index 1e22eaa8ea..705863dd6a 100644
@@ -17,7 +17,6 @@ ifeq ($(UBOOT_VERSION),custom)
 UBOOT_TARBALL = $(call qstrip,$(BR2_TARGET_UBOOT_CUSTOM_TARBALL_LOCATION))
 UBOOT_SITE = $(patsubst %/,%,$(dir $(UBOOT_TARBALL)))
 UBOOT_SOURCE = $(notdir $(UBOOT_TARBALL))
-BR_NO_CHECK_HASH_FOR += $(UBOOT_SOURCE)
 else ifeq ($(BR2_TARGET_UBOOT_CUSTOM_GIT),y)
 UBOOT_SITE = $(call qstrip,$(BR2_TARGET_UBOOT_CUSTOM_REPO_URL))
 UBOOT_SITE_METHOD = git

@@ -31,9 +30,10 @@ else
 # Handle stable official U-Boot versions
 UBOOT_SITE = ftp://ftp.denx.de/pub/u-boot
 UBOOT_SOURCE = u-boot-$(UBOOT_VERSION).tar.bz2
-ifeq ($(BR2_TARGET_UBOOT_CUSTOM_VERSION),y)
-BR_NO_CHECK_HASH_FOR += $(UBOOT_SOURCE)
 endif
+
+ifeq ($(BR2_TARGET_UBOOT)$(BR2_TARGET_UBOOT_LATEST_VERSION),y)
+BR_NO_CHECK_HASH_FOR += $(UBOOT_SOURCE)
 endif
 
 ifeq ($(BR2_TARGET_UBOOT_FORMAT_BIN),y)

2.11.0

Instead of excluding the hash specifically for the custom tarball and custom version cases, exclude it always except in the one case where we do have a hash: the latest version.

This simplifies the code a little because soon we will add hash checks for git as well, so we also need an exclusion in that case.

It is not needed to exclude it twice for barebox and barebox-aux, because they use the same source tarball and barebox-aux can only be enabled if barebox is enabled. So simplify even further by pulling the exclusion logic out of inner-barebox-package.

---
 boot/barebox/barebox.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

index 7a110d2145..dc19224c76 100644
@@ -22,7 +22,6 @@ ifeq ($$($(1)_VERSION),custom)
 $(1)_TARBALL = $$(call qstrip,$$(BR2_TARGET_BAREBOX_CUSTOM_TARBALL_LOCATION))
 $(1)_SITE = $$(patsubst %/,%,$$(dir $$($(1)_TARBALL)))
 $(1)_SOURCE = $$(notdir $$($(1)_TARBALL))
-BR_NO_CHECK_HASH_FOR += $$($(1)_SOURCE)
 else ifeq ($$(BR2_TARGET_BAREBOX_CUSTOM_GIT),y)
 $(1)_SITE = $$(call qstrip,$$(BR2_TARGET_BAREBOX_CUSTOM_GIT_REPO_URL))
 $(1)_SITE_METHOD = git

@@ -30,9 +29,6 @@ else
 # Handle stable official Barebox versions
 $(1)_SOURCE = barebox-$$($(1)_VERSION).tar.bz2
 $(1)_SITE = http://www.barebox.org/download
-ifeq ($$(BR2_TARGET_BAREBOX_CUSTOM_VERSION),y)
-BR_NO_CHECK_HASH_FOR += $$($(1)_SOURCE)
-endif
 endif
 
 $(1)_DEPENDENCIES = host-lzop

@@ -155,3 +151,7 @@ barebox-package=$(call inner-barebox-package,$(call UPPERCASE,$(pkgname)))
 
 include boot/barebox/barebox/barebox.mk
 include boot/barebox/barebox-aux/barebox-aux.mk
+
+ifeq ($(BR2_TARGET_BAREBOX)$(BR2_TARGET_BAREBOX_LATEST_VERSION),y)
+BR_NO_CHECK_HASH_FOR += $(BAREBOX_SOURCE)
+endif

2.11.0

Since we will enable hash checks for git downloads soon, the hash check for the custom git download should be disabled.

---
 boot/at91bootstrap3/at91bootstrap3.mk | 1 +
 1 file changed, 1 insertion(+)

index 32732f4758..685be5bbd7 100644
@@ -9,6 +9,7 @@ AT91BOOTSTRAP3_VERSION = $(call qstrip,$(BR2_TARGET_AT91BOOTSTRAP3_VERSION))
 ifeq ($(BR2_TARGET_AT91BOOTSTRAP3_CUSTOM_GIT),y)
 AT91BOOTSTRAP3_SITE = $(call qstrip,$(BR2_TARGET_AT91BOOTSTRAP3_CUSTOM_REPO_URL))
 AT91BOOTSTRAP3_SITE_METHOD = git
+BR_NO_CHECK_HASH_FOR += $(AT91BOOTSTRAP3_SOURCE)
 else
 AT91BOOTSTRAP3_SITE = $(call github,linux4sam,at91bootstrap,$(AT91BOOTSTRAP3_VERSION))
 endif

2.11.0

Since we will enable hash checks for git downloads soon, the hash check for the custom git download should be disabled.

---
Since mxs-bootlets is simpler, it is easier to do the exclusion explicitly for these two cases, and not use the more generic approach like for linux and uboot.

---
 boot/mxs-bootlets/mxs-bootlets.mk | 1 +
 1 file changed, 1 insertion(+)

index 17af6dc158..c7dd534ec0 100644
@@ -13,6 +13,7 @@ else ifeq ($(BR2_TARGET_MXS_BOOTLETS_CUSTOM_GIT),y)
 MXS_BOOTLETS_SITE = $(BR2_TARGET_MXS_BOOTLETS_CUSTOM_GIT_URL)
 MXS_BOOTLETS_SITE_METHOD = git
 MXS_BOOTLETS_VERSION = $(call qstrip,$(BR2_TARGET_MXS_BOOTLETS_CUSTOM_GIT_VERSION))
+BR_NO_CHECK_HASH_FOR += $(MXS_BOOTLETS_SOURCE)
 else
 MXS_BOOTLETS_VERSION = 10.12.01
 MXS_BOOTLETS_SITE = http://download.ossystems.com.br/bsp/freescale/source

2.11.0

Since we will enable hash checks for git downloads soon, the hash check for the custom git download should be disabled.

---
Since arm-trusted-firmware is simpler, it is easier to do the exclusion explicitly for these two cases, and not use the more generic approach like for linux and uboot.

---
 boot/arm-trusted-firmware/arm-trusted-firmware.mk | 1 +
 1 file changed, 1 insertion(+)

index 5acfab391f..b9fb3d84d3 100644
@@ -19,6 +19,7 @@ BR_NO_CHECK_HASH_FOR += $(ARM_TRUSTED_FIRMWARE_SOURCE)
 else ifeq ($(BR2_TARGET_ARM_TRUSTED_FIRMWARE_CUSTOM_GIT),y)
 ARM_TRUSTED_FIRMWARE_SITE = $(call qstrip,$(BR2_TARGET_ARM_TRUSTED_FIRMWARE_CUSTOM_REPO_URL))
 ARM_TRUSTED_FIRMWARE_SITE_METHOD = git
+BR_NO_CHECK_HASH_FOR += $(ARM_TRUSTED_FIRMWARE_SOURCE)
 else
 ARM_TRUSTED_FIRMWARE_SITE = $(call github,ARM-software,arm-trusted-firmware,$(ARM_TRUSTED_FIRMWARE_VERSION))
 endif

2.11.0

For a while already, our git downloaded tarballs are reproducible, so we can actually check the hash for them.

---
Tested 'make pkg-source' for all git-downloaded packages (with and without hash files).

---
 package/pkg-download.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index cfc550e77b..fbbc2d7391 100644
@@ -77,6 +77,7 @@ define DOWNLOAD_GIT
 	$(EXTRA_ENV) $(DL_WRAPPER) -b git \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		$(if $($(PKG)_GIT_SUBMODULES),-r) \
+		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
 		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \

@@ -225,7 +226,7 @@ define SOURCE_CHECK
 endef
 
 define DOWNLOAD_INNER
-	$(Q)$(if $(filter bzr cvs git hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
+	$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
 	if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
 		case "$(call geturischeme,$(BR2_PRIMARY_SITE))" in \
 			file) $(call $(3)_LOCALFILES,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \

2.11.0

Arnout,

----- Original Message -----

[snip]

fmc-source works fine for me but host-vboot-utils-source does not. Also arm-trusted-firmware-source falls back to sources.buildroot.net for me. See logs below.

git clean -ffdx && make defconfig
make fmc-source
----->8-----

Doing shallow clone
Cloning into 'fmc-fsl-sdk-v2.0'...
remote: Counting objects: 69, done.
remote: Compressing objects: 100% (65/65), done.
remote: Total 69 (delta 9), reused 26 (delta 2)
Receiving objects: 100% (69/69), 276.00 KiB | 138.00 KiB/s, done.
Resolving deltas: 100% (9/9), done.
Note: checking out 'a079d2c844edd85dff85a317a63198e7988bcd09'.
[snip git detached HEAD warning]
warning: refname 'fsl-sdk-v2.0' is ambiguous.
fmc-fsl-sdk-v2.0.tar.gz: OK (sha256: a91e0c9b7c7f238634c64a755c05671f33f2acdb6ae2d09cad4d683b364ee8e4) ----->8-----

make host-vboot-utils-source
----->8-----

Doing full clone
Cloning into 'vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656'...
remote: Sending approximately 38.99 MiB ...
remote: Total 21863 (delta 14206), reused 21863 (delta 14206)
Receiving objects: 100% (21863/21863), 38.99 MiB | 1.55 MiB/s, done.
Resolving deltas: 100% (14206/14206), done.
warning: refname 'bbdd62f9b030db7ad8eef789aaf58a7ff9a25656' is ambiguous.
[snip git warning]
ERROR: vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz has wrong sha256 hash:
ERROR: expected: e119782a374655117e3d9a4e667b0056c76961c4593ba907f860d1310f6fbc2a
ERROR: got : d95b64b1f1de4a3ffa5c2e446d7c8e92aa197aee10de24206b2ea2deb5a8b947
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
--2017-03-21 09:14:38-- http://sources.buildroot.net/vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 35645166 (34M) [application/x-gzip]
[snip long lines]
ERROR: vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz has wrong sha256 hash:
ERROR: expected: e119782a374655117e3d9a4e667b0056c76961c4593ba907f860d1310f6fbc2a
ERROR: got : 2c71c3d04b9397ccb4b18202ca83d507f227e1e39c2bab6c9be2c3859155a52b
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
----->8-----

git clean -ffdx
make arm_juno_defconfig
make arm-trusted-firmware-source
----->8-----
Doing shallow clone
Cloning into 'arm-trusted-firmware-v1.2'...
remote: Counting objects: 645, done.
remote: Compressing objects: 100% (550/550), done.
remote: Total 645 (delta 240), reused 288 (delta 46), pack-reused 0
Receiving objects: 100% (645/645), 1.96 MiB | 827.00 KiB/s, done.
Resolving deltas: 100% (240/240), done.
Note: checking out 'd0c104e1e1ad0102f0f4c70997b7ee6e6fbbe273'.
[snip git warning]
warning: refname 'v1.2' is ambiguous.
ERROR: arm-trusted-firmware-v1.2.tar.gz has wrong sha256 hash:
ERROR: expected: cbdd9b770ec1ab4933fc7f9f520daea5a364bb4dc964820fb017a0cf8c7df556
ERROR: got : 0eeba7a89028392a97fd64fc9052a36391af388ff716bd7c884cd50098a2f50c
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
--2017-03-21 09:04:41-- http://sources.buildroot.net/arm-trusted-firmware-v1.2.tar.gz
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1808700 (1,7M) [application/x-gzip]
Saving to: ?/tmp/git-hash/output/build/.arm-trusted-firmware-v1.2.tar.gz.clvxYW/output?
[snip long lines]
arm-trusted-firmware-v1.2.tar.gz: OK (sha256: cbdd9b770ec1ab4933fc7f9f520daea5a364bb4dc964820fb017a0cf8c7df556) arm-trusted-firmware-v1.2.tar.gz: OK (md5: fac2c08bd74337fec2e14a98fc9f748f)
----->8-----

cat /etc/os-release | grep VERSION= ; git --version ; tar --version | grep tar ----->8-----
VERSION="14.04.5 LTS, Trusty Tahr"
git version 2.11.0
tar (GNU tar) 1.27.1
----->8-----

git log --oneline --decorate -17
----->8-----
4b9c7077a6 (HEAD) pkg-download: enable hash check for git downloads 9183c9d31e arm-trusted-firmware: exclude from hash when downloading from git 6db2b0ba07 mxs-bootlets: exclude from hash when downloading from git 69f9d1b489 at91bootstrap3: exclude from hash when downloading from git 804053ad18 barebox: exclude from hash check except for latest version 989d9b77f5 uboot: exclude from hash check except for latest version 95d5d580ea linux-headers: rework hash exclusion
524a3f8aed linux: exclude from hash check except for latest version 8c3f8dc348 vboot-utils: add hash
bfe808e92d uhttpd: add hash
885bce3fec ubus: add hash
331d44fae2 squashfs: correct hash
7c4b32dfb7 linux-firmware: correct hash
d8e8a374e6 fmc: correct hash file
1a6e356b9d aer-inject: remove redundant _SITE_METHOD
0501fe2808 download/git: create GNU format tar files
1a83dda003 (upstream/master) package/ghostscript: new package ----->8-----

Best regards,
Ricardo

Hi Ricardo,

Thanks for testing!

On 21-03-17 13:19, Ricardo Martincoski wrote:
[snip]

This is worrisome - both your tarball and the one from sources.buildroot.net
differ from the one I have, and are also different between them. Well, the one on sources.buildroot.net can be explained: that tarball was created at a time that our git helper was not reproducible yet, so it will have the wrong hash.

Peter, could you do a refresh of all git-downloaded packages on
sources.buildroot.net? You can get them with

make printvars VARS="$(git grep -l 'git:\|METHOD = git' -- \*.mk | \
		sed 's%.*/\([^/]*\).mk%\1_SOURCE%g' | \
		tr 'a-z-\n' 'A-Z_ ')"

And that probably also explains why my hash is wrong. I downloaded the
vboot-utils tarball on Jan 7, 2016. However, the git archive creation was changed a couple of time and has only been stable since commit 4f87aa7db5ee of Oct 25. So I generated the hash based on the wrong tarball...

I'll have to recheck my patches that add hashes.

Hm, this one is a little bit annoying. arm-trusted-firmware-v1.2.tar.gz
actually does get added to BR_NO_CHECK_HASH_FOR. However, version v1.2 is the default version and it *does* have a hash. check-hash has the following code:

if [ ${nb_checks} -eq 0 ]; then

    case " ${BR_NO_CHECK_HASH_FOR} " in
    *" ${base} "*)
        # File explicitly has no hash
        exit 0
        ;;
    esac
    printf "ERROR: No hash found for %s\n" "${base}" >&2
    exit 3

fi

i.e. the hash check is only skipped when there is no hash in the hash file.

What happens here is that the tarball that is generated by github is slightly
different from the tarball that we generate. So using the hash of the github tarball to check the hash of our tarball is going to fail.

Yann, I think the solution is to move the BR_NO_CHECK_HASH_FOR out of the
${nb_checks}. Actually move it all the way to the beginning and skip the check entirely. Do you agree? The commit log that introduced it (8d2f4e6246) offers no hint as to why the check isn't skipped completely.

Note that this could also happen for e.g. U-Boot when downloading from git and
setting the version to 2017.03. It's not very likely, however; AFAICS arm-trusted-firmware is the only one which has a reasonable chance of ever hitting this issue.

Here it works because the tarball on sources.buildroot.net is the one from the
github helper.

Regards,
Arnout

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Hello,

So I've applied this one.

And all those ones.

Still needs to review those one carefully.

Thanks,

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Hello,

All those patches applied. Thanks!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com