toolchain: better handle hashes (branch yem/dl-hash-toolchains)

Hello All!

This series makes it so we can download custom external toolchains, and still have mandatory checks of hashes.

It also adds the long-awaited-for hashes for gcc and binutils.

Changes v6 -> v7:
- Minor fixes in commit messages (Arnout)
- Add Acks where appropriate (Arnout, Yann)
- Beautify the pattern in the check-hash script (Yann)

Changes v5 -> v6:
- Add patch 4/4 (uclibc: update hash for arc) (Arnout)
- Fix check-hash script so the pattern actually works (Arnout)

Changes v4 -> v5:
- drop patch #8, that errored-out when an excluded file did have one or more associated hashes (Arnout)
- typoes (Arnout)

Changes v3 -> v4:
- add missing fake-hash for uClibc eXtensa git snapshot; add uClibc daily snapshots to exclusion list (Arnout)
- drop using a magic value; use a list of excluded files (Arnout)

Changes v2 -> v3:
- add hashes for gcc and binutils
- fix a few more typoes

Changes v1 -> v2:
- fix a few typoes

Regards,
Yann E. MORIN.
----------------------------------------------------------------
Arnout Vandecappelle (1):
uclibc: update hash for arc

Yann E. MORIN (7):
package/gcc: add hashes
package/binutils: add hashes
package/uclibc: add missing hash for eXtensa support/download: add possibility to not fail on missing hash toolchain/external: ignore missing hash for custom downloaded toolchain package/uclibc: ignore missing hash for snapshots support/download: restore mandatory check of hashes

 package/binutils/binutils.hash                     |  7 +++++++
 package/gcc/gcc-final/gcc-final.hash               |  1 +
 package/gcc/gcc-initial/gcc-initial.hash           |  1 +
 package/gcc/gcc.hash                               | 10 ++++++++++
 package/pkg-download.mk                            |  3 +++
 package/uclibc/uclibc.hash                         |  9 ++++++---
 package/uclibc/uclibc.mk                           |  1 +
 support/download/check-hash                        |  8 +++++++-
 toolchain/toolchain-external/toolchain-external.mk |  2 ++
 9 files changed, 38 insertions(+), 4 deletions(-)

From: "Yann E. MORIN" <[email protected]>

---
Changes v4 -> v5:
- use alternate upstream as source of hashes (Arnout)

---
 package/gcc/gcc-final/gcc-final.hash     |  1 +
 package/gcc/gcc-initial/gcc-initial.hash |  1 +
 package/gcc/gcc.hash                     | 10 ++++++++++
 3 files changed, 12 insertions(+)
 create mode 120000 package/gcc/gcc-final/gcc-final.hash
 create mode 120000 package/gcc/gcc-initial/gcc-initial.hash
 create mode 100644 package/gcc/gcc.hash

new file mode 120000
index 0000000..7ac9361
@@ -0,0 +1 @@
+../gcc.hash
\ No newline at end of file

new file mode 120000
index 0000000..7ac9361
@@ -0,0 +1 @@
+../gcc.hash
\ No newline at end of file

new file mode 100644
index 0000000..5eec97f
@@ -0,0 +1,10 @@
+# From ftp://gcc.gnu.org/pub/gcc/releases/gcc-4.5.4/sha512.sum
+sha512  78696b287d46aacd6f150920da376ea32f58ad9f0dafd2d3b7fa6dbdd8dd7afe659108d1dfaa5807617fc5248719f50b39c37ade0173ea5b9fec2091f4656115  gcc-4.5.4.tar.bz2
+# From ftp://gcc.gnu.org/pub/gcc/releases/gcc-4.7.4/sha512.sum
+sha512  dfcb737073191e628231031a3571ec77ee760a59377630f4a6e4fdfa66f9ddad39fde47e3f0f227eb43cdf90e0d34cde5abdc9ac892c1e111a911062a66c9189  gcc-4.7.4.tar.bz2
+# From ftp://gcc.gnu.org/pub/gcc/releases/gcc-4.8.4/sha512.sum
+sha512  b4c84ebd754c026029d706ef52c4242df4dcadc6a22cc5669b3aa37ac9dc25bc9cd9651fda83fb24dd5d17b1710e6865a35bb6c5cd9b95d0971717453935ae8e  gcc-4.8.4.tar.bz2
+# From ftp://gcc.gnu.org/pub/gcc/releases/gcc-4.9.2/sha512.sum
+sha512  e96cba06d572dbe7f382a2b00fd9297dcf9f9219d46a9ad0bd904dc36d5e7db3335e422640b79a27fed6fde91b0731732b94a0e2fbcd47344d6779f8ba1f4f9c  gcc-4.9.2.tar.bz2
+# No hash for the ARC variant, comes from the github-helper:
+none  xxx  gcc-arc-2014.12.tar.gz

2.1.4

From: "Yann E. MORIN" <[email protected]>

---
Changes v4 -> v5:
- use alternate upstream as source for hashes (Arnout)

---
 package/binutils/binutils.hash | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 package/binutils/binutils.hash

new file mode 100644
index 0000000..028fc10
@@ -0,0 +1,7 @@
+# From ftp://gcc.gnu.org/pub/binutils/releases/sha512.sum
+sha512  ffe8ef263ef99183e8cc823fe8487ff7d0f7bf9a8efd2853b5f4636aca0023850d13de4eac7d77a5f69413d8a50e6f95bb14569be53df86c0bce38034525ab74  binutils-2.22.tar.bz2
+sha512  dec753bbba008f1526b89cf1bd85feba78f362f5333ffdf93953fd131eb755976dec82a0a4ba38c43d2434da007137780cfe674de5414be5cf7ce7fbc6af6d16  binutils-2.23.2.tar.bz2
+sha512  5ec95ad47d49b12c4558a8db0ca2109d3ee1955e3776057f3330c4506f8f4d1cf5e505fbf8a16b98403a0fcdeaaf986fe0a22be6456247dbdace63ce1f776b12  binutils-2.24.tar.bz2
+sha512  49438970642f379978aba988e3d2e7b63dab6fd0b46d91f4100f083e87a84d242089daca58572245fe620457a77bbbe2d78588ff3fb79314e500d1a39bf96f30  binutils-2.25.tar.bz2
+# No hash for the ARC variant, comes from the github-helper:
+none   xxx                                                               binutils-arc-2014.12.tar.gz

2.1.4

From: "Yann E. MORIN" <[email protected]>

This is a git snapshot, we can't have a hash for it.

---
 package/uclibc/uclibc.hash | 3 +++
 1 file changed, 3 insertions(+)

index 7bfe46f..729c643 100644
@@ -6,3 +6,6 @@ sha256	0a9085a11615296cfe7783f16ff29073393eb330749947adece0eeddb31705a4	uClibc-a
 
 # From http://www.uclibc-ng.org/trac/
 sha256  7e92e99995daa634cbdc63c297784df6e8fe39cc1d3bbd2d37eb5c3ced8a86e5        uClibc-ng-1.0.2.tar.xz
+
+# No hash for this git snapshot
+none  xxx  uClibc-7bf35c8b7d4a1f97174eb49f47f33946b282114c.tar.gz

2.1.4

From: Arnout Vandecappelle <[email protected]>

The old hash for 2014.08 was never updated. Since uclibc-arc-2014.12 is downloaded from the github helper, it is converted into a 'none' hash.

Reordered the hashes so the none-hashes are together.

---
 package/uclibc/uclibc.hash | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

index 729c643..fe60291 100644
@@ -1,11 +1,11 @@
 # Locally calculated after checking pgp signature
 sha256	62333167b79afb0b25a843513288c67b59547acf653e8fbe62ee64e71ebd1587	uClibc-0.9.33.2.tar.xz
 
-# Locally calculated
-sha256	0a9085a11615296cfe7783f16ff29073393eb330749947adece0eeddb31705a4	uClibc-arc-2014.08.tar.gz
-
 # From http://www.uclibc-ng.org/trac/
 sha256  7e92e99995daa634cbdc63c297784df6e8fe39cc1d3bbd2d37eb5c3ced8a86e5        uClibc-ng-1.0.2.tar.xz
 
+# No hash for the ARC variant, comes from the github-helper
+none	xxx	uClibc-arc-2014.12.tar.gz
+
 # No hash for this git snapshot
 none  xxx  uClibc-7bf35c8b7d4a1f97174eb49f47f33946b282114c.tar.gz

2.1.4

From: "Yann E. MORIN" <[email protected]>

In very constrained cases, it might be needed to not fail if a hash is missing. This is notably the case for custom external toolchains to be downloaded, because we do have a .hash file for external toolchains, but we obviously can not have hashes for all existing custom toolchains (he, "custom"!).

So, add a way to avoid failing in that case.

From the Makefile, we export the list of files for which not to check the hash. Then, from the check-hash script, if no check was done, and the file we were trying to match in in this exclusion list, we just exit without error.

changes v6 -> v7:
- /beautify/ the pattern in the case clause

Changed v5 -> v6: (Arnout)
- fix the pattern in the case clause

Changes v4 -> v5:
- micro-optimisation, use case-esac instead of a for-loop (Arnout)
- typoes (Arnout)

Changes v3 -> v4:
- drop the magic value, use a list of excluded files (Arnout)

Changes v1 -> v2:
- fix typoes in commit log

---
 package/pkg-download.mk     | 3 +++
 support/download/check-hash | 6 ++++++
 2 files changed, 9 insertions(+)

index e274712..1fc23c4 100644
@@ -60,6 +60,9 @@ domainseparator = $(if $(1),$(1),/)
 # github(user,package,version): returns site of GitHub repository
 github = https://github.com/$(1)/$(2)/archive/$(3)
 
+# Expressly do not check hashes for those files
+export BR_NO_CHECK_HASH_FOR
+
 ################################################################################
 # The DOWNLOAD_* helpers are in charge of getting a working copy
 # of the source repository for their corresponding SCM,

index 678a7ef..6b0372e 100755
@@ -99,6 +99,12 @@ while read t h f; do
 done <"${h_file}"
 
 if [ ${nb_checks} -eq 0 ]; then
+    case " ${BR_NO_CHECK_HASH_FOR} " in
+    *" ${base} "*)
+        # File explicitly has no hash
+        exit 0
+        ;;
+    esac
     printf "ERROR: No hash found for %s\n" "${base}" >&2
     exit 0
 fi

2.1.4

From: "Yann E. MORIN" <[email protected]>

We will *always* be missing a hash file for custom external toolchains that are downloaded.

So, just ignore that failure.

---
Chamges v3 -> v4:
- drop magic value, append to the exclusion list (Arnout)

Changes v1 -> v2:
- fix typoes in title

---
 toolchain/toolchain-external/toolchain-external.mk | 2 ++
 1 file changed, 2 insertions(+)

index 2427463..5576dcb 100644
@@ -400,6 +400,8 @@ else
 # Custom toolchain
 TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
 TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
+# We can't check hashes for custom downloaded toolchains
+BR_NO_CHECK_HASH_FOR += $(TOOLCHAIN_EXTERNAL_SOURCE)
 endif
 
 # In fact, we don't need to download the toolchain, since it is already

2.1.4

From: "Yann E. MORIN" <[email protected]>

We can't have a hash for snapshots, so add them to the exclusion list.

---
Changes v1 -> v6: (Arnout)
- use $(UCLIBC_SOURCE) instead of a wrong filename

---
 package/uclibc/uclibc.mk | 1 +
 1 file changed, 1 insertion(+)

index 07ff8e5..ce9b2b4 100644
@@ -11,6 +11,7 @@ UCLIBC_LICENSE_FILES = COPYING.LIB
 
 ifeq ($(BR2_UCLIBC_VERSION_SNAPSHOT),y)
 UCLIBC_SITE = http://www.uclibc.org/downloads/snapshots
+BR_NO_CHECK_HASH_FOR += $(UCLIBC_SOURCE)
 else ifeq ($(BR2_UCLIBC_VERSION_NG),y)
 UCLIBC_SITE = http://downloads.uclibc-ng.org/releases/$(UCLIBC_VERSION)
 UCLIBC_SOURCE = uClibc-ng-$(UCLIBC_VERSION).tar.xz

2.1.4

From: "Yann E. MORIN" <[email protected]>

Now that custom external toolchains to be downloaded properly instruct to not fail on a missing hash, restore the mandatory hash check for everything else.

---
 support/download/check-hash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 6b0372e..c1ff53c 100755
@@ -106,5 +106,5 @@ if [ ${nb_checks} -eq 0 ]; then
         ;;
     esac
     printf "ERROR: No hash found for %s\n" "${base}" >&2
-    exit 0
+    exit 3
 fi

2.1.4

Hello,

Series applied.

One thing that bothered me a little bit is that when the hash of the tarball downloaded from upstream doesn't match the one we have in BR, we gracefully fall back on downloading the tarball from sources.buildroot.net.

This is good for users as it means their build succeeds. But not so good from the point of view of noticing that upstream has for some reason changed the tarball.

Any opinion about this?

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

We could disable the SECONDARY site in the autobuilders.

Regards,
Arnout

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F