toolchain: better handle hashes (branch yem/dl-hash-toolchains)

PATCH Superseded Yann E. MORIN 3 participants 16 messages Apr 6, 2015 – Apr 7, 2015

lore

Share & tools

lore

Yann E. MORIN

author

2015-04-06 12:13 UTC #0 lore raw

v3 cover toolchain: better handle hashes (branch yem/dl-hash-toolchains)

Hello All!

This series makes it so we can download custom external toolchains, and still have mandatory checks of hashes.

It also adds the long-awaited-for hashes for gcc and binutils.

Finally, it fixes a slightly annoying warning message when downloading from a repository.

Changes v2 -> v3:
- add hashes for gcc and binutils
- fix a few more typoes

Changes v1 -> v2:
- fix a few typoes

Regards,
Yann E. MORIN.

The following changes since commit ca735d8a0b86b7ad274319774df2bca0abfe4ee9:

package/ipmiutil: do not build doc (2015-04-06 12:42:54 +0200)

are available in the git repository at:

git://git.busybox.net/~ymorin/git/buildroot yem/dl-hash-toolchains

for you to fetch changes up to 21c6a4cd9d79b9279ba50426a2d0baaa1bc3f0b0:

support/download: restore mandatory check of hashes (2015-04-06 14:11:45 +0200)

----------------------------------------------------------------
Yann E. MORIN (6):
support/download: do not warn about missing hash file for repositories package/gcc: add hashes
package/binutils: add hashes
support/download: add possibility to not fail on missing hash toolchain/external: ignore missing hash for custom downloaded toolchain support/download: restore mandatory check of hashes

 package/binutils/binutils.hash                     |  7 +++++++
 package/gcc/gcc-final/gcc-final.hash               |  1 +
 package/gcc/gcc-initial/gcc-initial.hash           |  1 +
 package/gcc/gcc.hash                               |  7 +++++++
 package/pkg-download.mk                            |  4 ++++
 support/download/check-hash                        | 12 ++++++++++--
 toolchain/toolchain-external/toolchain-external.mk |  2 ++
 7 files changed, 32 insertions(+), 2 deletions(-)
 create mode 100644 package/binutils/binutils.hash
 create mode 120000 package/gcc/gcc-final/gcc-final.hash
 create mode 120000 package/gcc/gcc-initial/gcc-initial.hash
 create mode 100644 package/gcc/gcc.hash

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Yann E. MORIN

author

2015-04-06 12:13 UTC #1 lore raw

PATCH v3 1/6 support/download: do not warn about missing hash file for repositories

When downloading from a repository, we explicitly pass no hash file, because we can't check hashes in that case.

However, we're still printing a message that there is a missign hash file.

Beside being a bit annoying (since we can't do anything about it), it may also be wrong, especially for packages for which we support multiple versions, with some being downloaded via a git clone and others as tarballs.

Just print no warning when the path to the hash file is empty.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Gustavo Zacarias <[email protected]>

---
Changes v1 -> v2;
- fix typoes in commit log

---
 support/download/check-hash | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

support/download/check-hash +5 -1

index 67c1780..d37f1cd 100755
@@ -31,8 +31,12 @@ h_file="${1}"
 file="${2}"
 base="${3}"
 
+# Bail early if no hash to check
+if [ -z "${h_file}" ]; then
+    exit 0
+fi
 # Does the hash-file exist?
-if [ -z "${h_file}" -o ! -f "${h_file}" ]; then
+if [ ! -f "${h_file}" ]; then
     printf "WARNING: no hash file for %s\n" "${base}" >&2
     exit 0
 fi

Signature · 1 line

1.9.1

Arnout Vandecappelle

2015-04-06 14:39 UTC #2 lore raw

PATCH v3 1/6 support/download: do not warn about missing hash file for repositories

Quoted 11 lines · Yann E. MORIN

When downloading from a repository, we explicitly pass no hash file, because we can't check hashes in that case.
However, we're still printing a message that there is a missign hash file.
Beside being a bit annoying (since we can't do anything about it), it may also be wrong, especially for packages for which we support multiple versions, with some being downloaded via a git clone and others as tarballs.

Quoted 5 lines

Just print no warning when the path to the hash file is empty.
Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Gustavo Zacarias <[email protected]>

Reviewed-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

Regards,
Arnout

Quoted 27 lines

---
Changes v1 -> v2;
- fix typoes in commit log

---
 support/download/check-hash | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

support/download/check-hash +5 -1

index 67c1780..d37f1cd 100755
@@ -31,8 +31,12 @@ h_file="${1}"
 file="${2}"
 base="${3}"
 
+# Bail early if no hash to check
+if [ -z "${h_file}" ]; then
+    exit 0
+fi
 # Does the hash-file exist?
-if [ -z "${h_file}" -o ! -f "${h_file}" ]; then
+if [ ! -f "${h_file}" ]; then
     printf "WARNING: no hash file for %s\n" "${base}" >&2
     exit 0
 fi

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Thomas Petazzoni

Applied

2015-04-06 21:29 UTC #3 lore raw

PATCH v3 1/6 support/download: do not warn about missing hash file for repositories

In reply to Yann E. MORIN

Dear Yann E. MORIN,

Quoted 11 lines · Yann E. MORIN

When downloading from a repository, we explicitly pass no hash file, because we can't check hashes in that case.
However, we're still printing a message that there is a missign hash file.
Beside being a bit annoying (since we can't do anything about it), it may also be wrong, especially for packages for which we support multiple versions, with some being downloaded via a git clone and others as tarballs.

Quoted 5 lines

Just print no warning when the path to the hash file is empty.
Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Gustavo Zacarias <[email protected]>

Applied, thanks!

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Yann E. MORIN

author

2015-04-06 12:13 UTC #4 lore raw

PATCH v3 2/6 package/gcc: add hashes

In reply to Yann E. MORIN

Signed-off-by "Yann E. MORIN" <[email protected]>

cc Thomas Petazzoni <[email protected]>

Cc Gustavo Zacarias <[email protected]>

---
 package/gcc/gcc-final/gcc-final.hash     | 1 +
 package/gcc/gcc-initial/gcc-initial.hash | 1 +
 package/gcc/gcc.hash                     | 7 +++++++
 3 files changed, 9 insertions(+)
 create mode 120000 package/gcc/gcc-final/gcc-final.hash
 create mode 120000 package/gcc/gcc-initial/gcc-initial.hash
 create mode 100644 package/gcc/gcc.hash

package/gcc/gcc-final/gcc-final.hash new file +1 -0

new file mode 120000
index 0000000..7ac9361
@@ -0,0 +1 @@
+../gcc.hash
\ No newline at end of file

package/gcc/gcc-initial/gcc-initial.hash new file +1 -0

new file mode 120000
index 0000000..7ac9361
@@ -0,0 +1 @@
+../gcc.hash
\ No newline at end of file

package/gcc/gcc.hash new file +7 -0

new file mode 100644
index 0000000..d9a04fc
@@ -0,0 +1,7 @@
+# Locally computed (upstream only has sigs):
+sha256  eef3f0456db8c3d992cbb51d5d32558190bc14f3bc19383dd93acc27acc6befc  gcc-4.5.4.tar.bz2
+sha256  92e61c6dc3a0a449e62d72a38185fda550168a86702dea07125ebd3ec3996282  gcc-4.7.4.tar.bz2
+sha256  4a80aa23798b8e9b5793494b8c976b39b8d9aa2e53cd5ed5534aff662a7f8695  gcc-4.8.4.tar.bz2
+sha256  2020c98295856aa13fda0f2f3a4794490757fc24bcca918d52cc8b4917b972dd  gcc-4.9.2.tar.bz2
+# No hash for the ARC variant, comes from the github-helper:
+none    xxx                                                               gcc-arc-2014.12.tar.gz

Signature · 1 line

1.9.1

Yann E. MORIN

author

2015-04-06 12:13 UTC #5 lore raw

PATCH v3 3/6 package/binutils: add hashes

In reply to Yann E. MORIN

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Gustavo Zacarias <[email protected]>

---
 package/binutils/binutils.hash | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 package/binutils/binutils.hash

package/binutils/binutils.hash new file +7 -0

new file mode 100644
index 0000000..4f99514
@@ -0,0 +1,7 @@
+# Locally computed:
+sha256 6c7af8ed1c8cf9b4b9d6e6fe09a3e1d3d479fe63984ba8b9b26bf356b6313ca9  binutils-2.22.tar.bz2
+sha256 fe914e56fed7a9ec2eb45274b1f2e14b0d8b4f41906a5194eac6883cfe5c1097  binutils-2.23.2.tar.bz2
+sha256 e5e8c5be9664e7f7f96e0d09919110ab5ad597794f5b1809871177a0f0f14137  binutils-2.24.tar.bz2
+sha256 22defc65cfa3ef2a3395faaea75d6331c6e62ea5dfacfed3e2ec17b08c882923  binutils-2.25.tar.bz2
+# No hash for the ARC variant, comes from the github-helper:
+none   xxx                                                               binutils-arc-2014.12.tar.gz

Signature · 1 line

1.9.1

Yann E. MORIN

author

2015-04-06 12:13 UTC #6 lore raw

PATCH v3 4/6 support/download: add possibility to not fail on missing hash

In reply to Yann E. MORIN

In very constrained cases, it might be needed to not fail if a hash is missing. This is notably the case for custom external toolchains to be downloaded, because we do have a .hash file for external toolchains, but we oviously can not have hashes for all existing cutom toolchains (he, "custom"!).

So, add a way to avoid failing in that case.

We use a magic value depending on the file to be downloaded, instead of yes/no, to avoid any accidental use. We also do not document it, because we do not want users to use it, and even more, to rely on it, since we still have hopes to get rid of that in the future.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Gustavo Zacarias <[email protected]>

---
Changes v1 -> v2:
- fix typoes in commit log

---
 package/pkg-download.mk     | 4 ++++
 support/download/check-hash | 4 ++++
 2 files changed, 8 insertions(+)

package/pkg-download.mk +4 -0

index e274712..1e080e5 100644
@@ -60,6 +60,10 @@ domainseparator = $(if $(1),$(1),/)
 # github(user,package,version): returns site of GitHub repository
 github = https://github.com/$(1)/$(2)/archive/$(3)
 
+# Compute magic value to ignore a missing hash
+# $1: filename, without path components
+ignore-missing-hash-magic = $(shell sha1sum <<<"$(1)" |cut -d ' ' -f 1)
+
 ################################################################################
 # The DOWNLOAD_* helpers are in charge of getting a working copy
 # of the source repository for their corresponding SCM,

support/download/check-hash +4 -0

index d37f1cd..a091c61 100755
@@ -99,6 +99,10 @@ while read t h f; do
 done <"${h_file}"
 
 if [ ${nb_checks} -eq 0 ]; then
+    if [ "${BR_MISSING_HASH_OK}" = "$(sha1sum <<<"${base}" |cut -d ' ' -f 1)" ]; then
+        printf "WARNING: ignored missing hash for %s\n" "${base}" >&2
+        exit 0
+    fi
     printf "ERROR: No hash found for %s\n" "${base}" >&2
     exit 0
 fi

Signature · 1 line

1.9.1

Arnout Vandecappelle

2015-04-06 20:47 UTC #7 lore raw

PATCH v3 4/6 support/download: add possibility to not fail on missing hash

Quoted 6 lines · Yann E. MORIN

In very constrained cases, it might be needed to not fail if a hash is missing. This is notably the case for custom external toolchains to be downloaded, because we do have a .hash file for external toolchains, but we oviously can not have hashes for all existing cutom toolchains (he, "custom"!).

Quoted 6 lines

So, add a way to avoid failing in that case.
We use a magic value depending on the file to be downloaded, instead of yes/no, to avoid any accidental use. We also do not document it, because we do not want users to use it, and even more, to rely on it, since we still have hopes to get rid of that in the future.

I don't think that complexity is really warranted. Something like
BR_MISSING_HASH=OK should be enough.

I do agree with not documenting it, however.

Quoted 33 lines

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Gustavo Zacarias <[email protected]>

---
Changes v1 -> v2:
- fix typoes in commit log

---
 package/pkg-download.mk     | 4 ++++
 support/download/check-hash | 4 ++++
 2 files changed, 8 insertions(+)

package/pkg-download.mk +4 -0

index e274712..1e080e5 100644
@@ -60,6 +60,10 @@ domainseparator = $(if $(1),$(1),/)
 # github(user,package,version): returns site of GitHub repository
 github = https://github.com/$(1)/$(2)/archive/$(3)
 
+# Compute magic value to ignore a missing hash
+# $1: filename, without path components
+ignore-missing-hash-magic = $(shell sha1sum <<<"$(1)" |cut -d ' ' -f 1)
+
 ################################################################################
 # The DOWNLOAD_* helpers are in charge of getting a working copy
 # of the source repository for their corresponding SCM,

support/download/check-hash +0 -0

index d37f1cd..a091c61 100755

support/download/check-hash while read t h f; do · line 103

Quoted 5 lines

support/download/check-hash +2 -0

@@ -99,6 +99,10 @@ while read t h f; do
 done <"${h_file}"
 
 if [ ${nb_checks} -eq 0 ]; then
+    if [ "${BR_MISSING_HASH_OK}" = "$(sha1sum <<<"${base}" |cut -d ' ' -f 1)" ]; then
+        printf "WARNING: ignored missing hash for %s\n" "${base}" >&2

There shouldn't be a warning in this case either, right? Neither end users nor
buildroot developers can do anything about it, so what would be the point...

I'd also make the branching more explicit:

if [ ${nb_checks} -eq 0 ]; then

    if [ "${BR_MISSING_HASH}" = "OK" ]; then
        exit 0
    else
        printf "ERROR: No hash found for %s\n" "${base}" >&2
        exit 0
    fi

Regards,
Arnout

Quoted 6 lines

support/download/check-hash +2 -0

+        exit 0
+    fi
     printf "ERROR: No hash found for %s\n" "${base}" >&2
     exit 0
 fi

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Yann E. MORIN

author

2015-04-06 12:13 UTC #8 lore raw

PATCH v3 5/6 toolchain/external: ignore missing hash for custom downloaded toolchain

In reply to Yann E. MORIN

We will *always* be missing a hash file for custom external toolchains that are downloaded.

So, just ignore that failure.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Gustavo Zacarias <[email protected]>

---
Changes v1 -> v2:
- fix typoes in title

---
 toolchain/toolchain-external/toolchain-external.mk | 2 ++
 1 file changed, 2 insertions(+)

toolchain/toolchain-external/toolchain-external.mk +2 -0

index c0429bb..3659511 100644
@@ -400,6 +400,8 @@ else
 # Custom toolchain
 TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
 TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
+# Magic value to tell the download helper to avoid failling on missing hash
+export BR_MISSING_HASH_OK := $(call ignore-missing-hash-magic,$(TOOLCHAIN_EXTERNAL_SOURCE))
 endif
 
 # In fact, we don't need to download the toolchain, since it is already

Signature · 1 line

1.9.1

Arnout Vandecappelle

2015-04-06 21:03 UTC #9 lore raw

PATCH v3 5/6 toolchain/external: ignore missing hash for custom downloaded toolchain

Quoted 21 lines · Yann E. MORIN

We will *always* be missing a hash file for custom external toolchains that are downloaded.
So, just ignore that failure.
Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Gustavo Zacarias <[email protected]>
---
Changes v1 -> v2:
- fix typoes in title
---
 toolchain/toolchain-external/toolchain-external.mk | 2 ++
 1 file changed, 2 insertions(+)
toolchain/toolchain-external/toolchain-external.mk +0 -0
index c0429bb..3659511 100644

toolchain/toolchain-external/toolchain-external.mk else · line 404

Quoted 5 lines

toolchain/toolchain-external/toolchain-external.mk +2 -0

@@ -400,6 +400,8 @@ else
 # Custom toolchain
 TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
 TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
+# Magic value to tell the download helper to avoid failling on missing hash
+export BR_MISSING_HASH_OK := $(call ignore-missing-hash-magic,$(TOOLCHAIN_EXTERNAL_SOURCE))

Urgh - this disables the hash check for all packages, not just the toolchain.
So NACK for exporting it.

But I see that it's not so simple to get it into the environment of the
download step...

Perhaps instead we should add a global variable with the files that should not
be hash-checked:

NOHASH_FILES += $(TOOLCHAIN_EXTERNAL_SOURCE)

and in pkg-download.mk

hasharg = $(if $(filter-out $(TOOLCHAIN_EXTERNAL_SOURCE),$(1)),

	-H $(PKGDIR)/$($(PKG)_RAWNAME).hash)

define DOWNLOAD_WGET

        $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
                -o $(DL_DIR)/$(2) \
                $(call hasharg,$(2)) \
                $(QUIET) \
                -- \
                '$(call qstrip,$(1))'

endef

(all completely untested, obviously :-)

Regards,
Arnout

Quoted 4 lines

toolchain/toolchain-external/toolchain-external.mk +0 -0
 endif
 
 # In fact, we don't need to download the toolchain, since it is already

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Yann E. MORIN

author

2015-04-06 21:20 UTC #10 lore raw

PATCH v3 5/6 toolchain/external: ignore missing hash for custom downloaded toolchain

Arnout, All,

On 2015-04-06 23:03 +0200, Arnout Vandecappelle spake thusly:

Quoted 17 lines · Yann E. MORIN

Quoted 16 lines
We will *always* be missing a hash file for custom external toolchains that are downloaded.
So, just ignore that failure.
Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Gustavo Zacarias <[email protected]>
---
Changes v1 -> v2:
- fix typoes in title
---
 toolchain/toolchain-external/toolchain-external.mk | 2 ++
 1 file changed, 2 insertions(+)

Quoted 12 lines

toolchain/toolchain-external/toolchain-external.mk else · line 404
Quoted 10 lines
toolchain/toolchain-external/toolchain-external.mk +2 -0
index c0429bb..3659511 100644
@@ -400,6 +400,8 @@ else
 # Custom toolchain
 TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
 TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
+# Magic value to tell the download helper to avoid failling on missing hash
+export BR_MISSING_HASH_OK := $(call ignore-missing-hash-magic,$(TOOLCHAIN_EXTERNAL_SOURCE))
Urgh - this disables the hash check for all packages, not just the toolchain.

No, that disable the check only for that one file, because we're not chcking whether it is set, but whether it is set to the correct value (i.e. the sha1 of the filename).

Quoted 1 line

So NACK for exporting it.

OK, this is indeed fishy.

Quoted 4 lines

But I see that it's not so simple to get it into the environment of the
download step...

Quoted 8 lines

Perhaps instead we should add a global variable with the files that should not
be hash-checked:
NOHASH_FILES += $(TOOLCHAIN_EXTERNAL_SOURCE)
and in pkg-download.mk
hasharg = $(if $(filter-out $(TOOLCHAIN_EXTERNAL_SOURCE),$(1)),

You probably meant s/TOOLCHAIN_EXTERNAL_SOURCE/NOHASH_FILES/ ?

Quoted 9 lines

	-H $(PKGDIR)/$($(PKG)_RAWNAME).hash)

define DOWNLOAD_WGET

        $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
                -o $(DL_DIR)/$(2) \
                $(call hasharg,$(2)) \
                $(QUIET) \
                -- \
                '$(call qstrip,$(1))'

Quoted 3 lines

endef
(all completely untested, obviously :-)

Well, that sounds a bit better than my proposal.

Still, I prefer:

- we offload the check in the dl-wrapper so we can do the check in a single place, rather than re-add extra code in each download macros (the dl-wrapper was added because the Makefile macros were too complex to handle);

- we pass obscur values (like a sha1), rather than the filenames, to make it even less easy to use.

I'll respin a series taking into account your comments. Thanks! :-)

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Arnout Vandecappelle

2015-04-06 23:24 UTC #11 lore raw

PATCH v3 5/6 toolchain/external: ignore missing hash for custom downloaded toolchain

Quoted 33 lines · Yann E. MORIN

Arnout, All,
On 2015-04-06 23:03 +0200, Arnout Vandecappelle spake thusly:
Quoted 17 lines · Yann E. MORIN
Quoted 16 lines
We will *always* be missing a hash file for custom external toolchains that are downloaded.
So, just ignore that failure.
Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Gustavo Zacarias <[email protected]>
---
Changes v1 -> v2:
- fix typoes in title
---
 toolchain/toolchain-external/toolchain-external.mk | 2 ++
 1 file changed, 2 insertions(+)
Quoted 12 lines
toolchain/toolchain-external/toolchain-external.mk else · line 404
Quoted 10 lines
toolchain/toolchain-external/toolchain-external.mk +2 -0
index c0429bb..3659511 100644
@@ -400,6 +400,8 @@ else
 # Custom toolchain
 TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
 TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
+# Magic value to tell the download helper to avoid failling on missing hash
+export BR_MISSING_HASH_OK := $(call ignore-missing-hash-magic,$(TOOLCHAIN_EXTERNAL_SOURCE))
Urgh - this disables the hash check for all packages, not just the toolchain.

Quoted 3 lines

No, that disable the check only for that one file, because we're not chcking whether it is set, but whether it is set to the correct value (i.e. the sha1 of the filename).

Ah of course, that's probably why you introduced that.

But it does mean that only a single file can ever be bypassed...

Quoted 12 lines

Quoted 2 lines
So NACK for exporting it.
OK, this is indeed fishy.
Quoted 7 lines
But I see that it's not so simple to get it into the environment of the
download step...
Perhaps instead we should add a global variable with the files that should not
be hash-checked:

Quoted 7 lines

Quoted 5 lines
NOHASH_FILES += $(TOOLCHAIN_EXTERNAL_SOURCE)
and in pkg-download.mk
hasharg = $(if $(filter-out $(TOOLCHAIN_EXTERNAL_SOURCE),$(1)),
You probably meant s/TOOLCHAIN_EXTERNAL_SOURCE/NOHASH_FILES/ ?

Untested and all :-)

Quoted 14 lines

Quoted 10 lines

	-H $(PKGDIR)/$($(PKG)_RAWNAME).hash)

define DOWNLOAD_WGET

        $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
                -o $(DL_DIR)/$(2) \
                $(call hasharg,$(2)) \
                $(QUIET) \
                -- \
                '$(call qstrip,$(1))'

Quoted 3 lines

endef
(all completely untested, obviously :-)

Quoted 8 lines

Well, that sounds a bit better than my proposal.
Still, I prefer:
- we offload the check in the dl-wrapper so we can do the check in a single place, rather than re-add extra code in each download macros (the dl-wrapper was added because the Makefile macros were too complex to handle);

Ack that, though I don't immediately see a way to do that.

Quoted 3 lines

- we pass obscur values (like a sha1), rather than the filenames, to make it even less easy to use.

I really don't see the point of that... If it's not documented, you have to
look at the source code to find out which variable to set. And when you look at the source code, you immediately see how this obscure value is calculated so it makes no real difference.

Regards,
Arnout

Quoted 6 lines

I'll respin a series taking into account your comments. Thanks! :-)
Regards,
Yann E. MORIN.

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Yann E. MORIN

author

2015-04-07 21:51 UTC #12 lore raw

PATCH v3 5/6 toolchain/external: ignore missing hash for custom downloaded toolchain

Arnout, All,

On 2015-04-07 01:24 +0200, Arnout Vandecappelle spake thusly:

Quoted 3 lines · Yann E. MORIN

Quoted 2 lines
On 2015-04-06 23:03 +0200, Arnout Vandecappelle spake thusly:
Quoted 1 line · Yann E. MORIN

[--SNIP--]

Quoted 11 lines

Quoted 7 lines
Quoted 3 lines
toolchain/toolchain-external/toolchain-external.mk else · line 404
Quoted 2 lines
toolchain/toolchain-external/toolchain-external.mk +2 -0
+# Magic value to tell the download helper to avoid failling on missing hash
+export BR_MISSING_HASH_OK := $(call ignore-missing-hash-magic,$(TOOLCHAIN_EXTERNAL_SOURCE))
Urgh - this disables the hash check for all packages, not just the toolchain.
No, that disable the check only for that one file, because we're not chcking whether it is set, but whether it is set to the correct value (i.e. the sha1 of the filename).
Ah of course, that's probably why you introduced that.
But it does mean that only a single file can ever be bypassed...

Right.

[--SNIP--]

Quoted 3 lines

Quoted 2 lines
Quoted 1 line
hasharg = $(if $(filter-out $(TOOLCHAIN_EXTERNAL_SOURCE),$(1)),
You probably meant s/TOOLCHAIN_EXTERNAL_SOURCE/NOHASH_FILES/ ?
Untested and all :-)

Hehe! :-)

Quoted 15 lines

Quoted 15 lines
Quoted 9 lines
	-H $(PKGDIR)/$($(PKG)_RAWNAME).hash)
define DOWNLOAD_WGET
        $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
                -o $(DL_DIR)/$(2) \
                $(call hasharg,$(2)) \
                $(QUIET) \
                -- \
                '$(call qstrip,$(1))'
Quoted 3 lines
endef
(all completely untested, obviously :-)
Well, that sounds a bit better than my proposal.

Quoted 7 lines

Quoted 6 lines
Still, I prefer:
- we offload the check in the dl-wrapper so we can do the check in a single place, rather than re-add extra code in each download macros (the dl-wrapper was added because the Makefile macros were too complex to handle);
Ack that, though I don't immediately see a way to do that.

Don;t worry, I'll find a way... ;-]

Quoted 7 lines

Quoted 2 lines
- we pass obscur values (like a sha1), rather than the filenames, to make it even less easy to use.
I really don't see the point of that... If it's not documented, you have to
look at the source code to find out which variable to set. And when you look at the source code, you immediately see how this obscure value is calculated so it makes no real difference.

OK, makes sense.

I will rework the series accordingly. Thanks for the suggestions!

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Yann E. MORIN

author

2015-04-06 12:13 UTC #13 lore raw

PATCH v3 6/6 support/download: restore mandatory check of hashes

In reply to Yann E. MORIN

Now that custom external toolchains to be downloaded proprely instruct to not fail on a missing hash, restore the mandatory hash check for everything else.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Gustavo Zacarias <[email protected]>

---
 support/download/check-hash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

support/download/check-hash +1 -1

index a091c61..96d12a5 100755
@@ -104,5 +104,5 @@ if [ ${nb_checks} -eq 0 ]; then
         exit 0
     fi
     printf "ERROR: No hash found for %s\n" "${base}" >&2
-    exit 0
+    exit 3
 fi

Signature · 1 line

1.9.1

Arnout Vandecappelle

2015-04-06 21:05 UTC #14 lore raw

PATCH v3 6/6 support/download: restore mandatory check of hashes

Quoted 3 lines · Yann E. MORIN

Now that custom external toolchains to be downloaded proprely instruct to not fail on a missing hash, restore the mandatory hash check for everything else.

There are also a couple of uclibc versions without hash.

Otherwise looks good :-)

Regards,
Arnout

Quoted 20 lines

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Gustavo Zacarias <[email protected]>
---
 support/download/check-hash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/download/check-hash +1 -1
index a091c61..96d12a5 100755
@@ -104,5 +104,5 @@ if [ ${nb_checks} -eq 0 ]; then
         exit 0
     fi
     printf "ERROR: No hash found for %s\n" "${base}" >&2
-    exit 0
+    exit 3
 fi

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Yann E. MORIN

author

2015-04-07 22:04 UTC #15 lore raw

PATCH v3 6/6 support/download: restore mandatory check of hashes

Arnout, All,

On 2015-04-06 23:05 +0200, Arnout Vandecappelle spake thusly:

Quoted 6 lines · Yann E. MORIN

Quoted 3 lines
Now that custom external toolchains to be downloaded proprely instruct to not fail on a missing hash, restore the mandatory hash check for everything else.
There are also a couple of uclibc versions without hash.

Then they should be added (unless they are git snapshots or the likes...) ;-)

I'll see which are missing and add them; thanks for noticing.

Regards,
Yann E. MORIN.

Quoted 38 lines

Otherwise looks good :-)

Regards,
Arnout

Quoted 20 lines

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Gustavo Zacarias <[email protected]>
---
 support/download/check-hash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/download/check-hash +1 -1
index a091c61..96d12a5 100755
@@ -104,5 +104,5 @@ if [ ${nb_checks} -eq 0 ]; then
         exit 0
     fi
     printf "ERROR: No hash found for %s\n" "${base}" >&2
-    exit 0
+    exit 3
 fi

Signature · 10 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'