support/download: accept user-provided list of extra hash files (branch yem/multi-hash)
v2 cover support/download: accept user-provided list of extra hash files (branch yem/multi-hash)
Hello All!
For packages where the version can be specified by the user (e.g. with a custom version string, a custom tarball location, a custom git tree and changeset...), Buildroot can't carry hashes for those, and thus does not check the integritiy of the downloads.
Add the possibility for users to provide hashes for those, by leveraging the global patch dir setting, to look up extra hash files in a way similar to how extra patches are looked up in there.
Users who provide such extra hashes will most probably be interested in ensuring that no download ever gets used without an actual integrity check, so also add an option the requires all downloads to have at least one valid hash (and no invalid ones, of course), rather than ignoring downloads for custom versions.
Changes v1 -> v2:
- restore legal-info (Peter K.)
- drop all.hash
Regards,
Yann E. MORIN.
----------------------------------------------------------------
Yann E. MORIN (3):
support/download: teach dl-wrapper to handle more than one hash file package/pkg-download: lookup hash files in global-patch-dir
pkg-download: add option to enforce hash checking
Config.in | 27 +++++++++++-- Makefile | 2 +- docs/manual/adding-packages-directory.adoc | 6 +++ docs/manual/customize-patches.adoc | 24 ++++++++++- package/pkg-download.mk | 7 ++-- package/pkg-generic.mk | 14 ++++--- package/pkg-utils.mk | 8 ++-- support/download/check-hash | 64 ++++++++++++++++-------------- support/download/dl-wrapper | 10 ++--- 9 files changed, 108 insertions(+), 54 deletions(-)
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 1/3 support/download: teach dl-wrapper to handle more than one hash file
Currently, we expect and only use hash files that lie within the package directory, alongside the .mk file. Those hash files are thus bundeld with Buildroot.
This implies that only what's known to Buildroot can ever get into those hash files. For packages where the version is fixed (or a static choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form entry, where the user can provide a custom location and/or version. like a custom VCS tree and revision, or a custom tarball URL. This means that Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is what was expected. For a sha1 in a git tree, this is a minor issue, because the sha1 by itself is already a hash of the expected content. But for custom tarballs URLs, or for a tag in a VCS, there is indeed no integrity check.
Buildroot can't provide such hashes, but interested users may want to provide those, and currently there is no (easy) way to do so.
So, we need our download helpers to be able to accept more than one hash file to lookup for hashes.
Extend the dl-wrapper and the check-hash helpers thusly, and update the legal-info accordingly.
Note that, to be able to pass more than one hash file, we also need to re-order the arguments passed to support/download/check-hash, which also impies some shuffling in the three places it is called:
- 2 in dl-wrapper
- 1 in the legal-info infra
That in turn also requires that the legal-license-file macro args get re-ordered to have the hash file last; we take the opportunity to also move the HOST/TARGET arg to be first, like in the other legal-info macros.
---
Changes v1 -> v2:
- restore legal-info (Peter K.)
--- Makefile | 2 +- package/pkg-generic.mk | 2 +- package/pkg-utils.mk | 8 ++--- support/download/check-hash | 64 ++++++++++++++++++++----------------- support/download/dl-wrapper | 10 +++--- 5 files changed, 45 insertions(+), 41 deletions(-)
Makefile +1 -1
@@ -836,7 +836,7 @@ legal-info-clean: .PHONY: legal-info-prepare legal-info-prepare: $(LEGAL_INFO_DIR) @$(call MESSAGE,"Buildroot $(BR2_VERSION_FULL) Collecting legal info") - @$(call legal-license-file,buildroot,buildroot,support/legal-info/buildroot.hash,COPYING,COPYING,HOST) + @$(call legal-license-file,HOST,buildroot,buildroot,COPYING,COPYING,support/legal-info/buildroot.hash) @$(call legal-manifest,TARGET,PACKAGE,VERSION,LICENSE,LICENSE FILES,SOURCE ARCHIVE,SOURCE SITE,DEPENDENCIES WITH LICENSES) @$(call legal-manifest,HOST,PACKAGE,VERSION,LICENSE,LICENSE FILES,SOURCE ARCHIVE,SOURCE SITE,DEPENDENCIES WITH LICENSES) @$(call legal-manifest,HOST,buildroot,$(BR2_VERSION_FULL),GPL-2.0+,COPYING,not saved,not saved)
package/pkg-generic.mk +1 -1
@@ -1138,7 +1138,7 @@ ifneq ($$(call qstrip,$$($(2)_SOURCE)),) ifeq ($$(call qstrip,$$($(2)_LICENSE_FILES)),) $(Q)$$(call legal-warning-pkg,$$($(2)_BASENAME_RAW),cannot save license ($(2)_LICENSE_FILES not defined)) else - $(Q)$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$($(2)_RAWNAME),$$($(2)_BASENAME_RAW),$$($(2)_HASH_FILE),$$(F),$$($(2)_DIR)/$$(F),$$(call UPPERCASE,$(4)))$$(sep)) + $(Q)$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$(call UPPERCASE,$(4)),$$($(2)_RAWNAME),$$($(2)_BASENAME_RAW),$$(F),$$($(2)_DIR)/$$(F),$$($(2)_HASH_FILE))$$(sep)) endif # license files ifeq ($$($(2)_REDISTRIBUTE),YES)
package/pkg-utils.mk +4 -4
@@ -280,13 +280,13 @@ define legal-manifest # {HOST|TARGET}, pkg, version, license, license-files, sou echo '"$(2)","$(3)","$(4)","$(5)","$(6)","$(7)","$(8)"' >>$(LEGAL_MANIFEST_CSV_$(1)) endef -define legal-license-file # pkgname, pkgname-pkgver, pkg-hashfile, filename, file-fullpath, {HOST|TARGET} - mkdir -p $(LICENSE_FILES_DIR_$(6))/$(2)/$(dir $(4)) && \ +define legal-license-file # {HOST|TARGET}, pkgname, pkgname-pkgver, filename, file-fullpath, pkg-hashfile + mkdir -p $(LICENSE_FILES_DIR_$(1))/$(3)/$(dir $(4)) && \ { \ - support/download/check-hash $(3) $(5) $(4); \ + support/download/check-hash $(5) $(4) $(6); \ case $${?} in (0|3) ;; (*) exit 1;; esac; \ } && \ - cp $(5) $(LICENSE_FILES_DIR_$(6))/$(2)/$(4) + cp $(5) $(LICENSE_FILES_DIR_$(1))/$(3)/$(4) endef non-virtual-deps = $(foreach p,$(1),$(if $($(call UPPERCASE,$(p))_IS_VIRTUAL),,$(p)))
support/download/check-hash +34 -30
101 lines not on this page · raw message
support/download/dl-wrapper +5 -5
@@ -21,8 +21,8 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:lru:qf:e" main() { local OPT OPTARG - local backend output hfile large_file recurse quiet rc - local -a uris + local backend output large_file recurse quiet rc + local -a uris hfiles # Parse our options; anything after '--' is for the backend while getopts ":c:d:D:o:n:N:H:lrf:u:qp:" OPT; do
@@ -33,7 +33,7 @@ main() { o) output="${OPTARG}";; n) raw_base_name="${OPTARG}";; N) base_name="${OPTARG}";; - H) hfile="${OPTARG}";; + H) hfiles+=( "${OPTARG}" );; l) large_file="-l";; r) recurse="-r";; f) filename="${OPTARG}";;
@@ -70,7 +70,7 @@ main() { # - fails at least one of its hashes: force a re-download # - there's no hash (but a .hash file): consider it a hard error if [ -e "${output}" ]; then - if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then + if support/download/check-hash ${quiet} "${output}" "${output##*/}" "${hfiles[@]}"; then exit 0 elif [ ${?} -ne 2 ]; then # Do not remove the file, otherwise it might get re-downloaded
@@ -154,7 +154,7 @@ main() { # Check if the downloaded file is sane, and matches the stored hashes # for that file - if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then + if support/download/check-hash ${quiet} "${tmpf}" "${output##*/}" "${hfiles[@]}"; then rc=0 else if [ ${?} -ne 3 ]; then
Signature · 6 lines
2.25.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 1/3 v2] support/download: teach dl-wrapper to handle more than one hash file
Quoted 41 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Currently, we expect and only use hash files that lie within the package directory, alongside the .mk file. Those hash files are thus bundeld with Buildroot.
This implies that only what's known to Buildroot can ever get into those hash files. For packages where the version is fixed (or a static choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form entry, where the user can provide a custom location and/or version. like a custom VCS tree and revision, or a custom tarball URL. This means that Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is what was expected. For a sha1 in a git tree, this is a minor issue, because the sha1 by itself is already a hash of the expected content. But for custom tarballs URLs, or for a tag in a VCS, there is indeed no integrity check.
Buildroot can't provide such hashes, but interested users may want to provide those, and currently there is no (easy) way to do so.
So, we need our download helpers to be able to accept more than one hash file to lookup for hashes.
Extend the dl-wrapper and the check-hash helpers thusly, and update the legal-info accordingly.
Note that, to be able to pass more than one hash file, we also need to re-order the arguments passed to support/download/check-hash, which also impies some shuffling in the three places it is called:
- 2 in dl-wrapper
- 1 in the legal-info infraThat in turn also requires that the legal-license-file macro args get re-ordered to have the hash file last; we take the opportunity to also move the HOST/TARGET arg to be first, like in the other legal-info macros.
Quoted 7 lines
Reported-by "Martin Zeiser (mzeiser)" <[email protected]>Signed-off-by Yann E. MORIN <[email protected]>Cc Peter Korsgaard <[email protected]>---
Changes v1 -> v2:
- restore legal-info (Peter K.)
Committed, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 1/3 v2] support/download: teach dl-wrapper to handle more than one hash file
Quoted 46 lines
Quoted 45 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
"Peter" == Peter Korsgaard <[email protected]> writes:
"Yann" == Yann E MORIN <[email protected]> writes:
Currently, we expect and only use hash files that lie within the package directory, alongside the .mk file. Those hash files are thus bundeld with Buildroot.
This implies that only what's known to Buildroot can ever get into those hash files. For packages where the version is fixed (or a static choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form entry, where the user can provide a custom location and/or version. like a custom VCS tree and revision, or a custom tarball URL. This means that Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is what was expected. For a sha1 in a git tree, this is a minor issue, because the sha1 by itself is already a hash of the expected content. But for custom tarballs URLs, or for a tag in a VCS, there is indeed no integrity check.
Buildroot can't provide such hashes, but interested users may want to provide those, and currently there is no (easy) way to do so.
So, we need our download helpers to be able to accept more than one hash file to lookup for hashes.
Extend the dl-wrapper and the check-hash helpers thusly, and update the legal-info accordingly.
Note that, to be able to pass more than one hash file, we also need to re-order the arguments passed to support/download/check-hash, which also impies some shuffling in the three places it is called:
- 2 in dl-wrapper
- 1 in the legal-info infraThat in turn also requires that the legal-license-file macro args get re-ordered to have the hash file last; we take the opportunity to also move the HOST/TARGET arg to be first, like in the other legal-info macros.
Reported-by "Martin Zeiser (mzeiser)" <[email protected]>Signed-off-by Yann E. MORIN <[email protected]>Cc Peter Korsgaard <[email protected]>
Quoted 5 lines
Quoted 3 lines
---
Changes v1 -> v2:
- restore legal-info (Peter K.)Committed, thanks.
Committed to 2023.02.x and 2023.08.x, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 2/3 package/pkg-download: lookup hash files in global-patch-dir
In reply to Yann E. MORINCurrently, we expect and only use hash files that lie within the package directory, alongside the .mk file. Those hash files are thus bundled with Buildroot.
This implies that only what's known to Buildroot can ever get into those hash files. For packages where the version is fixed (or a static choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form entry, where the user can provide a custom location and/or version. like a custom VCS tree and revision, or a custom tarball URL. This means that Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is what was expected. For a sha1 in a git tree, this is a minor issue, because the sha1 by itself is already a hash of the expected content. But for custom tarballs URLs, or for a tag in a VCS, there is indeed no integrity check.
Buildroot can't provide such hashes, but interested users may want to provide those, and currently there is no (easy) way to do so.
We leverage the existng global-patch-dir mechanism to look for extra hash files. We use the same heuristic that is used for bundled hash files, and for each global patch directory <dir>, we use the first file to exist among:
1. look into <dir>/<package>/<version>/<package>.hash
2. look into <dir>/<package>/<package>.hash
---
Changes v1 -> v2:
- drop <dir>/all.hash
--- Config.in | 10 +++++---- docs/manual/adding-packages-directory.adoc | 6 ++++++ docs/manual/customize-patches.adoc | 24 +++++++++++++++++++++- package/pkg-download.mk | 2 +- package/pkg-generic.mk | 14 ++++++++----- package/pkg-utils.mk | 2 +- 6 files changed, 46 insertions(+), 12 deletions(-)
Config.in +6 -4
@@ -674,12 +674,12 @@ config BR2_PACKAGE_OVERRIDE_FILE documentation for more details on this feature. config BR2_GLOBAL_PATCH_DIR - string "global patch directories" + string "global patch and hash directories" help You may specify a space separated list of one or more - directories containing global package patches. For a specific - version <packageversion> of a specific package <packagename>, - patches are applied as follows: + directories containing global package patches and/or hashes. + For a specific version <packageversion> of a specific package + <packagename>, patches are looked up as follows: First, the default Buildroot patch set for the package is applied from the package's directory in Buildroot.
@@ -693,6 +693,8 @@ config BR2_GLOBAL_PATCH_DIR exists, then all *.patch files in the directory will be applied. + The hash files are looked up similarly to the patches. + menu "Advanced" config BR2_FORCE_HOST_BUILD
docs/manual/adding-packages-directory.adoc +6 -0
@@ -540,6 +540,12 @@ mercurial) because Buildroot currently does not generate reproducible tarballs when source code is fetched from such version control systems. +Additionally, for packages for which it is possible to specify a custom +version (e.g. a custom version string, a remote tarball URL, or a VCS +repository location and changeset), Buildroot can't carry hashes for +those. It is however possible to xref:customize-hashes[provide a list of +extra hashes] that can cover such cases. + Hashes should only be added in +.hash+ files for files that are guaranteed to be stable. For example, patches auto-generated by Github are not guaranteed to be stable, and therefore their hashes can change
docs/manual/customize-patches.adoc +23 -1
@@ -1,8 +1,10 @@ // -*- mode:doc -*- ; // vim: set syntax=asciidoc: +=== Adding project-specific patches and hashes + [[customize-patches]] -=== Adding project-specific patches +==== Providing extra patches It is sometimes useful to apply 'extra' patches to packages - on top of those provided in Buildroot. This might be used to support custom
@@ -57,3 +59,23 @@ are available at a URL. *Note:* +BR2_LINUX_KERNEL_PATCH+ specifies kernel patches that are applied after patches available in +BR2_GLOBAL_PATCH_DIR+, as it is done from a post-patch hook of the Linux package. + +[[customize-hashes]] +==== Providing extra hashes + +Buildroot bundles a xref:adding-packages-hash[list of hashes] against +which it checks the integrity of the downloaded archives, or of those +it generates locally from VCS checkouts. However, it can only do so +for the known versions; for packages where it is possible to specify +a custom version (e.g. a custom version string, a remote tarball URL, +or a VCS repository location and changeset), Buildroot can't carry +hashes for those. + +For users concerned with the integrity of such downloads, it is possible +to provide a list of hashes that Buildroot can use to check arbitrary +downloaded files. Those extra hashes are looked up similarly to the +extra patches (above); for each directory in +BR2_GLOBAL_PATCH_DIR+, +the first file to exist is used to check a package download: + +* +<global-patch-dir>/<packagename>/<packageversion>/<packagename>.hash+ +* +<global-patch-dir>/<packagename>/<packagename>.hash+
package/pkg-download.mk +1 -1
@@ -115,7 +115,7 @@ define DOWNLOAD -d '$($(2)_DL_DIR)' \ -D '$(DL_DIR)' \ -f '$(notdir $(1))' \ - -H '$($(2)_HASH_FILE)' \ + $(foreach f,$($(2)_HASH_FILES),-H '$(f)') \ -n '$($(2)_BASENAME_RAW)' \ -N '$($(2)_RAWNAME)' \ -o '$($(2)_DL_DIR)/$(notdir $(1))' \
package/pkg-generic.mk +9 -5
@@ -509,11 +509,15 @@ else endif $(2)_VERSION := $$(call sanitize,$$($(2)_DL_VERSION)) -$(2)_HASH_FILE = \ +$(2)_HASH_FILES = \ $$(strip \ - $$(if $$(wildcard $$($(2)_PKGDIR)/$$($(2)_VERSION)/$$($(2)_RAWNAME).hash),\ - $$($(2)_PKGDIR)/$$($(2)_VERSION)/$$($(2)_RAWNAME).hash,\ - $$($(2)_PKGDIR)/$$($(2)_RAWNAME).hash)) + $$(foreach d, $$($(2)_PKGDIR) $$(addsuffix /$$($(2)_RAWNAME), $$(call qstrip,$$(BR2_GLOBAL_PATCH_DIR))),\ + $$(if $$(wildcard $$(d)/$$($(2)_VERSION)/$$($(2)_RAWNAME).hash),\ + $$(d)/$$($(2)_VERSION)/$$($(2)_RAWNAME).hash,\ + $$(d)/$$($(2)_RAWNAME).hash\ + )\ + )\ + ) ifdef $(3)_OVERRIDE_SRCDIR $(2)_OVERRIDE_SRCDIR ?= $$($(3)_OVERRIDE_SRCDIR)
@@ -1138,7 +1142,7 @@ ifneq ($$(call qstrip,$$($(2)_SOURCE)),) ifeq ($$(call qstrip,$$($(2)_LICENSE_FILES)),) $(Q)$$(call legal-warning-pkg,$$($(2)_BASENAME_RAW),cannot save license ($(2)_LICENSE_FILES not defined)) else - $(Q)$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$(call UPPERCASE,$(4)),$$($(2)_RAWNAME),$$($(2)_BASENAME_RAW),$$(F),$$($(2)_DIR)/$$(F),$$($(2)_HASH_FILE))$$(sep)) + $(Q)$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$(call UPPERCASE,$(4)),$$($(2)_RAWNAME),$$($(2)_BASENAME_RAW),$$(F),$$($(2)_DIR)/$$(F),$$($(2)_HASH_FILES))$$(sep)) endif # license files ifeq ($$($(2)_REDISTRIBUTE),YES)
package/pkg-utils.mk +1 -1
@@ -280,7 +280,7 @@ define legal-manifest # {HOST|TARGET}, pkg, version, license, license-files, sou echo '"$(2)","$(3)","$(4)","$(5)","$(6)","$(7)","$(8)"' >>$(LEGAL_MANIFEST_CSV_$(1)) endef -define legal-license-file # {HOST|TARGET}, pkgname, pkgname-pkgver, filename, file-fullpath, pkg-hashfile +define legal-license-file # {HOST|TARGET}, pkgname, pkgname-pkgver, filename, file-fullpath, pkg-hashfiles mkdir -p $(LICENSE_FILES_DIR_$(1))/$(3)/$(dir $(4)) && \ { \ support/download/check-hash $(5) $(4) $(6); \
Signature · 6 lines
2.25.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/3 v2] package/pkg-download: lookup hash files in global-patch-dir
Quoted 31 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Currently, we expect and only use hash files that lie within the package directory, alongside the .mk file. Those hash files are thus bundled with Buildroot.
This implies that only what's known to Buildroot can ever get into those hash files. For packages where the version is fixed (or a static choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form entry, where the user can provide a custom location and/or version. like a custom VCS tree and revision, or a custom tarball URL. This means that Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is what was expected. For a sha1 in a git tree, this is a minor issue, because the sha1 by itself is already a hash of the expected content. But for custom tarballs URLs, or for a tag in a VCS, there is indeed no integrity check.
Buildroot can't provide such hashes, but interested users may want to provide those, and currently there is no (easy) way to do so.
We leverage the existng global-patch-dir mechanism to look for extra hash files. We use the same heuristic that is used for bundled hash files, and for each global patch directory <dir>, we use the first file to exist among:
1. look into <dir>/<package>/<version>/<package>.hash
2. look into <dir>/<package>/<package>.hash
Quoted 6 lines
Reported-by "Martin Zeiser (mzeiser)" <[email protected]>Signed-off-by Yann E. MORIN <[email protected]>---
Changes v1 -> v2:
- drop <dir>/all.hash
Committed, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/3 v2] package/pkg-download: lookup hash files in global-patch-dir
In reply to Yann E. MORINQuoted 31 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Currently, we expect and only use hash files that lie within the package directory, alongside the .mk file. Those hash files are thus bundled with Buildroot.
This implies that only what's known to Buildroot can ever get into those hash files. For packages where the version is fixed (or a static choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form entry, where the user can provide a custom location and/or version. like a custom VCS tree and revision, or a custom tarball URL. This means that Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is what was expected. For a sha1 in a git tree, this is a minor issue, because the sha1 by itself is already a hash of the expected content. But for custom tarballs URLs, or for a tag in a VCS, there is indeed no integrity check.
Buildroot can't provide such hashes, but interested users may want to provide those, and currently there is no (easy) way to do so.
We leverage the existng global-patch-dir mechanism to look for extra hash files. We use the same heuristic that is used for bundled hash files, and for each global patch directory <dir>, we use the first file to exist among:
1. look into <dir>/<package>/<version>/<package>.hash
2. look into <dir>/<package>/<package>.hash
Quoted 6 lines
Reported-by "Martin Zeiser (mzeiser)" <[email protected]>Signed-off-by Yann E. MORIN <[email protected]>---
Changes v1 -> v2:
- drop <dir>/all.hash
Committed to 2023.02.x and 2023.08.x, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 3/3 pkg-download: add option to enforce hash checking
In reply to Yann E. MORINCurrently, when a pacakge is downloaded from a custom location or version, Buildroot excludes such a package from the mandatory integrity check with hashes, because it was until now not possible to have such hashes.
We now have a mechanism which users can leverage to provide additional hashes, and so custom versions or locations can now be checked too.
Buildroot has no way to know that hashes have indeed been provided for a custom location/version, and so will still happilly ignore an unchecked package.
However, users who do provide extra hashes most probably do expect that no download is done without an integrity check, and thus expect that a missing hash not be ignored.
Add an option that users can select to make Buildroot forcibly require at least one valid hash, and no invalid hash, for all downloads.
--- Config.in | 17 +++++++++++++++++ package/pkg-download.mk | 5 ++--- 2 files changed, 19 insertions(+), 3 deletions(-)
Config.in +17 -0
@@ -709,6 +709,23 @@ config BR2_FORCE_HOST_BUILD This option will increase build time. +config BR2_DOWNLOAD_FORCE_CHECK_HASHES + bool "Force all downloads to have a valid hash" + depends on BR2_GLOBAL_PATCH_DIR != "" + help + For packages where a custom version or location can be set, + Buildroot does not carry a hash for those custom versions or + locations, so the integrity of such downloads is not verified. + + Say 'y' here to enforce downloads to have at least one valid + hash (and of course, that all hashes be valid). + + Those hashes are looked in files in BR2_GLOBAL_PATCH_DIR, + see above. + +comment "Forcing all downloads to have a valid hash needs a global patch and hash directory" + depends on BR2_GLOBAL_PATCH_DIR = "" + config BR2_REPRODUCIBLE bool "Make the build reproducible (experimental)" # SOURCE_DATE_EPOCH support in toolchain-wrapper requires GCC 4.4
package/pkg-download.mk +2 -3
@@ -66,9 +66,7 @@ github = https://github.com/$(1)/$(2)/archive/$(3) gitlab = https://gitlab.com/$(1)/$(2)/-/archive/$(3) # Expressly do not check hashes for those files -# Exported variables default to immediately expanded in some versions of -# make, but we need it to be recursively-epxanded, so explicitly assign it. -export BR_NO_CHECK_HASH_FOR = +BR_NO_CHECK_HASH_FOR = ################################################################################ # DOWNLOAD_URIS - List the candidates URIs where to get the package from:
@@ -110,6 +108,7 @@ endif define DOWNLOAD $(Q)mkdir -p $($(2)_DL_DIR) $(Q)$(EXTRA_ENV) $($(2)_DL_ENV) \ + BR_NO_CHECK_HASH_FOR="$(if $(BR2_DOWNLOAD_FORCE_CHECK_HASHES),,$(BR_NO_CHECK_HASH_FOR))" \ flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \ -c '$($(2)_DL_VERSION)' \ -d '$($(2)_DL_DIR)' \
Signature · 6 lines
2.25.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 3/3 v2] pkg-download: add option to enforce hash checking
Quoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Currently, when a pacakge is downloaded from a custom location or version, Buildroot excludes such a package from the mandatory integrity check with hashes, because it was until now not possible to have such hashes.
We now have a mechanism which users can leverage to provide additional hashes, and so custom versions or locations can now be checked too.
Buildroot has no way to know that hashes have indeed been provided for a custom location/version, and so will still happilly ignore an unchecked package.
However, users who do provide extra hashes most probably do expect that no download is done without an integrity check, and thus expect that a missing hash not be ignored.
Quoted 5 lines
Add an option that users can select to make Buildroot forcibly require at least one valid hash, and no invalid hash, for all downloads.
Reported-by "Martin Zeiser (mzeiser)" <[email protected]>Signed-off-by Yann E. MORIN <[email protected]>
Committed, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 3/3 v2] pkg-download: add option to enforce hash checking
In reply to Yann E. MORINQuoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Currently, when a pacakge is downloaded from a custom location or version, Buildroot excludes such a package from the mandatory integrity check with hashes, because it was until now not possible to have such hashes.
We now have a mechanism which users can leverage to provide additional hashes, and so custom versions or locations can now be checked too.
Buildroot has no way to know that hashes have indeed been provided for a custom location/version, and so will still happilly ignore an unchecked package.
However, users who do provide extra hashes most probably do expect that no download is done without an integrity check, and thus expect that a missing hash not be ignored.
Quoted 5 lines
Add an option that users can select to make Buildroot forcibly require at least one valid hash, and no invalid hash, for all downloads.
Reported-by "Martin Zeiser (mzeiser)" <[email protected]>Signed-off-by Yann E. MORIN <[email protected]>
Committed to 2023.02.x and 2023.08.x, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot