support/download: add blake2b-256 hash support
PATCH v2 1/3 support/download: add blake2b-256 hash support
PyPI exposes a blake2b_256 digest for distribution files, and that same digest is used in the hash-based files.pythonhosted.org download paths. Add support for storing those hashes in Buildroot .hash files.
Teach the download hash checker to validate blake2b-256 entries using b2sum -l 256, and update check-package so the new hash type and length are accepted. Also add the CycloneDX hash spelling so generated SBOMs can include blake2b-256 hashes from .hash files.
Update scanpypi to write the PyPI-provided blake2b_256 digest alongside the existing md5 and sha256 entries.
--- docs/manual/adding-packages-directory.adoc | 3 ++- support/download/check-hash | 7 ++++++- support/testing/tests/utils/test_generate_cyclonedx.py | 9 +++++++++ utils/checkpackagelib/lib_hash.py | 2 +- utils/checkpackagelib/test_lib_hash.py | 4 ++++ utils/generate-cyclonedx | 1 + utils/scanpypi | 7 ++++++- 7 files changed, 29 insertions(+), 4 deletions(-)
docs/manual/adding-packages-directory.adoc +2 -1
@@ -464,7 +464,7 @@ The format of this file is one line for each file for which to check the hash, each line with the following three fields separated by two spaces: * the type of hash, one of: -** +md5+, +sha1+, +sha224+, +sha256+, +sha384+, +sha512+ +** +md5+, +sha1+, +sha224+, +sha256+, +sha384+, +sha512+, +blake2b-256+ * the hash of the file: ** for +md5+, 32 hexadecimal characters ** for +sha1+, 40 hexadecimal characters
@@ -472,6 +472,7 @@ hash, each line with the following three fields separated by two spaces: ** for +sha256+, 64 hexadecimal characters ** for +sha384+, 96 hexadecimal characters ** for +sha512+, 128 hexadecimal characters +** for +blake2b-256+, 64 hexadecimal characters * the name of the file: ** for a source archive: the basename of the file, without any directory component,
support/download/check-hash +6 -1
@@ -49,6 +49,7 @@ check_one_hash() { case "${_h}" in md5|sha1) ;; sha224|sha256|sha384|sha512) ;; + blake2b-256) ;; *) # Unknown hash, exit with error printf "ERROR: unknown hash '%s' for '%s'\n" \ "${_h}" "${base}" >&2
@@ -57,7 +58,11 @@ check_one_hash() { esac # Do the hashes match? - _hash="$( "${_h}sum" "${_file}" |cut -d ' ' -f 1 )" + if [ "${_h}" = "blake2b-256" ]; then + _hash="$( b2sum -l 256 "${_file}" |cut -d ' ' -f 1 )" + else + _hash="$( "${_h}sum" "${_file}" |cut -d ' ' -f 1 )" + fi if [ "${_hash}" = "${_known}" ]; then printf "%s: OK (%s: %s)\n" "${base}" "${_h}" "${_hash}" return 0
support/testing/tests/utils/test_generate_cyclonedx.py +9 -0
@@ -186,6 +186,7 @@ class TestGenerateCycloneDX(unittest.TestCase): "# source archive checksums\n" "sha256 1111111111111111111111111111111111111111111111111111111111111111 foo-1.2.tar.gz\n" "sha1 2222222222222222222222222222222222222222 foo-1.2.tar.gz\n" + "blake2b-256 3333333333333333333333333333333333333333333333333333333333333333 foo-1.2.tar.gz\n" "sha256 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa LICENSE\n" )
@@ -220,6 +221,10 @@ class TestGenerateCycloneDX(unittest.TestCase): "alg": "SHA-1", "content": "2222222222222222222222222222222222222222", }, + { + "alg": "BLAKE2b-256", + "content": "3333333333333333333333333333333333333333333333333333333333333333", + }, ] }, {
@@ -234,6 +239,10 @@ class TestGenerateCycloneDX(unittest.TestCase): "alg": "SHA-1", "content": "2222222222222222222222222222222222222222", }, + { + "alg": "BLAKE2b-256", + "content": "3333333333333333333333333333333333333333333333333333333333333333", + }, ], } ],
utils/checkpackagelib/lib_hash.py +1 -1
@@ -31,7 +31,7 @@ class HashNumberOfFields(_CheckFunction): class HashType(_CheckFunction): len_of_hash = {"md5": 32, "sha1": 40, "sha224": 56, "sha256": 64, - "sha384": 96, "sha512": 128} + "sha384": 96, "sha512": 128, "blake2b-256": 64} def check_line(self, lineno, text): if _empty_line_or_comment(text):
utils/checkpackagelib/test_lib_hash.py +4 -0
@@ -124,6 +124,10 @@ HashType = [ 'sha512 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678' '9012345678\n', []), + ('blake2b-256', + 'any', + 'blake2b-256 1234567890123456789012345678901234567890123456789012345678901234\n', + []), ]
utils/generate-cyclonedx +1 -0
@@ -298,6 +298,7 @@ def cyclonedx_source_hashes(comp, source): "sha256": "SHA-256", "sha512": "SHA-512", "md5": "MD5", + "blake2b-256": "BLAKE2b-256", } hashes = []
utils/scanpypi +6 -1
@@ -672,7 +672,7 @@ class BuildrootPackage(): print('Creating {filename}...'.format(filename=path_to_hash)) lines = [] if self.used_url['digests']['md5'] and self.used_url['digests']['sha256']: - hash_header = '# md5, sha256 from {url}\n'.format( + hash_header = '# md5, sha256, blake2b-256 from {url}\n'.format( url=self.metadata_url) lines.append(hash_header) hash_line = '{method} {digest} {filename}\n'.format(
@@ -685,6 +685,11 @@ class BuildrootPackage(): digest=self.used_url['digests']['sha256'], filename=self.filename) lines.append(hash_line) + hash_line = '{method} {digest} {filename}\n'.format( + method='blake2b-256', + digest=self.used_url['digests']['blake2b_256'], + filename=self.filename) + lines.append(hash_line) if self.license_files: lines.append('# Locally computed sha256 checksums\n')
Signature · 6 lines
2.53.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 2/3 package/pkg-python: derive PyPI site from blake2b-256 hash
Python packages downloaded from PyPI currently store the complete files.pythonhosted.org package path in their .mk file. That path is hash-based, so it changes for every version bump and adds churn to updates and backports.
When a Python package does not define _SITE, derive the hash-based files.pythonhosted.org site from the blake2b-256 entry matching the package source archive in the package .hash file before the generic package infrastructure validates _SITE.
Follow the usual target/host fallback pattern for _SITE, and raise an explicit error if no hash file is available or if the hash file does not contain a matching blake2b-256 entry.
Update scanpypi to omit generated _SITE assignments so newly generated PyPI packages can use the resolver.
---
Changes v1 -> v2:
- Follow the usual target/host _SITE fallback pattern.
- Drop the /dev/null awk input and guard missing hash files explicitly.
- Error out when no matching blake2b-256 entry is found.
--- docs/manual/adding-packages-python.adoc | 5 +++++ package/pkg-python.mk | 14 ++++++++++++++ utils/scanpypi | 9 --------- 3 files changed, 19 insertions(+), 9 deletions(-)
docs/manual/adding-packages-python.adoc +5 -0
@@ -82,6 +82,11 @@ All the package metadata information variables that exist in the xref:generic-package-reference[generic package infrastructure] also exist in the Python infrastructure. +For Python packages downloaded from PyPI, the +PYTHON_FOO_SITE+ variable +can be omitted when the package +.hash+ file contains a +blake2b-256+ +hash for the source archive. Buildroot derives the hash-based +files.pythonhosted.org URL from that hash. + Note that: * It is not necessary to add +python+ or +host-python+ in the
package/pkg-python.mk +14 -0
@@ -454,6 +454,20 @@ endif endif # host / target +ifndef $(2)_SITE + ifdef $(3)_SITE + $(2)_SITE = $$($(3)_SITE) + else + $(2)_SITE = $$(strip $$(if $$(strip $$($(2)_HASH_FILES)),\ + $$(or $$(shell \ + awk -v filename="$$(notdir $$($(2)_SOURCE))" \ + '$$$$1 == "blake2b-256" && $$$$3 == filename { h = $$$$2; printf "https://files.pythonhosted.org/packages/%s/%s/%s", substr(h, 1, 2), substr(h, 3, 2), substr(h, 5); exit }' \ + $$($(2)_HASH_FILES)),\ + $$(error $(2)_SITE must be set or $$($(2)_HASH_FILES) must contain a blake2b-256 hash for $$($(2)_SOURCE))),\ + $$(error $(2)_SITE must be set or a hash file must exist to derive the PyPI site))) + endif +endif + # Call the generic package infrastructure to generate the necessary # make targets $(call inner-generic-package,$(1),$(2),$(3),$(4))
utils/scanpypi +0 -9
@@ -495,15 +495,6 @@ class BuildrootPackage(): filename=targz) lines.append(targz_line) - if self.filename not in self.url: - # Sometimes the filename is in the url, sometimes it's not - site_url = self.url - else: - site_url = self.url[:self.url.find(self.filename)] - site_line = '{name}_SITE = {url}'.format(name=self.mk_name, - url=site_url) - site_line = site_line.rstrip('/') + '\n' - lines.append(site_line) return lines def __create_mk_setup(self):
Signature · 6 lines
2.53.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 3/3 package/python-async-timeout: use hash-based PyPI site
In reply to James HilliardAdd the blake2b-256 digest from the PyPI metadata and drop the explicit PYTHON_ASYNC_TIMEOUT_SITE assignment.
The Python package infrastructure can now derive the hash-based files.pythonhosted.org site from the package .hash file.
--- package/python-async-timeout/python-async-timeout.hash | 3 ++- package/python-async-timeout/python-async-timeout.mk | 1 - 2 files changed, 2 insertions(+), 2 deletions(-)
package/python-async-timeout/python-async-timeout.hash +2 -1
@@ -1,5 +1,6 @@ -# md5, sha256 from https://pypi.org/pypi/async-timeout/json +# md5, sha256, blake2b-256 from https://pypi.org/pypi/async-timeout/json md5 566a39011e87cb8044ee75369e69b327 async_timeout-5.0.1.tar.gz sha256 d9321a7a3d5a6a5e187e824d2fa0793ce379a202935782d555d6e9d2735677d3 async_timeout-5.0.1.tar.gz +blake2b-256 a5ae136395dfbfe00dfc94da3f3e136d0b13f394cba8f4841120e34226265780 async_timeout-5.0.1.tar.gz # Locally computed sha256 checksums sha256 e18d7bb8f513e2c46bb585c94b585bd30720dd3ccb21ddb0786f72d16658f92c LICENSE
package/python-async-timeout/python-async-timeout.mk +0 -1
@@ -6,7 +6,6 @@ PYTHON_ASYNC_TIMEOUT_VERSION = 5.0.1 PYTHON_ASYNC_TIMEOUT_SOURCE = async_timeout-$(PYTHON_ASYNC_TIMEOUT_VERSION).tar.gz -PYTHON_ASYNC_TIMEOUT_SITE = https://files.pythonhosted.org/packages/a5/ae/136395dfbfe00dfc94da3f3e136d0b13f394cba8f4841120e34226265780 PYTHON_ASYNC_TIMEOUT_LICENSE = Apache-2.0 PYTHON_ASYNC_TIMEOUT_LICENSE_FILES = LICENSE PYTHON_ASYNC_TIMEOUT_SETUP_TYPE = setuptools
Signature · 6 lines
2.53.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
James, All,
On 2026-06-17 11:42 -0600, James Hilliard spake thusly:
[--SNIP--]
Quoted 4 lines
Teach the download hash checker to validate blake2b-256 entries using b2sum -l 256, and update check-package so the new hash type and length are accepted. Also add the CycloneDX hash spelling so generated SBOMs can include blake2b-256 hashes from .hash files.
[--SNIP--]
support/download/check-hash check_one_hash() { · line 62
Quoted 2 lines
support/download/check-hash +2
-0+ if [ "${_h}" = "blake2b-256" ]; then + _hash="$( b2sum -l 256 "${_file}" |cut -d ' ' -f 1 )"
b2sum was only introduced with coreutils 8.26, but we support running on systems that have an older version: we already check that ln supports --relative (introduced in 8.16) and that realpath exists (introduced in 8.15).
So, we probably want to add b2sum to the list of tools to check in support/dependencies/check-host-coreutils.s and then add $(BR2_COREUTILS_HOST_DEPENDENCY) to the dependencies of packages that have a blake2 hash (the python infra can do so, for example).
Regards,
Yann E. MORIN.
Signature · 23 lines
____________ .-----------------.--------------------: _ :------------------. | Yann E. MORIN | Real-Time Embedded | __/ ) | /"\ ASCII RIBBON | | | Software Designer | _/ - /' | \ / CAMPAIGN | | +33 638.411.245 '--------------------: (_ `--, | X AGAINST | | yann.morin (at) orange.com |_=" ,--' | / \ HTML MAIL | '--------------------------------------:______/_____:------------------' ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] minimum system/coreutils requirements (was: support/download: add blake2b-256 hash support)
Hi Yann, James, (Peter et. all),
I wondered about the b2sum dependency as well but the conclusion was "a version from 10 years ago is probably fine".
Some discussion on this below.
Quoted 8 lines · On 6/18/26 07:51, [email protected] wrote:
James, All,
On 2026-06-17 11:42 -0600, James Hilliard spake thusly:
[--SNIP--]Quoted 4 lines
Teach the download hash checker to validate blake2b-256 entries using b2sum -l 256, and update check-package so the new hash type and length are accepted. Also add the CycloneDX hash spelling so generated SBOMs can include blake2b-256 hashes from .hash files.
Quoted 8 lines
[--SNIP--]
support/download/check-hash check_one_hash() { · line 62
Quoted 2 lines
support/download/check-hash +2
-0+ if [ "${_h}" = "blake2b-256" ]; then + _hash="$( b2sum -l 256 "${_file}" |cut -d ' ' -f 1 )"b2sum was only introduced with coreutils 8.26, but we support running on systems that have an older version: we already check that ln supports --relative (introduced in 8.16) and that realpath exists (introduced in 8.15).
Coreutils 8.26 was released almost 10 years ago, 8.15 was released (slightly) over 14 years ago. I wonder at what point we can assume that having a version with the required functionality is a given. 10 years is not thaaat long ago in lts distro terms, I suppose, but even debian old-old-stable has b2sum already. I also wonder if buildroot actually fully works on a 10 or 15 year old system. Without known users or actual automatic tests being run it feels kind of weird to assume it does.
Anyway, thoughts about this? Anyone got actual experience on running buildroot on these old systems? (I'm more on the side of running into breakage on the newest stuff on archlinux)
Quoted 5 lines
So, we probably want to add b2sum to the list of tools to check in
support/dependencies/check-host-coreutils.sand then add $(BR2_COREUTILS_HOST_DEPENDENCY) to the dependencies of packages that have a blake2 hash (the python infra can do so, for example).
James already sent a version which does that and it seems okay to merge if anyone actually needs this on old versions.
It's not super great though because if someone where to add a blake2b hash for a non-python package which then someone would try to build on an old distro it would still break, I think?
I guess I'd mostly like to know if this complexity is actually worth it and if we would want to at least define a minimum supported coreutils version which we could bump at some point to get rid of these checks again.
Opinions?
Quoted 4 lines
Regards,
Yann E. MORIN.
Best,
Marcus
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] minimum system/coreutils requirements
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Marcus" == Marcus Hoffmann <[email protected]> writes:
Hello,
Quoted 7 lines
James already sent a version which does that and it seems okay to merge if anyone actually needs this on old versions.
It's not super great though because if someone where to add a blake2b hash for a non-python package which then someone would try to build on an old distro it would still break, I think?
Quoted 6 lines
I guess I'd mostly like to know if this complexity is actually worth it and if we would want to at least define a minimum supported coreutils version which we could bump at some point to get rid of these checks again.
Opinions?
I would not be against just requiring b2sum on the build host. This is presumably not a feature we would want to backport to 2025.02.x, so "enterprise" users will most likely only run into it when moving to 2027.02.x.
In Debian terms, coreutils 8.26 was included in Debian 9 which was released 2017/06/17 and will go out of ELTS support on 2027/06/30, so it is not a suitable distribution to use for the duration of our 2027.02.x anyway.
I think time has also come to drop the check for realpath / relative ln.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] minimum system/coreutils requirements (was: support/download: add blake2b-256 hash support)
In reply to Marcus Hoffmann via buildrootMarcus, James, All,
On 2026-06-18 16:04 +0200, Marcus Hoffmann spake thusly:
Quoted 2 lines · On 6/18/26 07:51, [email protected] wrote:
Quoted 1 line
On 2026-06-17 11:42 -0600, James Hilliard spake thusly:
[--SNIP--]
Quoted 13 lines
Quoted 4 lines
b2sum was only introduced with coreutils 8.26, but we support running on systems that have an older version: we already check that ln supports --relative (introduced in 8.16) and that realpath exists (introduced in 8.15).
Coreutils 8.26 was released almost 10 years ago, 8.15 was released (slightly) over 14 years ago. I wonder at what point we can assume that having a version with the required functionality is a given. 10 years is not thaaat long ago in lts distro terms, I suppose, but even debian old-old-stable has b2sum already. I also wonder if buildroot actually fully works on a 10 or 15 year old system. Without known users or actual automatic tests being run it feels kind of weird to assume it does.
Quoted 3 lines
Anyway, thoughts about this? Anyone got actual experience on running buildroot on these old systems? (I'm more on the side of running into breakage on the newest stuff on archlinux)
Thing is, either we clain we can run on such systems, and so we extend the existing checks for those, or we don't claim so and we can drop the tests.
But since we have the tests for ln --relative and realpath, we either need to raise the requirement for coreutils on the host, or we also need to test for b2sum. We can't test for ln -relative and realpath on one hand, but not for b2sum on the other.
[--SNIP--]
Quoted 3 lines
It's not super great though because if someone where to add a blake2b hash for a non-python package which then someone would try to build on an old distro it would still break, I think?
Sure, but then it would be a bug in Buildroot; as I said, a package that adds a blake2 hash would have to have a dependency on BR2_COREUTILS_HOST_DEPENDENCY.
Unless of course we decide to raise the bar to require at least coreutils 8.26 or later... But since this is not currently the case, I think it is easier to also check for b2sum for now...
Quoted 4 lines
I guess I'd mostly like to know if this complexity is actually worth it and if we would want to at least define a minimum supported coreutils version which we could bump at some point to get rid of these checks again. Opinions?
Some people did complain in 2019-or-such that they were still running with system dating from before 8.15 (released 2012, 7 years before), so had issues with ln --relative and/or realpath. That makes for 14-year old systems today; even entreprise-grade distros that are that old and still in maintenance are few: RHEL 6, released 2011, got EOLed in 2024, while RHEL 7 was released 2014 and will get EOLed in 2028...
In any case I was just pointing to a discrepancy betweeni how we handle various tools provided by coreutils... Whatever the eventual outcome, I don't care much, as long as it is homogeneous.
Regards,
Yann E. MORIN.
Signature · 23 lines
____________ .-----------------.--------------------: _ :------------------. | Yann E. MORIN | Real-Time Embedded | __/ ) | /"\ ASCII RIBBON | | | Software Designer | _/ - /' | \ / CAMPAIGN | | +33 638.411.245 '--------------------: (_ `--, | X AGAINST | | yann.morin (at) orange.com |_=" ,--' | / \ HTML MAIL | '--------------------------------------:______/_____:------------------' ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] minimum system/coreutils requirements
Yann,
On 6/18/26 16:59, [email protected] wrote:
[...]
Quoted 4 lines
In any case I was just pointing to a discrepancy betweeni how we handle various tools provided by coreutils... Whatever the eventual outcome, I don't care much, as long as it is homogeneous.
Got it, makes sense!
So I think at this point it's easiest to apply James' series with the check included and then I send a follow-up to remove all the coreutils checks and add a note to the docs that we require 8.26 for the upcoming version.
Quoted 4 lines
Regards,
Yann E. MORIN.
Marcus
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot