Hash verification from GitHub
Hello folks
I'm trying to create a new buildroot package (my first one). This is what the makefile (tealdeer.mk) looks like:
TEALDEER_VERSION = 1.5.0 TEALDEER_SITE = $(call github,dbrgn,tealdeer,v$(TEALDEER_VERSION)) TEALDEER_LICENSE = Apache-2.0 or MIT TEALDEER_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT $(eval $(cargo-package))
The URL should expand to
https://github.com/dbrgn/tealdeer/archive/v1.5.0/tealdeer-1.5.0.tar.gz.
To generate the checksum, I ran:
$ sha256sum tealdeer-1.5.0.tar.gz 00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a tealdeer-1.5.0.tar.gz
I also added a hash file (tealdeer.hash):
# Locally generated
sha256 00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a tealdeer-1.5.0.tar.gz sha256 62c7a1e35f56406896d7aa7ca52d0cc0d272ac022b5d2796e7d6905db8a3636a LICENSE-APACHE sha256 a313b5e62b80a08f3aae0fa62ff3de8482ef55247299eb352ab44f87ef456b1b LICENSE-MIT
When building this package, checksum verification fails every time.
ERROR: tealdeer-1.5.0.tar.gz has wrong sha256 hash:
ERROR: expected: 00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a
ERROR: got : 42febf9ee84721b9230077d62e2fc51201fd59624d3c776ccc1a634788768a60
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
No matter how I download the file (via wget, through the GitHub web UI, etc), it always results in the SHA256 checksum starting with 009..., but buildroot always thinks it should be 42f... I also tried changing the TEALDEER_SITE variable as follows:
TEALDEER_SITE = https://github.com/dbrgn/tealdeer/archive/v$(TEALDEER_VERSION)
...to ensure that this URL is *really* being downloaded, but it fails every time.
Full build log can be found here:
https://gist.github.com/dbrgn/cc9e96051a079f5b63c531ca3c195954
Does someone have any pointers why the hash verification would fail in this case? It must be some obvious mistake I'm making, but I cannot figure out what it is.
Best regards,
Danilo Bargen
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 12 lines · Danilo Bargen
Hello folks
I'm trying to create a new buildroot package (my first one). This is what the makefile (tealdeer.mk) looks like:
TEALDEER_VERSION = 1.5.0 TEALDEER_SITE = $(call github,dbrgn,tealdeer,v$(TEALDEER_VERSION)) TEALDEER_LICENSE = Apache-2.0 or MIT TEALDEER_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT $(eval $(cargo-package))
Quoted 6 lines
The URL should expand to
https://github.com/dbrgn/tealdeer/archive/v1.5.0/tealdeer-1.5.0.tar.gz.
To generate the checksum, I ran:$ sha256sum tealdeer-1.5.0.tar.gz 00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a tealdeer-1.5.0.tar.gz
Just use the hash buildroot generated for you, it should be correct.
Quoted 8 lines
I also added a hash file (tealdeer.hash):
# Locally generated
sha256 00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a tealdeer-1.5.0.tar.gz sha256 62c7a1e35f56406896d7aa7ca52d0cc0d272ac022b5d2796e7d6905db8a3636a LICENSE-APACHE sha256 a313b5e62b80a08f3aae0fa62ff3de8482ef55247299eb352ab44f87ef456b1b LICENSE-MIT
Quoted 5 lines
When building this package, checksum verification fails every time.
ERROR: tealdeer-1.5.0.tar.gz has wrong sha256 hash:
ERROR: expected:00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a
ERROR: got :42febf9ee84721b9230077d62e2fc51201fd59624d3c776ccc1a634788768a60
Use this hash.
Quoted 6 lines
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
No matter how I download the file (via wget, through the GitHub web UI, etc), it always results in the SHA256 checksum starting with 009..., but buildroot always thinks it should be 42f... I also tried changing the
TEALDEER_SITEvariable as follows:
Yeah, it's not going to match since buildroot is doing stuff like vendoring the cargo deps
before generating the tarball(to ensure that the package can be built from the tarball without
having to download anything additional from cargo):
https://github.com/buildroot/buildroot/blob/master/support/download/cargo-post-process
Quoted 8 lines
TEALDEER_SITE = https://github.com/dbrgn/tealdeer/archive/v$(TEALDEER_VERSION)...to ensure that this URL is *really* being downloaded, but it fails every time.
Full build log can be found here:
https://gist.github.com/dbrgn/cc9e96051a079f5b63c531ca3c195954
You can see it vendoring the deps in the log here:
https://gist.github.com/dbrgn/cc9e96051a079f5b63c531ca3c195954#file-gistfile1-txt-L23-L200
Quoted 11 lines
Does someone have any pointers why the hash verification would fail in this case? It must be some obvious mistake I'm making, but I cannot figure out what it is.
Best regards,
Danilo Bargen
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Hello James
Quoted 4 lines
Just use the hash buildroot generated for you, it should be correct. (...)
Yeah, it's not going to match since buildroot is doing stuff like vendoring the cargo deps before generating the tarball
Ahh, that explains it. Does this mean that for Cargo based packages, it will always be TOFU and checksums provided by the project / maintainer cannot be used (because they will mismatch)?
Danilo
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 11 lines · Danilo Bargen
Hello James
Quoted 4 lines
Just use the hash buildroot generated for you, it should be correct. (...)
Yeah, it's not going to match since buildroot is doing stuff like vendoring the cargo deps before generating the tarballAhh, that explains it. Does this mean that for Cargo based packages, it will always be TOFU and checksums provided by the project / maintainer cannot be used (because they will mismatch)?
A maintainer could sign the tarball generated by buildroot and upload it to github releases if they want.
As you can see the script will bail out if the tarball already has vendored deps:
https://github.com/buildroot/buildroot/blob/6e4791b751c0f8e0ba218da2c22e71d3e1436b5d/support/download/cargo-post-process#L16-L19
So as long as the upstream release tarball has vendored deps the hash should not change AFAIU.
Quoted 2 lines
Danilo
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
James, Danilo, All,
On 2022-01-16 15:51 -0700, James Hilliard spake thusly:
Quoted 10 lines · Danilo Bargen
Quoted 7 lines
Quoted 4 lines
Just use the hash buildroot generated for you, it should be correct. (...)
Yeah, it's not going to match since buildroot is doing stuff like vendoring the cargo deps before generating the tarballAhh, that explains it. Does this mean that for Cargo based packages, it will always be TOFU and checksums provided by the project / maintainer cannot be used (because they will mismatch)?
A maintainer could sign the tarball generated by buildroot and upload it to github releases if they want.
To be exact: an upstream maintainer (i.e. not a Buildroot maintainer).
Quoted 3 lines
As you can see the script will bail out if the tarball already has vendored deps:
https://github.com/buildroot/buildroot/blob/6e4791b751c0f8e0ba218da2c22e71d3e1436b5d/support/download/cargo-post-process#L16-L19
The script will not "bail out" on pre-vendored tarballs; it will just exit without touching the tarball at all.
I.e. it means that we prefer using tarballs as-is from their upstreams, when they are vendored; we only vendor packages which upstreams have not.
Quoted 2 lines
So as long as the upstream release tarball has vendored deps the hash should not change AFAIU.
Yes, this is the goal.
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hello James, Yann
Quoted 10 lines · Yann E. MORIN
Quoted 2 lines
A maintainer could sign the tarball generated by buildroot and upload it to github releases if they want.
To be exact: an upstream maintainer (i.e. not a Buildroot maintainer).
(...)
I.e. it means that we prefer using tarballs as-is from their upstreams, when they are vendored; we only vendor packages which upstreams have not.
That makes sense! I am the maintainer of tealdeer, and I'll provide a vendored source tarball for the next release. (I've heard of "cargo vendor" before, but I haven't used it so far.)
Maybe a corresponding paragraph could be added to the docs at https://nightly.buildroot.org/#_integration_of_cargo_based_packages ?
Thanks for the helpful comments,
Danilo
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Danilo, All,
On 2022-01-17 11:24 +0100, Danilo Bargen spake thusly:
Quoted 11 lines · Yann E. MORIN
Quoted 10 lines
Quoted 2 lines
A maintainer could sign the tarball generated by buildroot and upload it to github releases if they want.
To be exact: an upstream maintainer (i.e. not a Buildroot maintainer).
(...)
I.e. it means that we prefer using tarballs as-is from their upstreams, when they are vendored; we only vendor packages which upstreams have not.
Quoted 6 lines
That makes sense! I am the maintainer of tealdeer, and I'll provide a vendored source tarball for the next release. (I've heard of "cargo vendor" before, but I haven't used it so far.)
Maybe a corresponding paragraph could be added to the docs at
https://nightly.buildroot.org/#_integration_of_cargo_based_packages ?
Yeah, so I already noticed that the nightly manual is not up to date...
It indeed points to the correct sha1 in master, but the Cargo part is not what it should be; it is still the older text from before we added the cargo infra. We're looking into the issue...
You can hack on the manual too, it's in: docs/manual/
And you can generate the manual locally: make manual-html
Patches welcome! ;-)
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Danilo, All,
On 2022-01-17 11:32 +0100, Yann E. MORIN spake thusly:
Quoted 1 line
On 2022-01-17 11:24 +0100, Danilo Bargen spake thusly:
[--SNIP--]
Quoted 3 lines
Quoted 2 lines
Maybe a corresponding paragraph could be added to the docs at
https://nightly.buildroot.org/#_integration_of_cargo_based_packages ?Yeah, so I already noticed that the nightly manual is not up to date...
It appears that the local tree that serves to generate the manual was somehow corrupted, and we were only ever building the same old manual over and over again...
It has now presumably been resolved, and [0] now reads:
A crate can depend on other libraries from crates.io or git repositories, listed in its Cargo.toml file. Buildroot automatically takes care of downloading such dependencies as part of the download step of packages that use the cargo-package infrastructure. Such dependencies are then kept together with the package source code in the tarball cached in Buildroot’s DL_DIR, and therefore the hash of the package’s tarball includes such dependencies.
[0] https://nightly.buildroot.org/#_infrastructure_for_cargo_based_packages
Maybe it is not so obvious and prominently advertised, indeed; patches to improve the situation are most welcome! :-)
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Danilo, All,
On 2022-01-17 11:24 +0100, Danilo Bargen spake thusly:
Quoted 7 lines · Yann E. MORIN
Quoted 3 lines
I.e. it means that we prefer using tarballs as-is from their upstreams, when they are vendored; we only vendor packages which upstreams have not.
That makes sense! I am the maintainer of tealdeer, and I'll provide a vendored source tarball for the next release. (I've heard of "cargo vendor" before, but I haven't used it so far.)
Note that vendoring is definitely not a requirement we impose on upstreams.
We do prefer when the vendoring has been done by upstream, because it means (at least we hope it does!) that upstream has validated the fully vendored package, and thus we have some confidence everything works as expected.
It also avoids the case where an uppstreams for a dependency mucks around with their releases: we already noticed the case where an upstream for a dependency did a re-tag of their release, thus breaking the vendoring of the dependees because it would no longer match the expected hashes in the cargo.toml (or go.mod?). Also (but that's mostly for go, IIRC), we also already noticed that some of the points of distribution (goproxies?) are serving some incorrect archives, thus causing download issues...
However, if an upstream decides to not vendor (for whatever reason), then this is perfectly fine for Buildroot; this is exactly why the cargo infra has been made to support doing the vendoring.
All in all, it is better that upstream vendors their releases, because it avoids any of the pitfals I mention above.
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot