linux-firmware: bump version to latest 1baa348

package/linux-firmware/linux-firmware.hash line 2

Quoted 3 lines

package/linux-firmware/linux-firmware.hash +1 -1

-sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz
+sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz

I am sorry, but this hash is still not good for me, I get:

5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

I.e:

ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash:
ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755
ERROR: got : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

And this morning, Yann E. Morin reported having the same hash as me:

08:07 < y_morin> kos_tom: I also have a different hash here.
08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

My system tar is 1.29, which is considered as a "good" version by support/dependencies/check-host-tar.sh. I have nonetheless forced building host-tar, and I still get the same hash.

I have uploaded the tarball at
https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz.
Could you upload the tarball that was generated on your side so that we can compare them, and see where the problem lies ?

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Marcin, Thomas, All,

On 2018-11-09 21:57 +0100, Thomas Petazzoni spake thusly:

Quoted 15 lines · Marcin Niestroj

package/linux-firmware/linux-firmware.hash line 2

Quoted 3 lines

package/linux-firmware/linux-firmware.hash +1 -1

-sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz
+sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz

I am sorry, but this hash is still not good for me, I get:

5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

I.e:

ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash:
ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755
ERROR: got : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

Quoted 4 lines

And this morning, Yann E. Morin reported having the same hash as me:

08:07 < y_morin> kos_tom: I also have a different hash here.
08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

Right.

Quoted 4 lines

My system tar is 1.29, which is considered as a "good" version by support/dependencies/check-host-tar.sh. I have nonetheless forced building host-tar, and I still get the same hash.

Recently, another user reported hash issues as well, and it turned out that they had changed gzip to be really pigz (a parallel gzip). Can you check if that is not your case too?

Regards,
Yann E. MORIN.

Quoted 4 lines

I have uploaded the tarball at
https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz.
Could you upload the tarball that was generated on your side so that we can compare them, and see where the problem lies ?

Quoted 8 lines

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Hi All,

Quoted 18 lines · Yann E. MORIN

Marcin, Thomas, All,

On 2018-11-09 21:57 +0100, Thomas Petazzoni spake thusly:

Quoted 15 lines · Marcin Niestroj

package/linux-firmware/linux-firmware.hash line 2

Quoted 3 lines

package/linux-firmware/linux-firmware.hash +1 -1

-sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz
+sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz

I am sorry, but this hash is still not good for me, I get:

5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

I.e:

ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash:
ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755
ERROR: got : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

Quoted 6 lines

Quoted 4 lines

And this morning, Yann E. Morin reported having the same hash as me:

08:07 < y_morin> kos_tom: I also have a different hash here.
08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

Right.

Thank you both for testing.

Quoted 8 lines

Quoted 4 lines

My system tar is 1.29, which is considered as a "good" version by support/dependencies/check-host-tar.sh. I have nonetheless forced building host-tar, and I still get the same hash.

Recently, another user reported hash issues as well, and it turned out that they had changed gzip to be really pigz (a parallel gzip). Can you check if that is not your case too?

I have investigated the issue on my side. It turns out that gzip is really the issue here.

I have two PCs with Arch Linux. PC_1 is the one I have prepared linux-firmware. Here gzip package info on that machine:

[mniestroj at gm ~]$ LANG=C yaourt -Si gzip

  Repository      : core
  Name            : gzip
  Version         : 1.9-1
  Description     : GNU compression utility
  Architecture    : x86_64
  URL             : https://www.gnu.org/software/gzip/
  Licenses        : GPL3
  Groups          : base  base-devel
  Provides        : None
  Depends On      : glibc  bash  less
  Optional Deps   : None
  Conflicts With  : None
  Replaces        : None
  Download Size   : 77.78 KiB
  Installed Size  : 150.00 KiB
  Packager        : S
  Build Date      : Mon Jan 22 00:52:54 2018
  Validated By    : MD5 Sum  SHA-256 Sum  Signature

PC_2 is also Arch Linux, but with slightly more up-to-date packages. gzip package info looks like this:

[macius at zm ~]$ LANG=C yaourt -Si gzip

  Repository      : core
  Name            : gzip
  Version         : 1.9-2
  Description     : GNU compression utility
  Architecture    : x86_64
  URL             : https://www.gnu.org/software/gzip/
  Licenses        : GPL3
  Groups          : base  base-devel
  Provides        : None
  Depends On      : glibc  bash  less
  Optional Deps   : None
  Conflicts With  : None
  Replaces        : None
  Download Size   : 78.14 KiB
  Installed Size  : 185.00 KiB
  Packager        : Allan McRae <[email protected]>
  Build Date      : Sat Nov 3 23:10:39 2018
  Validated By    : MD5 Sum  SHA-256 Sum  Signature

You can find differences in package here:
https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip

I have also checked output of gzip command on another PC with pigz configured as gzip drop-in replacement. It outputs even different file, with different sha256 hash.

I think the overall conclusion is that a host-gzip package is needed, just like host-tar. In the meantime I will send v3 of this patch with proper hash (the same as you calculated above).

Quoted 8 lines

Regards,
Yann E. MORIN.

Quoted 4 lines

I have uploaded the tarball at
https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz.
Could you upload the tarball that was generated on your side so that we can compare them, and see where the problem lies ?

Thank for sharing. After gunzipping your version and mine I ended with the same *.tar files. So the difference was clearly because of gzip.

Regards,
Marcin

Quoted 8 lines

Quoted 8 lines

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

I have investigated the issue on my side. It turns out that gzip is really the issue here.

You can find differences in package here:
https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip

I have also checked output of gzip command on another PC with pigz configured as gzip drop-in replacement. It outputs even different file, with different sha256 hash.

I think the overall conclusion is that a host-gzip package is needed, just like host-tar. In the meantime I will send v3 of this patch with proper hash (the same as you calculated above).

Quoted 3 lines

I think the overall conclusion is that a host-gzip package is needed, just like host-tar. In the meantime I will send v3 of this patch with proper hash (the same as you calculated above).

This is getting really horrible.

On my side, I have updated to Fedora 29, which ships tar 1.30, that we consider as broken, and therefore now for everything single small build that I do, I'm paying the price of building tar, which is super annoying. This is especially annoying as tar 1.30 is only problematic when *creating* tarballs of git-fetched packages, which clearly does not happen at every build.

It would be truly horrible to also have to build our own gzip. Maybe we need to somehow revisit our requirement that we need to have reproducible tarballs for git-fetched packages ?

Quoted 6 lines

On my side, I have updated to Fedora 29, which ships tar 1.30, that we consider as broken, and therefore now for everything single small build that I do, I'm paying the price of building tar, which is super annoying. This is especially annoying as tar 1.30 is only problematic when *creating* tarballs of git-fetched packages, which clearly does not happen at every build.

Well, we only need host-tar and host-gzip when creating tarballs, not extracting them, correct? So, we could very well envision the fact that host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when FOO_SITE_METHOD == git (and they are needed, of course), no?

Quoted 3 lines

It would be truly horrible to also have to build our own gzip. Maybe we need to somehow revisit our requirement that we need to have reproducible tarballs for git-fetched packages ?

I am ready to pay the price for reproducibility any time.

Thomas, All,

On 2018-11-13 21:02 +0100, Thomas Petazzoni spake thusly:

Quoted 1 line · Marcin Niestr?j

[--SNIP--]

Quoted 4 lines

Quoted 3 lines

I think the overall conclusion is that a host-gzip package is needed, just like host-tar. In the meantime I will send v3 of this patch with proper hash (the same as you calculated above).

Quoted 8 lines

This is getting really horrible.

On my side, I have updated to Fedora 29, which ships tar 1.30, that we consider as broken, and therefore now for everything single small build that I do, I'm paying the price of building tar, which is super annoying. This is especially annoying as tar 1.30 is only problematic when *creating* tarballs of git-fetched packages, which clearly does not happen at every build.

Well, we only need host-tar and host-gzip when creating tarballs, not extracting them, correct? So, we could very well envision the fact that host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when FOO_SITE_METHOD == git (and they are needed, of course), no?

(note: tar is ugly, because its configure step takes ~26s, while the build is only ~4s).

Quoted 3 lines

It would be truly horrible to also have to build our own gzip. Maybe we need to somehow revisit our requirement that we need to have reproducible tarballs for git-fetched packages ?

I am ready to pay the price for reproducibility any time.

Regards,
Yann E. MORIN.

Quoted 10 lines

On 2018-11-13 21:02 +0100, Thomas Petazzoni spake thusly:

Quoted 5 lines · Marcin Niestr?j

Quoted 3 lines

I think the overall conclusion is that a host-gzip package is needed, just like host-tar. In the meantime I will send v3 of this patch with proper hash (the same as you calculated above).

This is getting really horrible.

Well, we only need host-tar and host-gzip when creating tarballs, not extracting them, correct? So, we could very well envision the fact that host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when FOO_SITE_METHOD == git (and they are needed, of course), no?

Since we have the same problems sometimes with github tarballs, I think we need
a more fundamental solution. I would propose a new tarsha256 hash type, which extracts the tarball to calculate the hash. In a simple version it's not so complicated, something like

tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum -

where tarsha256 contains:

{ echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \
	sha256sum - | cut -f 1 -d ' '

As usually, entirely untested.

Quoted 5 lines

Since we have the same problems sometimes with github tarballs, I think we need
a more fundamental solution. I would propose a new tarsha256 hash type, which extracts the tarball to calculate the hash. In a simple version it's not so complicated, something like

Quoted 8 lines

tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum -

where tarsha256 contains:

{ echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \
	sha256sum - | cut -f 1 -d ' '

As usually, entirely untested.

I don't like it, because this is totally non-standard. People expect to be able to check hashes by running the *usual* XXXsum commands, directly on the shipped/received files.

Introducing our own hash mechanism, how reliable or simple as it would be, breaks this assumption, and the tool to actually check them is not available at all except internally to Buildroot, so it is not possible to reproduce the checks outside of Buildroot.

This goes counter one of the initial goal of hashes, which is to be able to track archives and their validity across a supply chain, inbound (as sent by a provider to Buildroot, to do the build) or outbound (as received by a recepient, from Buildroot, for compliance) alike.

Quoted 23 lines

Quoted 5 lines

Since we have the same problems sometimes with github tarballs, I think we need
a more fundamental solution. I would propose a new tarsha256 hash type, which extracts the tarball to calculate the hash. In a simple version it's not so complicated, something like

Quoted 8 lines

tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum -

where tarsha256 contains:

{ echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \
	sha256sum - | cut -f 1 -d ' '

As usually, entirely untested.

I don't like it, because this is totally non-standard. People expect to be able to check hashes by running the *usual* XXXsum commands, directly on the shipped/received files.

Introducing our own hash mechanism, how reliable or simple as it would be, breaks this assumption, and the tool to actually check them is not available at all except internally to Buildroot, so it is not possible to reproduce the checks outside of Buildroot.

Quoted 4 lines

This goes counter one of the initial goal of hashes, which is to be able to track archives and their validity across a supply chain, inbound (as sent by a provider to Buildroot, to do the build) or outbound (as received by a recepient, from Buildroot, for compliance) alike.

I understand this argument, but do you have some alternative solution ?

Even building our own host-tar and host-gzip doesn't solve entirely the problem, because it doesn't solve the case of tarballs fetched from github, that tend to change in subtle ways once in a while.

Thomas, All,

On 2018-11-16 09:35 +0100, Thomas Petazzoni spake thusly:

Quoted 24 lines · Yann E. MORIN

Quoted 23 lines

Quoted 5 lines

Since we have the same problems sometimes with github tarballs, I think we need
a more fundamental solution. I would propose a new tarsha256 hash type, which extracts the tarball to calculate the hash. In a simple version it's not so complicated, something like

Quoted 8 lines

tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum -

where tarsha256 contains:

{ echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \
	sha256sum - | cut -f 1 -d ' '

As usually, entirely untested.

I don't like it, because this is totally non-standard. People expect to be able to check hashes by running the *usual* XXXsum commands, directly on the shipped/received files.

Introducing our own hash mechanism, how reliable or simple as it would be, breaks this assumption, and the tool to actually check them is not available at all except internally to Buildroot, so it is not possible to reproduce the checks outside of Buildroot.

Quoted 6 lines

Quoted 4 lines

This goes counter one of the initial goal of hashes, which is to be able to track archives and their validity across a supply chain, inbound (as sent by a provider to Buildroot, to do the build) or outbound (as received by a recepient, from Buildroot, for compliance) alike.

I understand this argument, but do you have some alternative solution ?

No.

Also, the tarshasum thingy would not work either, because it is not enough to hash the content of files. There are cases where the metadat also matters, for example if a tarball is regenerated with new dates in it. This could trigger a rebuild of parts of the package, for example if configure.ac is now younger than configure. Obviously, we could add that info to be hashed...

Until we find anotehr type of info we can't get via the --to-command, like extended attributes for example...

So, I am still opposed to the tarshasum idea.

Quoted 3 lines

Even building our own host-tar and host-gzip doesn't solve entirely the problem, because it doesn't solve the case of tarballs fetched from github, that tend to change in subtle ways once in a while.

For that, we can simply decide to ditch the github macro, and revert to using Github for what it is: a git repository, and switch to the git method.

Yes, yes, I know, this is a shame, especially since there are big repositories out there. But OTOH, we now have the git cache, so it should not be such a hassle in the end.

Regards,
Yann E. MORIN.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Arnout" == Arnout Vandecappelle <[email protected]> writes:

BTW, turns out that https://github.com/mikemccabe/tarsump is not usable - it
doesn't sort the files, and it doesn't compute an overall hash.

Hello,

Quoted 2 lines · Marcin Niestr?j

I have investigated the issue on my side. It turns out that gzip is really the issue here.

Thanks for the additional investigation!

Quoted 11 lines

You can find differences in package here:
https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip

I have also checked output of gzip command on another PC with pigz configured as gzip drop-in replacement. It outputs even different file, with different sha256 hash.

I think the overall conclusion is that a host-gzip package is needed, just like host-tar. In the meantime I will send v3 of this patch with proper hash (the same as you calculated above).