New: Packages get downloaded uncompressed with wget
https://bugs.busybox.net/show_bug.cgi?id=10511
Bug ID: 10511
Summary: Packages get downloaded uncompressed with wget
Product: buildroot
Version: 2017.08
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P5
Component: Other
Assignee: unassigned at buildroot.uclibc.org
Reporter: jean.parpaillon at free.fr
CC: buildroot at uclibc.org
Target Milestone: ---
Using buildroot with wget version 1.19.2 on Debian, through nerves toolchain.
(https://github.com/nerves-project/nerves_system_br)
When downloading otp (http://erlang.org/download/otp_src_20.1.tar.gz), wget (through pkg-download.mk recipes) uncompress the data, resulting with an gunzipe'd .tar.gz, hence md5 mismatch.
Downloading the package with 'wget --compress=none' fixed the problem, but may depend on wget versions.
Signature · 2 lines
You are receiving this mail because: You are on the CC list for the bug.
[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget
https://bugs.busybox.net/show_bug.cgi?id=10511
--- Comment #1 from Thomas Petazzoni <[email protected]> --- Could you indicate how to reproduce the problem you're seeing with the official Buildroot, and provide a minimal Buildroot .config / scenario that exhibits the issue ? Thanks!
Signature · 2 lines
You are receiving this mail because: You are on the CC list for the bug.
[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget
In reply to bugzilla at busybox.nethttps://bugs.busybox.net/show_bug.cgi?id=10511
--- Comment #2 from Frank Hunleth <[email protected]> --- (In reply to Thomas Petazzoni from comment #1)
This is due to a change in behavior with wget 1.19.2. This discussion describes the issue: http://lists.gnu.org/archive/html/bug-wget/2017-11/msg00000.html. To reproduce, I installed wget 1.19.2 on my system and added the following line to qemu_arm_versatile_defconfig:
BR2_PACKAGE_ERLANG=y
Here's the result of running "make source". You can see the MD5 mismatch from the erlang.org copy. That's because it was automatically decompressed by wget.
umask 0022 && make -C /home/fhunleth/buildroot.upstream O=/home/fhunleth/buildroot.upstream/o/wgettest/. source
Quoted 1 line
Quoted 1 line
Quoted 1 line
erlang 20.0 Downloading
--2017-12-18 08:46:07-- http://www.erlang.org/download/otp_src_20.0.tar.gz
Resolving www.erlang.org... 31.172.186.60
Connecting to www.erlang.org|31.172.186.60|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://erlang.org/download/otp_src_20.0.tar.gz [following] --2017-12-18 08:46:08-- http://erlang.org/download/otp_src_20.0.tar.gz
Resolving erlang.org... 192.121.151.106
Connecting to erlang.org|192.121.151.106|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 87346046 (83M) [application/x-tar]
Saving to:
?/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.5GQ6qy/output?
/home/fhunleth/buildroot.upstream/o/ 100%[======================================================================>]
83.30M 582KB/s in 2m 27s
2017-12-18 08:48:35 (582 KB/s) -
?/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.5GQ6qy/output? saved [237383680]
ERROR: otp_src_20.0.tar.gz has wrong md5 hash:
ERROR: expected: 2faed2c3519353e6bc2501ed4d8e6ae7
ERROR: got : 6d1210c2a3a9b178a751809016ef51d2
ERROR: Incomplete download, or man-in-the-middle (MITM) attack --2017-12-18 08:48:35-- http://sources.buildroot.net/otp_src_20.0.tar.gz
Resolving sources.buildroot.net... 176.9.16.109
Connecting to sources.buildroot.net|176.9.16.109|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 87346046 (83M) [application/x-gzip]
Saving to:
?/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.hY7Oae/output?
/home/fhunleth/buildroot.upstream/o/ 100%[======================================================================>]
83.30M 3.56MB/s in 23s
2017-12-18 08:48:59 (3.59 MB/s) -
?/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.hY7Oae/output? saved [87346046/87346046]
otp_src_20.0.tar.gz: OK (md5: 2faed2c3519353e6bc2501ed4d8e6ae7) otp_src_20.0.tar.gz: OK (sha256:fe80e1e14a2772901be717694bb30ac4e9a07eee0cc7a28988724cbd21476811) otp_src_20.0.tar.gz: OK (md5: 2faed2c3519353e6bc2501ed4d8e6ae7) otp_src_20.0.tar.gz: OK (sha256:fe80e1e14a2772901be717694bb30ac4e9a07eee0cc7a28988724cbd21476811)
Verify that the invalidate MD5 was done on the uncompressed file:
$ gunzip otp_src_20.0.tar.gz $ md5sum otp_src_20.0.tar 6d1210c2a3a9b178a751809016ef51d2 otp_src_20.0.tar
It seems like a configuration change on erlang.org could fix the problem as well, but I'm not sure that's the right answer for Buildroot.
Signature · 2 lines
You are receiving this mail because: You are on the CC list for the bug.
[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget
In reply to bugzilla at busybox.nethttps://bugs.busybox.net/show_bug.cgi?id=10511
--- Comment #3 from Thomas Petazzoni <[email protected]> --- Thanks for the investigation. This problem has been fixed/worked-around upstream by commit
http://git.savannah.gnu.org/cgit/wget.git/commit/?id=047746eb765d212cbf05afac16f4bd73257d8186
which ensures that .gz and .tgz files don't get uncompressed.
However wget 1.19.2 remains affected.
Options:
- Ignore the problem, and leave users of 1.19.2 affected by the problem.
- Bail out if wget 1.19.2 is used, and leave it to the user to use a suitable
wget version
- Detect if wget >= 1.19.2, and pass --compression=none
Thoughts?
Signature · 2 lines
You are receiving this mail because: You are on the CC list for the bug.
[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget
In reply to bugzilla at busybox.nethttps://bugs.busybox.net/show_bug.cgi?id=10511
Yann E. MORIN <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |yann.morin.1998 at free.fr
--- Comment #4 from Yann E. MORIN <[email protected]> --- Thomas, Jean, All,
Quoted 1 line
- Detect if wget >= 1.19.2, and pass --compression=none
That looks like the best solution IMHO. Unfortunately, wget versions < 1.19.2 do not accept the --compression option, so we really must make it conditional.
My 2ct.
Regards,
Yann E. MORIN.
Signature · 2 lines
You are receiving this mail because: You are on the CC list for the bug.
[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget
In reply to bugzilla at busybox.nethttps://bugs.busybox.net/show_bug.cgi?id=10511
Thomas Petazzoni <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |WONTFIX
Status|NEW |RESOLVED--- Comment #5 from Thomas Petazzoni <[email protected]> --- This problem only affects wget 1.19.2. In 1.19.3, a patch was merged that disables auto-decompression for files ending in .gz and .tgz, and then in 1.19.4, the auto-decompression was disabled entirely, as it was causing too many problems.
Since the problem has been fixed in dot-releases of wget, most reasonable distributions that were using 1.19.2 have probably upgraded to 1.19.3 or later.
In addition, for those stuck with wget 1.19.2, the fallback to sources.buildroot.net works (even if admittedly the bogus hashes can be scary). But considering all of this, we don't want to introduce extra complexity in Buildroot just to workaround a problem affecting only wget 1.19.2 when downloading files only from HTTP servers having a specific configuration.
Signature · 2 lines
You are receiving this mail because: You are on the CC list for the bug.
[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget
In reply to bugzilla at busybox.nethttps://bugs.busybox.net/show_bug.cgi?id=10511
--- Comment #6 from Jean Parpaillon <[email protected]> --- Thanks for your time !
Signature · 2 lines
You are receiving this mail because: You are on the CC list for the bug.