Skip to contentSkip to conversation

configure: Check bzip2 is available

PATCH In review Philippe Mathieu-Daudé 7 participants 16 messages Nov 8, 2019 – Nov 11, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 configure: Check bzip2 is available

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

    BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2

/bin/sh: bzip2: command not found
make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

---
 configure | 7 +++++++
 1 file changed, 7 insertions(+)
configure +7 -0
index efe165edf9..9957e913e8 100755
@@ -1851,6 +1851,13 @@ python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0]
 # Suppress writing compiled files
 python="$python -B"
 
+# Some firmware binaries are compressed with bzip2
+if has bzip2; then
+  :
+else
+  error_exit "bzip2 program not found. Please install it"
+fi
+
 # Check that the C compiler works. Doing this here before testing
 # the host CPU ensures that we had a valid CC to autodetect the
 # $cpu var (and we should bail right here if that's not the case).
Signature · 1 line
2.21.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 24 lines · Philippe Mathieu-Daudé

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

    BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2

/bin/sh: bzip2: command not found
make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

---
 configure | 7 +++++++
 1 file changed, 7 insertions(+)
configure +0 -0
index efe165edf9..9957e913e8 100755

configure python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0] · line 1859

Quoted 9 lines
configure +6 -0
@@ -1851,6 +1851,13 @@ python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0]
 # Suppress writing compiled files
 python="$python -B"
 
+# Some firmware binaries are compressed with bzip2
+if has bzip2; then
+  :
+else
+  error_exit "bzip2 program not found. Please install it"
+fi

Nitpick, I'd phrase this as

"The bzip2 program is required for building QEMU"

Either way though

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#2 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 24 lines · Philippe Mathieu-Daudé

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

     BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2

/bin/sh: bzip2: command not found
make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

---
  configure | 7 +++++++
  1 file changed, 7 insertions(+)
configure +0 -0
index efe165edf9..9957e913e8 100755

configure python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0] · line 1859

Quoted 9 lines
configure +6 -0
@@ -1851,6 +1851,13 @@ python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0]
  # Suppress writing compiled files
  python="$python -B"
 
+# Some firmware binaries are compressed with bzip2
+if has bzip2; then
+  :
+else
+  error_exit "bzip2 program not found. Please install it"
+fi

It's only required for the edk2 binaries, isn't it? So should this maybe also check whether we build any of the i386-softmmu, x86_64-softmmu arm-softmmu or aarch64-softmmu targets and pass otherwise?

Thomas

#3 lore raw
Quoted 20 lines · Thomas Huth
Quoted 19 lines · Philippe Mathieu-Daudé

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

     BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2
   /bin/sh: bzip2: command not found
   make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
   make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
   make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

---
  configure | 7 +++++++

  1 file changed, 7 insertions(+)
Quoted 16 lines
Quoted 8 lines
configure +0 -0
index efe165edf9..9957e913e8 100755
@@ -1851,6 +1851,13 @@ python_version=$($python -c 'import sys; 

print("%d.%d.%d" % (sys.version_info[0]
  # Suppress writing compiled files
  python="$python -B"

configure python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0] · line 1859

Quoted 6 lines
configure +6 -0
+# Some firmware binaries are compressed with bzip2
+if has bzip2; then
+  :
+else
+  error_exit "bzip2 program not found. Please install it"
+fi

It's only required for the edk2 binaries, isn't it? So should this maybe also check whether we build any of the i386-softmmu, x86_64-softmmu arm-softmmu or aarch64-softmmu targets and pass otherwise?

I have this on my TODO somewhere, because we extract the edk2 firmwares even if we only build MIPS/PowerPC.

#4 lore raw
Quoted 40 lines · Philippe Mathieu-Daudé
Quoted 30 lines · Thomas Huth
Quoted 29 lines · Philippe Mathieu-Daudé

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

     BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2
   /bin/sh: bzip2: command not found
   make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
   make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
   make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

---
  configure | 7 +++++++

  1 file changed, 7 insertions(+)
configure +0 -0
index efe165edf9..9957e913e8 100755
@@ -1851,6 +1851,13 @@ python_version=$($python -c 'import sys;

print("%d.%d.%d" % (sys.version_info[0]
  # Suppress writing compiled files
  python="$python -B"

configure +2 -0
+# Some firmware binaries are compressed with bzip2
+if has bzip2; then
Quoted 8 lines

configure python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0] · line 1859

Quoted 4 lines

+  :

configure +3 -0
+else
+  error_exit "bzip2 program not found. Please install it"
+fi

It's only required for the edk2 binaries, isn't it? So should this maybe also check whether we build any of the i386-softmmu, x86_64-softmmu arm-softmmu or aarch64-softmmu targets and pass otherwise?

Quoted 2 lines

I have this on my TODO somewhere, because we extract the edk2 firmwares even if we only build MIPS/PowerPC.

But that applies to all of "BLOBS" in the root-level Makefile, doesn't it? We also install, say, "vgabios-qxl.bin", when only the MIPS/PowerPC system emulators are built. The only aspect that's specific to edk2 binaries is that they are kept compressed until installation time, to save space in the git repo and in the source tarball.

I'm reminded of the "external QEMU blob / firmware repo" idea.

Thanks
Laszlo

#5 lore raw
Quoted 41 lines · Laszlo Ersek
Quoted 40 lines · Philippe Mathieu-Daudé
Quoted 38 lines · Thomas Huth
Quoted 27 lines · Philippe Mathieu-Daudé

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

     BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2
   /bin/sh: bzip2: command not found
   make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
   make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
   make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

---
   configure | 7 +++++++
   1 file changed, 7 insertions(+)
configure +0 -0
index efe165edf9..9957e913e8 100755
@@ -1851,6 +1851,13 @@ python_version=$($python -c 'import sys;

print("%d.%d.%d" % (sys.version_info[0]
  # Suppress writing compiled files
  python="$python -B"

configure python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0] · line 1859

Quoted 6 lines
configure +6 -0
+# Some firmware binaries are compressed with bzip2
+if has bzip2; then
+  :
+else
+  error_exit "bzip2 program not found. Please install it"
+fi

It's only required for the edk2 binaries, isn't it? So should this maybe also check whether we build any of the i386-softmmu, x86_64-softmmu arm-softmmu or aarch64-softmmu targets and pass otherwise?

Quoted 8 lines
Quoted 2 lines

I have this on my TODO somewhere, because we extract the edk2 firmwares even if we only build MIPS/PowerPC.

But that applies to all of "BLOBS" in the root-level Makefile, doesn't it? We also install, say, "vgabios-qxl.bin", when only the MIPS/PowerPC system emulators are built. The only aspect that's specific to edk2 binaries is that they are kept compressed until installation time, to save space in the git repo and in the source tarball.

You are right, other targets could improve this too.
Since this add quite complexity to the buildsys and nobody complained about that previously, I suggest we clean that *after* we switch the build machinery to Meson.

Quoted 1 line

I'm reminded of the "external QEMU blob / firmware repo" idea.

Do you mind starting a new thread asking about it? It would be nice we clear this during the next dev cycle.

Regards,

Phil.

#6 lore raw
In reply to Laszlo Ersek
Quoted 41 lines · Laszlo Ersek
Quoted 40 lines · Philippe Mathieu-Daudé
Quoted 38 lines · Thomas Huth
Quoted 27 lines · Philippe Mathieu-Daudé

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

     BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2
   /bin/sh: bzip2: command not found
   make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
   make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
   make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

---
   configure | 7 +++++++
   1 file changed, 7 insertions(+)
configure +0 -0
index efe165edf9..9957e913e8 100755
@@ -1851,6 +1851,13 @@ python_version=$($python -c 'import sys;

print("%d.%d.%d" % (sys.version_info[0]
  # Suppress writing compiled files
  python="$python -B"

configure python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0] · line 1859

Quoted 6 lines
configure +6 -0
+# Some firmware binaries are compressed with bzip2
+if has bzip2; then
+  :
+else
+  error_exit "bzip2 program not found. Please install it"
+fi

It's only required for the edk2 binaries, isn't it? So should this maybe also check whether we build any of the i386-softmmu, x86_64-softmmu arm-softmmu or aarch64-softmmu targets and pass otherwise?

Quoted 6 lines
Quoted 2 lines

I have this on my TODO somewhere, because we extract the edk2 firmwares even if we only build MIPS/PowerPC.

But that applies to all of "BLOBS" in the root-level Makefile, doesn't it? We also install, say, "vgabios-qxl.bin", when only the MIPS/PowerPC system emulators are built.

IIRC there was another odd dependency that the PCI devices need their ROM also on non-x86 systems... but that's another story...

Quoted 3 lines

The only aspect that's specific to edk2
binaries is that they are kept compressed until installation time, to save space in the git repo and in the source tarball.

I noticed that there are also some iotests that use bzip2 ... they are not used during the build process, but still, it might be better to simply always require bzip2, also in case some other architectures want to use it. So I'm also fine if we simply always require bzip2 for the build.

Thomas

PS: Anybody interested in writing a patch to compress the other big binaries in the pc-bios folder, too? ... skiboot.lid ... openbios-sparc64 ... ppc_rom.bin ...

Reviewed-by on v1 1/1
#7 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 11 lines · Philippe Mathieu-Daudé

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

    BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2

/bin/sh: bzip2: command not found
make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

We've come full circle. Let me explain:

Quoted 2 lines
Fixes 536d2173b2b

So this makes me kinda grumpy. If you look at the v3 posting of the patch that would later become commit 536d2173b2b:

http://mid.mail-archive.com/[email protected]

you see the following note in the changelog:

- compress FD files with bzip2 rather than xz, so that decompression at "make install" time succeed on older build OSes too [Peter]

So I couldn't use xz because that was "too new" for some build OSes, but now we also can't take bzip2 for granted because that's "too old" for some other build OSes? This is ridiculous.

To be clear, my disagreement is only with the "Fixes" tag. For me, "Fixes" stands for something that, in retrospect, can be proven to have been a bug at the time the code was *originally* committed. But, at the time, taking "bzip2" for granted was *not* a bug. The conditions / circumstances have changed more recently, and the assumption about bzip2 has been invalidated *after* adding a dependency on bzip2.

Nonetheless, thank you for adapting the code to the potential absence of bzip2. Can you perhaps go in some details in the commit message, near "not included in default installations" and "freshly installed systems"? If we can, we should identify the exact distro release where this problem has been encountered (and I wouldn't mind a link to the BZ or ticket under which people agreed to remove bzip2 from the default package set).

Thanks
Laszlo

Quoted 25 lines
---
 configure | 7 +++++++
 1 file changed, 7 insertions(+)
configure +7 -0
index efe165edf9..9957e913e8 100755
@@ -1851,6 +1851,13 @@ python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0]
 # Suppress writing compiled files
 python="$python -B"
 
+# Some firmware binaries are compressed with bzip2
+if has bzip2; then
+  :
+else
+  error_exit "bzip2 program not found. Please install it"
+fi
+
 # Check that the C compiler works. Doing this here before testing
 # the host CPU ensures that we had a valid CC to autodetect the
 # $cpu var (and we should bail right here if that's not the case).
#8 lore raw
Quoted 20 lines · Laszlo Ersek
Quoted 11 lines · Philippe Mathieu-Daudé

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

    BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2

/bin/sh: bzip2: command not found
make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

We've come full circle. Let me explain:

Quoted 2 lines
Fixes 536d2173b2b

So this makes me kinda grumpy. If you look at the v3 posting of the patch that would later become commit 536d2173b2b:

Quoted 8 lines

http://mid.mail-archive.com/[email protected]

you see the following note in the changelog:

- compress FD files with bzip2 rather than xz, so that decompression at "make install" time succeed on older build OSes too [Peter]

So I couldn't use xz because that was "too new" for some build OSes, but now we also can't take bzip2 for granted because that's "too old" for some other build OSes? This is ridiculous.

We're not saying bzip2 is too old / missing from the OS. Every OS we care about has bzip2. The problem is that a minimal installation migt not have installed it. This kind of problem is increasingly common with use of minimal container images for example. So we're just ensuring we validate that what we want is actuall present.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#9 lore raw
In reply to Laszlo Ersek
Quoted 24 lines · Laszlo Ersek
Quoted 11 lines · Philippe Mathieu-Daudé

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

     BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2

/bin/sh: bzip2: command not found
make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

We've come full circle. Let me explain:

Quoted 2 lines
Fixes 536d2173b2b

So this makes me kinda grumpy. If you look at the v3 posting of the patch that would later become commit 536d2173b2b:

   http://mid.mail-archive.com/[email protected]

you see the following note in the changelog:

Quoted 8 lines

- compress FD files with bzip2 rather than xz, so that decompression at "make install" time succeed on older build OSes too [Peter]

So I couldn't use xz because that was "too new" for some build OSes, but now we also can't take bzip2 for granted because that's "too old" for some other build OSes? This is ridiculous.

To be clear, my disagreement is only with the "Fixes" tag. For me, "Fixes" stands for something that, in retrospect, can be proven to have been a bug at the time the code was *originally* committed. But, at the time, taking "bzip2" for granted was *not* a bug. The conditions / circumstances have changed more recently, and the assumption about bzip2 has been invalidated *after* adding a dependency on bzip2.

Nonetheless, thank you for adapting the code to the potential absence of bzip2. Can you perhaps go in some details in the commit message, near "not included in default installations" and "freshly installed systems"? If we can, we should identify the exact distro release where this problem has been encountered (and I wouldn't mind a link to the BZ or ticket under which people agreed to remove bzip2 from the default package set).

I am just reading this and already sent a v2.

I can amend these details. Thomas, what distro release were you using?

Quoted 32 lines

Thanks
Laszlo

Quoted 25 lines
---
  configure | 7 +++++++
  1 file changed, 7 insertions(+)
configure +7 -0
index efe165edf9..9957e913e8 100755
@@ -1851,6 +1851,13 @@ python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0]
  # Suppress writing compiled files
  python="$python -B"
 
+# Some firmware binaries are compressed with bzip2
+if has bzip2; then
+  :
+else
+  error_exit "bzip2 program not found. Please install it"
+fi
+
  # Check that the C compiler works. Doing this here before testing
  # the host CPU ensures that we had a valid CC to autodetect the
  # $cpu var (and we should bail right here if that's not the case).
#10 lore raw
Quoted 30 lines · Philippe Mathieu-Daudé
Quoted 29 lines · Laszlo Ersek
Quoted 11 lines · Philippe Mathieu-Daudé

The bzip2 tool is not included in default installations. On freshly installed systems, ./configure succeeds but 'make' might fail later:

     BUNZIP2 pc-bios/edk2-i386-secure-code.fd.bz2
   /bin/sh: bzip2: command not found
   make: *** [Makefile:305: pc-bios/edk2-i386-secure-code.fd] Error 127
   make: *** Deleting file 'pc-bios/edk2-i386-secure-code.fd'
   make: *** Waiting for unfinished jobs....

Add a check in ./configure to warn the user if bzip2 is missing.

We've come full circle. Let me explain:

Quoted 2 lines
Fixes 536d2173b2b

So this makes me kinda grumpy. If you look at the v3 posting of the patch that would later become commit 536d2173b2b:

   http://mid.mail-archive.com/[email protected]

you see the following note in the changelog:

     - compress FD files with bzip2 rather than xz, so that decompression at "make install" time succeed on older build OSes too [Peter]

So I couldn't use xz because that was "too new" for some build OSes, but now we also can't take bzip2 for granted because that's "too old" for some other build OSes? This is ridiculous.

Quoted 7 lines
Quoted 3 lines

To be clear, my disagreement is only with the "Fixes" tag. For me, "Fixes" stands for something that, in retrospect, can be proven to have been a bug at the time the code was *originally* committed. But, at the time, taking "bzip2" for granted was *not* a bug. The conditions / circumstances have changed more recently, and the assumption about bzip2 has been invalidated *after* adding a dependency on bzip2.

Nonetheless, thank you for adapting the code to the potential absence of bzip2. Can you perhaps go in some details in the commit message, near "not included in default installations" and "freshly installed systems"? If we can, we should identify the exact distro release where this problem has been encountered (and I wouldn't mind a link to the BZ or ticket under which people agreed to remove bzip2 from the default package set).

I am just reading this and already sent a v2.

I can amend these details. Thomas, what distro release were you using?

I encountered this problem with a freshly installed Fedora 31.

Thomas

#11 lore raw
In reply to Laszlo Ersek
Quoted 14 lines · Laszlo Ersek
Quoted 2 lines

Add a check in ./configure to warn the user if bzip2 is missing.

We've come full circle. Let me explain:

Quoted 2 lines
Fixes 536d2173b2b

So this makes me kinda grumpy. If you look at the v3 posting of the patch that would later become commit 536d2173b2b:

   http://mid.mail-archive.com/[email protected]

you see the following note in the changelog:

Quoted 8 lines

- compress FD files with bzip2 rather than xz, so that decompression at "make install" time succeed on older build OSes too [Peter]

So I couldn't use xz because that was "too new" for some build OSes, but now we also can't take bzip2 for granted because that's "too old" for some other build OSes? This is ridiculous.

To be clear, my disagreement is only with the "Fixes" tag. For me, "Fixes" stands for something that, in retrospect, can be proven to have been a bug at the time the code was *originally* committed. But, at the time, taking "bzip2" for granted was *not* a bug. The conditions / circumstances have changed more recently, and the assumption about bzip2 has been invalidated *after* adding a dependency on bzip2.

Nonetheless, thank you for adapting the code to the potential absence of bzip2. Can you perhaps go in some details in the commit message, near "not included in default installations" and "freshly installed systems"? If we can, we should identify the exact distro release where this problem has been encountered (and I wouldn't mind a link to the BZ or ticket under which people agreed to remove bzip2 from the default package set).

bzip2 is no longer a favored compression. If we are trying to pick a compression that is most likely to be present on any system, go with gzip. If we are trying to pick a compression that packs tighter and uncompresses faster, pick xz or zstd. But bzip2 does neither: it packs slightly tighter than gzip but has slower performance in doing so, and thus is no longer used as a default compression.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#12 lore raw
Quoted 19 lines · Eric Blake
Quoted 17 lines · Laszlo Ersek
Quoted 2 lines

Add a check in ./configure to warn the user if bzip2 is missing.

We've come full circle. Let me explain:

Quoted 2 lines
Fixes 536d2173b2b

So this makes me kinda grumpy. If you look at the v3 posting of the patch that would later become commit 536d2173b2b:

   http://mid.mail-archive.com/[email protected]

you see the following note in the changelog:

     - compress FD files with bzip2 rather than xz, so that decompression at "make install" time succeed on older build OSes too [Peter]

Quoted 7 lines
Quoted 5 lines

So I couldn't use xz because that was "too new" for some build OSes, but now we also can't take bzip2 for granted because that's "too old" for some other build OSes? This is ridiculous.

To be clear, my disagreement is only with the "Fixes" tag. For me, "Fixes" stands for something that, in retrospect, can be proven to have been a bug at the time the code was *originally* committed. But, at the time, taking "bzip2" for granted was *not* a bug. The conditions / circumstances have changed more recently, and the assumption about bzip2 has been invalidated *after* adding a dependency on bzip2.

Nonetheless, thank you for adapting the code to the potential absence of bzip2. Can you perhaps go in some details in the commit message, near "not included in default installations" and "freshly installed systems"? If we can, we should identify the exact distro release where this problem has been encountered (and I wouldn't mind a link to the BZ or ticket under which people agreed to remove bzip2 from the default package set).

bzip2 is no longer a favored compression.  If we are trying to pick a compression that is most likely to be present on any system, go with gzip.  If we are trying to pick a compression that packs tighter and uncompresses faster, pick xz or zstd.  But bzip2 does neither: it packs slightly tighter than gzip but has slower performance in doing so, and thus is no longer used as a default compression.

The problem was with OpenBSD 6.1 which hadn't xz available.

In commit 12745eaa02 Gerd updated the VM to OpenBSD 6.5 and we now have access to xz. IIRC OSX supported versions also provide xz.

If we want to revert Laszlo's patches and apply his first version (using xz), we should do that during 5.0 dev cycle, now it is too late. I'd prefer we simply fix bzip2 for the next release.

#13 lore raw
Quoted 8 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Eric Blake

bzip2 is no longer a favored compression. If we are trying to pick a compression that is most likely to be present on any system, go with gzip. If we are trying to pick a compression that packs tighter and uncompresses faster, pick xz or zstd. But bzip2 does neither: it packs slightly tighter than gzip but has slower performance in doing so, and thus is no longer used as a default compression.

Quoted 8 lines

The problem was with OpenBSD 6.1 which hadn't xz available.

In commit 12745eaa02 Gerd updated the VM to OpenBSD 6.5 and we now have access to xz. IIRC OSX supported versions also provide xz.

If we want to revert Laszlo's patches and apply his first version (using xz), we should do that during 5.0 dev cycle, now it is too late. I'd prefer we simply fix bzip2 for the next release.

I don't think we should try to use 'xz' because I don't see the point. We should use something that's generally available, whether that's bzip2 or gzip. Life's too short to deal with yet another file compression tool and format.

thanks
-- PMM

#14 lore raw
Quoted 14 lines · Peter Maydell
Quoted 13 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Eric Blake

bzip2 is no longer a favored compression. If we are trying to pick a compression that is most likely to be present on any system, go with gzip. If we are trying to pick a compression that packs tighter and uncompresses faster, pick xz or zstd. But bzip2 does neither: it packs slightly tighter than gzip but has slower performance in doing so, and thus is no longer used as a default compression.

The problem was with OpenBSD 6.1 which hadn't xz available.

In commit 12745eaa02 Gerd updated the VM to OpenBSD 6.5 and we now have access to xz. IIRC OSX supported versions also provide xz.

Quoted 8 lines
Quoted 3 lines

If we want to revert Laszlo's patches and apply his first version (using xz), we should do that during 5.0 dev cycle, now it is too late. I'd prefer we simply fix bzip2 for the next release.

I don't think we should try to use 'xz' because I don't see the point. We should use something that's generally available, whether that's bzip2 or gzip. Life's too short to deal with yet another file compression tool and format.

FWIW, on the weekend, I accidentially came accross this page:

https://www.nongnu.org/lzip/xz_inadequate.html

After reading that, I also don't think anymore that we should switch to 'xz'.

Thomas

#15 lore raw
Quoted 25 lines · Thomas Huth
Quoted 1 line · Peter Maydell

On Fri, 8 Nov 2019 at 17:07, Philippe Mathieu-Daudé <[email protected]>

Quoted 13 lines · wrote:
Quoted 13 lines · Eric Blake
Quoted 6 lines

bzip2 is no longer a favored compression. If we are trying to pick a compression that is most likely to be present on any system, go with gzip. If we are trying to pick a compression that packs tighter and uncompresses faster, pick xz or zstd. But bzip2 does neither: it packs slightly tighter than gzip but has slower performance in doing so, and thus is no longer used as a default compression.

The problem was with OpenBSD 6.1 which hadn't xz available.

In commit 12745eaa02 Gerd updated the VM to OpenBSD 6.5 and we now have access to xz. IIRC OSX supported versions also provide xz.

Quoted 8 lines
Quoted 3 lines

If we want to revert Laszlo's patches and apply his first version (using xz), we should do that during 5.0 dev cycle, now it is too late. I'd prefer we simply fix bzip2 for the next release.

I don't think we should try to use 'xz' because I don't see the point. We should use something that's generally available, whether that's bzip2 or gzip. Life's too short to deal with yet another file compression tool and format.

Quoted 6 lines

FWIW, on the weekend, I accidentially came accross this page:

https://www.nongnu.org/lzip/xz_inadequate.html

After reading that, I also don't think anymore that we should switch to 'xz'.

Off topic, but, from the page you linked to:

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.
-- C.A.R. Hoare

Perfection is reached, not when there is no longer anything to add, but when there is no longer anything to take away.
-- Antoine de Saint-Exupery

Perhaps we should display these quotes at some highly visible place on our dev QEMU website. ;)

Aleksandar

Quoted 5 lines

Thomas