Skip to contentSkip to conversation

scripts/make-release: Decrease size of the release tarballs

PATCH In review Thomas Huth 6 participants 22 messages Nov 28, 2022 – Dec 9, 2022

lore
Share & tools
lore
#0 lore raw

v2 cover scripts/make-release: Decrease size of the release tarballs

Our release tarballs are huge - qemu-7.2.0-rc2.tar.xz has a size of 116 MiB. If you look at the contents, approx. 80% of the size is used for the firmware sources that we ship along to provide the sources for the ROM binaries. This feels very wrong, why do we urge users to download such huge tarballs while 99.9% of them never will rebuilt the firmware sources? We were also struggeling a bit in the past already with server load and costs, so we should really try to decrease the size of our release tarballs to a saner level.

So let's split the firmware sources into a separate tarball to decrease the size of the main QEMU sources tarball a lot (which should help us to safe a lot of traffic on the server).

Additional improvements for the make-release script add a little help text and speed it up by downloading less data from the various git repositories.

v2:
- Move the firmware sources into a separate tarball instead of dropping the edk2 and skiboot sources.

Thomas Huth (5):
scripts/make-release: Add a simple help text for the script scripts/make-release: Only clone single branches to speed up the
script
scripts/make-release: Remove CI yaml and more git files from the
tarball
roms: Add a README file with some basic information scripts/make-release: Move roms into separate tarball

 roms/README.rst      | 24 ++++++++++++++++++++++++
 scripts/make-release | 34 +++++++++++++++++++++++++++++-----
 2 files changed, 53 insertions(+), 5 deletions(-)
 create mode 100644 roms/README.rst
Signature · 1 line
2.31.1
#1 lore raw

PATCH v2 1/5 scripts/make-release: Add a simple help text for the script

Print a simple help text if the script has been called with the wrong amount of parameters.

---
 scripts/make-release | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/make-release +6 -0
index 05b14ecc95..4be9b3b9ce 100755
@@ -10,6 +10,12 @@
 # This work is licensed under the terms of the GNU GPLv2 or later.
 # See the COPYING file in the top-level directory.
 
+if [ $# -ne 2 ]; then
+    echo "Usage:"
+    echo " $0 gitrepo version"
+    exit 0
+fi
+
 src="$1"
 version="$2"
 destination=qemu-${version}
Signature · 1 line
2.31.1
#3 lore raw

PATCH v2 2/5 scripts/make-release: Only clone single branches to speed up the script

In reply to Thomas Huth

Using --single-branch and --depth 1 here helps to speed up the process a little bit and helps to save some networking bandwidth.

---
 scripts/make-release | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
scripts/make-release +6 -4
index 4be9b3b9ce..44a9d86a04 100755
@@ -20,10 +20,12 @@ src="$1"
 version="$2"
 destination=qemu-${version}
 
-git clone "${src}" ${destination}
+git clone --single-branch -b "v${version}" -c advice.detachedHead=false \
+    "${src}" ${destination}
+
 pushd ${destination}
-git checkout "v${version}"
-git submodule update --init
+
+git submodule update --init --single-branch
 (cd roms/seabios && git describe --tags --long --dirty > .version)
 (cd roms/skiboot && ./make_version.sh > .version)
 # Fetch edk2 submodule's submodules, since it won't have access to them via
@@ -34,7 +36,7 @@ git submodule update --init
 # submodule dependencies, so we continue to handle these on a case-by-case
 # basis for now.
 (cd roms/edk2 && \
-    git submodule update --init -- \
+    git submodule update --init --depth 1 -- \
         ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \
         BaseTools/Source/C/BrotliCompress/brotli \
         CryptoPkg/Library/OpensslLib/openssl \
Signature · 1 line
2.31.1
Reviewed-by on v2 2/5
#4 lore raw

Re: [PATCH v2 for-8.0 2/5] scripts/make-release: Only clone single branches to speed up the script

Quoted 4 lines · Thomas Huth

Using --single-branch and --depth 1 here helps to speed up the process a little bit and helps to save some networking bandwidth.

Signature · 1 line
Alex Bennée
#5 lore raw

PATCH v2 3/5 scripts/make-release: Remove CI yaml and more git files from the tarball

In reply to Thomas Huth

These files are of no use in a normal tarball and thus should not be included here.

---
 scripts/make-release | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
scripts/make-release +4 -1
index 44a9d86a04..febeb6cb36 100755
@@ -41,6 +41,9 @@ git submodule update --init --single-branch
         BaseTools/Source/C/BrotliCompress/brotli \
         CryptoPkg/Library/OpensslLib/openssl \
         MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
+
+rm -v .*.yml
 popd
-tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
+
+tar --exclude=".git*" -cjf ${destination}.tar.bz2 ${destination}
 rm -rf ${destination}
Signature · 1 line
2.31.1
#6 lore raw

Re: [PATCH v2 for-8.0 3/5] scripts/make-release: Remove CI yaml and more git files from the tarball

Quoted 13 lines · Thomas Huth

These files are of no use in a normal tarball and thus should not be included here.

---
 scripts/make-release | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
scripts/make-release +0 -0
index 44a9d86a04..febeb6cb36 100755

scripts/make-release git submodule update --init --single-branch · line 45

Quoted 5 lines
scripts/make-release +2 -0
@@ -41,6 +41,9 @@ git submodule update --init --single-branch
         BaseTools/Source/C/BrotliCompress/brotli \
         CryptoPkg/Library/OpensslLib/openssl \
         MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
+
+rm -v .*.yml

Why remove them instead of excluding them like you do for .git* bellow.

scripts/make-release git submodule update --init --single-branch · line 48

Quoted 4 lines
scripts/make-release +2 -1
 popd
-tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
+
+tar --exclude=".git*" -cjf ${destination}.tar.bz2 ${destination}

In fact you could probably instead use a file (release.exclude?) and put all those patterns in there?

Quoted 1 line
scripts/make-release +0 -0
 rm -rf ${destination}
Signature · 1 line
Alex Bennée
#7 lore raw

PATCH v2 4/5 roms: Add a README file with some basic information

In reply to Thomas Huth

We're going to ship the contents of the "roms" folder as a separate tarball, so we should have at least a short README in this folder for this.

---
 roms/README.rst | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
 create mode 100644 roms/README.rst
roms/README.rst new file +24 -0
new file mode 100644
index 0000000000..0a80d780e9
@@ -0,0 +1,24 @@
+====================
+QEMU firmware images
+====================
+
+This folder contains the collection of sources for firmware (ROM / BIOS)
+images which are used for the various machines that are emulated by QEMU.
+See the individual sub-folders for more information like requirements for
+building and license statements.
+
+Pre-built binaries of these firmwares can be found in the "pc-bios" folder
+of the main QEMU source tree. It can be browsed online here:
+
+ https://gitlab.com/qemu-project/qemu/-/tree/master/pc-bios
+
+
+Building
+========
+
+The main Makefile provides some targets for building the various firmware
+images in an easy way. Run "make help" in this directory to get a list of
+available build targets.
+
+Note that you might need to install an appropriate cross-compiler for
+compiling certain targets first.
Signature · 1 line
2.31.1
Reviewed-by on v2 4/5
#8 lore raw

Re: [PATCH v2 for-8.0 4/5] roms: Add a README file with some basic information

Quoted 5 lines · Thomas Huth

We're going to ship the contents of the "roms" folder as a separate tarball, so we should have at least a short README in this folder for this.

Signature · 1 line
Alex Bennée
#9 lore raw

PATCH v2 5/5 scripts/make-release: Move roms into separate tarball

In reply to Thomas Huth

Our current release tarballs are huge and caused already some trouble with the server traffic in the past. However, the biggest chunk (~80%) of the tarball is caused by the firmware sources - which most users don't need anyway (assuming that they just want to compile QEMU itself on their own). So we can certainly safe a lot of traffic by moving the firmware sources into a separate tarball, so that only people who really want to rebuild the firmware images on their own have to download it.

This decreases the size of the main QEMU xz tarball from ca. 116 MiB to only 27 MiB!

---
 scripts/make-release | 13 +++++++++++++
 1 file changed, 13 insertions(+)
scripts/make-release +13 -0
index febeb6cb36..44408c24a2 100755
@@ -42,6 +42,19 @@ git submodule update --init --single-branch
         CryptoPkg/Library/OpensslLib/openssl \
         MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
 
+tar --exclude=".git*" -cjf ../qemu-rom-sources-${version}.tar.bz2 roms
+rm -r roms/*
+
+cat > roms/README.rst <<EOF
+Please download the qemu-rom-sources-${version}.tar.bz2 tarball
+and unpack it in the topmost directory of the QEMU sources in
+case you want to recompile the QEMU ROM images.
+EOF
+cat > roms/Makefile <<EOF
+%:
+	@cat README.rst
+EOF
+
 rm -v .*.yml
 popd
 
Signature · 1 line
2.31.1
#10 lore raw

Re: [PATCH v2 for-8.0 5/5] scripts/make-release: Move roms into separate tarball

Quoted 28 lines · Thomas Huth

Our current release tarballs are huge and caused already some trouble with the server traffic in the past. However, the biggest chunk (~80%) of the tarball is caused by the firmware sources - which most users don't need anyway (assuming that they just want to compile QEMU itself on their own). So we can certainly safe a lot of traffic by moving the firmware sources into a separate tarball, so that only people who really want to rebuild the firmware images on their own have to download it.

This decreases the size of the main QEMU xz tarball from ca. 116 MiB to only 27 MiB!

---
 scripts/make-release | 13 +++++++++++++
 1 file changed, 13 insertions(+)
scripts/make-release +2 -0
index febeb6cb36..44408c24a2 100755
@@ -42,6 +42,19 @@ git submodule update --init --single-branch
         CryptoPkg/Library/OpensslLib/openssl \
         MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
 
+tar --exclude=".git*" -cjf ../qemu-rom-sources-${version}.tar.bz2 roms
+rm -r roms/*

scripts/make-release git submodule update --init --single-branch · line 56

Quoted 10 lines
scripts/make-release +10 -0
+
+cat > roms/README.rst <<EOF
+Please download the qemu-rom-sources-${version}.tar.bz2 tarball
+and unpack it in the topmost directory of the QEMU sources in
+case you want to recompile the QEMU ROM images.
+EOF
+cat > roms/Makefile <<EOF
+%:
+       @cat README.rst
+EOF

Replacing files with different content worries me a little. If someone is trying to match a git tree with a release these files will be different and require manual inspection.

How about having no roms/ directory in the main sources tarball and including a check for an empty directory in the build scripts? Then there's no need to patch the roms/ directory.

Stefan

#11 lore raw

Re: [PATCH v2 for-8.0 5/5] scripts/make-release: Move roms into separate tarball

Quoted 44 lines · Stefan Hajnoczi
Quoted 28 lines · Thomas Huth

Our current release tarballs are huge and caused already some trouble with the server traffic in the past. However, the biggest chunk (~80%) of the tarball is caused by the firmware sources - which most users don't need anyway (assuming that they just want to compile QEMU itself on their own). So we can certainly safe a lot of traffic by moving the firmware sources into a separate tarball, so that only people who really want to rebuild the firmware images on their own have to download it.

This decreases the size of the main QEMU xz tarball from ca. 116 MiB to only 27 MiB!

---
  scripts/make-release | 13 +++++++++++++
  1 file changed, 13 insertions(+)
scripts/make-release +2 -0
index febeb6cb36..44408c24a2 100755
@@ -42,6 +42,19 @@ git submodule update --init --single-branch
          CryptoPkg/Library/OpensslLib/openssl \
          MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
 
+tar --exclude=".git*" -cjf ../qemu-rom-sources-${version}.tar.bz2 roms
+rm -r roms/*

scripts/make-release git submodule update --init --single-branch · line 56

Quoted 10 lines
scripts/make-release +10 -0
+
+cat > roms/README.rst <<EOF
+Please download the qemu-rom-sources-${version}.tar.bz2 tarball
+and unpack it in the topmost directory of the QEMU sources in
+case you want to recompile the QEMU ROM images.
+EOF
+cat > roms/Makefile <<EOF
+%:
+       @cat README.rst
+EOF

Replacing files with different content worries me a little. If someone is trying to match a git tree with a release these files will be different and require manual inspection.

Quoted 2 lines

How about having no roms/ directory in the main sources tarball and including a check for an empty directory in the build scripts?

Not sure where to add such a check - do we have a way to build the roms from the main build machinery yet? ... last time I asked, I was told that "make -C roms ..." is the way to go...

Thomas

Reviewed-by Tested-by on v2 5/5
#12 lore raw

Re: [PATCH v2 for-8.0 5/5] scripts/make-release: Move roms into separate tarball

In reply to Thomas Huth
Quoted 28 lines · Thomas Huth

Our current release tarballs are huge and caused already some trouble with the server traffic in the past. However, the biggest chunk (~80%) of the tarball is caused by the firmware sources - which most users don't need anyway (assuming that they just want to compile QEMU itself on their own). So we can certainly safe a lot of traffic by moving the firmware sources into a separate tarball, so that only people who really want to rebuild the firmware images on their own have to download it.

This decreases the size of the main QEMU xz tarball from ca. 116 MiB to only 27 MiB!

---
 scripts/make-release | 13 +++++++++++++
 1 file changed, 13 insertions(+)
scripts/make-release +3 -0
index febeb6cb36..44408c24a2 100755
@@ -42,6 +42,19 @@ git submodule update --init --single-branch
         CryptoPkg/Library/OpensslLib/openssl \
         MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
 
+tar --exclude=".git*" -cjf ../qemu-rom-sources-${version}.tar.bz2 roms
+rm -r roms/*
+

scripts/make-release git submodule update --init --single-branch · line 57

Quoted 10 lines
scripts/make-release +10 -0
+cat > roms/README.rst <<EOF
+Please download the qemu-rom-sources-${version}.tar.bz2 tarball
+and unpack it in the topmost directory of the QEMU sources in
+case you want to recompile the QEMU ROM images.
+EOF
+cat > roms/Makefile <<EOF
+%:
+	@cat README.rst
+EOF
+

Hmmm I guess this is marginally more magic than an exclude file. Anyway it works:

Quoted 2 lines
scripts/make-release +0 -0
 rm -v .*.yml
 popd
Signature · 1 line
Alex Bennée
Reviewed-by on the v2 series
#13 lore raw
In reply to Thomas Huth

This is great. It will reduce qemu.org's network bandwidth consumption and make QEMU release tarballs nicer to use due to reduced size.

I left a comment because I don't like patching the roms/ directory, but if I'm the only one who doesn't like it then feel free to keep that approach.

#14 lore raw
In reply to Thomas Huth
Quoted 8 lines · Thomas Huth

Our release tarballs are huge - qemu-7.2.0-rc2.tar.xz has a size of 116 MiB. If you look at the contents, approx. 80% of the size is used for the firmware sources that we ship along to provide the sources for the ROM binaries. This feels very wrong, why do we urge users to download such huge tarballs while 99.9% of them never will rebuilt the firmware sources? We were also struggeling a bit in the past already with server load and costs, so we should really try to decrease the size of our release tarballs to a saner level.

The main reason for shipping the source in the tarball was to guarantee license compliance for anyone who is distributing qemu release tarballs, including ourselves.

Splitting off the firmware source, but not the firmware binaries, means people are now at risk of not complying with the license but failing to provide complete and corresponding source.

Technically the license requirement is only critical for GPL licenses ROMs, but as good practice we do it for all our ROMs.

Quoted 3 lines

So let's split the firmware sources into a separate tarball to decrease the size of the main QEMU sources tarball a lot (which should help us to safe a lot of traffic on the server).

With my distro maintainer hat I would rather QEMU ship neither the ROM source, nor the ROM binaries.

Still the binaries are convenient for people doing their own QEMU builds from source.

How about shipping two distinct options:

qemu-x.y.z.tar.xz (QEMU source only)
qemu-bundled-x.y.z.tar.xz (QEMU source + bundled ROM binaries + ROM sources)

though I'm not sure how much of an impact that will have on the download traffic - depends what is causing the traffic.

Another option is

qemu-x.y.z.tar.xz (QEMU source only)
qemu-roms-x.y.z.tar.xz (bundled ROM binaries + ROM sources)

though this is slightly more inconvenient for users, and there's the risk they'll use new QEMU with old ROMs.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#15 lore raw

Il lun 28 nov 2022, 18:04 Daniel P. Berrangé <[email protected]> ha scritto:

Quoted 3 lines

With my distro maintainer hat I would rather QEMU ship neither the ROM source, nor the ROM binaries.

Annd since QEMU can finally cross compile its embedded firmware modules, too, it is now easier for distros not to use any prebuilt binary.

However some firmware sources are only available from QEMU's submodules. So either we distribute those submodules as separate tarballs, or distros would need to use the bundled tarball as well.

Separately, I am not even sure what compiler is needed for the old Macintosh ROMs...

Paolo

Quoted 32 lines

Still the binaries are convenient for people doing their own QEMU builds from source.

How about shipping two distinct options:

qemu-x.y.z.tar.xz (QEMU source only)
qemu-bundled-x.y.z.tar.xz (QEMU source + bundled ROM binaries + ROM
sources)

though I'm not sure how much of an impact that will have on the download traffic - depends what is causing the traffic.

Another option is

qemu-x.y.z.tar.xz (QEMU source only)
qemu-roms-x.y.z.tar.xz (bundled ROM binaries + ROM sources)

though this is slightly more inconvenient for users, and there's the risk they'll use new QEMU with old ROMs.

With regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|

#16 lore raw
Quoted 7 lines · Paolo Bonzini

Il lun 28 nov 2022, 18:04 Daniel P. Berrangé <[email protected]> ha scritto:

Quoted 3 lines

With my distro maintainer hat I would rather QEMU ship neither the ROM source, nor the ROM binaries.

Quoted 6 lines

Annd since QEMU can finally cross compile its embedded firmware modules, too, it is now easier for distros not to use any prebuilt binary.

However some firmware sources are only available from QEMU's submodules. So either we distribute those submodules as separate tarballs, or distros would need to use the bundled tarball as well.

If the firmware doesn't exist as a standalone project, IMHO, it is fine to bundle their sources with QEMU. If they move off into a separate project some time in arbitrary future, they can be unbundled at that point.

Quoted 2 lines

Separately, I am not even sure what compiler is needed for the old Macintosh ROMs...

That we're not sure how to build some ROMS is exactly why distros have their build everything from source policy !

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#17 lore raw

Il mar 29 nov 2022, 09:12 Daniel P. Berrangé <[email protected]> ha scritto:

Quoted 9 lines
Quoted 1 line

However some firmware sources are only available from QEMU's submodules.

So

Quoted 2 lines

either we distribute those submodules as separate tarballs, or distros would need to use the bundled tarball as well.

If the firmware doesn't exist as a standalone project, IMHO, it is fine to bundle their sources with QEMU

Quoted 8 lines

They're not small though, for example we have a fork of uboot.

Separately, I am not even sure what compiler is needed for the old

Quoted 1 line

Macintosh ROMs...

That we're not sure how to build some ROMS is exactly why distros have their build everything from source policy !

What I mean is that it might be that an ancient proprietary compiler is needed. I am not sure though.

Paolo

Quoted 11 lines

With regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|

#18 lore raw
In reply to Paolo Bonzini
Quoted 6 lines · Paolo Bonzini

Il lun 28 nov 2022, 18:04 Daniel P. Berrangé <[email protected] <mailto:[email protected]>> ha scritto:

With my distro maintainer hat I would rather QEMU ship neither the ROM source, nor the ROM binaries.

Quoted 7 lines

Annd since QEMU can finally cross compile its embedded firmware modules, too, it is now easier for distros not to use any prebuilt binary.

However some firmware sources are only available from QEMU's submodules. So either we distribute those submodules as separate tarballs, or distros would need to use the bundled tarball as well.

Separately, I am not even sure what compiler is needed for the old Macintosh ROMs...

Paolo

I can quickly summarise what is needed for the ROMs that I'm involved with:

- OpenBIOS is currently built using the builder docker image hosted at https://github.com/openbios/openbios/pkgs/container/openbios-builder which is based upon the kernel crosstool archives.

I initially tried using the QEMU docker cross-compilers, however whilst the builds work the binaries generated would crash early for PPC/SPARC64. I suspect it's likely because of specific patches applied by Debian or the options used for the build but didn't really dig into it further.

- qemu_vga.ndrv is a tricky one: it's actually built by Metrowerks CodeWarrior running in an OS 9 VM in QEMU!

There is a project called Retro68 (https://github.com/autc04/Retro68) which is a port of gcc for 68k/ppc Macs which could potentially be used, however it would need conversion of the driver from Metrowerks to gcc, and also likely require the use of Apple's MPW headers which were free to distribute but whose copyright status is currently undetermined.

ATB,

Mark.

#19 lore raw
In reply to Daniel P. Berrangé
Quoted 39 lines · Daniel P. Berrangé
Quoted 8 lines · Thomas Huth

Our release tarballs are huge - qemu-7.2.0-rc2.tar.xz has a size of 116 MiB. If you look at the contents, approx. 80% of the size is used for the firmware sources that we ship along to provide the sources for the ROM binaries. This feels very wrong, why do we urge users to download such huge tarballs while 99.9% of them never will rebuilt the firmware sources? We were also struggeling a bit in the past already with server load and costs, so we should really try to decrease the size of our release tarballs to a saner level.

The main reason for shipping the source in the tarball was to guarantee license compliance for anyone who is distributing qemu release tarballs, including ourselves.

Splitting off the firmware source, but not the firmware binaries, means people are now at risk of not complying with the license but failing to provide complete and corresponding source.

Technically the license requirement is only critical for GPL licenses ROMs, but as good practice we do it for all our ROMs.

Quoted 3 lines

So let's split the firmware sources into a separate tarball to decrease the size of the main QEMU sources tarball a lot (which should help us to safe a lot of traffic on the server).

With my distro maintainer hat I would rather QEMU ship neither the ROM source, nor the ROM binaries.

Still the binaries are convenient for people doing their own QEMU builds from source.

How about shipping two distinct options:

qemu-x.y.z.tar.xz (QEMU source only)
qemu-bundled-x.y.z.tar.xz (QEMU source + bundled ROM binaries + ROM sources)

though I'm not sure how much of an impact that will have on the download traffic - depends what is causing the traffic.

Quoted 7 lines

Another option is

qemu-x.y.z.tar.xz (QEMU source only)
qemu-roms-x.y.z.tar.xz (bundled ROM binaries + ROM sources)

though this is slightly more inconvenient for users, and there's the risk they'll use new QEMU with old ROMs.

Maybe that would work for distros, but I don't think that these are good options for the average users who just want to download and recompile the latest version of QEMU on their own.
I assume that most users don't have an environment with cross-compilers or for running things in a container, so I think they still want to use the pre-built binaries. Thus, if you bundle the binaries along with their sources, people will still continue to download the big tarball and we haven't gained anything.

So do you really really think shipping the binaries in the main tarball is a problem? Honestly, it's not a problem for us as long as we publish both tarballs together - and if someone wants to mirror the main tarball to their webserver and fails to mirror the rom-sources tarball, too, it's their fault, not ours.

Anyway, what about splitting the binaries into a separate tarball, so we would have three tarballs:

    qemu-x.y.z.tar.xz               (QEMU source only)
    qemu-roms-x.y.z.tar.xz          (ROM binaries)
    qemu-roms-sources-x.y.z.tar.xz  (ROM sources)

That should make it hopefully obvious that the two qemu-roms* tarballs belong together. Would that be OK for you?

Thomas

#20 lore raw
Quoted 57 lines · Thomas Huth
Quoted 39 lines · Daniel P. Berrangé
Quoted 8 lines · Thomas Huth

Our release tarballs are huge - qemu-7.2.0-rc2.tar.xz has a size of 116 MiB. If you look at the contents, approx. 80% of the size is used for the firmware sources that we ship along to provide the sources for the ROM binaries. This feels very wrong, why do we urge users to download such huge tarballs while 99.9% of them never will rebuilt the firmware sources? We were also struggeling a bit in the past already with server load and costs, so we should really try to decrease the size of our release tarballs to a saner level.

The main reason for shipping the source in the tarball was to guarantee license compliance for anyone who is distributing qemu release tarballs, including ourselves.

Splitting off the firmware source, but not the firmware binaries, means people are now at risk of not complying with the license but failing to provide complete and corresponding source.

Technically the license requirement is only critical for GPL licenses ROMs, but as good practice we do it for all our ROMs.

Quoted 3 lines

So let's split the firmware sources into a separate tarball to decrease the size of the main QEMU sources tarball a lot (which should help us to safe a lot of traffic on the server).

With my distro maintainer hat I would rather QEMU ship neither the ROM source, nor the ROM binaries.

Still the binaries are convenient for people doing their own QEMU builds from source.

How about shipping two distinct options:

qemu-x.y.z.tar.xz (QEMU source only)
qemu-bundled-x.y.z.tar.xz (QEMU source + bundled ROM binaries + ROM sources)

though I'm not sure how much of an impact that will have on the download traffic - depends what is causing the traffic.

Quoted 7 lines

Another option is

qemu-x.y.z.tar.xz (QEMU source only)
qemu-roms-x.y.z.tar.xz (bundled ROM binaries + ROM sources)

though this is slightly more inconvenient for users, and there's the risk they'll use new QEMU with old ROMs.

Maybe that would work for distros, but I don't think that these are good options for the average users who just want to download and recompile the latest version of QEMU on their own.
I assume that most users don't have an environment with cross-compilers or for running things in a container, so I think they still want to use the pre-built binaries. Thus, if you bundle the binaries along with their sources, people will still continue to download the big tarball and we haven't gained anything.

Quoted 5 lines

So do you really really think shipping the binaries in the main tarball is a problem? Honestly, it's not a problem for us as long as we publish both tarballs together - and if someone wants to mirror the main tarball to their webserver and fails to mirror the rom-sources tarball, too, it's their fault, not ours.

I think we would be contributing to mistakes by providing a tarball that contains a mixture of sources and binaries, but not all the sources for all the binaries.

Quoted 9 lines

Anyway, what about splitting the binaries into a separate tarball, so we would have three tarballs:

    qemu-x.y.z.tar.xz               (QEMU source only)
    qemu-roms-x.y.z.tar.xz          (ROM binaries)
    qemu-roms-sources-x.y.z.tar.xz  (ROM sources)

That should make it hopefully obvious that the two qemu-roms* tarballs belong together. Would that be OK for you?

Yes, I think that's better, as it is cleanly separating the binaries and sources.

Even bikeshedding a bit more I woud have probably suggested 'qemu-roms-prebuilt-x.y.z.tar.xz' for ROM binaries and 'qemu-roms-x.y.z.tar.xz for the ROM sources, since sources is the normal tarball content.

The downside is that there's the risk of ROMS not matching the QEMU version, if people updates to latest qemu tarball but forgot the corresponding ROMs tarball. Most of the time a mismatch would not matter, but we should think about if there is a way to make it easier to diagnose such a mismatch if only for easier bug triage.

Perhaps the ROMs should install into a versioned subdir of /usr/share/qemu, instead of the root of it, and the QEMU binary preferentially look at the versioned subdir Maybe that's overthinking things though, and we would suffic to have a /usr/share/qemu/ROM-VERSION.txt file

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#21 lore raw
Quoted 75 lines · Daniel P. Berrangé
Quoted 57 lines · Thomas Huth
Quoted 39 lines · Daniel P. Berrangé
Quoted 8 lines · Thomas Huth

Our release tarballs are huge - qemu-7.2.0-rc2.tar.xz has a size of 116 MiB. If you look at the contents, approx. 80% of the size is used for the firmware sources that we ship along to provide the sources for the ROM binaries. This feels very wrong, why do we urge users to download such huge tarballs while 99.9% of them never will rebuilt the firmware sources? We were also struggeling a bit in the past already with server load and costs, so we should really try to decrease the size of our release tarballs to a saner level.

The main reason for shipping the source in the tarball was to guarantee license compliance for anyone who is distributing qemu release tarballs, including ourselves.

Splitting off the firmware source, but not the firmware binaries, means people are now at risk of not complying with the license but failing to provide complete and corresponding source.

Technically the license requirement is only critical for GPL licenses ROMs, but as good practice we do it for all our ROMs.

Quoted 3 lines

So let's split the firmware sources into a separate tarball to decrease the size of the main QEMU sources tarball a lot (which should help us to safe a lot of traffic on the server).

With my distro maintainer hat I would rather QEMU ship neither the ROM source, nor the ROM binaries.

Still the binaries are convenient for people doing their own QEMU builds from source.

How about shipping two distinct options:

qemu-x.y.z.tar.xz (QEMU source only)
qemu-bundled-x.y.z.tar.xz (QEMU source + bundled ROM binaries + ROM sources)

though I'm not sure how much of an impact that will have on the download traffic - depends what is causing the traffic.

Quoted 7 lines

Another option is

qemu-x.y.z.tar.xz (QEMU source only)
qemu-roms-x.y.z.tar.xz (bundled ROM binaries + ROM sources)

though this is slightly more inconvenient for users, and there's the risk they'll use new QEMU with old ROMs.

Maybe that would work for distros, but I don't think that these are good options for the average users who just want to download and recompile the latest version of QEMU on their own.
I assume that most users don't have an environment with cross-compilers or for running things in a container, so I think they still want to use the pre-built binaries. Thus, if you bundle the binaries along with their sources, people will still continue to download the big tarball and we haven't gained anything.

Quoted 5 lines

So do you really really think shipping the binaries in the main tarball is a problem? Honestly, it's not a problem for us as long as we publish both tarballs together - and if someone wants to mirror the main tarball to their webserver and fails to mirror the rom-sources tarball, too, it's their fault, not ours.

I think we would be contributing to mistakes by providing a tarball that contains a mixture of sources and binaries, but not all the sources for all the binaries.

Quoted 7 lines

Anyway, what about splitting the binaries into a separate tarball, so we would have three tarballs:

     qemu-x.y.z.tar.xz               (QEMU source only)
     qemu-roms-x.y.z.tar.xz          (ROM binaries)
     qemu-roms-sources-x.y.z.tar.xz  (ROM sources)
Quoted 5 lines
Quoted 2 lines

That should make it hopefully obvious that the two qemu-roms* tarballs belong together. Would that be OK for you?

Yes, I think that's better, as it is cleanly separating the binaries and sources.

I pondered about this idea quite a while now, but I think that will definitely be a decrease in the user's experience, e.g. "make install" won't work by default anymore if the user did not download the binaries before. Sure, we can instruct the users to download and uncompress the roms tarball here or there ... but still, it's getting more cumbersome for the users this way. Thus I think I'd rather would like to avoid packaging the binaries in an extra tarball.

Quoted 6 lines

The downside is that there's the risk of ROMS not matching the QEMU version, if people updates to latest qemu tarball but forgot the corresponding ROMs tarball. Most of the time a mismatch would not matter, but we should think about if there is a way to make it easier to diagnose such a mismatch if only for easier bug triage.

Yes, that's another disadvantage which will lead to very subtle bug reports if we don't prevent it from happening somehow.

Quoted 5 lines

Perhaps the ROMs should install into a versioned subdir of /usr/share/qemu, instead of the root of it, and the QEMU binary preferentially look at the versioned subdir Maybe that's overthinking things though, and we would suffic to have a /usr/share/qemu/ROM-VERSION.txt file

I don't think that a text file will be sufficient to prevent people running into the problem. Having versioned subdirs might help, but it has the disadvantage that it might cause lots of old files lying around in the file system if the users forget to delete the old versions after an upgrade.

... Ok, let's start again from scratch. So what we are trying to solve is that our tarballs are too huge and cause a non-neglectable traffic on the server. A huge part of the tarball size is caused by the edk2 sources. And we don't want to separate the GPL firmware sources from the binaries. ... so what if we just move the edk2 sources (and maybe skiboot sources) into a separate tarball? These sources are not licensed under a strong copyleft license, so it should be OK if we don't ship the sources for these binaries in the main tarball (similar to my v1 series, but I was rather completely removing the sources there instead). We could call the tarball with the separate source like qemu-extra-rom-sources.tar.xz or so?

Thomas