scripts/make-release: Decrease the size of the release tarballs
cover scripts/make-release: Decrease the size of the release tarballs
Our release tarballs are huge - qemu-7.0.0.tar.xz has a size of 119 MiB. If you look at the contents, more than half of the size is used for the edk2 sources that we ship along to provide the sources for the firmware binaries, too. This feels very wrong, why do we urge users to download such huge tarballs while 99.9% of them never will rebuilt the firmware sources? We were also struggeling a bit in the past already with server load and costs, so we should really try to decrease the size of our release tarballs to a saner level.
Fortunately, edk2 has a permissive BSD license, so we are not forced to distribute the sources for this. Thus instead of packaging the whole edk2 source tree in our tarballs, let's just do the bare minimum and provide the license information and a pointer to where the users can download the edk2 sources instead. This decreases the size of our tarballs already to the half of the original size.
Some few additional MiBs can be saved by omitting the sources of the skiboot firmware, which has a permissive license, too (see second patch). The final patch is rather cosmetics only - it drops some additional .yml and .git files from the tarball that are of no use for the normal user without the corresponding git repository.
Thomas Huth (3):
scripts/make-release: Do not include the edk2 sources in the tarball
anymore
scripts/make-release: Do not include the skiboot sources in the
tarball anymore
scripts/make-release: Remove CI yaml and more git files from the
tarball
scripts/make-release | 46 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 8 deletions(-)
Signature · 1 line
2.31.1
PATCH 1/3 scripts/make-release: Do not include the edk2 sources in the tarball anymore
The edk2 sources are bigger than the sources of QEMU - so they double the size of our release tarballs if we include them. Fortunately, edk2 has a permissive license, so there is no need for us to do this as long as we continue to distribute the edk2 license information in our release tarball.
--- scripts/make-release | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)
scripts/make-release +21 -6
@@ -20,19 +20,34 @@ git checkout "v${version}" git submodule update --init (cd roms/seabios && git describe --tags --long --dirty > .version) (cd roms/skiboot && ./make_version.sh > .version) -# Fetch edk2 submodule's submodules, since it won't have access to them via -# the tarball later. + +# Fetch edk2 submodule's submodules to get their license information. +# Edk2 is distributed under permissive licenses, so we don't have to +# include the full source tree here (which is bigger than QEMU's sources) +# as long as we provide the license information in the tarball. # # A more uniform way to handle this sort of situation would be nice, but we # don't necessarily have much control over how a submodule handles its # submodule dependencies, so we continue to handle these on a case-by-case # basis for now. -(cd roms/edk2 && \ - git submodule update --init -- \ +cd roms/edk2 +git submodule update --init --depth 1 -- \ ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \ BaseTools/Source/C/BrotliCompress/brotli \ - CryptoPkg/Library/OpensslLib/openssl \ - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli) + CryptoPkg/Library/OpensslLib/openssl +mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \ + License-softfloat.txt +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt +edk2rev=$(git rev-parse --short=12 HEAD) +rm -r $(ls | grep -v License) .[a-z]* +cat > sources.txt <<EOF +The edk2 sources can be downloaded from: + +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz +EOF +cd ../.. + popd tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination} rm -rf ${destination}
Signature · 1 line
2.31.1
Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore
Quoted 40 lines · Thomas Huth
The edk2 sources are bigger than the sources of QEMU - so they double the size of our release tarballs if we include them. Fortunately, edk2 has a permissive license, so there is no need for us to do this as long as we continue to distribute the edk2 license information in our release tarball.
Signed-off-by Thomas Huth <[email protected]>--- scripts/make-release | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)scripts/make-release +8
-6@@ -20,19 +20,34 @@ git checkout "v${version}" git submodule update --init (cd roms/seabios && git describe --tags --long --dirty > .version) (cd roms/skiboot && ./make_version.sh > .version) -# Fetch edk2 submodule's submodules, since it won't have access to them via -# the tarball later. + +# Fetch edk2 submodule's submodules to get their license information. +# Edk2 is distributed under permissive licenses, so we don't have to +# include the full source tree here (which is bigger than QEMU's sources) +# as long as we provide the license information in the tarball. # # A more uniform way to handle this sort of situation would be nice, but we # don't necessarily have much control over how a submodule handles its # submodule dependencies, so we continue to handle these on a case-by-case # basis for now. -(cd roms/edk2 && \ - git submodule update --init -- \ +cd roms/edk2 +git submodule update --init --depth 1 -- \ ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \ BaseTools/Source/C/BrotliCompress/brotli \ - CryptoPkg/Library/OpensslLib/openssl \ - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli) + CryptoPkg/Library/OpensslLib/openssl
scripts/make-release git checkout "v${version}" · line 47
Quoted 10 lines
scripts/make-release +10
-0+mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \ + License-softfloat.txt +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt +edk2rev=$(git rev-parse --short=12 HEAD) +rm -r $(ls | grep -v License) .[a-z]* +cat > sources.txt <<EOF +The edk2 sources can be downloaded from: + +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz
Please use QEMU's edk2 mirror:
https://gitlab.com/qemu-project/edk2
QEMU mirrors all dependencies so that even if upstream projects go offline we can still rebuild all of QEMU from source.
Thanks,
Stefan
Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore
Quoted 55 lines · Stefan Hajnoczi
Quoted 40 lines · Thomas Huth
The edk2 sources are bigger than the sources of QEMU - so they double the size of our release tarballs if we include them. Fortunately, edk2 has a permissive license, so there is no need for us to do this as long as we continue to distribute the edk2 license information in our release tarball.
Signed-off-by Thomas Huth <[email protected]>--- scripts/make-release | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)scripts/make-release +8
-6@@ -20,19 +20,34 @@ git checkout "v${version}" git submodule update --init (cd roms/seabios && git describe --tags --long --dirty > .version) (cd roms/skiboot && ./make_version.sh > .version) -# Fetch edk2 submodule's submodules, since it won't have access to them via -# the tarball later. + +# Fetch edk2 submodule's submodules to get their license information. +# Edk2 is distributed under permissive licenses, so we don't have to +# include the full source tree here (which is bigger than QEMU's sources) +# as long as we provide the license information in the tarball. # # A more uniform way to handle this sort of situation would be nice, but we # don't necessarily have much control over how a submodule handles its # submodule dependencies, so we continue to handle these on a case-by-case # basis for now. -(cd roms/edk2 && \ - git submodule update --init -- \ +cd roms/edk2 +git submodule update --init --depth 1 -- \ ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \ BaseTools/Source/C/BrotliCompress/brotli \ - CryptoPkg/Library/OpensslLib/openssl \ - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli) + CryptoPkg/Library/OpensslLib/opensslscripts/make-release git checkout "v${version}" · line 47
Quoted 10 lines
scripts/make-release +10
-0+mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \ + License-softfloat.txt +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt +edk2rev=$(git rev-parse --short=12 HEAD) +rm -r $(ls | grep -v License) .[a-z]* +cat > sources.txt <<EOF +The edk2 sources can be downloaded from: + +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gzPlease use QEMU's edk2 mirror:
https://gitlab.com/qemu-project/edk2
Quoted 2 lines
QEMU mirrors all dependencies so that even if upstream projects go offline we can still rebuild all of QEMU from source.
Sure, that makes sense, indeed. I'll wait a little bit longer for other comments, then I'll respin the series with the URLs updated (also in the second patch).
Thomas
Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore
In reply to Stefan HajnocziQuoted 55 lines · Stefan Hajnoczi
Quoted 40 lines · Thomas Huth
The edk2 sources are bigger than the sources of QEMU - so they double the size of our release tarballs if we include them. Fortunately, edk2 has a permissive license, so there is no need for us to do this as long as we continue to distribute the edk2 license information in our release tarball.
Signed-off-by Thomas Huth <[email protected]>--- scripts/make-release | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)scripts/make-release +8
-6@@ -20,19 +20,34 @@ git checkout "v${version}" git submodule update --init (cd roms/seabios && git describe --tags --long --dirty > .version) (cd roms/skiboot && ./make_version.sh > .version) -# Fetch edk2 submodule's submodules, since it won't have access to them via -# the tarball later. + +# Fetch edk2 submodule's submodules to get their license information. +# Edk2 is distributed under permissive licenses, so we don't have to +# include the full source tree here (which is bigger than QEMU's sources) +# as long as we provide the license information in the tarball. # # A more uniform way to handle this sort of situation would be nice, but we # don't necessarily have much control over how a submodule handles its # submodule dependencies, so we continue to handle these on a case-by-case # basis for now. -(cd roms/edk2 && \ - git submodule update --init -- \ +cd roms/edk2 +git submodule update --init --depth 1 -- \ ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \ BaseTools/Source/C/BrotliCompress/brotli \ - CryptoPkg/Library/OpensslLib/openssl \ - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli) + CryptoPkg/Library/OpensslLib/opensslscripts/make-release git checkout "v${version}" · line 47
Quoted 10 lines
scripts/make-release +10
-0+mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \ + License-softfloat.txt +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt +edk2rev=$(git rev-parse --short=12 HEAD) +rm -r $(ls | grep -v License) .[a-z]* +cat > sources.txt <<EOF +The edk2 sources can be downloaded from: + +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gzPlease use QEMU's edk2 mirror:
https://gitlab.com/qemu-project/edk2
Quoted 2 lines
QEMU mirrors all dependencies so that even if upstream projects go offline we can still rebuild all of QEMU from source.
Note that the github/lab generated tarballs are not signed, while QEMU's release tarballs are gpg signed, so from that POV this would be a regression no matter which site we point to.
Also it would need more guidance on what to actually do with the tarball, as if you merely unpack it into this dir, it won't work as it will be one level of dirs nesting too deep for QEMU's build scripts to work.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore
Quoted 2 lines · Daniel P. Berrangé
Quoted 1 line · Stefan Hajnoczi
On Mon, 4 Jul 2022 at 07:45, Thomas Huth <[email protected]> wrote:
...
Quoted 8 lines
Quoted 8 lines
scripts/make-release git checkout "v${version}" · line 47
Quoted 4 lines
scripts/make-release +4
-0+cat > sources.txt <<EOF +The edk2 sources can be downloaded from: + +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gzPlease use QEMU's edk2 mirror:
https://gitlab.com/qemu-project/edk2
Quoted 6 lines
Quoted 2 lines
QEMU mirrors all dependencies so that even if upstream projects go offline we can still rebuild all of QEMU from source.
Note that the github/lab generated tarballs are not signed, while QEMU's release tarballs are gpg signed, so from that POV this would be a regression no matter which site we point to.
Maybe we should point to the edk2 release page instead? I.e.:
https://github.com/tianocore/edk2/releases
Anyway, it's IMHO certainly not *our* job to provide signed edk2 sources here.
Quoted 4 lines
Also it would need more guidance on what to actually do with the tarball, as if you merely unpack it into this dir, it won't work as it will be one level of dirs nesting too deep for QEMU's build scripts to work.
I could add some wording how to use the edk2-build.sh script with a downloaded edk2 tarball if that helps ... not sure whether it is really required, though, we also don't provide instruction for building any other firmware yet, as far as I know.
Thomas
Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore
Quoted 9 lines
Quoted 4 lines
Also it would need more guidance on what to actually do with the tarball, as if you merely unpack it into this dir, it won't work as it will be one level of dirs nesting too deep for QEMU's build scripts to work.
I could add some wording how to use the
edk2-build.shscript with a downloaded edk2 tarball if that helps ... not sure whether it is really required, though, we also don't provide instruction for building any other firmware yet, as far as I know.
The instructions are printed by "make -C roms help".
We could teach roms/Makefile to either automatically fetch the sources (be that the upstream edk2 tarball or a qemu-firmware-src tarball should we go with the approach suggested by Daniel elsewhere in this thread), or throw an error with instructions when they are not present.
take care,
Gerd
Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore
In reply to Thomas HuthQuoted 30 lines · Thomas Huth
Quoted 2 lines · Daniel P. Berrangé
Quoted 1 line · Stefan Hajnoczi
On Mon, 4 Jul 2022 at 07:45, Thomas Huth <[email protected]> wrote:
...
Quoted 8 lines
Quoted 8 lines
scripts/make-release git checkout "v${version}" · line 47
Quoted 4 lines
scripts/make-release +4
-0+cat > sources.txt <<EOF +The edk2 sources can be downloaded from: + +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gzPlease use QEMU's edk2 mirror:
https://gitlab.com/qemu-project/edk2Quoted 6 lines
Quoted 2 lines
QEMU mirrors all dependencies so that even if upstream projects go offline we can still rebuild all of QEMU from source.
Note that the github/lab generated tarballs are not signed, while QEMU's release tarballs are gpg signed, so from that POV this would be a regression no matter which site we point to.
Maybe we should point to the edk2 release page instead? I.e.:
https://github.com/tianocore/edk2/releases
Anyway, it's IMHO certainly not *our* job to provide signed edk2 sources here.
Quoted 4 lines
Also it would need more guidance on what to actually do with the tarball, as if you merely unpack it into this dir, it won't work as it will be one level of dirs nesting too deep for QEMU's build scripts to work.
Quoted 4 lines
I could add some wording how to use the
edk2-build.shscript with a downloaded edk2 tarball if that helps ... not sure whether it is really required, though, we also don't provide instruction for building any other firmware yet, as far as I know.
Well my thought is that currently you can do 'make -C roms efi' from the unpacked release tarball, and with this change, that no longer works. So we ought to explain what to do to get that working again.
This doesn't neccessarily means docs. It could just be 'make' prints out an error telling people what to do, or figure it out itself.
If we split off the entire of the 'roms' directory into a self-contained firmware-src tarball, the 'roms/Makefile' would still "just work".
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore
Quoted 40 lines · Daniel P. Berrangé
Quoted 30 lines · Thomas Huth
Quoted 2 lines · Daniel P. Berrangé
Quoted 1 line · Stefan Hajnoczi
On Mon, 4 Jul 2022 at 07:45, Thomas Huth <[email protected]> wrote:
...
Quoted 8 lines
Quoted 8 lines
scripts/make-release git checkout "v${version}" · line 47
Quoted 4 lines
scripts/make-release +4
-0+cat > sources.txt <<EOF +The edk2 sources can be downloaded from: + +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gzPlease use QEMU's edk2 mirror:
https://gitlab.com/qemu-project/edk2Quoted 6 lines
Quoted 2 lines
QEMU mirrors all dependencies so that even if upstream projects go offline we can still rebuild all of QEMU from source.
Note that the github/lab generated tarballs are not signed, while QEMU's release tarballs are gpg signed, so from that POV this would be a regression no matter which site we point to.
Maybe we should point to the edk2 release page instead? I.e.:
https://github.com/tianocore/edk2/releases
Anyway, it's IMHO certainly not *our* job to provide signed edk2 sources here.
Quoted 4 lines
Also it would need more guidance on what to actually do with the tarball, as if you merely unpack it into this dir, it won't work as it will be one level of dirs nesting too deep for QEMU's build scripts to work.
Quoted 4 lines
I could add some wording how to use the
edk2-build.shscript with a downloaded edk2 tarball if that helps ... not sure whether it is really required, though, we also don't provide instruction for building any other firmware yet, as far as I know.Well my thought is that currently you can do 'make -C roms efi' from the unpacked release tarball, and with this change, that no longer works. So we ought to explain what to do to get that working again.
Quoted 5 lines
This doesn't neccessarily means docs. It could just be 'make' prints out an error telling people what to do, or figure it out itself.
If we split off the entire of the 'roms' directory into a self-contained firmware-src tarball, the 'roms/Makefile' would still "just work".
Sorry for the late reply, I've been pondering about this for a while and was busy with other stuff...
So I think I tend to agree - if we still want to ship the edk2 (and skiboot) sources, it's likely best if we put all roms/* sources into a separate tarball. Maybe even split the roms folder into a separate git repository.
However, there's at least one minor obstacle: The sources of the s390-ccw bios (in pc-bios/s390-ccw) currently partly depend on the roms/SLOF sources to be available for the full build. So if we slice out the roms/SLOF stuff, we should likely move the s390-ccw bios also to same location...
I guess it's now too late for 7.1, so I'll postpone my further work on this to the 7.2 release cycle.
Thomas
PATCH 2/3 scripts/make-release: Do not include the skiboot sources in the tarball anymore
In reply to Thomas HuthThe skiboot sources are licensed under the Apache license, so we don't have to include them in our tarball as long as we continue to distribute the skiboot license information in our release tarball.
--- scripts/make-release | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
scripts/make-release +14 -1
@@ -19,7 +19,6 @@ pushd ${destination} git checkout "v${version}" git submodule update --init (cd roms/seabios && git describe --tags --long --dirty > .version) -(cd roms/skiboot && ./make_version.sh > .version) # Fetch edk2 submodule's submodules to get their license information. # Edk2 is distributed under permissive licenses, so we don't have to
@@ -48,6 +47,20 @@ https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz EOF cd ../.. +# Same for skiboot - it has a permissive Apache license, so we don't +# need to ship the sources for this. +cd roms/skiboot +skibootrev=$(git rev-parse --short=12 HEAD) +head -n 38 libstb/tss2/ibmtpm20tss/utils/tss.c > LICENSE-tss.txt +head -n 50 libfdt/fdt.c > LICENSE-libfdt.txt +rm -r $(ls | grep -v LICEN) .[a-z]* +cat > sources.txt <<EOF +The skiboot sources can be downloaded from: + +https://github.com/open-power/skiboot/archive/${skibootrev}.tar.gz +EOF +cd ../.. + popd tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination} rm -rf ${destination}
Signature · 1 line
2.31.1
Re: [PATCH 2/3] scripts/make-release: Do not include the skiboot sources in the tarball anymore
Quoted 5 lines · Thomas Huth
The skiboot sources are licensed under the Apache license, so we don't have to include them in our tarball as long as we continue to distribute the skiboot license information in our release tarball.
Signed-off-by Thomas Huth <[email protected]>
Thanks,
C.
Quoted 37 lines
--- scripts/make-release | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)scripts/make-release +14
-1@@ -19,7 +19,6 @@ pushd ${destination} git checkout "v${version}" git submodule update --init (cd roms/seabios && git describe --tags --long --dirty > .version) -(cd roms/skiboot && ./make_version.sh > .version) # Fetch edk2 submodule's submodules to get their license information. # Edk2 is distributed under permissive licenses, so we don't have to@@ -48,6 +47,20 @@ https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz EOF cd ../.. +# Same for skiboot - it has a permissive Apache license, so we don't +# need to ship the sources for this. +cd roms/skiboot +skibootrev=$(git rev-parse --short=12 HEAD) +head -n 38 libstb/tss2/ibmtpm20tss/utils/tss.c > LICENSE-tss.txt +head -n 50 libfdt/fdt.c > LICENSE-libfdt.txt +rm -r $(ls | grep -v LICEN) .[a-z]* +cat > sources.txt <<EOF +The skiboot sources can be downloaded from: + +https://github.com/open-power/skiboot/archive/${skibootrev}.tar.gz +EOF +cd ../.. + popd tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination} rm -rf ${destination}
PATCH 3/3 scripts/make-release: Remove CI yaml and more git files from the tarball
In reply to Thomas HuthThese files are of no use in a normal tarball and thus should not be included here.
--- scripts/make-release | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
scripts/make-release +3 -1
@@ -61,6 +61,8 @@ https://github.com/open-power/skiboot/archive/${skibootrev}.tar.gz EOF cd ../.. +rm .*.yml popd -tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination} + +tar --exclude=".git*" -cjf ${destination}.tar.bz2 ${destination} rm -rf ${destination}
Signature · 1 line
2.31.1
Re: [PATCH 3/3] scripts/make-release: Remove CI yaml and more git files from the tarball
Quoted 14 lines · Thomas Huth
These files are of no use in a normal tarball and thus should not be included here.
Signed-off-by Thomas Huth <[email protected]>--- scripts/make-release | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)scripts/make-release +0
-0
scripts/make-release https://github.com/open-power/skiboot/archive/${skibootrev}.tar.gz · line 67
Quoted 8 lines
scripts/make-release +3
-1@@ -61,6 +61,8 @@ https://github.com/open-power/skiboot/archive/${skibootrev}.tar.gz EOF cd ../.. +rm .*.yml popd -tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination} + +tar --exclude=".git*" -cjf ${destination}.tar.bz2 ${destination}
If the excludes become more elaborate in the future we can use tar --exclude-from=FILE and keep a separate file with all the patterns instead of using both rm and --exclude= patterns.
Thanks for doing this! I haven't reviewed the licensing impact.
Quoted 9 lines · Thomas Huth
Our release tarballs are huge - qemu-7.0.0.tar.xz has a size of 119 MiB. If you look at the contents, more than half of the size is used for the edk2 sources that we ship along to provide the sources for the firmware binaries, too. This feels very wrong, why do we urge users to download such huge tarballs while 99.9% of them never will rebuilt the firmware sources? We were also struggeling a bit in the past already with server load and costs, so we should really try to decrease the size of our release tarballs to a saner level.
Quoted 6 lines
Fortunately, edk2 has a permissive BSD license, so we are not forced to distribute the sources for this. Thus instead of packaging the whole edk2 source tree in our tarballs, let's just do the bare minimum and provide the license information and a pointer to where the users can download the edk2 sources instead. This decreases the size of our tarballs already to the half of the original size.
Regardless of license, we are not required to bundle the source code and binaries in the same tarball. We've merely done that because it was a convenient & easy way to approach the problem historically.
It would be valid to not ship *any* of the source for the pre-built roms in the main qemu-x.y.z.tar.xz file /provided/ we ensure that we *always* have a qemu-firmware-src-x.y.z.tar.xz file alongside it.
Quoted 5 lines
Some few additional MiBs can be saved by omitting the sources of the skiboot firmware, which has a permissive license, too (see second patch). The final patch is rather cosmetics only - it drops some additional .yml and .git files from the tarball that are of no use for the normal user without the corresponding git repository.
I look at a few more scenarios
* Current tarball: 119 MB
* minus edk/skiboot source: 54 MB
* also minus edk2 binaries: 45 MB
* also minus pc-bios/ + roms/: 19 MB
* minus roms/ only: 31 MB
IOW, cutting the tarball in half is great, but if we split off firmware binaries and source into completely separated tarballs we would win big. If we fully split off only the firmware source we still win quite alot.
IOW, rather than special casing edk/skiboot, I would prefer to see us have make a consistent approach to firmware.
Either
* qemu-x.y.z.tar.gz (only qemu maintained src, 19 MB)
* qemu-firmware-x.y.z.tar.gz (pre-built blobs aka pc-bios/, 13 MB)
* qemu-firmware-src-x.y.z.tar.gz (source for pre-built blobs aka roms/, 92 MB)
Or
* qemu-x.y.z.tar.gz (qemu maintained src and pre-built blobs, 31 MB)
* qemu-firmware-src-x.y.z.tar.gz (source for pre-built blobs aka roms/, 92 MB)
The second option is probably the least disruptive option for end users building QEMU directly, while still giving distros most of the benefits they desire. And probably easiest to put into practice for us.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|