populate_sdk_base: add support for zstd compression (and default to it)
RFC PATCH 1/1 populate_sdk_base: add support for zstd compression (and default to it)
This allows significantly faster do_populate_sdk() (5 min -> 3 min for the sato sdk on my machine) and decompression, at the cost of SDK file taking up more disk space.
If that's a significant concern, it's possible to increase the zstd compression level or even go back to xz compression.
--- meta/classes/populate_sdk_base.bbclass | 10 ++++++++-- meta/files/toolchain-shar-extract.sh | 9 ++++----- 2 files changed, 12 insertions(+), 7 deletions(-)
meta/classes/populate_sdk_base.bbclass +8 -2
@@ -50,17 +50,23 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" # Default archived SDK's suffix -SDK_ARCHIVE_TYPE ?= "tar.xz" +SDK_ARCHIVE_TYPE ?= "tar.zst" SDK_XZ_COMPRESSION_LEVEL ?= "-9" SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${SDK_XZ_COMPRESSION_LEVEL}" -# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip and tar.xz +SDK_ZSTD_COMPRESSION_LEVEL ?= "" +SDK_ZSTD_OPTIONS ?= "-T${ZSTD_THREADS} ${SDK_ZSTD_COMPRESSION_LEVEL}" + +# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip, tar.zst and tar.xz python () { if d.getVar('SDK_ARCHIVE_TYPE') == 'zip': d.setVar('SDK_ARCHIVE_DEPENDS', 'zip-native') # SDK_ARCHIVE_CMD used to generate archived sdk ${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} from input dir ${SDK_OUTPUT}/${SDKPATH} to output dir ${SDKDEPLOYDIR} # recommand to cd into input dir first to avoid archive with buildpath d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r -y ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .') + elif d.getVar('SDK_ARCHIVE_TYPE') == 'tar.zst': + d.setVar('SDK_ARCHIVE_DEPENDS', 'zstd-native') + d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | zstd ${SDK_ZSTD_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}') else: d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native') d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
meta/files/toolchain-shar-extract.sh +4 -5
@@ -51,11 +51,6 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then fi fi -if ! xz -V > /dev/null 2>&1; then - echo "Error: xz is required for installation of this SDK, please install it first" - exit 1 -fi - SDK_BUILD_PATH="@SDKPATH@" DEFAULT_INSTALL_DIR="@SDKPATHINSTALL@" SUDO_EXEC=""
@@ -121,6 +116,8 @@ if [ "$listcontents" = "1" ] ; then else rm sdk.zip && exit 1 fi + elif [ @SDK_ARCHIVE_TYPE@ = "tar.zst" ]; then + tail -n +$payload_offset "$0"| tar tv --zstd || exit 1 else tail -n +$payload_offset "$0"| tar tvJ || exit 1 fi
@@ -249,6 +246,8 @@ if [ @SDK_ARCHIVE_TYPE@ = "zip" ]; then else rm sdk.zip && exit 1 fi +elif [ @SDK_ARCHIVE_TYPE@ = "tar.zst" ]; then + tail -n +$payload_offset "$0"| $SUDO_EXEC tar mx -I "pzstd -p 8" -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1 else tail -n +$payload_offset "$0"| $SUDO_EXEC tar mxJ -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1 fi
Signature · 1 line
2.20.1
Quoted 3 lines · Alexander Kanavin
This allows significantly faster
do_populate_sdk()(5 min -> 3 min for the sato sdk on my machine) and decompression, at the cost of SDK file taking up more disk space.
Can you quantify "more"?
I'd be interested to know if a higher compression helps and what the time/cost tradeoff is too. Not sure I'm keep to have it increase in size. This likely increases the eSDK size a lot too?
Quoted 44 lines
If that's a significant concern, it's possible to increase the zstd compression level or even go back to xz compression.
Signed-off-by Alexander Kanavin <[email protected]>--- meta/classes/populate_sdk_base.bbclass | 10 ++++++++-- meta/files/toolchain-shar-extract.sh | 9 ++++----- 2 files changed, 12 insertions(+), 7 deletions(-)meta/classes/populate_sdk_base.bbclass +8
-2@@ -50,17 +50,23 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" # Default archived SDK's suffix -SDK_ARCHIVE_TYPE ?= "tar.xz" +SDK_ARCHIVE_TYPE ?= "tar.zst" SDK_XZ_COMPRESSION_LEVEL ?= "-9" SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${SDK_XZ_COMPRESSION_LEVEL}" -# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip and tar.xz +SDK_ZSTD_COMPRESSION_LEVEL ?= "" +SDK_ZSTD_OPTIONS ?= "-T${ZSTD_THREADS} ${SDK_ZSTD_COMPRESSION_LEVEL}" + +# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip, tar.zst and tar.xz python () { if d.getVar('SDK_ARCHIVE_TYPE') == 'zip': d.setVar('SDK_ARCHIVE_DEPENDS', 'zip-native') # SDK_ARCHIVE_CMD used to generate archived sdk ${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} from input dir ${SDK_OUTPUT}/${SDKPATH} to output dir ${SDKDEPLOYDIR} # recommand to cd into input dir first to avoid archive with buildpath d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r -y ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .') + elif d.getVar('SDK_ARCHIVE_TYPE') == 'tar.zst': + d.setVar('SDK_ARCHIVE_DEPENDS', 'zstd-native') + d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | zstd ${SDK_ZSTD_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}') else: d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native') d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')meta/files/toolchain-shar-extract.sh +0
-0
meta/files/toolchain-shar-extract.sh if [ "$INST_ARCH" != "$SDK_ARCH" ]; then · line 55
Quoted 10 lines
meta/files/toolchain-shar-extract.sh +0
-5@@ -51,11 +51,6 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then fi fi -if ! xz -V > /dev/null 2>&1; then - echo "Error: xz is required for installation of this SDK, please install it first" - exit 1 -fi -
Deleting that is fine for testing but I think the final patch needs to do better! ;-)
Cheers,
Richard
Quoted 10 lines · Richard Purdie
Quoted 3 lines · Alexander Kanavin
This allows significantly faster
do_populate_sdk()(5 min -> 3 min for the sato sdk on my machine) and decompression, at the cost of SDK file taking up more disk space.Can you quantify "more"?
I'd be interested to know if a higher compression helps and what the time/cost tradeoff is too. Not sure I'm keep to have it increase in size. This likely increases the eSDK size a lot too?
since eSDKs are generate once use many times compression time is not as important, decompression time is important, however, they are also transmitted via network which means we might lose out the decompression gain towards network transfer time. Perhaps it will be good to get a wholesome picture.
Quoted 73 lines
Quoted 45 lines
If that's a significant concern, it's possible to increase the zstd compression level or even go back to xz compression.
Signed-off-by Alexander Kanavin <[email protected]>--- meta/classes/populate_sdk_base.bbclass | 10 ++++++++-- meta/files/toolchain-shar-extract.sh | 9 ++++----- 2 files changed, 12 insertions(+), 7 deletions(-)meta/classes/populate_sdk_base.bbclass +8
-2@@ -50,17 +50,23 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" # Default archived SDK's suffix -SDK_ARCHIVE_TYPE ?= "tar.xz" +SDK_ARCHIVE_TYPE ?= "tar.zst" SDK_XZ_COMPRESSION_LEVEL ?= "-9" SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${SDK_XZ_COMPRESSION_LEVEL}" -# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip and tar.xz +SDK_ZSTD_COMPRESSION_LEVEL ?= "" +SDK_ZSTD_OPTIONS ?= "-T${ZSTD_THREADS} ${SDK_ZSTD_COMPRESSION_LEVEL}" + +# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip, tar.zst and tar.xz python () { if d.getVar('SDK_ARCHIVE_TYPE') == 'zip': d.setVar('SDK_ARCHIVE_DEPENDS', 'zip-native') # SDK_ARCHIVE_CMD used to generate archived sdk ${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} from input dir ${SDK_OUTPUT}/${SDKPATH} to output dir ${SDKDEPLOYDIR} # recommand to cd into input dir first to avoid archive with buildpath d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r -y ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .') + elif d.getVar('SDK_ARCHIVE_TYPE') == 'tar.zst': + d.setVar('SDK_ARCHIVE_DEPENDS', 'zstd-native') + d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | zstd ${SDK_ZSTD_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}') else: d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native') d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')meta/files/toolchain-shar-extract.sh +0
-0meta/files/toolchain-shar-extract.sh if [ "$INST_ARCH" != "$SDK_ARCH" ]; then · line 55
Quoted 10 lines
meta/files/toolchain-shar-extract.sh +0
-5@@ -51,11 +51,6 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then fi fi -if ! xz -V > /dev/null 2>&1; then - echo "Error: xz is required for installation of this SDK, please install it first" - exit 1 -fi -Deleting that is fine for testing but I think the final patch needs to do better! ;-)
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157005): https://lists.openembedded.org/g/openembedded-core/message/157005
Mute This Topic: https://lists.openembedded.org/mt/86353954/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 8 lines · Richard Purdie
Can you quantify "more"?
I'd be interested to know if a higher compression helps and what the time/cost
tradeoff is too. Not sure I'm keep to have it increase in size. This likely increases the eSDK size a lot too?
The benefit is indeed not unquestionable, hence the RFC.
1. The eSDK is mostly made of already compressed sstate artifacts, so the effect on size is minimal and the zstd's time benefit is clear:
$ bitbake -c populate_sdk_ext core-image-sato (to have everything ready) $ bitbake -c cleansstate core-image-sato buildtools-tarball $ time bitbake -c populate_sdk_ext core-image-sato
xz time/size
4m46s 1.7G
zstd compression level/time/size
19 3m4s 1.7G
10 2m44s 1.8G
2. For the basic SDK (bitbake commands are similar), xz's better (but slower) compression does show clearly, but zstd's time wins are similar :
xz time/size
5m20s 581M
zstd compression level/time/size:
19 4m5s 837M
15 3m44s 979M
10 3m22s 993M
default(3) 3m19s 1.1G
Alex
Quoted 5 lines · Alexander Kanavin
Quoted 3 lines · Richard Purdie
Can you quantify "more"?
Quoted 8 lines
Quoted 5 lines
I'd be interested to know if a higher compression helps and what the time/cost
tradeoff is too. Not sure I'm keep to have it increase in size. This likely increases the eSDK size a lot too?The benefit is indeed not unquestionable, hence the RFC.
Thanks, I think we need data to be able to comment effectively! :)
Quoted 7 lines
1. The eSDK is mostly made of already compressed sstate artifacts, so the effect on size is minimal and the zstd's time benefit is clear:
$ bitbake -c populate_sdk_ext core-image-sato (to have everything ready) $ bitbake -c cleansstate core-image-sato buildtools-tarball $ time bitbake -c populate_sdk_ext core-image-sato
Quoted 6 lines
xz time/size
4m46s 1.7Gzstd compression level/time/size
19 3m4s 1.7G
10 2m44s 1.8G
For the eSDK this looks like an ok change.
Quoted 11 lines
2. For the basic SDK (bitbake commands are similar), xz's better (but slower) compression does show clearly, but zstd's time wins are similar :
xz time/size
5m20s 581Mzstd compression level/time/size:
19 4m5s 837M
15 3m44s 979M
10 3m22s 993M
default(3) 3m19s 1.1G
but nearly doubling the size of the standard SDK isn't going to be well received by people using it, those numbers don't look like they can work even if it is faster. The key things are primarily size and secondly decompression time, compression time is in third place and unfortunate but something the build systems can afford to pay.
Cheers,
Richard
Quoted 32 lines · Alexander Kanavin
Quoted 9 lines · Richard Purdie
Can you quantify "more"?
I'd be interested to know if a higher compression helps and what the time/cost
tradeoff is too. Not sure I'm keep to have it increase in size. This likely
increases the eSDK size a lot too?The benefit is indeed not unquestionable, hence the RFC.
1. The eSDK is mostly made of already compressed sstate artifacts, so the effect on size is minimal and the zstd's time benefit is clear:
$ bitbake -c populate_sdk_ext core-image-sato (to have everything ready) $ bitbake -c cleansstate core-image-sato buildtools-tarball $ time bitbake -c populate_sdk_ext core-image-satoxz time/size
4m46s 1.7Gzstd compression level/time/size
19 3m4s 1.7G
10 2m44s 1.8G2. For the basic SDK (bitbake commands are similar), xz's better (but slower) compression does show clearly, but zstd's time wins are similar :
Quoted 9 lines
xz time/size
5m20s 581Mzstd compression level/time/size:
19 4m5s 837M
15 3m44s 979M
10 3m22s 993M
default(3) 3m19s 1.1G
Is this the maximum compression level for zstd? Also, are any of these using parallel compression?
Quoted 13 lines
Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#157009):
https://lists.openembedded.org/g/openembedded-core/message/157009
Mute This Topic: https://lists.openembedded.org/mt/86353954/3616693
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 4 lines · Joshua Watt
Is this the maximum compression level for zstd? Also, are any of these using parallel compression?
Yes, they all run with the same standard cpu cores setting from bitbake.conf. xz additionally has a RAM cap.
zstd has further compression levels, but I didn't try them - you're weclome to:
--ultra : enable levels beyond 19, up to 22 (requires more memory)
Alex