Skip to contentSkip to conversation

populate_sdk_base: add support for zstd compression (and default to it)

RFC PATCH In review Alexander Kanavin 4 participants 7 messages Oct 15, 2021 – Oct 17, 2021

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 populate_sdk_base: add support for zstd compression (and default to it)

This allows significantly faster do_populate_sdk() (5 min -> 3 min for the sato sdk on my machine) and decompression, at the cost of SDK file taking up more disk space.

If that's a significant concern, it's possible to increase the zstd compression level or even go back to xz compression.

---
 meta/classes/populate_sdk_base.bbclass | 10 ++++++++--
 meta/files/toolchain-shar-extract.sh   |  9 ++++-----
 2 files changed, 12 insertions(+), 7 deletions(-)
meta/classes/populate_sdk_base.bbclass +8 -2
index 49e166e697..92a23e2325 100644
@@ -50,17 +50,23 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
 TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
 
 # Default archived SDK's suffix
-SDK_ARCHIVE_TYPE ?= "tar.xz"
+SDK_ARCHIVE_TYPE ?= "tar.zst"
 SDK_XZ_COMPRESSION_LEVEL ?= "-9"
 SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${SDK_XZ_COMPRESSION_LEVEL}"
 
-# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip and tar.xz
+SDK_ZSTD_COMPRESSION_LEVEL ?= ""
+SDK_ZSTD_OPTIONS ?= "-T${ZSTD_THREADS} ${SDK_ZSTD_COMPRESSION_LEVEL}"
+
+# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip, tar.zst and tar.xz
 python () {
     if d.getVar('SDK_ARCHIVE_TYPE') == 'zip':
        d.setVar('SDK_ARCHIVE_DEPENDS', 'zip-native')
        # SDK_ARCHIVE_CMD used to generate archived sdk ${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} from input dir ${SDK_OUTPUT}/${SDKPATH} to output dir ${SDKDEPLOYDIR}
        # recommand to cd into input dir first to avoid archive with buildpath
        d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r -y ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .')
+    elif d.getVar('SDK_ARCHIVE_TYPE') == 'tar.zst':
+       d.setVar('SDK_ARCHIVE_DEPENDS', 'zstd-native')
+       d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | zstd ${SDK_ZSTD_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
     else:
        d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native')
        d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
meta/files/toolchain-shar-extract.sh +4 -5
index 4386b985bb..4b8892af79 100644
@@ -51,11 +51,6 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then
 	fi
 fi
 
-if ! xz -V > /dev/null 2>&1; then
-	echo "Error: xz is required for installation of this SDK, please install it first"
-	exit 1
-fi
-
 SDK_BUILD_PATH="@SDKPATH@"
 DEFAULT_INSTALL_DIR="@SDKPATHINSTALL@"
 SUDO_EXEC=""
@@ -121,6 +116,8 @@ if [ "$listcontents" = "1" ] ; then
         else
             rm sdk.zip && exit 1
         fi
+    elif [ @SDK_ARCHIVE_TYPE@ = "tar.zst" ]; then
+        tail -n +$payload_offset "$0"| tar tv --zstd || exit 1
     else
         tail -n +$payload_offset "$0"| tar tvJ || exit 1
     fi
@@ -249,6 +246,8 @@ if [ @SDK_ARCHIVE_TYPE@ = "zip" ]; then
     else
         rm sdk.zip && exit 1
     fi
+elif [ @SDK_ARCHIVE_TYPE@ = "tar.zst" ]; then
+    tail -n +$payload_offset "$0"| $SUDO_EXEC tar mx -I "pzstd -p 8" -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1
 else
     tail -n +$payload_offset "$0"| $SUDO_EXEC tar mxJ -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1
 fi
Signature · 1 line
2.20.1
#1 lore raw
Quoted 3 lines · Alexander Kanavin

This allows significantly faster do_populate_sdk() (5 min -> 3 min for the sato sdk on my machine) and decompression, at the cost of SDK file taking up more disk space.

Can you quantify "more"?

I'd be interested to know if a higher compression helps and what the time/cost tradeoff is too. Not sure I'm keep to have it increase in size. This likely increases the eSDK size a lot too?

Quoted 44 lines

If that's a significant concern, it's possible to increase the zstd compression level or even go back to xz compression.

---
 meta/classes/populate_sdk_base.bbclass | 10 ++++++++--
 meta/files/toolchain-shar-extract.sh   |  9 ++++-----
 2 files changed, 12 insertions(+), 7 deletions(-)
meta/classes/populate_sdk_base.bbclass +8 -2
index 49e166e697..92a23e2325 100644
@@ -50,17 +50,23 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
 TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
 
 # Default archived SDK's suffix
-SDK_ARCHIVE_TYPE ?= "tar.xz"
+SDK_ARCHIVE_TYPE ?= "tar.zst"
 SDK_XZ_COMPRESSION_LEVEL ?= "-9"
 SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${SDK_XZ_COMPRESSION_LEVEL}"
 
-# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip and tar.xz
+SDK_ZSTD_COMPRESSION_LEVEL ?= ""
+SDK_ZSTD_OPTIONS ?= "-T${ZSTD_THREADS} ${SDK_ZSTD_COMPRESSION_LEVEL}"
+
+# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip, tar.zst and tar.xz
 python () {
     if d.getVar('SDK_ARCHIVE_TYPE') == 'zip':
        d.setVar('SDK_ARCHIVE_DEPENDS', 'zip-native')
        # SDK_ARCHIVE_CMD used to generate archived sdk ${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} from input dir ${SDK_OUTPUT}/${SDKPATH} to output dir ${SDKDEPLOYDIR}
        # recommand to cd into input dir first to avoid archive with buildpath
        d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r -y ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .')
+    elif d.getVar('SDK_ARCHIVE_TYPE') == 'tar.zst':
+       d.setVar('SDK_ARCHIVE_DEPENDS', 'zstd-native')
+       d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | zstd ${SDK_ZSTD_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
     else:
        d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native')
        d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
meta/files/toolchain-shar-extract.sh +0 -0
index 4386b985bb..4b8892af79 100644

meta/files/toolchain-shar-extract.sh if [ "$INST_ARCH" != "$SDK_ARCH" ]; then · line 55

Quoted 10 lines
meta/files/toolchain-shar-extract.sh +0 -5
@@ -51,11 +51,6 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then
 	fi
 fi
 
-if ! xz -V > /dev/null 2>&1; then
-	echo "Error: xz is required for installation of this SDK, please install it first"
-	exit 1
-fi
-
 
 

Deleting that is fine for testing but I think the final patch needs to do better! ;-)

Cheers,

Richard

#2 lore raw
Quoted 10 lines · Richard Purdie
Quoted 3 lines · Alexander Kanavin

This allows significantly faster do_populate_sdk() (5 min -> 3 min for the sato sdk on my machine) and decompression, at the cost of SDK file taking up more disk space.

Can you quantify "more"?

I'd be interested to know if a higher compression helps and what the time/cost tradeoff is too. Not sure I'm keep to have it increase in size. This likely increases the eSDK size a lot too?

since eSDKs are generate once use many times compression time is not as important, decompression time is important, however, they are also transmitted via network which means we might lose out the decompression gain towards network transfer time. Perhaps it will be good to get a wholesome picture.

Quoted 73 lines
Quoted 45 lines

If that's a significant concern, it's possible to increase the zstd compression level or even go back to xz compression.

---
  meta/classes/populate_sdk_base.bbclass | 10 ++++++++--
  meta/files/toolchain-shar-extract.sh   |  9 ++++-----
  2 files changed, 12 insertions(+), 7 deletions(-)
meta/classes/populate_sdk_base.bbclass +8 -2
index 49e166e697..92a23e2325 100644
@@ -50,17 +50,23 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
  TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
 
  # Default archived SDK's suffix
-SDK_ARCHIVE_TYPE ?= "tar.xz"
+SDK_ARCHIVE_TYPE ?= "tar.zst"
  SDK_XZ_COMPRESSION_LEVEL ?= "-9"
  SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${SDK_XZ_COMPRESSION_LEVEL}"
 
-# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip and tar.xz
+SDK_ZSTD_COMPRESSION_LEVEL ?= ""
+SDK_ZSTD_OPTIONS ?= "-T${ZSTD_THREADS} ${SDK_ZSTD_COMPRESSION_LEVEL}"
+
+# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip, tar.zst and tar.xz
  python () {
      if d.getVar('SDK_ARCHIVE_TYPE') == 'zip':
         d.setVar('SDK_ARCHIVE_DEPENDS', 'zip-native')
         # SDK_ARCHIVE_CMD used to generate archived sdk ${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} from input dir ${SDK_OUTPUT}/${SDKPATH} to output dir ${SDKDEPLOYDIR}
         # recommand to cd into input dir first to avoid archive with buildpath
         d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r -y ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .')
+    elif d.getVar('SDK_ARCHIVE_TYPE') == 'tar.zst':
+       d.setVar('SDK_ARCHIVE_DEPENDS', 'zstd-native')
+       d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | zstd ${SDK_ZSTD_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
      else:
         d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native')
         d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
meta/files/toolchain-shar-extract.sh +0 -0
index 4386b985bb..4b8892af79 100644

meta/files/toolchain-shar-extract.sh if [ "$INST_ARCH" != "$SDK_ARCH" ]; then · line 55

Quoted 10 lines
meta/files/toolchain-shar-extract.sh +0 -5
@@ -51,11 +51,6 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then
  	fi
  fi
 
-if ! xz -V > /dev/null 2>&1; then
-	echo "Error: xz is required for installation of this SDK, please install it first"
-	exit 1
-fi
-
 
 

Deleting that is fine for testing but I think the final patch needs to do better! ;-)

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157005): https://lists.openembedded.org/g/openembedded-core/message/157005
Mute This Topic: https://lists.openembedded.org/mt/86353954/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw
In reply to Richard Purdie
Quoted 8 lines · Richard Purdie

Can you quantify "more"?

I'd be interested to know if a higher compression helps and what the time/cost
tradeoff is too. Not sure I'm keep to have it increase in size. This likely increases the eSDK size a lot too?

The benefit is indeed not unquestionable, hence the RFC.

1. The eSDK is mostly made of already compressed sstate artifacts, so the effect on size is minimal and the zstd's time benefit is clear:

$ bitbake -c populate_sdk_ext core-image-sato (to have everything ready)
$ bitbake -c cleansstate core-image-sato buildtools-tarball
$ time bitbake -c populate_sdk_ext core-image-sato

xz time/size
4m46s 1.7G

zstd compression level/time/size
19 3m4s 1.7G
10 2m44s 1.8G

2. For the basic SDK (bitbake commands are similar), xz's better (but slower) compression does show clearly, but zstd's time wins are similar :

xz time/size
5m20s 581M

zstd compression level/time/size:
19 4m5s 837M
15 3m44s 979M
10 3m22s 993M
default(3) 3m19s 1.1G

Alex

#4 lore raw
Quoted 5 lines · Alexander Kanavin
Quoted 3 lines · Richard Purdie

Can you quantify "more"?

Quoted 8 lines
Quoted 5 lines

I'd be interested to know if a higher compression helps and what the time/cost
tradeoff is too. Not sure I'm keep to have it increase in size. This likely increases the eSDK size a lot too?

The benefit is indeed not unquestionable, hence the RFC.

Thanks, I think we need data to be able to comment effectively! :)

Quoted 7 lines

1. The eSDK is mostly made of already compressed sstate artifacts, so the effect on size is minimal and the zstd's time benefit is clear:

$ bitbake -c populate_sdk_ext core-image-sato (to have everything ready)
$ bitbake -c cleansstate core-image-sato buildtools-tarball
$ time bitbake -c populate_sdk_ext core-image-sato
Quoted 6 lines

xz time/size
4m46s 1.7G

zstd compression level/time/size
19 3m4s 1.7G
10 2m44s 1.8G

For the eSDK this looks like an ok change.

Quoted 11 lines

2. For the basic SDK (bitbake commands are similar), xz's better (but slower) compression does show clearly, but zstd's time wins are similar :

xz time/size
5m20s 581M

zstd compression level/time/size:
19 4m5s 837M
15 3m44s 979M
10 3m22s 993M
default(3) 3m19s 1.1G

but nearly doubling the size of the standard SDK isn't going to be well received by people using it, those numbers don't look like they can work even if it is faster. The key things are primarily size and secondly decompression time, compression time is in third place and unfortunate but something the build systems can afford to pay.

Cheers,

Richard

#5 lore raw
In reply to Alexander Kanavin
Quoted 32 lines · Alexander Kanavin
Quoted 9 lines · Richard Purdie

Can you quantify "more"?

I'd be interested to know if a higher compression helps and what the time/cost
tradeoff is too. Not sure I'm keep to have it increase in size. This likely
increases the eSDK size a lot too?

The benefit is indeed not unquestionable, hence the RFC.

1. The eSDK is mostly made of already compressed sstate artifacts, so the effect on size is minimal and the zstd's time benefit is clear:

$ bitbake -c populate_sdk_ext core-image-sato (to have everything ready)
$ bitbake -c cleansstate core-image-sato buildtools-tarball
$ time bitbake -c populate_sdk_ext core-image-sato

xz time/size
4m46s 1.7G

zstd compression level/time/size
19 3m4s 1.7G
10 2m44s 1.8G

2. For the basic SDK (bitbake commands are similar), xz's better (but slower) compression does show clearly, but zstd's time wins are similar :

Quoted 9 lines

xz time/size
5m20s 581M

zstd compression level/time/size:
19 4m5s 837M
15 3m44s 979M
10 3m22s 993M
default(3) 3m19s 1.1G

Is this the maximum compression level for zstd? Also, are any of these using parallel compression?

Quoted 13 lines

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#157009):
https://lists.openembedded.org/g/openembedded-core/message/157009
Mute This Topic: https://lists.openembedded.org/mt/86353954/3616693
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#6 lore raw
Quoted 4 lines · Joshua Watt

Is this the maximum compression level for zstd? Also, are any of these using parallel compression?

Yes, they all run with the same standard cpu cores setting from bitbake.conf. xz additionally has a RAM cap.

zstd has further compression levels, but I didn't try them - you're weclome to:
--ultra : enable levels beyond 19, up to 22 (requires more memory)

Alex