Skip to contentSkip to conversation

classes/populate_sdk_base: Implement xz compression options

PATCH Superseded Mike Looijmans 2 participants 3 messages Mar 18, 2020 – Mar 20, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 classes/populate_sdk_base: Implement xz compression options

Building an SDK on a machine with 8GB RAM resulted in excessive swapping due to the xz compressor using ~20GB of memory. This is because xz is being called with "-T 0 -9".

To allow tuning the compression versus memory usage, introduce a variable named SDK_XZ_OPTIONS that defaults to a more sane default:
SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}" Thus, inherit any XZ tuning already done, and allow users to specify overrides for this task in their config by supplying SDK_XZ_OPTIONS. Since XZ_COMPRESSION_LEVEL defaults to -9 this does not change the standard behavior.

---
 meta/classes/populate_sdk_base.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes/populate_sdk_base.bbclass +2 -1
index d03465b6fc..13247e42e5 100644
@@ -48,6 +48,7 @@ TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
 
 # Default archived SDK's suffix
 SDK_ARCHIVE_TYPE ?= "tar.xz"
+SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}"
 
 # To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip and tar.xz
 python () {
@@ -58,7 +59,7 @@ python () {
        d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .')
     else:
        d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native')
-       d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz -T 0 -9 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
+       d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
 }
 
 SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
Signature · 1 line
2.17.1
#1 lore raw
Quoted 4 lines · Mike Looijmans

Building an SDK on a machine with 8GB RAM resulted in excessive swapping due to the xz compressor using ~20GB of memory. This is because xz is being called with "-T 0 -9".

Quoted 9 lines

To allow tuning the compression versus memory usage, introduce a variable named SDK_XZ_OPTIONS that defaults to a more sane default:
SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}" Thus, inherit any XZ tuning already done, and allow users to specify overrides for this task in their config by supplying SDK_XZ_OPTIONS. Since XZ_COMPRESSION_LEVEL defaults to -9 this does not change the standard behavior.
...
+SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}"

A problem is that XZ_COMPRESSION_LEVEL compression would now be used for unrelated usecases, and lowering the compression for being able to boot on low-end targets (<= 64 MB RAM) shouldn't impact the SDK.

meta/classes/populate_sdk_base.bbclass python () { · line 62

Quoted 4 lines

...

meta/classes/populate_sdk_base.bbclass +1 -1
-       d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz -T 0 -9 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
+       d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
...

Replacing "-T 0" with ${XZ_DEFAULTS} should solve your problem. This already takes both cpus and memory into account.

cu
Adrian

#2 lore raw
Quoted 15 lines · Adrian Bunk
Quoted 4 lines · Mike Looijmans

Building an SDK on a machine with 8GB RAM resulted in excessive swapping due to the xz compressor using ~20GB of memory. This is because xz is being called with "-T 0 -9".

Quoted 9 lines

To allow tuning the compression versus memory usage, introduce a variable named SDK_XZ_OPTIONS that defaults to a more sane default:
SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}" Thus, inherit any XZ tuning already done, and allow users to specify overrides for this task in their config by supplying SDK_XZ_OPTIONS. Since XZ_COMPRESSION_LEVEL defaults to -9 this does not change the standard behavior.
...
+SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}"

Quoted 3 lines

A problem is that XZ_COMPRESSION_LEVEL compression would now be used for unrelated usecases, and lowering the compression for being able to boot on low-end targets (<= 64 MB RAM) shouldn't impact the SDK.

Clear. I wasn't aware that this was for the targets...

Quoted 8 lines

meta/classes/populate_sdk_base.bbclass python () { · line 62

Quoted 5 lines

...

meta/classes/populate_sdk_base.bbclass +1 -1
-       d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz -T 0 -9 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
+       d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
...

Replacing "-T 0" with ${XZ_DEFAULTS} should solve your problem. This already takes both cpus and memory into account.

I'd still want to have control over that -9 though.

It's not worth it to spend 15 minutes compressing just a few megabytes better in a development environment.

Signature · 1 line
Mike Looijmans