classes/populate_sdk_base: Implement xz compression options
PATCH 1/1 classes/populate_sdk_base: Implement xz compression options
Building an SDK on a machine with 8GB RAM resulted in excessive swapping due to the xz compressor using ~20GB of memory. This is because xz is being called with "-T 0 -9".
To allow tuning the compression versus memory usage, introduce a variable named SDK_XZ_OPTIONS that defaults to a more sane default:SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}" Thus, inherit any XZ tuning already done, and allow users to specify overrides for this task in their config by supplying SDK_XZ_OPTIONS. Since XZ_COMPRESSION_LEVEL defaults to -9 this does not change the standard behavior.
--- meta/classes/populate_sdk_base.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes/populate_sdk_base.bbclass +2 -1
@@ -48,6 +48,7 @@ TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" # Default archived SDK's suffix SDK_ARCHIVE_TYPE ?= "tar.xz" +SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}" # To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip and tar.xz python () {
@@ -58,7 +59,7 @@ python () { d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .') else: d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native') - d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz -T 0 -9 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}') + d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}') } SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
Signature · 1 line
2.17.1
Quoted 4 lines · Mike Looijmans
Building an SDK on a machine with 8GB RAM resulted in excessive swapping due to the xz compressor using ~20GB of memory. This is because xz is being called with "-T 0 -9".
Quoted 9 lines
To allow tuning the compression versus memory usage, introduce a variable named
SDK_XZ_OPTIONSthat defaults to a more sane default:SDK_XZ_OPTIONS?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}" Thus, inherit any XZ tuning already done, and allow users to specify overrides for this task in their config by supplyingSDK_XZ_OPTIONS. SinceXZ_COMPRESSION_LEVELdefaults to -9 this does not change the standard behavior.
...
+SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}"
A problem is that XZ_COMPRESSION_LEVEL compression would now be used for unrelated usecases, and lowering the compression for being able to boot on low-end targets (<= 64 MB RAM) shouldn't impact the SDK.
meta/classes/populate_sdk_base.bbclass python () { · line 62
Quoted 4 lines
...
meta/classes/populate_sdk_base.bbclass +1
-1- d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz -T 0 -9 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}') + d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
Replacing "-T 0" with ${XZ_DEFAULTS} should solve your problem. This already takes both cpus and memory into account.
cu
Adrian
Quoted 15 lines · Adrian Bunk
Quoted 4 lines · Mike Looijmans
Building an SDK on a machine with 8GB RAM resulted in excessive swapping due to the xz compressor using ~20GB of memory. This is because xz is being called with "-T 0 -9".
Quoted 9 lines
To allow tuning the compression versus memory usage, introduce a variable named
SDK_XZ_OPTIONSthat defaults to a more sane default:SDK_XZ_OPTIONS?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}" Thus, inherit any XZ tuning already done, and allow users to specify overrides for this task in their config by supplyingSDK_XZ_OPTIONS. SinceXZ_COMPRESSION_LEVELdefaults to -9 this does not change the standard behavior.
...
+SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${XZ_COMPRESSION_LEVEL}"
Quoted 3 lines
A problem is that
XZ_COMPRESSION_LEVELcompression would now be used for unrelated usecases, and lowering the compression for being able to boot on low-end targets (<= 64 MB RAM) shouldn't impact the SDK.
Clear. I wasn't aware that this was for the targets...
Quoted 8 lines
meta/classes/populate_sdk_base.bbclass python () { · line 62
Quoted 5 lines
...
meta/classes/populate_sdk_base.bbclass +1
-1- d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz -T 0 -9 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}') + d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')Replacing "-T 0" with ${XZ_DEFAULTS} should solve your problem. This already takes both cpus and memory into account.
I'd still want to have control over that -9 though.
It's not worth it to spend 15 minutes compressing just a few megabytes better in a development environment.
Signature · 1 line
Mike Looijmans