Set XZ_COMPRESSION_LEVEL to -9
PATCH 1/1 Set XZ_COMPRESSION_LEVEL to -9
It is consistent with other compressors also using their best compression.
xz is currently the option that offers best compression, using settings even lower than the default is surprising and makes it harder to choose the best available compression.
For anyone who cares about compression time using a different option like gzip or bzip2 is the logical choice.
For decompression, better compressed is actually faster to decompress.
--- meta/classes/image_types.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types.bbclass +1 -1
@@ -54,7 +54,7 @@ def imagetypes_getdepends(d): # Sort the set so that ordering is consistant return " ".join(sorted(deps)) -XZ_COMPRESSION_LEVEL ?= "-3" +XZ_COMPRESSION_LEVEL ?= "-9" XZ_INTEGRITY_CHECK ?= "crc32" ZIP_COMPRESSION_LEVEL ?= "-9"
Signature · 1 line
2.17.1
Quoted 10 lines · Adrian Bunk
It is consistent with other compressors also using their best compression.
xz is currently the option that offers best compression, using settings even lower than the default is surprising and makes it harder to choose the best available compression.
For anyone who cares about compression time using a different option like gzip or bzip2 is the logical choice.
The concern with xz's highest compression level is memory usage rather than CPU time. Quoting from the manpage:
Preset DictSize CompCPU CompMem DecMem
-0 256 KiB 0 3 MiB 1 MiB
-1 1 MiB 1 9 MiB 2 MiB
-2 2 MiB 2 17 MiB 3 MiB
-3 4 MiB 3 32 MiB 5 MiB
-4 4 MiB 4 48 MiB 5 MiB
-5 8 MiB 5 94 MiB 9 MiB
-6 8 MiB 6 94 MiB 9 MiB
-7 16 MiB 6 186 MiB 17 MiB
-8 32 MiB 6 370 MiB 33 MiB
-9 64 MiB 6 674 MiB 65 MiBGiven that these are per-thread the memory requirements may start to get a little unreasonable with xz -9. We do limit usage to 50% of physical memory via XZ_DEFAULTS though so maybe it's not a big concern?
Either way, perhaps now would be a good time to remove XZ_COMPRESSION_LEVEL from image_types.bbclass and instead just include the compression level option in XZ_DEFAULTS so that it gets used consistently everywhere.
Quoted 29 lines
For decompression, better compressed is actually faster to decompress.
Signed-off-by Adrian Bunk <[email protected]>--- meta/classes/image_types.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/classes/image_types.bbclass +1
-2@@ -54,7 +54,7 @@ def imagetypes_getdepends(d): # Sort the set so that ordering is consistant return " ".join(sorted(deps)) -XZ_COMPRESSION_LEVEL ?= "-3" +XZ_COMPRESSION_LEVEL ?= "-9" XZ_INTEGRITY_CHECK ?= "crc32" ZIP_COMPRESSION_LEVEL ?= "-9" --2.17.1
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 27 lines · Andre McCurdy
Quoted 10 lines · Adrian Bunk
It is consistent with other compressors also using their best compression.
xz is currently the option that offers best compression, using settings even lower than the default is surprising and makes it harder to choose the best available compression.
For anyone who cares about compression time using a different option like gzip or bzip2 is the logical choice.
The concern with xz's highest compression level is memory usage rather than CPU time. Quoting from the manpage:
Preset DictSize CompCPU CompMem DecMem
-0 256 KiB 0 3 MiB 1 MiB -1 1 MiB 1 9 MiB 2 MiB -2 2 MiB 2 17 MiB 3 MiB -3 4 MiB 3 32 MiB 5 MiB -4 4 MiB 4 48 MiB 5 MiB -5 8 MiB 5 94 MiB 9 MiB -6 8 MiB 6 94 MiB 9 MiB -7 16 MiB 6 186 MiB 17 MiB -8 32 MiB 6 370 MiB 33 MiB -9 64 MiB 6 674 MiB 65 MiB
Quoted 4 lines
Given that these are per-thread the memory requirements may start to get a little unreasonable with xz -9. We do limit usage to 50% of physical memory via
XZ_DEFAULTSthough so maybe it's not a big concern?
Due to the limit, and also due to the fact that build machines with very low RAM/core are likely to have bigger problems earlier:
The compile task of a recipe spawns the same number of gcc processes, and there can be several compile tasks running.
gcc memory usage when compiling C++ code like webkit can be worse than these per-thread numbers.
Quoted 4 lines
Either way, perhaps now would be a good time to remove
XZ_COMPRESSION_LEVELfrom image_types.bbclass and instead just include the compression level option inXZ_DEFAULTSso that it gets used consistently everywhere.
OPKGBUILDCMD uses XZ_DEFAULTS, and the decompression memory requirement on the target for -9 might be a problem if someone uses package feeds on low-memory targets.
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 8 lines · Adrian Bunk
It is consistent with other compressors also using their best compression.
xz is currently the option that offers best compression, using settings even lower than the default is surprising and makes it harder to choose the best available compression.
Quoted 5 lines
For anyone who cares about compression time using a different option like gzip or bzip2 is the logical choice.
For decompression, better compressed is actually faster to decompress.
I worry about resources we need to spend when creating these archives. Can you post some comparisons
along to assess the performance.
Quoted 27 lines
Signed-off-by Adrian Bunk <[email protected]>--- meta/classes/image_types.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/classes/image_types.bbclass +1
-2@@ -54,7 +54,7 @@ def imagetypes_getdepends(d): # Sort the set so that ordering is consistant return " ".join(sorted(deps)) -XZ_COMPRESSION_LEVEL ?= "-3" +XZ_COMPRESSION_LEVEL ?= "-9" XZ_INTEGRITY_CHECK ?= "crc32" ZIP_COMPRESSION_LEVEL ?= "-9" --2.17.1
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 12 lines · Khem Raj
Quoted 11 lines · Adrian Bunk
It is consistent with other compressors also using their best compression.
xz is currently the option that offers best compression, using settings even lower than the default is surprising and makes it harder to choose the best available compression.
For anyone who cares about compression time using a different option like gzip or bzip2 is the logical choice.
Quoted 6 lines
Quoted 2 lines
For decompression, better compressed is actually faster to decompress.
I worry about resources we need to spend when creating these archives. Can you post some comparisons
along to assess the performance.
On my laptop single-threaded compression of an image I am frequently building takes 20s with -3 and 55s with -9.
Building the whole image from scratch takes more than 2 hours.
If you care about resources spent, I remember it saved me around 10 minutes build time when I changed OPKGBUILDCMD from the default "xz -6" to "gzip -1" - for the common case where package feeds are not used there is no point in spending time on compression and decompression.
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed