Skip to contentSkip to conversation

Set XZ_COMPRESSION_LEVEL to -9

PATCH In review Adrian Bunk 3 participants 5 messages Apr 12, 2019 – Apr 13, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Set XZ_COMPRESSION_LEVEL to -9

It is consistent with other compressors also using their best compression.

xz is currently the option that offers best compression, using settings even lower than the default is surprising and makes it harder to choose the best available compression.

For anyone who cares about compression time using a different option like gzip or bzip2 is the logical choice.

For decompression, better compressed is actually faster to decompress.

---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types.bbclass +1 -1
index 95aa1674cf..1c44ec4a80 100644
@@ -54,7 +54,7 @@ def imagetypes_getdepends(d):
     # Sort the set so that ordering is consistant
     return " ".join(sorted(deps))
 
-XZ_COMPRESSION_LEVEL ?= "-3"
+XZ_COMPRESSION_LEVEL ?= "-9"
 XZ_INTEGRITY_CHECK ?= "crc32"
 
 ZIP_COMPRESSION_LEVEL ?= "-9"
Signature · 1 line
2.17.1
#1 lore raw
Quoted 10 lines · Adrian Bunk

It is consistent with other compressors also using their best compression.

xz is currently the option that offers best compression, using settings even lower than the default is surprising and makes it harder to choose the best available compression.

For anyone who cares about compression time using a different option like gzip or bzip2 is the logical choice.

The concern with xz's highest compression level is memory usage rather than CPU time. Quoting from the manpage:

Preset DictSize CompCPU CompMem DecMem

    -0     256 KiB     0      3 MiB    1 MiB
    -1       1 MiB     1      9 MiB    2 MiB
    -2       2 MiB     2     17 MiB    3 MiB
    -3       4 MiB     3     32 MiB    5 MiB
    -4       4 MiB     4     48 MiB    5 MiB
    -5       8 MiB     5     94 MiB    9 MiB
    -6       8 MiB     6     94 MiB    9 MiB
    -7      16 MiB     6    186 MiB   17 MiB
    -8      32 MiB     6    370 MiB   33 MiB
    -9      64 MiB     6    674 MiB   65 MiB

Given that these are per-thread the memory requirements may start to get a little unreasonable with xz -9. We do limit usage to 50% of physical memory via XZ_DEFAULTS though so maybe it's not a big concern?

Either way, perhaps now would be a good time to remove XZ_COMPRESSION_LEVEL from image_types.bbclass and instead just include the compression level option in XZ_DEFAULTS so that it gets used consistently everywhere.

Quoted 29 lines

For decompression, better compressed is actually faster to decompress.

---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types.bbclass +1 -2
index 95aa1674cf..1c44ec4a80 100644
@@ -54,7 +54,7 @@ def imagetypes_getdepends(d):
     # Sort the set so that ordering is consistant
     return " ".join(sorted(deps))
 
-XZ_COMPRESSION_LEVEL ?= "-3"
+XZ_COMPRESSION_LEVEL ?= "-9"
 XZ_INTEGRITY_CHECK ?= "crc32"
 
 ZIP_COMPRESSION_LEVEL ?= "-9"
--

2.17.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#2 lore raw
Quoted 27 lines · Andre McCurdy
Quoted 10 lines · Adrian Bunk

It is consistent with other compressors also using their best compression.

xz is currently the option that offers best compression, using settings even lower than the default is surprising and makes it harder to choose the best available compression.

For anyone who cares about compression time using a different option like gzip or bzip2 is the logical choice.

The concern with xz's highest compression level is memory usage rather than CPU time. Quoting from the manpage:

Preset DictSize CompCPU CompMem DecMem

    -0     256 KiB     0      3 MiB    1 MiB
    -1       1 MiB     1      9 MiB    2 MiB
    -2       2 MiB     2     17 MiB    3 MiB
    -3       4 MiB     3     32 MiB    5 MiB
    -4       4 MiB     4     48 MiB    5 MiB
    -5       8 MiB     5     94 MiB    9 MiB
    -6       8 MiB     6     94 MiB    9 MiB
    -7      16 MiB     6    186 MiB   17 MiB
    -8      32 MiB     6    370 MiB   33 MiB
    -9      64 MiB     6    674 MiB   65 MiB
Quoted 4 lines

Given that these are per-thread the memory requirements may start to get a little unreasonable with xz -9. We do limit usage to 50% of physical memory via XZ_DEFAULTS though so maybe it's not a big concern?

Due to the limit, and also due to the fact that build machines with very low RAM/core are likely to have bigger problems earlier:

The compile task of a recipe spawns the same number of gcc processes, and there can be several compile tasks running.

gcc memory usage when compiling C++ code like webkit can be worse than these per-thread numbers.

Quoted 4 lines

Either way, perhaps now would be a good time to remove XZ_COMPRESSION_LEVEL from image_types.bbclass and instead just include the compression level option in XZ_DEFAULTS so that it gets used consistently everywhere.

OPKGBUILDCMD uses XZ_DEFAULTS, and the decompression memory requirement on the target for -9 might be a problem if someone uses package feeds on low-memory targets.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#3 lore raw
In reply to Adrian Bunk
Quoted 8 lines · Adrian Bunk

It is consistent with other compressors also using their best compression.

xz is currently the option that offers best compression, using settings even lower than the default is surprising and makes it harder to choose the best available compression.

Quoted 5 lines

For anyone who cares about compression time using a different option like gzip or bzip2 is the logical choice.

For decompression, better compressed is actually faster to decompress.

I worry about resources we need to spend when creating these archives. Can you post some comparisons
along to assess the performance.

Quoted 27 lines
---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types.bbclass +1 -2
index 95aa1674cf..1c44ec4a80 100644
@@ -54,7 +54,7 @@ def imagetypes_getdepends(d):
     # Sort the set so that ordering is consistant
     return " ".join(sorted(deps))
 
-XZ_COMPRESSION_LEVEL ?= "-3"
+XZ_COMPRESSION_LEVEL ?= "-9"
 XZ_INTEGRITY_CHECK ?= "crc32"
 
 ZIP_COMPRESSION_LEVEL ?= "-9"
--

2.17.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#4 lore raw
Quoted 12 lines · Khem Raj
Quoted 11 lines · Adrian Bunk

It is consistent with other compressors also using their best compression.

xz is currently the option that offers best compression, using settings even lower than the default is surprising and makes it harder to choose the best available compression.

For anyone who cares about compression time using a different option like gzip or bzip2 is the logical choice.

Quoted 6 lines
Quoted 2 lines

For decompression, better compressed is actually faster to decompress.

I worry about resources we need to spend when creating these archives. Can you post some comparisons
along to assess the performance.

On my laptop single-threaded compression of an image I am frequently building takes 20s with -3 and 55s with -9.

Building the whole image from scratch takes more than 2 hours.

If you care about resources spent, I remember it saved me around 10 minutes build time when I changed OPKGBUILDCMD from the default "xz -6" to "gzip -1" - for the common case where package feeds are not used there is no point in spending time on compression and decompression.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed