Fix xz compression command and optimize compression time

Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.

-T threads, --threads=threads
Specify the number of worker threads to use. Setting threads to a special value 0 makes xz use as many threads as there are CPU cores on the system. The actual number of threads can be less than threads if the input file is not big enough for threading with the given settings or if using more threads would exceed the memory usage limit.

-------------------

Memory: 7.8 GiB
Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4

File to compress: 1.9G

xz -f -k -e -9 --check=crc32 (current configuration)
Memory: 673Mb

real    6m37.170s
user    6m35.389s
sys     0m0.884s

Compressed file size: 3.4Mb

xz -f -k -e -9 -T 4 --check=crc32
Memory: Cannot allocate memory

xz -f -k -e -8 -T 4 --check=crc32
Memory: 1.8G

real    3m24.462s
user    12m8.502s
sys     0m2.180s

Compressed file size: 3.4Mb

xz -f -k -T 4 --check=crc32 (-e defaults to -6)
Memory: 471Mb

real    1m9.265s
user    4m8.972s
sys     0m0.944s

Compressed file size: 3.4Mb

So my conclusion would be to use the default -e -6 with -T 0.

The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:

upstream_tracking: update boost (2012-07-09 17:21:38 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ag/xz
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz

Andrei Gherzan (3):
image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
data to file
image_types.bbclass: Add XZ variable to set number of threads to be
used while compressing
image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6

meta/classes/image_types.bbclass | 5 +++--

1 file changed, 3 insertions(+), 2 deletions(-)

Signature · 1 line

1.7.9.5

Any volunteers to test this on a system with >4 real cores?

Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven:

Quoted 59 lines

Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.

-T threads, --threads=threads
Specify the number of worker threads to use. Setting threads to a special value 0 makes xz use as many threads as there are CPU cores on the system. The actual number of threads can be less than threads if the input file is not big enough for threading with the given settings or if using more threads would exceed the memory usage limit.

-------------------

Memory: 7.8 GiB
Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4

File to compress: 1.9G

xz -f -k -e -9 --check=crc32 (current configuration)
Memory: 673Mb

real    6m37.170s
user    6m35.389s
sys     0m0.884s

Compressed file size: 3.4Mb

xz -f -k -e -9 -T 4 --check=crc32
Memory: Cannot allocate memory

xz -f -k -e -8 -T 4 --check=crc32
Memory: 1.8G

real    3m24.462s
user    12m8.502s
sys     0m2.180s

Compressed file size: 3.4Mb

xz -f -k -T 4 --check=crc32 (-e defaults to -6)
Memory: 471Mb

real    1m9.265s
user    4m8.972s
sys     0m0.944s

Compressed file size: 3.4Mb

So my conclusion would be to use the default -e -6 with -T 0.

The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:

upstream_tracking: update boost (2012-07-09 17:21:38 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ag/xz
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz

Andrei Gherzan (3):
image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
data to file
image_types.bbclass: Add XZ variable to set number of threads to be
used while compressing
image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6

Quoted 6 lines

meta/classes/image_types.bbclass | 5 +++--

1 file changed, 3 insertions(+), 2 deletions(-)

--
1.7.9.5

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Any volunteers to test this on a system with >4 real cores?

Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven:

Quoted 59 lines

Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.

-T threads, --threads=threads
Specify the number of worker threads to use. Setting threads to a special value 0 makes xz use as many threads as there are CPU cores on the system. The actual number of threads can be less than threads if the input file is not big enough for threading with the given settings or if using more threads would exceed the memory usage limit.

-------------------

Memory: 7.8 GiB
Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4

File to compress: 1.9G

xz -f -k -e -9 --check=crc32 (current configuration)
Memory: 673Mb

real    6m37.170s
user    6m35.389s
sys     0m0.884s

Compressed file size: 3.4Mb

xz -f -k -e -9 -T 4 --check=crc32
Memory: Cannot allocate memory

xz -f -k -e -8 -T 4 --check=crc32
Memory: 1.8G

real    3m24.462s
user    12m8.502s
sys     0m2.180s

Compressed file size: 3.4Mb

xz -f -k -T 4 --check=crc32 (-e defaults to -6)
Memory: 471Mb

real    1m9.265s
user    4m8.972s
sys     0m0.944s

Compressed file size: 3.4Mb

So my conclusion would be to use the default -e -6 with -T 0.

The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:

upstream_tracking: update boost (2012-07-09 17:21:38 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ag/xz
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz

Andrei Gherzan (3):
image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
data to file
image_types.bbclass: Add XZ variable to set number of threads to be
used while compressing
image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6

Quoted 6 lines

meta/classes/image_types.bbclass | 5 +++--

1 file changed, 3 insertions(+), 2 deletions(-)

--
1.7.9.5

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Quoted 3 lines · Koen Kooi

Any volunteers to test this on a system with >4 real cores?

Koen,

Does OE-Core or Poky have an image setup for using .xz by default?

I will run it on a machine I have, just want to make sure I am doing the same tests that Andrei is running.

I was about to ask what msm asked about the Memory info.

Quoted 6 lines · Saul Wold

Quoted 3 lines · Koen Kooi

Any volunteers to test this on a system with >4 real cores?

Quoted 8 lines

Koen,

Does OE-Core or Poky have an image setup for using .xz by default?

I will run it on a machine I have, just want to make sure I am doing the same tests that Andrei is running.

I was about to ask what msm asked about the Memory info.

That field in results is not from "time". It's from gnome-system-monitor. Sorry for being a little confusing.

@g

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Quoted 2 lines

Any volunteers to test this on a system with >4 real cores?

Koen,

Does OE-Core or Poky have an image setup for using .xz by default?

I will run it on a machine I have, just want to make sure I am doing the same tests that Andrei is running.

I was about to ask what msm asked about the Memory info.

Sau!

Quoted 75 lines

Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven:

Quoted 59 lines

Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.

-T threads, --threads=threads
Specify the number of worker threads to use. Setting threads to a special value 0 makes xz use as many threads as there are CPU cores on the system. The actual number of threads can be less than threads if the input file is not big enough for threading with the given settings or if using more threads would exceed the memory usage limit.

-------------------

Memory: 7.8 GiB
Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4

File to compress: 1.9G

xz -f -k -e -9 --check=crc32 (current configuration)
Memory: 673Mb

real    6m37.170s
user    6m35.389s
sys     0m0.884s

Compressed file size: 3.4Mb

xz -f -k -e -9 -T 4 --check=crc32
Memory: Cannot allocate memory

xz -f -k -e -8 -T 4 --check=crc32
Memory: 1.8G

real    3m24.462s
user    12m8.502s
sys     0m2.180s

Compressed file size: 3.4Mb

xz -f -k -T 4 --check=crc32 (-e defaults to -6)
Memory: 471Mb

real    1m9.265s
user    4m8.972s
sys     0m0.944s

Compressed file size: 3.4Mb

So my conclusion would be to use the default -e -6 with -T 0.

The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:

upstream_tracking: update boost (2012-07-09 17:21:38 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ag/xz
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz

Andrei Gherzan (3):
image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
data to file
image_types.bbclass: Add XZ variable to set number of threads to be
used while compressing
image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6

Quoted 6 lines

meta/classes/image_types.bbclass | 5 +++--

1 file changed, 3 insertions(+), 2 deletions(-)

--
1.7.9.5

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven:

Quoted 6 lines · Koen Kooi

Quoted 2 lines

Any volunteers to test this on a system with >4 real cores?

Koen,

Quoted 1 line

Does OE-Core or Poky have an image setup for using .xz by default?

No, and as you can see from Andrei's patches, it would have never worked :)

regards,

Koen

Quoted 84 lines

I will run it on a machine I have, just want to make sure I am doing the same tests that Andrei is running.

I was about to ask what msm asked about the Memory info.

Sau!

Quoted 75 lines

Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven:

Quoted 59 lines

Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.

-T threads, --threads=threads
Specify the number of worker threads to use. Setting threads to a special value 0 makes xz use as many threads as there are CPU cores on the system. The actual number of threads can be less than threads if the input file is not big enough for threading with the given settings or if using more threads would exceed the memory usage limit.

-------------------

Memory: 7.8 GiB
Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4

File to compress: 1.9G

xz -f -k -e -9 --check=crc32 (current configuration)
Memory: 673Mb

real    6m37.170s
user    6m35.389s
sys     0m0.884s

Compressed file size: 3.4Mb

xz -f -k -e -9 -T 4 --check=crc32
Memory: Cannot allocate memory

xz -f -k -e -8 -T 4 --check=crc32
Memory: 1.8G

real    3m24.462s
user    12m8.502s
sys     0m2.180s

Compressed file size: 3.4Mb

xz -f -k -T 4 --check=crc32 (-e defaults to -6)
Memory: 471Mb

real    1m9.265s
user    4m8.972s
sys     0m0.944s

Compressed file size: 3.4Mb

So my conclusion would be to use the default -e -6 with -T 0.

The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:

upstream_tracking: update boost (2012-07-09 17:21:38 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ag/xz
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz

Andrei Gherzan (3):
image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
data to file
image_types.bbclass: Add XZ variable to set number of threads to be
used while compressing
image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6

Quoted 6 lines

meta/classes/image_types.bbclass | 5 +++--

1 file changed, 3 insertions(+), 2 deletions(-)

--
1.7.9.5

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Quoted 4 lines · Koen Kooi

Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven:

Quoted 1 line · Koen Kooi

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

Any volunteers to test this on a system with >4 real cores?

Koen,

Does OE-Core or Poky have an image setup for using .xz by default?

No, and as you can see from Andrei's patches, it would have never worked

:)

Quoted 5 lines

regards,

Koen

This leads to a similar remark for lzma:

The xz/lzma have been added with commit 38334ac and at that time both where using XZ_COMPRESSION_LEVEL.

The code has been refactored afterwards and what happened is that lzma now defaults to compression 7 (the default) while xz is way too high (default is 6).

There is a well hidden pitfall with that, being that any image, including the initramfs, will use that compression factor.

As you can notice in that (old) man xz [2] the memory requirements

are very high and what did happen to me was that the cpio.lzma could not be decompressed thus kernel could not boot (we override now the value setting "-e2" in our BSP).

Quoted 27 lines · Andrea Adami

Quoted 9 lines · Koen Kooi

Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven:

Quoted 6 lines · Koen Kooi

Quoted 2 lines

Any volunteers to test this on a system with >4 real cores?

Koen,

Quoted 9 lines

Quoted 1 line

Does OE-Core or Poky have an image setup for using .xz by default?

No, and as you can see from Andrei's patches, it would have never worked :)

regards,

Koen

This leads to a similar remark for lzma:

The xz/lzma have been added with commit 38334ac and at that time both where using XZ_COMPRESSION_LEVEL.

Quoted 4 lines

The code has been refactored afterwards and what happened is that lzma now defaults to compression 7 (the default) while xz is way too high (default is 6).

Nope. It's default value is -9.

Quoted 5 lines

There is a well hidden pitfall with that, being that any image, including the initramfs, will use that compression factor.

As you can notice in that (old) man xz [2] the memory requirements

What old version?

Quoted 5 lines

are very high and what did happen to me was that the cpio.lzma could not be decompressed thus kernel could not boot (we override now the value setting "-e2" in our BSP).

Without threads, memory demand was around 700mb. So it should be ok for most of machines.

Quoted 2 lines

Adding more threads will multiply that figures.

Indeed. With 8GB memory at my disposal i couldn't compress an image with -e -9 and -T 0 (-T 4)/

@g

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Well, in oe-core maybe.
Upstream they are more conservative, see that man.

"The default preset level in LZMA Utils is -7 while in XZ Utils it is -6, so both use 8 MiB dictionary by default."

Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.

-T threads, --threads=threads
Specify the number of worker threads to use. Setting threads to a special value 0 makes xz use as many threads as there are CPU cores on the system. The actual number of threads can be less than threads if the input file is not big enough for threading with the given settings or if using more threads would exceed the memory usage limit.

-------------------

Memory: 7.8 GiB
Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4

File to compress: 1.9G

xz -f -k -e -9 --check=crc32 (current configuration)
Memory: 673Mb

real    6m37.170s
user    6m35.389s
sys     0m0.884s

Compressed file size: 3.4Mb

xz -f -k -e -9 -T 4 --check=crc32
Memory: Cannot allocate memory

xz -f -k -e -8 -T 4 --check=crc32
Memory: 1.8G

real    3m24.462s
user    12m8.502s
sys     0m2.180s

Compressed file size: 3.4Mb

xz -f -k -T 4 --check=crc32 (-e defaults to -6)
Memory: 471Mb

real    1m9.265s
user    4m8.972s
sys     0m0.944s

Compressed file size: 3.4Mb

So my conclusion would be to use the default -e -6 with -T 0.

The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:

   upstream_tracking: update boost (2012-07-09 17:21:38 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ag/xz
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz

Andrei Gherzan (3):
image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
data to file
image_types.bbclass: Add XZ variable to set number of threads to be
used while compressing
image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6

  meta/classes/image_types.bbclass |    5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)