Skip to contentSkip to conversation

lib: enable lzma decompression support for SPL build

PATCH New Weijie Gao 3 participants 6 messages Jan 8, 2020 – Jan 10, 2020 Library code

lore
Share & tools
lore

Weijie Gao

author
#0 lore raw

PATCH 11/16 lib: enable lzma decompression support for SPL build

This patch enables LZMA decompression support for SPL build

---
 lib/Kconfig  | 5 +++++
 lib/Makefile | 1 +
 2 files changed, 6 insertions(+)
lib/Kconfig +5 -0
index 965cf7bc03..2f38e37802 100644
@@ -418,6 +418,11 @@ config SPL_LZ4
 	  fast compression and decompression speed. It belongs to the LZ77
 	  family of byte-oriented compression schemes.
 
+config SPL_LZMA
+	bool "Enable LZMA decompression support for SPL build"
+	help
+	  This enables support for LZMA compression altorithm for SPL boot.
+
 config SPL_LZO
 	bool "Enable LZO decompression support in SPL"
 	help
lib/Makefile +1 -0
index 1fb650cd90..8e51eec01c 100644
@@ -64,6 +64,7 @@ obj-$(CONFIG_$(SPL_)ZLIB) += zlib/
 obj-$(CONFIG_$(SPL_)ZSTD) += zstd/
 obj-$(CONFIG_$(SPL_)GZIP) += gunzip.o
 obj-$(CONFIG_$(SPL_)LZO) += lzo/
+obj-$(CONFIG_$(SPL_)LZMA) += lzma/
 obj-$(CONFIG_$(SPL_)LZ4) += lz4_wrapper.o
 
 obj-$(CONFIG_LIBAVB) += libavb/
Signature · 1 line
2.17.1
#1 lore raw

PATCH 11/16 lib: enable lzma decompression support for SPL build

Quoted 13 lines · Weijie Gao

This patch enables LZMA decompression support for SPL build

---
  lib/Kconfig  | 5 +++++
  lib/Makefile | 1 +
  2 files changed, 6 insertions(+)
lib/Kconfig +0 -0
index 965cf7bc03..2f38e37802 100644

lib/Kconfig config SPL_LZ4 · line 424

Quoted 7 lines
lib/Kconfig +4 -0
@@ -418,6 +418,11 @@ config SPL_LZ4
  	  fast compression and decompression speed. It belongs to the LZ77
  	  family of byte-oriented compression schemes.
 
+config SPL_LZMA
+	bool "Enable LZMA decompression support for SPL build"
+	help
+	  This enables support for LZMA compression altorithm for SPL boot.

%s/altorithm/algorithm/g

Why do we need yet another algorithm for compression?

Doesn't Zstandard provide a much faster decompression? Cf. https://sites.google.com/site/powturbo/home/web-compression

Best regards

Heinrich

Quoted 17 lines
lib/Kconfig +1 -0
+
  config SPL_LZO
  	bool "Enable LZO decompression support in SPL"
  	help
lib/Makefile +1 -0
index 1fb650cd90..8e51eec01c 100644
@@ -64,6 +64,7 @@ obj-$(CONFIG_$(SPL_)ZLIB) += zlib/
  obj-$(CONFIG_$(SPL_)ZSTD) += zstd/
  obj-$(CONFIG_$(SPL_)GZIP) += gunzip.o
  obj-$(CONFIG_$(SPL_)LZO) += lzo/
+obj-$(CONFIG_$(SPL_)LZMA) += lzma/
  obj-$(CONFIG_$(SPL_)LZ4) += lz4_wrapper.o
 
  obj-$(CONFIG_LIBAVB) += libavb/
author
#2 lore raw

PATCH 11/16 lib: enable lzma decompression support for SPL build

Quoted 49 lines · Heinrich Schuchardt
Quoted 13 lines · Weijie Gao

This patch enables LZMA decompression support for SPL build

---
  lib/Kconfig  | 5 +++++
  lib/Makefile | 1 +
  2 files changed, 6 insertions(+)
lib/Kconfig +0 -0
index 965cf7bc03..2f38e37802 100644

lib/Kconfig config SPL_LZ4 · line 424

Quoted 7 lines
lib/Kconfig +4 -0
@@ -418,6 +418,11 @@ config SPL_LZ4
  	  fast compression and decompression speed. It belongs to the LZ77
  	  family of byte-oriented compression schemes.
 
+config SPL_LZMA
+	bool "Enable LZMA decompression support for SPL build"
+	help
+	  This enables support for LZMA compression altorithm for SPL boot.

%s/altorithm/algorithm/g

Why do we need yet another algorithm for compression?

Doesn't Zstandard provide a much faster decompression? Cf. https://sites.google.com/site/powturbo/home/web-compression

Best regards

Heinrich

Quoted 15 lines
lib/Kconfig +1 -0
+
  config SPL_LZO
  	bool "Enable LZO decompression support in SPL"
  	help
lib/Makefile +1 -0
index 1fb650cd90..8e51eec01c 100644
@@ -64,6 +64,7 @@ obj-$(CONFIG_$(SPL_)ZLIB) += zlib/
  obj-$(CONFIG_$(SPL_)ZSTD) += zstd/
  obj-$(CONFIG_$(SPL_)GZIP) += gunzip.o
  obj-$(CONFIG_$(SPL_)LZO) += lzo/
+obj-$(CONFIG_$(SPL_)LZMA) += lzma/
  obj-$(CONFIG_$(SPL_)LZ4) += lz4_wrapper.o
 
Quoted 2 lines

lib/Makefile obj-$(CONFIG_$(SPL_)ZLIB) += zlib/ · line 70

Quoted 2 lines
lib/Makefile +0 -0
  obj-$(CONFIG_LIBAVB) += libavb/

Hi Heinrich,

I need an algorithm with the maximum compression ratio to reduce the size of the u-boot payload as much as possible.

I believe currently lzma and xz are the best choices. Refer to:
https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

Although they will take longer time for compression and decompression, considering that the u-boot payload is usually small (<= 1MiB), there will not be significant effect on decompression time.

Best Regards,

Weijie

#3 lore raw

PATCH 11/16 lib: enable lzma decompression support for SPL build

Quoted 58 lines · Weijie Gao
Quoted 49 lines · Heinrich Schuchardt
Quoted 13 lines · Weijie Gao

This patch enables LZMA decompression support for SPL build

---
  lib/Kconfig  | 5 +++++
  lib/Makefile | 1 +
  2 files changed, 6 insertions(+)
lib/Kconfig +0 -0
index 965cf7bc03..2f38e37802 100644

lib/Kconfig config SPL_LZ4 · line 424

Quoted 7 lines
lib/Kconfig +4 -0
@@ -418,6 +418,11 @@ config SPL_LZ4
  	  fast compression and decompression speed. It belongs to the LZ77
  	  family of byte-oriented compression schemes.
 
+config SPL_LZMA
+	bool "Enable LZMA decompression support for SPL build"
+	help
+	  This enables support for LZMA compression altorithm for SPL boot.

%s/altorithm/algorithm/g

Why do we need yet another algorithm for compression?

Doesn't Zstandard provide a much faster decompression? Cf. https://sites.google.com/site/powturbo/home/web-compression

Best regards

Heinrich

Quoted 15 lines
lib/Kconfig +1 -0
+
  config SPL_LZO
  	bool "Enable LZO decompression support in SPL"
  	help
lib/Makefile +1 -0
index 1fb650cd90..8e51eec01c 100644
@@ -64,6 +64,7 @@ obj-$(CONFIG_$(SPL_)ZLIB) += zlib/
  obj-$(CONFIG_$(SPL_)ZSTD) += zstd/
  obj-$(CONFIG_$(SPL_)GZIP) += gunzip.o
  obj-$(CONFIG_$(SPL_)LZO) += lzo/
+obj-$(CONFIG_$(SPL_)LZMA) += lzma/
  obj-$(CONFIG_$(SPL_)LZ4) += lz4_wrapper.o
 
Quoted 2 lines

lib/Makefile obj-$(CONFIG_$(SPL_)ZLIB) += zlib/ · line 70

Quoted 2 lines
lib/Makefile +0 -0
  obj-$(CONFIG_LIBAVB) += libavb/

Hi Heinrich,

I need an algorithm with the maximum compression ratio to reduce the size of the u-boot payload as much as possible.

Quoted 7 lines

I believe currently lzma and xz are the best choices. Refer to:
https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

Although they will take longer time for compression and decompression, considering that the u-boot payload is usually small (<= 1MiB), there will not be significant effect on decompression time.

Where does zstd fall in the list however? Can you please try that and report back? Thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200109/744b4b1c/attachment.sig>
author
#4 lore raw

PATCH 11/16 lib: enable lzma decompression support for SPL build

Quoted 63 lines · Tom Rini
Quoted 62 lines · Weijie Gao
Quoted 49 lines · Heinrich Schuchardt
Quoted 13 lines · Weijie Gao

This patch enables LZMA decompression support for SPL build

---
  lib/Kconfig  | 5 +++++
  lib/Makefile | 1 +
  2 files changed, 6 insertions(+)
lib/Kconfig +0 -0
index 965cf7bc03..2f38e37802 100644

lib/Kconfig config SPL_LZ4 · line 424

Quoted 7 lines
lib/Kconfig +4 -0
@@ -418,6 +418,11 @@ config SPL_LZ4
  	  fast compression and decompression speed. It belongs to the LZ77
  	  family of byte-oriented compression schemes.
 
+config SPL_LZMA
+	bool "Enable LZMA decompression support for SPL build"
+	help
+	  This enables support for LZMA compression altorithm for SPL boot.

%s/altorithm/algorithm/g

Why do we need yet another algorithm for compression?

Doesn't Zstandard provide a much faster decompression? Cf. https://sites.google.com/site/powturbo/home/web-compression

Best regards

Heinrich

Quoted 15 lines
lib/Kconfig +1 -0
+
  config SPL_LZO
  	bool "Enable LZO decompression support in SPL"
  	help
lib/Makefile +1 -0
index 1fb650cd90..8e51eec01c 100644
@@ -64,6 +64,7 @@ obj-$(CONFIG_$(SPL_)ZLIB) += zlib/
  obj-$(CONFIG_$(SPL_)ZSTD) += zstd/
  obj-$(CONFIG_$(SPL_)GZIP) += gunzip.o
  obj-$(CONFIG_$(SPL_)LZO) += lzo/
+obj-$(CONFIG_$(SPL_)LZMA) += lzma/
  obj-$(CONFIG_$(SPL_)LZ4) += lz4_wrapper.o
 
Quoted 2 lines

lib/Makefile obj-$(CONFIG_$(SPL_)ZLIB) += zlib/ · line 70

Quoted 2 lines
lib/Makefile +0 -0
  obj-$(CONFIG_LIBAVB) += libavb/

Hi Heinrich,

I need an algorithm with the maximum compression ratio to reduce the size of the u-boot payload as much as possible.

I believe currently lzma and xz are the best choices. Refer to:
https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

Quoted 7 lines
Quoted 3 lines

Although they will take longer time for compression and decompression, considering that the u-boot payload is usually small (<= 1MiB), there will not be significant effect on decompression time.

Where does zstd fall in the list however? Can you please try that and report back? Thanks!

My purpose is to reduce the size of u-boot binary, and my plan is to compress the payload and the SPL decompress it at runtime. I need smallest size here, not fastest decompression speed.

So there are two points must be taken into consideration:
the size of the decompressor (i.e. u-boot-spl.bin) and the size of the compressed payload (u-boot.bin.gz/bz2/...).

Below is my test results, including all compression algorithms supported by uImage, plus zstd:

Platform: mips - mt7628

Uncompressed size of u-boot.bin: 302736 (295KiB)

Commands for compression:
gzip -c -f -k -9
bzip2 -c -f -k -9
lzma -c -z -k -9
lzop -c -f -9
lz4 -f -9
zstd --ultra -f

Results:
algo. | spl.bin size | compressed size | % of uncomp | total size
------------------------------------------------------------------------
gzip | 26232 (25.6K) | 141207 (137KiB) | 46.64% | 167439 (163KiB)
------------------------------------------------------------------------
bzip2 | 30616 (29.8K) | 135403 (132KiB) | 44.72% | 166019 (162KiB)
------------------------------------------------------------------------
lzma | 18920 (18.4K) | 112660 (110KiB) | 37.21% | 131580 (128KiB)
------------------------------------------------------------------------
lzo | 13288 (12.9K) | 152409 (148KiB) | 50.34% | 165697 (161KiB)
------------------------------------------------------------------------
lz4 | 12408 (12.1K) | 167311 (163KiB) | 55.27% | 179719 (175KiB)
------------------------------------------------------------------------
zstd | 44824 (43.7K) | 149765 (146KiB) | 49.47% | 194589 (190KiB)

As you can see lzma has the best compression ratio. The is the reason I chose lzma here.

Best Regards

#5 lore raw

PATCH 11/16 lib: enable lzma decompression support for SPL build

Quoted 110 lines · Weijie Gao
Quoted 63 lines · Tom Rini
Quoted 62 lines · Weijie Gao
Quoted 49 lines · Heinrich Schuchardt
Quoted 13 lines · Weijie Gao

This patch enables LZMA decompression support for SPL build

---
  lib/Kconfig  | 5 +++++
  lib/Makefile | 1 +
  2 files changed, 6 insertions(+)
lib/Kconfig +0 -0
index 965cf7bc03..2f38e37802 100644

lib/Kconfig config SPL_LZ4 · line 424

Quoted 7 lines
lib/Kconfig +4 -0
@@ -418,6 +418,11 @@ config SPL_LZ4
  	  fast compression and decompression speed. It belongs to the LZ77
  	  family of byte-oriented compression schemes.
 
+config SPL_LZMA
+	bool "Enable LZMA decompression support for SPL build"
+	help
+	  This enables support for LZMA compression altorithm for SPL boot.

%s/altorithm/algorithm/g

Why do we need yet another algorithm for compression?

Doesn't Zstandard provide a much faster decompression? Cf. https://sites.google.com/site/powturbo/home/web-compression

Best regards

Heinrich

Quoted 15 lines
lib/Kconfig +1 -0
+
  config SPL_LZO
  	bool "Enable LZO decompression support in SPL"
  	help
lib/Makefile +1 -0
index 1fb650cd90..8e51eec01c 100644
@@ -64,6 +64,7 @@ obj-$(CONFIG_$(SPL_)ZLIB) += zlib/
  obj-$(CONFIG_$(SPL_)ZSTD) += zstd/
  obj-$(CONFIG_$(SPL_)GZIP) += gunzip.o
  obj-$(CONFIG_$(SPL_)LZO) += lzo/
+obj-$(CONFIG_$(SPL_)LZMA) += lzma/
  obj-$(CONFIG_$(SPL_)LZ4) += lz4_wrapper.o
 
Quoted 2 lines

lib/Makefile obj-$(CONFIG_$(SPL_)ZLIB) += zlib/ · line 70

Quoted 2 lines
lib/Makefile +0 -0
  obj-$(CONFIG_LIBAVB) += libavb/

Hi Heinrich,

I need an algorithm with the maximum compression ratio to reduce the size of the u-boot payload as much as possible.

I believe currently lzma and xz are the best choices. Refer to:
https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

Quoted 7 lines
Quoted 3 lines

Although they will take longer time for compression and decompression, considering that the u-boot payload is usually small (<= 1MiB), there will not be significant effect on decompression time.

Where does zstd fall in the list however? Can you please try that and report back? Thanks!

My purpose is to reduce the size of u-boot binary, and my plan is to compress the payload and the SPL decompress it at runtime. I need smallest size here, not fastest decompression speed.

So there are two points must be taken into consideration:
the size of the decompressor (i.e. u-boot-spl.bin) and the size of the compressed payload (u-boot.bin.gz/bz2/...).

Below is my test results, including all compression algorithms supported by uImage, plus zstd:

Platform: mips - mt7628

Uncompressed size of u-boot.bin: 302736 (295KiB)

Commands for compression:
gzip -c -f -k -9
bzip2 -c -f -k -9
lzma -c -z -k -9
lzop -c -f -9
lz4 -f -9
zstd --ultra -f

Results:
algo. | spl.bin size | compressed size | % of uncomp | total size
------------------------------------------------------------------------
gzip | 26232 (25.6K) | 141207 (137KiB) | 46.64% | 167439 (163KiB)
------------------------------------------------------------------------
bzip2 | 30616 (29.8K) | 135403 (132KiB) | 44.72% | 166019 (162KiB)
------------------------------------------------------------------------
lzma | 18920 (18.4K) | 112660 (110KiB) | 37.21% | 131580 (128KiB)
------------------------------------------------------------------------
lzo | 13288 (12.9K) | 152409 (148KiB) | 50.34% | 165697 (161KiB)
------------------------------------------------------------------------
lz4 | 12408 (12.1K) | 167311 (163KiB) | 55.27% | 179719 (175KiB)
------------------------------------------------------------------------
zstd | 44824 (43.7K) | 149765 (146KiB) | 49.47% | 194589 (190KiB)

Quoted 2 lines

As you can see lzma has the best compression ratio. The is the reason I chose lzma here.

Thanks, that's exactly the kind of data I was looking for.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200110/e5894b33/attachment.sig>