Skip to contentSkip to conversation

Add support for XZ decompression algorithm.

PATCH New Luigi 'Comio' Mantellini 6 participants 8 messages Dec 5, 2010 – Dec 17, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 Add support for XZ decompression algorithm.

XZ (aka LZMA2) is the new version of lzma compression format.
The following patch add a cut-down version of XZ Embedded library (v20100702) that supports only single-call API.

In order to enable XZ support, the CONFIG_XZ must be defined by board configuration file.

For any details, please refer to XZ Embedded homesite
(http://tukaani.org/xz/embedded.html)

---
 Makefile                 |    1 +
 README                   |    9 +
 include/unxz.h           |   16 +
 include/xz.h             |  237 ++++++++++
 lib/xz/Makefile          |   51 ++
 lib/xz/decompress_unxz.c |  231 +++++++++
 lib/xz/xz_dec_bcj.c      |  564 ++++++++++++++++++++++
 lib/xz/xz_dec_lzma2.c    | 1175 ++++++++++++++++++++++++++++++++++++++++++++++
 lib/xz/xz_dec_stream.c   |  823 ++++++++++++++++++++++++++++++++
 lib/xz/xz_lzma2.h        |  204 ++++++++
 lib/xz/xz_private.h      |  154 ++++++
 lib/xz/xz_stream.h       |   50 ++
 12 files changed, 3515 insertions(+), 0 deletions(-)
 create mode 100644 include/unxz.h
 create mode 100644 include/xz.h
 create mode 100644 lib/xz/Makefile
 create mode 100644 lib/xz/decompress_unxz.c
 create mode 100644 lib/xz/xz_dec_bcj.c
 create mode 100644 lib/xz/xz_dec_lzma2.c
 create mode 100644 lib/xz/xz_dec_stream.c
 create mode 100644 lib/xz/xz_lzma2.h
 create mode 100644 lib/xz/xz_private.h
 create mode 100644 lib/xz/xz_stream.h
Makefile +1 -0
index 87a383d..ba832a4 100644
@@ -183,6 +183,7 @@ OBJS := $(addprefix $(obj),$(OBJS))
 LIBS  = lib/libgeneric.o
 LIBS += lib/lzma/liblzma.o
 LIBS += lib/lzo/liblzo.o
+LIBS += lib/xz/libxz.o
 LIBS += $(shell if [ -f board/$(VENDOR)/common/Makefile ]; then echo \
 	"board/$(VENDOR)/common/lib$(VENDOR).o"; fi)
 ifdef CONFIG_FSL_DIU_FB
README +9 -0
index 68f5fb0..90c19a9 100644
@@ -220,6 +220,7 @@ Directory Hierarchy:
   /libfdt		Library files to support flattened device trees
   /lzma			Library files to support LZMA decompression
   /lzo			Library files to support LZO decompression
+  /xz			Library files to support XZ/LZMA2 decompression
 /net			Networking code
 /post			Power On Self Test
 /rtc			Real Time Clock drivers
@@ -1178,6 +1179,14 @@ The following options need to be configured:
 		then calculate the amount of needed dynamic memory (ensuring
 		the appropriate CONFIG_SYS_MALLOC_LEN value).
 
+		CONFIG_XZ
+
+		If this option is set, support for xz compresed
+		images is included.
+
+		XZ format is the next version of LZMA format. See also
+		the CONFIG_LZMA
+
 - MII/PHY support:
 		CONFIG_PHY_ADDR
 
include/unxz.h new file +16 -0
new file mode 100644
index 0000000..db4d58a
@@ -0,0 +1,16 @@
+/*
+ * Wrapper for XZ decompressor to make it usable for kernel and initramfs
+ * decompression
+ *
+ * Author: Lasse Collin <[email protected]>
+ *
+ * This file has been put into the public domain.
+ * You can do whatever you want with this file.
+ */
+
+#ifndef DECOMPRESS_UNXZ_H
+#define DECOMPRESS_UNXZ_H
+
+int unxz(unsigned char *in, int in_size, unsigned char *out, int *in_used);
+
+#endif
include/xz.h new file +237 -0

237 lines not on this page · raw message

lib/xz/Makefile new file +51 -0

51 lines not on this page · raw message

lib/xz/decompress_unxz.c new file +231 -0

231 lines not on this page · raw message

lib/xz/xz_dec_bcj.c new file +564 -0

564 lines not on this page · raw message

lib/xz/xz_dec_lzma2.c new file +1175 -0

1,175 lines not on this page · raw message

lib/xz/xz_dec_stream.c new file +823 -0

698 lines not on this page · raw message

lib/xz/xz_lzma2.h new file +204 -0

204 lines not on this page · raw message

lib/xz/xz_private.h new file +154 -0

154 lines not on this page · raw message

lib/xz/xz_stream.h new file +50 -0

50 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
1.7.3

Part of the diff is not shown here. View the raw message.

#1 lore raw

PATCH 2/2 Add XZ compressed uImage support

This patch permits to use XZ compressed uImages using the XZ embedded library code.

---
 common/cmd_bootm.c |   27 +++++++++++++++++++++++++--
 common/image.c     |    1 +
 include/image.h    |    1 +
 3 files changed, 27 insertions(+), 2 deletions(-)
common/cmd_bootm.c +25 -2
index 1a024f1..066b908 100644
@@ -61,6 +61,10 @@
 #include <linux/lzo.h>
 #endif /* CONFIG_LZO */
 
+#ifdef CONFIG_XZ
+#include <unxz.h>
+#endif /* CONFIG_XZ */
+
 DECLARE_GLOBAL_DATA_PTR;
 
 #ifndef CONFIG_SYS_BOOTM_LEN
@@ -338,9 +342,9 @@ static int bootm_load_os(image_info_t os, ulong *load_end, int boot_progress)
 	ulong image_start = os.image_start;
 	ulong image_len = os.image_len;
 	uint unc_len = CONFIG_SYS_BOOTM_LEN;
-#if defined(CONFIG_LZMA) || defined(CONFIG_LZO)
+#if defined(CONFIG_LZMA) || defined(CONFIG_LZO) || defined(CONFIG_XZ)
 	int ret;
-#endif /* defined(CONFIG_LZMA) || defined(CONFIG_LZO) */
+#endif /* defined(CONFIG_LZMA) || defined(CONFIG_LZO) || defined(CONFIG_XZ) */
 
 	const char *type_name = genimg_get_type_name (os.type);
 
@@ -430,6 +434,25 @@ static int bootm_load_os(image_info_t os, ulong *load_end, int boot_progress)
 		*load_end = load + unc_len;
 		break;
 #endif /* CONFIG_LZO */
+#ifdef CONFIG_XZ
+	case IH_COMP_XZ: {
+		int in_used = unc_len;
+		printf ("   Uncompressing %s ... ", type_name);
+		ret = unxz(
+			(unsigned char *)image_start, image_len,
+			(unsigned char *)load, &in_used);
+		unc_len = in_used;
+		if (ret != 0) {
+			printf ("XZ: uncompress or overwrite error %d "
+				"- must RESET board to recover\n", ret);
+			show_boot_progress (-6);
+			return BOOTM_ERR_RESET;
+		}
+		*load_end = load + unc_len;
+		break;
+	}
+		break;
+#endif /* CONFIG_XZ */
 	default:
 		printf ("Unimplemented compression type %d\n", comp);
 		return BOOTM_ERR_UNIMPLEMENTED;
common/image.c +1 -0
index 42f5b79..4e31691 100644
@@ -150,6 +150,7 @@ static table_entry_t uimage_comp[] = {
 	{	IH_COMP_GZIP,	"gzip",		"gzip compressed",	},
 	{	IH_COMP_LZMA,	"lzma",		"lzma compressed",	},
 	{	IH_COMP_LZO,	"lzo",		"lzo compressed",	},
+	{	IH_COMP_XZ,	"xz",		"xz compressed",	},
 	{	-1,		"",		"",			},
 };
 
include/image.h +1 -0
index 49d6280..5ba5afa 100644
@@ -166,6 +166,7 @@
 #define IH_COMP_BZIP2		2	/* bzip2 Compression Used	*/
 #define IH_COMP_LZMA		3	/* lzma  Compression Used	*/
 #define IH_COMP_LZO		4	/* lzo   Compression Used	*/
+#define IH_COMP_XZ		5	/* xz    Compression Used       */
 
 #define IH_MAGIC	0x27051956	/* Image Magic Number		*/
 #define IH_NMLEN		32	/* Image Name Length		*/
Signature · 1 line
1.7.3
#2 lore raw

PATCH 1/2 Add support for XZ decompression algorithm.

In reply to Luigi 'Comio' Mantellini
Quoted 13 lines

XZ (aka LZMA2) is the new version of lzma compression format.
The following patch add a cut-down version of XZ Embedded library (v20100702) that supports only single-call API.

In order to enable XZ support, the CONFIG_XZ must be defined by board configuration file.

For any details, please refer to XZ Embedded homesite
(http://tukaani.org/xz/embedded.html)

---

[SNIP]

lib/xz/decompress_unxz.c line 124

Quoted 5 lines
lib/xz/decompress_unxz.c new file +5 -0
+/*
+ * Use the internal CRC32 code instead of kernel's CRC32 module, which
+ * is not available in early phase of booting.
+ */
+#define XZ_INTERNAL_CRC32 1

I really hope the text is misleading. Why use your own crc32 when u-boot has a pretty good one?

Jocke

#3 lore raw

PATCH 1/2 Add support for XZ decompression algorithm.

In reply to Luigi 'Comio' Mantellini

Hi Luigi,

Quoted 9 lines · Luigi 'Comio' Mantellini

XZ (aka LZMA2) is the new version of lzma compression format.
The following patch add a cut-down version of XZ Embedded library (v20100702) that supports only single-call API.

In order to enable XZ support, the CONFIG_XZ must be defined by board configuration file.

For any details, please refer to XZ Embedded homesite
(http://tukaani.org/xz/embedded.html)

If I understand the xz format correctly, its nearly identical to lzma, and the 'xz' utility can decompress .lzma files too. Any chance the xz code you're adding can be used to handle lzma-compressed archives? If so, can we deprecate CONFIG_LZMA and have the new CONFIG_XZ support handle both .xz and .lzma archives? Or at least share a significant chunk of code between the two formats?

Best,
Peter

#4 lore raw

PATCH 1/2 Add support for XZ decompression algorithm.

Quoted 11 lines · On Monday, December 06, 2010 12:58:37 Peter Tyser wrote:
Quoted 9 lines · Luigi 'Comio' Mantellini

XZ (aka LZMA2) is the new version of lzma compression format. The following patch add a cut-down version of XZ Embedded library (v20100702) that supports only single-call API.

In order to enable XZ support, the CONFIG_XZ must be defined by board configuration file.

For any details, please refer to XZ Embedded homesite
(http://tukaani.org/xz/embedded.html)

Quoted 6 lines

If I understand the xz format correctly, its nearly identical to lzma, and the 'xz' utility can decompress .lzma files too. Any chance the xz code you're adding can be used to handle lzma-compressed archives? If so, can we deprecate CONFIG_LZMA and have the new CONFIG_XZ support handle both .xz and .lzma archives? Or at least share a significant chunk of code between the two formats?

my understanding is that the compression algorithm is the same, but the ".xz" file format adds nice/detectable header of metadata where as ".lzma" does not.

so yes, i believe the two should have quite a lot of [undesirable] overlap. -mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20101206/8702d35a/attachment.pgp

#5 lore raw

PATCH 1/2 Add support for XZ decompression algorithm.

Hi All,

Quoted 19 lines · Mike Frysinger
Quoted 11 lines · On Monday, December 06, 2010 12:58:37 Peter Tyser wrote:
Quoted 9 lines · Luigi 'Comio' Mantellini

XZ (aka LZMA2) is the new version of lzma compression format. The following patch add a cut-down version of XZ Embedded library (v20100702) that supports only single-call API.

In order to enable XZ support, the CONFIG_XZ must be defined by board configuration file.

For any details, please refer to XZ Embedded homesite
(http://tukaani.org/xz/embedded.html)

Quoted 6 lines

If I understand the xz format correctly, its nearly identical to lzma, and the 'xz' utility can decompress .lzma files too. ?Any chance the xz code you're adding can be used to handle lzma-compressed archives? ?If so, can we deprecate CONFIG_LZMA and have the new CONFIG_XZ support handle both .xz and .lzma archives? ?Or at least share a significant chunk of code between the two formats?

Quoted 2 lines

my understanding is that the compression algorithm is the same, but the ".xz" file format adds nice/detectable header of metadata where as ".lzma" does not.

XZ is essentially a sequence of filters. LZMA2 is just a filter (a function that translate) and is made on top of LZMA algorithm. Furthermore the XZ file is not a LZMA file but the XZ header permits to distinguish a raw LZMA stream from a complex XZ. This is the "file format" point of view. Regarding the performance, the XZ, using a LZMA algorithm to compress the blocks, has performance very close the raw LZMA.
Regarding the XZ tools point of view, a compatibility are provided by xz-tools in order to permit an easy transition to xz.

http://tukaani.org/xz/xz-file-format-1.0.4.txt

Quoted 4 lines

so yes, i believe the two should have quite a lot of [undesirable] overlap. -mike

The LZMA and XZ overlap on the internal LZMA code (each one has the own implementation). We can decide 1) to ignore this patch (and future rework) to not support XZ 2) have both XZ and LZMA considering them as different algorithms 3) Deprecate LZMA and (in the future) support just XZ.

My idea is to have 2) and in the future switch to 3).

should be interesting to check XZ library in order to use just the internal LZMA decoder to drop the lzma lib.

my 2Eurocents,

luigi

Quoted 6 lines

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Signature · 10 lines
Luigi 'Comio' Mantellini
R&D - Software
Industrie Dial Face S.p.A.
Via Canzo, 4
20068 Peschiera Borromeo (MI), Italy

Tel.: +39 02 5167 2813
Fax: +39 02 5167 2459
web: www.idf-hit.com
mail: luigi.mantellini at idf-hit.com
#6 lore raw

PATCH 1/2 Add support for XZ decompression algorithm.

Dear Luigi Mantellini,

Quoted 8 lines · In message <[email protected]> you…

The LZMA and XZ overlap on the internal LZMA code (each one has the own implementation). We can decide 1) to ignore this patch (and future rework) to not support XZ 2) have both XZ and LZMA considering them as different algorithms 3) Deprecate LZMA and (in the future) support just XZ.

My idea is to have 2) and in the future switch to 3).

History has shown that the "let me get this in now and I will clean up later" seldom works. Usually there is just toolittle time for the cleanup, and what for? It's working, isn't it?

So please me this 3).

Thanks.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
You!  What PLANET is this!
	-- McCoy, "The City on the Edge of Forever", stardate 3134.0
#7 lore raw

PATCH 1/2 Add support for XZ decompression algorithm.

In reply to Luigi 'Comio' Mantellini

Dear Luigi 'Comio' Mantellini,

Quoted 9 lines · In message <[email protected]>…

XZ (aka LZMA2) is the new version of lzma compression format.
The following patch add a cut-down version of XZ Embedded library (v20100702) that supports only single-call API.

In order to enable XZ support, the CONFIG_XZ must be defined by board configuration file.

For any details, please refer to XZ Embedded homesite
(http://tukaani.org/xz/embedded.html)

...

README The following options need to be configured: · line 1188

Quoted 11 lines
README +7 -0
@@ -1178,6 +1179,14 @@ The following options need to be configured:
 		then calculate the amount of needed dynamic memory (ensuring
 		the appropriate CONFIG_SYS_MALLOC_LEN value).
 
+		CONFIG_XZ
+
+		If this option is set, support for xz compresed
+		images is included.
+
+		XZ format is the next version of LZMA format. See also
+		the CONFIG_LZMA

s/compresed/compressed/

lib/xz/decompress_unxz.c line 124

Quoted 5 lines
lib/xz/decompress_unxz.c new file +5 -0
+/*
+ * Use the internal CRC32 code instead of kernel's CRC32 module, which
+ * is not available in early phase of booting.
+ */
+#define XZ_INTERNAL_CRC32 1

This has already caused confusion, but appears to be unused.

Please clean up the code not to contain such dead definitions.

I did not see a reply yet to the request to deprecate CONFIG_LZMA and have the new CONFIG_XZ support handle both .xz and .lzma archives?

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
There has been an alarming increase in the number of things you  know
nothing about.