Skip to contentSkip to conversation

gunzip: add gzwrite routine for extracting compresed images to block device

PATCH Superseded Eric Nelson 4 participants 12 messages Feb 15, 2015 – Apr 23, 2015 Library code

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 gunzip: add gzwrite routine for extracting compresed images to block device

Initial filesystem images are generally highly compressible.

Add a routine gzwrite that allows gzip-compressed images to be written to block devices.

---
 include/common.h |  39 +++++++++++
 lib/gunzip.c     | 194 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 232 insertions(+), 1 deletion(-)
include/common.h +39 -0
index 9129454..f96baea 100644
@@ -713,6 +713,45 @@ int gunzip(void *, int, unsigned char *, unsigned long *);
 int zunzip(void *dst, int dstlen, unsigned char *src, unsigned long *lenp,
 						int stoponerr, int offset);
 
+/**
+ * gzwrite progress indicators: defined weak to allow board-specific
+ * overrides:
+ *
+ *	gzwrite_progress_init called on startup
+ *	gzwrite_progress called during decompress/write loop
+ *	gzwrite_progress_finish called at end of loop to
+ *		indicate success (retcode=0) or failure
+ */
+void gzwrite_progress_init(u64 expected_size);
+
+void gzwrite_progress(int iteration,
+		     u64 bytes_written,
+		     u64 total_bytes);
+
+void gzwrite_progress_finish(int retcode,
+			     u64 totalwritten,
+			     u64 totalsize,
+			     u32 expected_crc,
+			     u32 calculated_crc);
+
+/**
+ * decompress and write gzipped image from memory to block device
+ *
+ * @param	src		compressed image address
+ * @param	len		compressed image length in bytes
+ * @param	dev		block device descriptor
+ * @param	szwritebuf	bytes per write (pad to erase size)
+ * @param	startoffs	offset in bytes of first write
+ * @param	szexpected	expected uncompressed length
+ *				may be zero to use gzip trailer
+ *				for files under 4GiB
+ */
+int gzwrite(unsigned char *src, int len,
+	    struct block_dev_desc *dev,
+	    unsigned long szwritebuf,
+	    u64 startoffs,
+	    u64 szexpected);
+
 /* lib/qsort.c */
 void qsort(void *base, size_t nmemb, size_t size,
 	   int(*compar)(const void *, const void *));
lib/gunzip.c +193 -1

212 lines not on this page · raw message

Signature · 1 line
1.9.1
#1 lore raw

PATCH 2/2 unzip: add gzwrite command to write compressed image to block device

Add gzwrite command to write gzip-compressed images to block devices.

Input must be gzip-compressed according to RFC1952, since the crc and file size in the trailer will be confirmed during operation. The decompressed file size must be specified on the command line for images with decompressed sizes >= 4GiB because the trailer only contains the low 32 bits of the original file size.

---
 common/cmd_unzip.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
common/cmd_unzip.c +47 -0
index b02c69e..0686be6 100644
@@ -39,3 +39,50 @@ U_BOOT_CMD(
 	"unzip a memory region",
 	"srcaddr dstaddr [dstsize]"
 );
+
+static int do_gzwrite(cmd_tbl_t *cmdtp, int flag,
+		      int argc, char * const argv[])
+{
+	block_dev_desc_t *bdev;
+	int ret;
+	unsigned char *addr;
+	unsigned long length;
+	unsigned long writebuf = 1<<20;
+	u64 startoffs = 0;
+	u64 szexpected = 0;
+
+	if (argc < 5)
+		return CMD_RET_USAGE;
+	ret = get_device(argv[1], argv[2], &bdev);
+	if (ret < 0)
+		return CMD_RET_FAILURE;
+
+	addr = (unsigned char *)simple_strtoul(argv[3], NULL, 16);
+	length = simple_strtoul(argv[4], NULL, 16);
+
+	if (5 < argc) {
+		writebuf = simple_strtoul(argv[5], NULL, 16);
+		if (6 < argc) {
+			startoffs = simple_strtoull(argv[6], NULL, 16);
+			if (7 < argc)
+				szexpected = simple_strtoull(argv[7],
+							     NULL, 16);
+		}
+	}
+
+	ret = gzwrite(addr, length, bdev, writebuf, startoffs, szexpected);
+
+	return ret ? CMD_RET_FAILURE : CMD_RET_SUCCESS;
+}
+
+U_BOOT_CMD(
+	gzwrite, 8, 0, do_gzwrite,
+	"unzip and write memory to block device",
+	"<interface> <dev> <addr> length [wbuf=1M [offs=0 [outsize=0]]]\n"
+	"\twbuf is the size in bytes (hex) of write buffer\n"
+	"\t\tand should be padded to erase size for SSDs\n"
+	"\toffs is the output start offset in bytes (hex)\n"
+	"\toutsize is the size of the expected output (hex bytes)\n"
+	"\t\tand is required for files with uncompressed lengths\n"
+	"\t\t4 GiB or larger\n"
+);
Signature · 1 line
1.9.1
#2 lore raw

PATCH 2/2 unzip: add gzwrite command to write compressed image to block device

Quoted 9 lines · Eric Nelson

Add gzwrite command to write gzip-compressed images to block devices.

Input must be gzip-compressed according to RFC1952, since the crc and file size in the trailer will be confirmed during operation. The decompressed file size must be specified on the command line for images with decompressed sizes >= 4GiB because the trailer only contains the low 32 bits of the original file size.

Applied to u-boot/master, thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20150423/e1fb4c85/attachment.sig>
#3 lore raw

PATCH 1/2 gunzip: add gzwrite routine for extracting compresed images to block device

In reply to Eric Nelson
Quoted 6 lines · Eric Nelson

Initial filesystem images are generally highly compressible.

Add a routine gzwrite that allows gzip-compressed images to be written to block devices.

Hi!

Stupid question -- can't you compress the thing in DRAM and then use fatwrite or ext4write to write it to FS? Or are you really after writing the data to a raw block device (in which case, you can use similar commands for raw block devices) ?

Best regards,
Marek Vasut

#4 lore raw

PATCH 1/2 gunzip: add gzwrite routine for extracting compresed images to block device

Quoted 6 lines · Marek Vasut
Quoted 5 lines · Eric Nelson

Initial filesystem images are generally highly compressible.

Add a routine gzwrite that allows gzip-compressed images to be written to block devices.

Quoted 8 lines
Quoted 1 line

Hi!

Stupid question -- can't you compress the thing in DRAM and then use fatwrite or ext4write to write it to FS? Or are you really after writing the data to a raw block device (in which case, you can use similar commands for raw block devices) ?

I _think_ (and I really hope so otherwise yes, this series needs more expanation) that was this adds is the ability to {de,}compress on the fly rather than need to duplicate in DDR which could be hard-to-impossible depending on the size of the data in question.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20150216/30c3807c/attachment.sig>
#5 lore raw

PATCH 1/2 gunzip: add gzwrite routine for extracting compresed images to block device

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Tom and Marek,

Quoted 11 lines · Tom Rini
Quoted 10 lines · Marek Vasut
Quoted 6 lines · Eric Nelson

Initial filesystem images are generally highly compressible.

Add a routine gzwrite that allows gzip-compressed images to be written to block devices.

Hi!

Quoted 8 lines
Quoted 4 lines

Stupid question -- can't you compress the thing in DRAM and then use fatwrite or ext4write to write it to FS? Or are you really after writing the data to a raw block device (in which case, you can use similar commands for raw block devices) ?

I _think_ (and I really hope so otherwise yes, this series needs more expanation) that was this adds is the ability to {de,}compress on the

(or explanation ;))

Sometimes words fail. I thought that was clear from the commit message but apparently not.

Quoted 3 lines

fly rather than need to duplicate in DDR which could be hard-to-impossible depending on the size of the data in question.

That's exactly right.

The purpose of this is to aid in loading images onto storage devices like eMMC where the storage size usually exceeds the size of RAM, but the compressed image size doesn't.

Even if the compressed image size does exceed RAM, the gzwrite routine and command give you the ability to do things piecewise, and save lots of read transfer time.

To give a quick concrete example, we were looking at programming a relatively small (100's) batch of boards that use a very light O/S, but have 4GiB of eMMC.

Using ums takes over 25 minutes per board, but loading board.img.gz and using gzwrite takes 5-6, which is pretty close to optimal given the speed of the eMMC chip.

My hope is that this is useful as is, and also that the gzwrite routine can be worked into the fastboot protocol.

Transferring gigabytes of data is slow over USB 2.0 and storage sizes keep getting bigger.

Regards,

Eric
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJU4invAAoJEFUqXmm9AiVrdAIH/0oP9xvgOOcSM3cI1i6TabDX YDoQRrDjCwmJ7gSBJ6PDpaA8rR/bstdaaEHnbkldNr+r+1KK35zQ20sKT2LfaKtK qoYbFDsflKuBng91m4qdIhrYRRNv6GgP/PtBFOSF9LOdHYTK18E11UD7cmXWKQga j0oEoMXLg67Ye1FHKqmyPqmpcIq66k6slIVla4p+BJnkBfzb0Cw5GnuqVk1l212a vrZIP/xBhCoxRcumzrI8Hh3WpND6CVfepe0EF6s9LNBbhYfFVZYH3lwMoZNM9v/S EMi5bPqcQphmrM7aJ4/M3QXBr52Ffz5LWM85OwENqH2P9SH8L0nNhagNW/tSIQE= =Tu5S
-----END PGP SIGNATURE-----

#6 lore raw

PATCH 1/2 gunzip: add gzwrite routine for extracting compresed images to block device

Quoted 49 lines · Eric Nelson

Hi Tom and Marek,

Quoted 11 lines · Tom Rini
Quoted 10 lines · Marek Vasut
Quoted 6 lines · Eric Nelson

Initial filesystem images are generally highly compressible.

Add a routine gzwrite that allows gzip-compressed images to be written to block devices.

Hi!

Quoted 8 lines
Quoted 4 lines

Stupid question -- can't you compress the thing in DRAM and then use fatwrite or ext4write to write it to FS? Or are you really after writing the data to a raw block device (in which case, you can use similar commands for raw block devices) ?

I _think_ (and I really hope so otherwise yes, this series needs more expanation) that was this adds is the ability to {de,}compress on the

(or explanation ;))

Sometimes words fail. I thought that was clear from the commit message but apparently not.

Quoted 2 lines

fly rather than need to duplicate in DDR which could be hard-to-impossible depending on the size of the data in question.

That's exactly right.

The purpose of this is to aid in loading images onto storage devices like eMMC where the storage size usually exceeds the size of RAM, but the compressed image size doesn't.

Even if the compressed image size does exceed RAM, the gzwrite routine and command give you the ability to do things piecewise, and save lots of read transfer time.

To give a quick concrete example, we were looking at programming a relatively small (100's) batch of boards that use a very light O/S, but have 4GiB of eMMC.

Using ums takes over 25 minutes per board, but loading board.img.gz and using gzwrite takes 5-6, which is pretty close to optimal given the speed of the eMMC chip.

Quoted 5 lines

My hope is that this is useful as is, and also that the gzwrite routine can be worked into the fastboot protocol.

Transferring gigabytes of data is slow over USB 2.0 and storage sizes keep getting bigger.

Cool, thanks for explaining :)

Best regards,
Marek Vasut

#7 lore raw

PATCH v2 1/2 gunzip: add gzwrite routine for extracting compresed images to block device

In reply to Eric Nelson

Initial filesystem images are generally highly compressible.

Add a routine gzwrite that allows gzip-compressed images to be written to block devices.

---
V2 removes floating point references from u64 division

 include/common.h |  39 +++++++++++
 lib/gunzip.c     | 195 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 233 insertions(+), 1 deletion(-)
include/common.h +39 -0
index 9129454..f96baea 100644
@@ -713,6 +713,45 @@ int gunzip(void *, int, unsigned char *, unsigned long *);
 int zunzip(void *dst, int dstlen, unsigned char *src, unsigned long *lenp,
 						int stoponerr, int offset);
 
+/**
+ * gzwrite progress indicators: defined weak to allow board-specific
+ * overrides:
+ *
+ *	gzwrite_progress_init called on startup
+ *	gzwrite_progress called during decompress/write loop
+ *	gzwrite_progress_finish called at end of loop to
+ *		indicate success (retcode=0) or failure
+ */
+void gzwrite_progress_init(u64 expected_size);
+
+void gzwrite_progress(int iteration,
+		     u64 bytes_written,
+		     u64 total_bytes);
+
+void gzwrite_progress_finish(int retcode,
+			     u64 totalwritten,
+			     u64 totalsize,
+			     u32 expected_crc,
+			     u32 calculated_crc);
+
+/**
+ * decompress and write gzipped image from memory to block device
+ *
+ * @param	src		compressed image address
+ * @param	len		compressed image length in bytes
+ * @param	dev		block device descriptor
+ * @param	szwritebuf	bytes per write (pad to erase size)
+ * @param	startoffs	offset in bytes of first write
+ * @param	szexpected	expected uncompressed length
+ *				may be zero to use gzip trailer
+ *				for files under 4GiB
+ */
+int gzwrite(unsigned char *src, int len,
+	    struct block_dev_desc *dev,
+	    unsigned long szwritebuf,
+	    u64 startoffs,
+	    u64 szexpected);
+
 /* lib/qsort.c */
 void qsort(void *base, size_t nmemb, size_t size,
 	   int(*compar)(const void *, const void *));
lib/gunzip.c +194 -1

214 lines not on this page · raw message

Signature · 1 line
1.9.1
#8 lore raw

PATCH v2 1/2 gunzip: add gzwrite routine for extracting compresed images to block device

Hi all,

Quoted 8 lines · Eric Nelson

Initial filesystem images are generally highly compressible.

Add a routine gzwrite that allows gzip-compressed images to be written to block devices.

---
V2 removes floating point references from u64 division

Ping.

Is this falling through the cracks because there's no maintainer for 'gunzip'?

Please advise,

Eric

#9 lore raw

PATCH v2 1/2 gunzip: add gzwrite routine for extracting compresed images to block device

In reply to Eric Nelson
Quoted 8 lines · Eric Nelson

Initial filesystem images are generally highly compressible.

Add a routine gzwrite that allows gzip-compressed images to be written to block devices.

---
V2 removes floating point references from u64 division

Seems fine so:

Missed the merge window so I'll pick this up in a long while or do a next again, not sure.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20150223/eb5318dc/attachment.sig>
#10 lore raw

PATCH v2 1/2 gunzip: add gzwrite routine for extracting compresed images to block device

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thanks Tom,

Quoted 7 lines · Tom Rini
Quoted 5 lines · Eric Nelson

Initial filesystem images are generally highly compressible.

Add a routine gzwrite that allows gzip-compressed images to be written to block devices.

Quoted 8 lines
Quoted 2 lines

V2 removes floating point references from u64 division

Seems fine so: Reviewed-by: Tom Rini <[email protected]>

Missed the merge window so I'll pick this up in a long while or do a next again, not sure.

No worries about the timing here. I was just hoping to get an ack before using this syntax and such for downstream things.

And before looking at how it could be glued to fastboot!

Regards,

Eric

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJU68klAAoJEFUqXmm9AiVrTj8IALggPPlO3ff2ZOhT6Xw+NPmi 2CNU8yzgPHa1AmEUdRLA9Gsnc8YF2VD9uZMT2qJ8K10sVP14ICjnnuFpBCw5U/PN I9eLovFJCIBZLumlrMlHxTnKCDKB7P5it5X+sp0BD64VopS6kY2UEuSRDh2tV76m W+qEvBz2kxYp+ur6KWVq7HvQoOevNJubXOjgZUoe5brIXZRhuHm7LMxjClxmrBbb zyGQVu0lvf1wSENjhpRf177aeUV37dQpzKEwMrNA6KIHKUkvxXaRozuANC78ttvY bV1MO+/N5auNJ5+MaDerFW2GnvOsOzqHIWQpIVlcq+YZzbCzYhIpRsM28Wkxrr4= =1xS/
-----END PGP SIGNATURE-----

#11 lore raw

PATCH v2 1/2 gunzip: add gzwrite routine for extracting compresed images to block device

In reply to Eric Nelson
Quoted 7 lines · Eric Nelson

Initial filesystem images are generally highly compressible.

Add a routine gzwrite that allows gzip-compressed images to be written to block devices.

Applied to u-boot/master, thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20150423/a31d928c/attachment.sig>