gunzip: add gzwrite routine for extracting compresed images to block device
PATCH 1/2 gunzip: add gzwrite routine for extracting compresed images to block device
Initial filesystem images are generally highly compressible.
Add a routine gzwrite that allows gzip-compressed images to be written to block devices.
--- include/common.h | 39 +++++++++++ lib/gunzip.c | 194 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 232 insertions(+), 1 deletion(-)
include/common.h +39 -0
@@ -713,6 +713,45 @@ int gunzip(void *, int, unsigned char *, unsigned long *);int zunzip (void *dst,int dstlen,unsigned char *src,unsigned long *lenp,int stoponerr,int offset); +/** +* gzwrite progress indicators: defined weak to allow board-specific +* overrides: +* +* gzwrite_progress_init called on startup +* gzwrite_progress called during decompress/write loop +* gzwrite_progress_finish called at end of loop to +* indicate success (retcode=0) or failure +*/ +void gzwrite_progress_init (u64 expected_size); + +void gzwrite_progress (int iteration, +u64 bytes_written, +u64 total_bytes); + +void gzwrite_progress_finish (int retcode, +u64 totalwritten, +u64 totalsize, +u32 expected_crc, +u32 calculated_crc); + +/** +* decompress and write gzipped image from memory to block device +* +* @param src compressed image address +* @param len compressed image length in bytes +* @param dev block device descriptor +* @param szwritebuf bytes per write (pad to erase size) +* @param startoffs offset in bytes of first write +* @param szexpected expected uncompressed length +* may be zero to use gzip trailer +* for files under 4GiB +*/ +int gzwrite (unsigned char *src,int len, +struct block_dev_desc *dev, +unsigned long szwritebuf, +u64 startoffs, +u64 szexpected); +/* lib/qsort.c */ void qsort (void *base,size_t nmemb,size_t size,int (*compar)(const void *,const void *));
lib/gunzip.c +193 -1
212 lines not on this page · raw message
Signature · 1 line
1.9.1
PATCH 2/2 unzip: add gzwrite command to write compressed image to block device
Add gzwrite command to write gzip-compressed images to block devices.
Input must be gzip-compressed according to RFC1952, since the crc and file size in the trailer will be confirmed during operation. The decompressed file size must be specified on the command line for images with decompressed sizes >= 4GiB because the trailer only contains the low 32 bits of the original file size.
--- common/cmd_unzip.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
common/cmd_unzip.c +47 -0
@@ -39,3 +39,50 @@ U_BOOT_CMD("unzip a memory region" ,"srcaddr dstaddr [dstsize]" ); + +static int do_gzwrite (cmd_tbl_t *cmdtp,int flag, +int argc,char *const argv[]) +{ +block_dev_desc_t *bdev; +int ret; +unsigned char *addr; +unsigned long length; +unsigned long writebuf =1 <<20 ; +u64 startoffs =0 ; +u64 szexpected =0 ; + +if (argc <5 ) +return CMD_RET_USAGE ; + ret =get_device (argv[1 ], argv[2 ], &bdev); +if (ret <0 ) +return CMD_RET_FAILURE ; + + addr = (unsigned char *)simple_strtoul (argv[3 ],NULL ,16 ); + length =simple_strtoul (argv[4 ],NULL ,16 ); + +if (5 < argc) { + writebuf =simple_strtoul (argv[5 ],NULL ,16 ); +if (6 < argc) { + startoffs =simple_strtoull (argv[6 ],NULL ,16 ); +if (7 < argc) + szexpected =simple_strtoull (argv[7 ], +NULL ,16 ); + } + } + + ret =gzwrite (addr, length, bdev, writebuf, startoffs, szexpected); + +return ret ?CMD_RET_FAILURE :CMD_RET_SUCCESS ; +} + +U_BOOT_CMD ( + gzwrite,8 ,0 , do_gzwrite, +"unzip and write memory to block device" , +"<interface> <dev> <addr> length [wbuf=1M [offs=0 [outsize=0]]]\n" +"\twbuf is the size in bytes (hex) of write buffer\n" +"\t\tand should be padded to erase size for SSDs\n" +"\toffs is the output start offset in bytes (hex)\n" +"\toutsize is the size of the expected output (hex bytes)\n" +"\t\tand is required for files with uncompressed lengths\n" +"\t\t4 GiB or larger\n" +);
Signature · 1 line
1.9.1
PATCH 2/2 unzip: add gzwrite command to write compressed image to block device
Quoted 9 lines · Eric Nelson
Add gzwrite command to write gzip-compressed images to block devices.
Input must be gzip-compressed according to RFC1952, since the crc and file size in the trailer will be confirmed during operation. The decompressed file size must be specified on the command line for images with decompressed sizes >= 4GiB because the trailer only contains the low 32 bits of the original file size.
Signed-off-by Eric Nelson <[email protected]>
Applied to u-boot/master, thanks!
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20150423/e1fb4c85/attachment.sig>
PATCH 1/2 gunzip: add gzwrite routine for extracting compresed images to block device
In reply to Eric NelsonQuoted 6 lines · Eric Nelson
Initial filesystem images are generally highly compressible.
Add a routine gzwrite that allows gzip-compressed images to be written to block devices.
Signed-off-by Eric Nelson <[email protected]>
Hi!
Stupid question -- can't you compress the thing in DRAM and then use fatwrite or ext4write to write it to FS? Or are you really after writing the data to a raw block device (in which case, you can use similar commands for raw block devices) ?
Best regards,
Marek Vasut
PATCH 1/2 gunzip: add gzwrite routine for extracting compresed images to block device
Quoted 6 lines · Marek Vasut
Quoted 5 lines · Eric Nelson
Initial filesystem images are generally highly compressible.
Add a routine gzwrite that allows gzip-compressed images to be written to block devices.
Quoted 8 lines
Quoted 1 line
Signed-off-by Eric Nelson <[email protected]>Hi!
Stupid question -- can't you compress the thing in DRAM and then use fatwrite or ext4write to write it to FS? Or are you really after writing the data to a raw block device (in which case, you can use similar commands for raw block devices) ?
I _think_ (and I really hope so otherwise yes, this series needs more expanation) that was this adds is the ability to {de,}compress on the fly rather than need to duplicate in DDR which could be hard-to-impossible depending on the size of the data in question.
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20150216/30c3807c/attachment.sig>
PATCH 1/2 gunzip: add gzwrite routine for extracting compresed images to block device
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Tom and Marek,
Quoted 11 lines · Tom Rini
Quoted 10 lines · Marek Vasut
Quoted 6 lines · Eric Nelson
Initial filesystem images are generally highly compressible.
Add a routine gzwrite that allows gzip-compressed images to be written to block devices.
Signed-off-by Eric Nelson <[email protected]>Hi!
Quoted 8 lines
Quoted 4 lines
Stupid question -- can't you compress the thing in DRAM and then use fatwrite or ext4write to write it to FS? Or are you really after writing the data to a raw block device (in which case, you can use similar commands for raw block devices) ?
I _think_ (and I really hope so otherwise yes, this series needs more expanation) that was this adds is the ability to {de,}compress on the
(or explanation ;))
Sometimes words fail. I thought that was clear from the commit message but apparently not.
Quoted 3 lines
fly rather than need to duplicate in DDR which could be hard-to-impossible depending on the size of the data in question.
That's exactly right.
The purpose of this is to aid in loading images onto storage devices like eMMC where the storage size usually exceeds the size of RAM, but the compressed image size doesn't.
Even if the compressed image size does exceed RAM, the gzwrite routine and command give you the ability to do things piecewise, and save lots of read transfer time.
To give a quick concrete example, we were looking at programming a relatively small (100's) batch of boards that use a very light O/S, but have 4GiB of eMMC.
Using ums takes over 25 minutes per board, but loading board.img.gz and using gzwrite takes 5-6, which is pretty close to optimal given the speed of the eMMC chip.
My hope is that this is useful as is, and also that the gzwrite routine can be worked into the fastboot protocol.
Transferring gigabytes of data is slow over USB 2.0 and storage sizes keep getting bigger.
Regards,
Eric
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU4invAAoJEFUqXmm9AiVrdAIH/0oP9xvgOOcSM3cI1i6TabDX YDoQRrDjCwmJ7gSBJ6PDpaA8rR/bstdaaEHnbkldNr+r+1KK35zQ20sKT2LfaKtK qoYbFDsflKuBng91m4qdIhrYRRNv6GgP/PtBFOSF9LOdHYTK18E11UD7cmXWKQga j0oEoMXLg67Ye1FHKqmyPqmpcIq66k6slIVla4p+BJnkBfzb0Cw5GnuqVk1l212a vrZIP/xBhCoxRcumzrI8Hh3WpND6CVfepe0EF6s9LNBbhYfFVZYH3lwMoZNM9v/S EMi5bPqcQphmrM7aJ4/M3QXBr52Ffz5LWM85OwENqH2P9SH8L0nNhagNW/tSIQE= =Tu5S
-----END PGP SIGNATURE-----
PATCH 1/2 gunzip: add gzwrite routine for extracting compresed images to block device
Quoted 49 lines · Eric Nelson
Hi Tom and Marek,
Quoted 11 lines · Tom Rini
Quoted 10 lines · Marek Vasut
Quoted 6 lines · Eric Nelson
Initial filesystem images are generally highly compressible.
Add a routine gzwrite that allows gzip-compressed images to be written to block devices.
Signed-off-by Eric Nelson <[email protected]>Hi!
Quoted 8 lines
Quoted 4 lines
Stupid question -- can't you compress the thing in DRAM and then use fatwrite or ext4write to write it to FS? Or are you really after writing the data to a raw block device (in which case, you can use similar commands for raw block devices) ?
I _think_ (and I really hope so otherwise yes, this series needs more expanation) that was this adds is the ability to {de,}compress on the
(or explanation ;))
Sometimes words fail. I thought that was clear from the commit message but apparently not.
Quoted 2 lines
fly rather than need to duplicate in DDR which could be hard-to-impossible depending on the size of the data in question.
That's exactly right.
The purpose of this is to aid in loading images onto storage devices like eMMC where the storage size usually exceeds the size of RAM, but the compressed image size doesn't.
Even if the compressed image size does exceed RAM, the gzwrite routine and command give you the ability to do things piecewise, and save lots of read transfer time.
To give a quick concrete example, we were looking at programming a relatively small (100's) batch of boards that use a very light O/S, but have 4GiB of eMMC.
Using ums takes over 25 minutes per board, but loading board.img.gz and using gzwrite takes 5-6, which is pretty close to optimal given the speed of the eMMC chip.
Quoted 5 lines
My hope is that this is useful as is, and also that the gzwrite routine can be worked into the fastboot protocol.
Transferring gigabytes of data is slow over USB 2.0 and storage sizes keep getting bigger.
Cool, thanks for explaining :)
Best regards,
Marek Vasut
PATCH v2 1/2 gunzip: add gzwrite routine for extracting compresed images to block device
In reply to Eric NelsonInitial filesystem images are generally highly compressible.
Add a routine gzwrite that allows gzip-compressed images to be written to block devices.
---
V2 removes floating point references from u64 division
include/common.h | 39 +++++++++++ lib/gunzip.c | 195 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 233 insertions(+), 1 deletion(-)
include/common.h +39 -0
@@ -713,6 +713,45 @@ int gunzip(void *, int, unsigned char *, unsigned long *);int zunzip (void *dst,int dstlen,unsigned char *src,unsigned long *lenp,int stoponerr,int offset); +/** +* gzwrite progress indicators: defined weak to allow board-specific +* overrides: +* +* gzwrite_progress_init called on startup +* gzwrite_progress called during decompress/write loop +* gzwrite_progress_finish called at end of loop to +* indicate success (retcode=0) or failure +*/ +void gzwrite_progress_init (u64 expected_size); + +void gzwrite_progress (int iteration, +u64 bytes_written, +u64 total_bytes); + +void gzwrite_progress_finish (int retcode, +u64 totalwritten, +u64 totalsize, +u32 expected_crc, +u32 calculated_crc); + +/** +* decompress and write gzipped image from memory to block device +* +* @param src compressed image address +* @param len compressed image length in bytes +* @param dev block device descriptor +* @param szwritebuf bytes per write (pad to erase size) +* @param startoffs offset in bytes of first write +* @param szexpected expected uncompressed length +* may be zero to use gzip trailer +* for files under 4GiB +*/ +int gzwrite (unsigned char *src,int len, +struct block_dev_desc *dev, +unsigned long szwritebuf, +u64 startoffs, +u64 szexpected); +/* lib/qsort.c */ void qsort (void *base,size_t nmemb,size_t size,int (*compar)(const void *,const void *));
lib/gunzip.c +194 -1
214 lines not on this page · raw message
Signature · 1 line
1.9.1
PATCH v2 1/2 gunzip: add gzwrite routine for extracting compresed images to block device
Hi all,
Quoted 8 lines · Eric Nelson
Initial filesystem images are generally highly compressible.
Add a routine gzwrite that allows gzip-compressed images to be written to block devices.
Signed-off-by Eric Nelson <[email protected]>---
V2 removes floating point references from u64 division
Ping.
Is this falling through the cracks because there's no maintainer for 'gunzip'?
Please advise,
Eric
PATCH v2 1/2 gunzip: add gzwrite routine for extracting compresed images to block device
In reply to Eric NelsonQuoted 8 lines · Eric Nelson
Initial filesystem images are generally highly compressible.
Add a routine gzwrite that allows gzip-compressed images to be written to block devices.
Signed-off-by Eric Nelson <[email protected]>---
V2 removes floating point references from u64 division
Seems fine so:
Missed the merge window so I'll pick this up in a long while or do a next again, not sure.
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20150223/eb5318dc/attachment.sig>
PATCH v2 1/2 gunzip: add gzwrite routine for extracting compresed images to block device
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Thanks Tom,
Quoted 7 lines · Tom Rini
Quoted 5 lines · Eric Nelson
Initial filesystem images are generally highly compressible.
Add a routine gzwrite that allows gzip-compressed images to be written to block devices.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Eric Nelson <[email protected]> ---V2 removes floating point references from u64 division
Seems fine so: Reviewed-by: Tom Rini <[email protected]>
Missed the merge window so I'll pick this up in a long while or do a next again, not sure.
No worries about the timing here. I was just hoping to get an ack before using this syntax and such for downstream things.
And before looking at how it could be glued to fastboot!
Regards,
Eric
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU68klAAoJEFUqXmm9AiVrTj8IALggPPlO3ff2ZOhT6Xw+NPmi 2CNU8yzgPHa1AmEUdRLA9Gsnc8YF2VD9uZMT2qJ8K10sVP14ICjnnuFpBCw5U/PN I9eLovFJCIBZLumlrMlHxTnKCDKB7P5it5X+sp0BD64VopS6kY2UEuSRDh2tV76m W+qEvBz2kxYp+ur6KWVq7HvQoOevNJubXOjgZUoe5brIXZRhuHm7LMxjClxmrBbb zyGQVu0lvf1wSENjhpRf177aeUV37dQpzKEwMrNA6KIHKUkvxXaRozuANC78ttvY bV1MO+/N5auNJ5+MaDerFW2GnvOsOzqHIWQpIVlcq+YZzbCzYhIpRsM28Wkxrr4= =1xS/
-----END PGP SIGNATURE-----
PATCH v2 1/2 gunzip: add gzwrite routine for extracting compresed images to block device
In reply to Eric NelsonQuoted 7 lines · Eric Nelson
Initial filesystem images are generally highly compressible.
Add a routine gzwrite that allows gzip-compressed images to be written to block devices.
Signed-off-by Eric Nelson <[email protected]>Reviewed-by Tom Rini <[email protected]>
Applied to u-boot/master, thanks!
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20150423/a31d928c/attachment.sig>