fastboot: unpacking initramfs faster
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Andreas Robinson
I tried running the initramfs unpacking asynchronously (and earlier) but ran into a curious problem: The parent thread stalled until decompression was done. I don't know why - I made sure that the two threads ran on different cores. Perhaps the parent was cache starved.
I could not stall my system by gzip</dev/urandom>/dev/null. Maybe you accidentally used vfork?
Quoted 8 lines
I then looked at multithreaded decompression and found that neither gzip files nor deflate streams provide enough information (eg block lengths) to let you share the decompression among threads.
It can be done of course, but obviously not without modifying file formats and userspace tools.
(Thoughts of snowballs and hell presented themselves at this point.)
Why not make a new set? http://7eggert.dyndns.org:8080/l/bgz.tar.gz
Quoted 9 lines · Bodo Eggert
Quoted 4 lines · Andreas Robinson
I tried running the initramfs unpacking asynchronously (and earlier) but ran into a curious problem: The parent thread stalled until decompression was done. I don't know why - I made sure that the two threads ran on different cores. Perhaps the parent was cache starved.
I could not stall my system by gzip</dev/urandom>/dev/null. Maybe you accidentally used vfork?
No, this was an in-kernel test. I used the Arjan's new async facility which calls kthread_create().
Quoted 10 lines
Quoted 8 lines
I then looked at multithreaded decompression and found that neither gzip files nor deflate streams provide enough information (eg block lengths) to let you share the decompression among threads.
It can be done of course, but obviously not without modifying file formats and userspace tools.
(Thoughts of snowballs and hell presented themselves at this point.)
Why not make a new set? http://7eggert.dyndns.org:8080/l/bgz.tar.gz
Hmm, well if you think there is a chance of it being accepted, then sure! I'll cook something up, but will be looking to use LZO first since it's a bit faster to begin with.
Quoted 10 lines · Andreas Robinson
Quoted 9 lines · Bodo Eggert
Quoted 7 lines · Andreas Robinson
I then looked at multithreaded decompression and found that neither gzip files nor deflate streams provide enough information (eg block lengths) to let you share the decompression among threads.
It can be done of course, but obviously not without modifying file formats and userspace tools.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
(Thoughts of snowballs and hell presented themselves at this point.)
Why not make a new set? http://7eggert.dyndns.org:8080/l/bgz.tar.gz
Hmm, well if you think there is a chance of it being accepted, then sure! I'll cook something up, but will be looking to use LZO first since it's a bit faster to begin with.
It's small enough, and now I've updated it to support lzo and put a license into the files.
Thinking about it: It's now very kernel-specific, other users might like a block signature to make it robust against errors - e.g. in the middle of a .tar.bgz file. I think I'll change that soon and put 0x55aa in front of each block.
Signature · 1 line
Borges sumus. Resistere inutile est.
Quoted 2 lines · Bodo Eggert
Quoted 1 line · Andreas Robinson
On Tue, 2009-03-24 at 23:11 +0100, Bodo Eggert wrote:
[snip]
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Why not make a new set? http://7eggert.dyndns.org:8080/l/bgz.tar.gz
Hmm, well if you think there is a chance of it being accepted, then sure! I'll cook something up, but will be looking to use LZO first since it's a bit faster to begin with.
It's small enough, and now I've updated it to support lzo and put a license into the files.
The file permissions are wrong, I'm not allowed to download it.
Quoted 4 lines
Thinking about it: It's now very kernel-specific, other users might like a block signature to make it robust against errors - e.g. in the middle of a .tar.bgz file. I think I'll change that soon and put
0x55aain front of each block.
I think we'll want an optional checksum of the compressed data. Suppose bit 32 in the compressed block size field is set, the next field is the block's Adler32 checksum.
(the adler32 source is in include/linux/zutil.h)
Quoted 8 lines · Andreas Robinson
Quoted 2 lines · Bodo Eggert
Quoted 1 line · Andreas Robinson
On Tue, 2009-03-24 at 23:11 +0100, Bodo Eggert wrote:
[snip]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Why not make a new set? http://7eggert.dyndns.org:8080/l/bgz.tar.gz
The file permissions are wrong, I'm not allowed to download it.
fixed
Quoted 9 lines
Quoted 4 lines
Thinking about it: It's now very kernel-specific, other users might like a block signature to make it robust against errors - e.g. in the middle of a .tar.bgz file. I think I'll change that soon and put
0x55aain front of each block.I think we'll want an optional checksum of the compressed data. Suppose bit 32 in the compressed block size field is set, the next field is the block's Adler32 checksum.
(the adler32 source is ininclude/linux/zutil.h)
I will think about it.
Signature · 2 lines
Funny quotes: 5. 99 percent of lawyers give the rest a bad name.
Here is a quick and dirty patch that works for me. Applies to current git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-x86.git
Work on monolithic-kernel decompression and multithreading is next.
Quoted 1 line
From
27d64e5fca08d695282da219eb71e5fca978a9d2Mon Sep 17 00:00:00 2001
From: Andreas Robinson <[email protected]>
Date: Thu, 26 Mar 2009 16:44:14 +0100
Subject: [PATCH] lib: support lzo-compressed initramfs images
Impact: Allows faster unpacking of the initial ramdisk.
LZO files are marginally larger than gzip, about 7% for the best compression method lzo1x_999, but decompress about 30% faster on x86 hardware.
This patch supports a new stream format generated by a userspace tool named blzo.
Patch content:
* necessary additions to Kconfig and makefiles
* a simple initramfs decompressor
--- include/linux/decompress/ublzo.h | 12 ++++++ lib/Kconfig | 4 ++ lib/Makefile | 1 + lib/decompress.c | 5 +++ lib/decompress_ublzo.c | 75 ++++++++++++++++++++++++++++++++++++++ usr/Kconfig | 9 +++++ 6 files changed, 106 insertions(+), 0 deletions(-) create mode 100644 include/linux/decompress/ublzo.h create mode 100644 lib/decompress_ublzo.c
include/linux/decompress/ublzo.h new file +12 -0
@@ -0,0 +1,12 @@ +#ifndef DECOMPRESS_UBLZO_H +#define DECOMPRESS_UBLZO_H + +int ublzo (unsigned char *,int , +int (*fill)(void *,unsigned int ), +int (*flush)(void *,unsigned int ), +unsigned char *output, +int *posp, +void (*error)(char *x) + ); + +#endif
lib/Kconfig +4 -0
@@ -104,6 +104,10 @@ config LZO_DECOMPRESS # These all provide a common interface (hence the apparent duplication with # ZLIB_INFLATE; DECOMPRESS_GZIP is just a wrapper.) # +config DECOMPRESS_BLZO + select LZO_DECOMPRESS + tristate + config DECOMPRESS_GZIP select ZLIB_INFLATE tristate
lib/Makefile +1 -0
@@ -65,6 +65,7 @@ obj-$(CONFIG_REED_SOLOMON) += reed_solomon/ obj-$(CONFIG_LZO_COMPRESS) += lzo/ obj-$(CONFIG_LZO_DECOMPRESS) += lzo/ +lib-$(CONFIG_DECOMPRESS_BLZO) += decompress_ublzo.o lib-$(CONFIG_DECOMPRESS_GZIP) += decompress_inflate.o lib-$(CONFIG_DECOMPRESS_BZIP2) += decompress_bunzip2.o lib-$(CONFIG_DECOMPRESS_LZMA) += decompress_unlzma.o
lib/decompress.c +5 -0
@@ -9,10 +9,14 @@#include <linux/decompress/bunzip2.h> #include <linux/decompress/unlzma.h> #include <linux/decompress/inflate.h> +#include <linux/decompress/ublzo.h> #include <linux/types.h> #include <linux/string.h> +#ifndef CONFIG_DECOMPRESS_BLZO +# define ublzo NULL +#endif #ifndef CONFIG_DECOMPRESS_GZIP # define gunzip NULL#endif
@@ -28,6 +32,7 @@ static const struct compress_format {const char *name;decompress_fn decompressor; } compressed_formats[] = { + { {'B', 'L'}, "blzo", ublzo }, { {037, 0213}, "gzip", gunzip }, { {037, 0236}, "gzip", gunzip }, { {0x42, 0x5a}, "bzip2", bunzip2 },
lib/decompress_ublzo.c new file +75 -0
75 lines not on this page · raw message
usr/Kconfig +9 -0
@@ -45,6 +45,15 @@ config INITRAMFS_ROOT_GID If you are not sure, leave it set to "0". +config RD_BLZO + bool "Initial ramdisk compressed using blzo" + default y + depends on BLK_DEV_INITRD=y + select DECOMPRESS_BLZO + help + Support loading of a blzo encoded initial ramdisk or cpio buffer. + If unsure, say Y. + config RD_GZIP bool "Initial ramdisk compressed using gzip" default y
Signature · 1 line
1.5.6.3