Skip to contentSkip to conversation

fastboot: unpacking initramfs faster

DISCUSSION Bodo Eggert 2 participants 6 messages Mar 24, 2009 – Mar 26, 2009 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 4 lines · Andreas Robinson

I tried running the initramfs unpacking asynchronously (and earlier) but ran into a curious problem: The parent thread stalled until decompression was done. I don't know why - I made sure that the two threads ran on different cores. Perhaps the parent was cache starved.

I could not stall my system by gzip</dev/urandom>/dev/null. Maybe you accidentally used vfork?

Quoted 8 lines

I then looked at multithreaded decompression and found that neither gzip files nor deflate streams provide enough information (eg block lengths) to let you share the decompression among threads.

It can be done of course, but obviously not without modifying file formats and userspace tools.

(Thoughts of snowballs and hell presented themselves at this point.)

Why not make a new set? http://7eggert.dyndns.org:8080/l/bgz.tar.gz

#2 lore raw
Quoted 9 lines · Bodo Eggert
Quoted 4 lines · Andreas Robinson

I tried running the initramfs unpacking asynchronously (and earlier) but ran into a curious problem: The parent thread stalled until decompression was done. I don't know why - I made sure that the two threads ran on different cores. Perhaps the parent was cache starved.

I could not stall my system by gzip</dev/urandom>/dev/null. Maybe you accidentally used vfork?

No, this was an in-kernel test. I used the Arjan's new async facility which calls kthread_create().

Quoted 10 lines
Quoted 8 lines

I then looked at multithreaded decompression and found that neither gzip files nor deflate streams provide enough information (eg block lengths) to let you share the decompression among threads.

It can be done of course, but obviously not without modifying file formats and userspace tools.

(Thoughts of snowballs and hell presented themselves at this point.)

Why not make a new set? http://7eggert.dyndns.org:8080/l/bgz.tar.gz

Hmm, well if you think there is a chance of it being accepted, then sure! I'll cook something up, but will be looking to use LZO first since it's a bit faster to begin with.

#3 lore raw
Quoted 10 lines · Andreas Robinson
Quoted 9 lines · Bodo Eggert
Quoted 7 lines · Andreas Robinson

I then looked at multithreaded decompression and found that neither gzip files nor deflate streams provide enough information (eg block lengths) to let you share the decompression among threads.

It can be done of course, but obviously not without modifying file formats and userspace tools.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

(Thoughts of snowballs and hell presented themselves at this point.)

Why not make a new set? http://7eggert.dyndns.org:8080/l/bgz.tar.gz

Hmm, well if you think there is a chance of it being accepted, then sure! I'll cook something up, but will be looking to use LZO first since it's a bit faster to begin with.

It's small enough, and now I've updated it to support lzo and put a license into the files.

Thinking about it: It's now very kernel-specific, other users might like a block signature to make it robust against errors - e.g. in the middle of a .tar.bgz file. I think I'll change that soon and put 0x55aa in front of each block.

Signature · 1 line
Borges sumus. Resistere inutile est.
#4 lore raw
Quoted 2 lines · Bodo Eggert
Quoted 1 line · Andreas Robinson

On Tue, 2009-03-24 at 23:11 +0100, Bodo Eggert wrote:

[snip]

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Why not make a new set? http://7eggert.dyndns.org:8080/l/bgz.tar.gz

Hmm, well if you think there is a chance of it being accepted, then sure! I'll cook something up, but will be looking to use LZO first since it's a bit faster to begin with.

It's small enough, and now I've updated it to support lzo and put a license into the files.

The file permissions are wrong, I'm not allowed to download it.

Quoted 4 lines

Thinking about it: It's now very kernel-specific, other users might like a block signature to make it robust against errors - e.g. in the middle of a .tar.bgz file. I think I'll change that soon and put 0x55aa in front of each block.

I think we'll want an optional checksum of the compressed data. Suppose bit 32 in the compressed block size field is set, the next field is the block's Adler32 checksum.
(the adler32 source is in include/linux/zutil.h)

#5 lore raw
Quoted 8 lines · Andreas Robinson
Quoted 2 lines · Bodo Eggert
Quoted 1 line · Andreas Robinson

On Tue, 2009-03-24 at 23:11 +0100, Bodo Eggert wrote:

[snip]

Quoted 1 line
Quoted 1 line
Quoted 1 line

Why not make a new set? http://7eggert.dyndns.org:8080/l/bgz.tar.gz

The file permissions are wrong, I'm not allowed to download it.

fixed

Quoted 9 lines
Quoted 4 lines

Thinking about it: It's now very kernel-specific, other users might like a block signature to make it robust against errors - e.g. in the middle of a .tar.bgz file. I think I'll change that soon and put 0x55aa in front of each block.

I think we'll want an optional checksum of the compressed data. Suppose bit 32 in the compressed block size field is set, the next field is the block's Adler32 checksum.
(the adler32 source is in include/linux/zutil.h)

I will think about it.

Signature · 2 lines
Funny quotes:
5. 99 percent of lawyers give the rest a bad name.
#6 lore raw

Here is a quick and dirty patch that works for me. Applies to current git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-x86.git

Work on monolithic-kernel decompression and multithreading is next.

Quoted 1 line

From 27d64e5fca08d695282da219eb71e5fca978a9d2 Mon Sep 17 00:00:00 2001

From: Andreas Robinson <[email protected]>
Date: Thu, 26 Mar 2009 16:44:14 +0100
Subject: [PATCH] lib: support lzo-compressed initramfs images

Impact: Allows faster unpacking of the initial ramdisk.

LZO files are marginally larger than gzip, about 7% for the best compression method lzo1x_999, but decompress about 30% faster on x86 hardware.

This patch supports a new stream format generated by a userspace tool named blzo.

Patch content:
* necessary additions to Kconfig and makefiles
* a simple initramfs decompressor

---
 include/linux/decompress/ublzo.h |   12 ++++++
 lib/Kconfig                      |    4 ++
 lib/Makefile                     |    1 +
 lib/decompress.c                 |    5 +++
 lib/decompress_ublzo.c           |   75 ++++++++++++++++++++++++++++++++++++++
 usr/Kconfig                      |    9 +++++
 6 files changed, 106 insertions(+), 0 deletions(-)
 create mode 100644 include/linux/decompress/ublzo.h
 create mode 100644 lib/decompress_ublzo.c
include/linux/decompress/ublzo.h new file +12 -0
new file mode 100644
index 0000000..e453d07
@@ -0,0 +1,12 @@
+#ifndef DECOMPRESS_UBLZO_H
+#define DECOMPRESS_UBLZO_H
+
+int ublzo(unsigned char *, int,
+	  int(*fill)(void*, unsigned int),
+	  int(*flush)(void*, unsigned int),
+	  unsigned char *output,
+	  int *posp,
+	  void(*error)(char *x)
+	);
+
+#endif
lib/Kconfig +4 -0
index e3b7d7a..9e5c7f1 100644
@@ -104,6 +104,10 @@ config LZO_DECOMPRESS
 # These all provide a common interface (hence the apparent duplication with
 # ZLIB_INFLATE; DECOMPRESS_GZIP is just a wrapper.)
 #
+config DECOMPRESS_BLZO
+	select LZO_DECOMPRESS
+	tristate
+
 config DECOMPRESS_GZIP
 	select ZLIB_INFLATE
 	tristate
lib/Makefile +1 -0
index b24222c..9ee48e8 100644
@@ -65,6 +65,7 @@ obj-$(CONFIG_REED_SOLOMON) += reed_solomon/
 obj-$(CONFIG_LZO_COMPRESS) += lzo/
 obj-$(CONFIG_LZO_DECOMPRESS) += lzo/
 
+lib-$(CONFIG_DECOMPRESS_BLZO) += decompress_ublzo.o
 lib-$(CONFIG_DECOMPRESS_GZIP) += decompress_inflate.o
 lib-$(CONFIG_DECOMPRESS_BZIP2) += decompress_bunzip2.o
 lib-$(CONFIG_DECOMPRESS_LZMA) += decompress_unlzma.o
lib/decompress.c +5 -0
index d2842f5..8c189a0 100644
@@ -9,10 +9,14 @@
 #include <linux/decompress/bunzip2.h>
 #include <linux/decompress/unlzma.h>
 #include <linux/decompress/inflate.h>
+#include <linux/decompress/ublzo.h>
 
 #include <linux/types.h>
 #include <linux/string.h>
 
+#ifndef CONFIG_DECOMPRESS_BLZO
+# define ublzo NULL
+#endif
 #ifndef CONFIG_DECOMPRESS_GZIP
 # define gunzip NULL
 #endif
@@ -28,6 +32,7 @@ static const struct compress_format {
 	const char *name;
 	decompress_fn decompressor;
 } compressed_formats[] = {
+	{ {'B', 'L'}, "blzo", ublzo },
 	{ {037, 0213}, "gzip", gunzip },
 	{ {037, 0236}, "gzip", gunzip },
 	{ {0x42, 0x5a}, "bzip2", bunzip2 },
lib/decompress_ublzo.c new file +75 -0

75 lines not on this page · raw message

usr/Kconfig +9 -0
index 43a3a0f..ff2b15c 100644
@@ -45,6 +45,15 @@ config INITRAMFS_ROOT_GID
 
 	  If you are not sure, leave it set to "0".
 
+config RD_BLZO
+	bool "Initial ramdisk compressed using blzo"
+	default y
+	depends on BLK_DEV_INITRD=y
+	select DECOMPRESS_BLZO
+	help
+	  Support loading of a blzo encoded initial ramdisk or cpio buffer.
+	  If unsure, say Y.
+
 config RD_GZIP
 	bool "Initial ramdisk compressed using gzip"
 	default y
Signature · 1 line
1.5.6.3