Easy trick to reduce kernel footprint
Subsystems (3)
Hi,
Here's a simple trick for all those who try to squeeze their kernels to the absolute smallest size.
I recently discovered p7zip which comes with the LZMA compression algorithm, which is somewhat better than gzip and bzip2 on most datasets, and I also noticed that this tool provides support for gzip and bzip2 outputs. So I tried to produce some of those standard outputs, and observed a slight gain compared to the default tools. The reason is that we can change the number of passes and the dictionnary size.
So as an experiment, I used it to compress my kernel+initramfs and I could gain about 2% (23 kB) which is not bad at all for embedded systems. Don't ask about the '.' after '-si' in the '7za' command line, it's just that the tool expects a file name, and I didn't managed to fix it, the source is some sort of obfuscated c++ code (who said "pleonasm" ?) ported from windows, but at least it works. Out of curiosity, I also tried both bzip2 and the LZMA algorithm, although that's not fair because the decompressor code would have to be changed to support them.
Size of the 2.6.12-rc4-mm2 vmlinux compiled without the initramfs image :
bash-3.00$ size vmlinux
text data bss dec hex filename
1024458 123712 73580 1221750 12a476 vmlinux
Size of the initramfs :
-rw-rw-r-- 1 willy users 1014272 2005-05-27 16:31 usr/initramfs_data.cpio
Now the resulting bzImage size with various compression methods
size command 1197277 make 1197275 make cmd_gzip="gzip -9c <\$< >\$@" 1173550 make cmd_gzip="7za a -tgzip -mx9 -mpass=4 -so -si . <\$< >\$@" 1207599 make cmd_gzip="bzip2 -c9 <\$< >\$@" 1051705 make cmd_gzip="rm -f \$@;7za a -t7z -mx9 -md=64m -si \$@ < \$<"
Surprizingly, bzip2 makes a bigger kernel. The gzip implementation in 7za saves 23 kB (2%) on the overall image without touching any code. The LZMA implementation could save 145 kB (12%), but would require a different extraction code (I've already seen patches to bring LZMA support on 2.4).
Note: I don't know who implemented the if_changed macro in the makefile which allows us to pass cmd_gzip here, but it was an excellent idea.
Regards,
Willy
Quoted 5 lines · Willy Tarreau
I recently discovered p7zip which comes with the LZMA compression algorithm, which is somewhat better than gzip and bzip2 on most datasets, and I also noticed that this tool provides support for gzip and bzip2 outputs. So I tried to produce some of those standard outputs, and observed a slight gain compared to the default tools.
But the kernel gunzip can't decompress everything that the normal gzip could decompress, i found some options it didn't work with. (but this was long ago when there wasn't a linux version and i used wine for 7za, but i don't think anyone even thinks about fixing gunzip())
Quoted 4 lines
Surprizingly, bzip2 makes a bigger kernel. The gzip implementation in 7za saves 23 kB (2%) on the overall image without touching any code. The LZMA implementation could save 145 kB (12%), but would require a different extraction code (I've already seen patches to bring LZMA support on 2.4).
Somehow this is not so easy, i'll post some stuff in another mail, perhaps there is just a stupid tiny error.
But at least for initrds lzma works nicely, therefore a patch:
This patch adds the possibility to compress a initrd with lzma, you need ths CONFIG_BLK_DEV_RAM_LZMA build option for this. (Device Drivers -> Block devices)
For example the debian installer initrd:
-rw-r--r-- 1 ijuz ijuz 2870355 Mar 5 20:00 initrd.gz -rw-r--r-- 1 ijuz ijuz 2158769 May 8 02:09 initrd.lzma
Building the lzma tool cut and paste style:
mkdir build-lzma
wget http://www.7-zip.org/dl/lzma417.tar.bz2
tar xjf lzma417.tar.bz2
cd SRC/7zip/Compress/LZMA_Alone/
make
cp lzma ../../../../../
cd ../../../../../
drivers/block/Kconfig +37 -0
@@ -398,6 +398,43 @@ what are you doing. If you are using IBM S/390, then set this to 8192. +config BLK_DEV_RAM_GZ + bool "Gzip compressed ramdisk support" + depends on BLK_DEV_RAM=y || BLK_DEV_RAM=m + default y + help + This option enables support for gzip compressed ramdisk images. + An image can be loaded as initrd (see above) or as a normal + ramdisk at boot time. + Enabling this option is good for people who use an initrd or + a normal ramdisk and who are low on available disk space. + The decompression process at boot time needs about 2 MB additional + RAM to the space the unpacked ramdisk needs. + + Due to the fact that this feature is normally included in the + original ramdisk, you should say Y to this option. + +config BLK_DEV_RAM_LZMA + bool "Lzma compressed ramdisk support (EXPERIMENTAL)" + depends on BLK_DEV_RAM=y || BLK_DEV_RAM=m && EXPERIMENTAL + default n + help + This option enables support for lzma compressed ramdisk images. + An image can be loaded as initrd (see above) or as a normal + ramdisk at boot time. + Enabling this option is good for people who use an initrd or + a normal ramdisk and who are very low on available disk space + (eg. for embedded systems with rom image and plenty of RAM). + This option was implemented, because lzma has better compression + than gzip. If you want to use this feature and disk space is a + matter, say N to gzip support and compress your ramdisk using + lzma. + + This is an optional feature. Only enable this option if you need + lzma support for your ramdisk (e.g. for boot+root floppies). So + if you are unsure say N. + + config BLK_DEV_INITRD bool "Initial RAM disk (initrd) support" depends on BLK_DEV_RAM=y
init/do_mounts.h +32 -0
@@ -90,3 +90,35 @@static inline void md_run_setup (void ) {}#endif + + +/* +* +* The following lines define some constants to support different +* compressed ramdisk types. +* Each compressed ramdisk type is bound to a _negative_ value, +* since positive values give us the number of blocks. +* +*/ + +#define RD_ERROR (-65535) +#define CRAMDISK_LZMA (-1) +#define CRAMDISK_GZ (-2) + +#if defined(CONFIG_BLK_DEV_RAM_GZ ) || defined(CONFIG_BLK_DEV_RAM_LZMA ) +# define BUILD_CRAMDISK +#endif /* (CONFIG_BLK_DEV_RAM_GZ) || defined(CONFIG_BLK_DEV_RAM_BZ2) */ + +#ifdef CONFIG_BLK_DEV_RAM_GZ +int __init gz_load (int in_fd,int out_fd); +#else +static inline int __init gz_load (int in_fd,int out_fd) {return RD_ERROR ; } +#endif /* CONFIG_BLK_DEV_RAM_GZ */ + +#ifdef CONFIG_BLK_DEV_RAM_LZMA +int __init lzma_load (int in_fd,int out_fd); +#else +static inline int __init lzma_load (int in_fd,int out_fd) {return RD_ERROR ; } +#endif /* CONFIG_BLK_DEV_RAM_BZ2 */ + +
init/do_mounts_rd.c +189 -29
282 lines not on this page · raw message
lib/LzmaDecode.c +586 -0
586 lines not on this page · raw message
lib/LzmaDecode.h +100 -0
100 lines not on this page · raw message
Signature · 1 line
http://www.nosoftwarepatents.com
Quoted 9 lines · Willy Tarreau
Here's a simple trick for all those who try to squeeze their kernels to the absolute smallest size.
I recently discovered p7zip which comes with the LZMA compression algorithm, which is somewhat better than gzip and bzip2 on most datasets, and I also noticed that this tool provides support for gzip and bzip2 outputs. So I tried to produce some of those standard outputs, and observed a slight gain compared to the default tools. The reason is that we can change the number of passes and the dictionnary size.
Is it any smaller than a UPX'd kernel? (I think you need the beta version. I know the upx-ucl in debian won't compress but upx-ucl-beta will if you force). I got a significant reduction using it.
Signature · 1 line
Lab tests show that use of micro$oft causes cancer in lab animals
Quoted 11 lines · Wakko Warner
Quoted 9 lines · Willy Tarreau
Here's a simple trick for all those who try to squeeze their kernels to the absolute smallest size.
I recently discovered p7zip which comes with the LZMA compression algorithm, which is somewhat better than gzip and bzip2 on most datasets, and I also noticed that this tool provides support for gzip and bzip2 outputs. So I tried to produce some of those standard outputs, and observed a slight gain compared to the default tools. The reason is that we can change the number of passes and the dictionnary size.
Quoted 3 lines
Is it any smaller than a UPX'd kernel? (I think you need the beta version. I know the upx-ucl in debian won't compress but upx-ucl-beta will if you force). I got a significant reduction using it.
It's not better at all, but unfortunately, UPX cannot compress a kernel which embeds a big initramfs. The problem is that the compressed initramfs is embedded into vmlinux, which is then compressed into bzImage, and UPX only replaces the bzImage compression.
May be it would work if we did not compress the initramfs before including it into the vmlinux.
Willy
Keep me CC.
Quoted 5 lines · Willy Tarreau
Quoted 3 lines · Wakko Warner
Is it any smaller than a UPX'd kernel? (I think you need the beta version. I know the upx-ucl in debian won't compress but upx-ucl-beta will if you force). I got a significant reduction using it.
Quoted 7 lines
It's not better at all, but unfortunately, UPX cannot compress a kernel which embeds a big initramfs. The problem is that the compressed initramfs is embedded into vmlinux, which is then compressed into bzImage, and UPX only replaces the bzImage compression.
May be it would work if we did not compress the initramfs before including it into the vmlinux.
My initramfs is passed via initrd so that I can change any aspect of it with out recompiling the kernel (or maybe i could use a better understanding of initramfs) I compared bzImage to bupxImage and the savings I got was around 50kb difference IIRC.
Signature · 1 line
Lab tests show that use of micro$oft causes cancer in lab animals
Quoted 4 lines · Wakko Warner
My initramfs is passed via initrd so that I can change any aspect of it with out recompiling the kernel (or maybe i could use a better understanding of initramfs) I compared bzImage to bupxImage and the savings I got was around 50kb difference IIRC.
With what algo and what kernel size ?
With the close-source upx-1.93 linked with the NRV compression, I often observe an average 15-20% gain on the bzImage size if it does not embed an initramfs. I remember that the UCL library was not as good as the NRV, so I've stopped using it a long time ago.
Regards,
Willy
Quoted 7 lines · Willy Tarreau
Quoted 4 lines · Wakko Warner
My initramfs is passed via initrd so that I can change any aspect of it with out recompiling the kernel (or maybe i could use a better understanding of initramfs) I compared bzImage to bupxImage and the savings I got was around 50kb difference IIRC.
Quoted 5 lines
With what algo and what kernel size ?
With the close-source upx-1.93 linked with the NRV compression, I often observe an average 15-20% gain on the bzImage size if it does not embed an initramfs. I remember that the UCL library was not as good as the NRV, so I've stopped using it a long time ago.
I'm using upx-ucl-beta v1.91+0.20030910cvs-2 debian package. Command line:
upx-ucl-beta -9 -f bupxImage-2.6.12-rc6
-rw-r--r-- 3 root root 618442 Jun 6 16:14 bupxImage-2.6.12-rc6 -rwxrwxrwx 1 root root 669106 Jun 6 16:14 bzImage-2.6.12-rc6*
50664 bytes savings.
the kernel I have there is 100% modular except that ramdisk is compiled in. It was also compiled with -Os (Optimize for size config option) using gcc 3.3.5.
Signature · 1 line
Lab tests show that use of micro$oft causes cancer in lab animals
Quoted 3 lines · Willy Tarreau
saves 23 kB (2%) on the overall image without touching any code. The LZMA implementation could save 145 kB (12%), but would require a different extraction code (I've already seen patches to bring LZMA support on 2.4).
The patch for 2.4 is here:
http://www.zelow.no/floppyfw/download/Development/Patches/kernel/040-lzma-vmlinuz.diff
and it actually works.
My not working patch is here:
http://debian.christian-leber.de/kernel_lzma/lzma_image.patch
you have to apply the patch from the other mail before.
(or here: http://debian.christian-leber.de/kernel_lzma/lzma_ramdisk.patch)
If you just want to see how it doesn't work, here is a kernel image to see it failing in qemu, basically it just decompresses again and again.
http://debian.christian-leber.de/kernel_lzma/bzImage
The decompression works correctly(checksum), i think the problem is the address where the decompressed kernel image should be, i just can't get out how this piece of code is supposed to work.
Therefore it could be a small problem only.
Another minor problem of lzma is that you need slightly more memory than with gzip. (at least when you can't have the output in a continuous piece of memory)
On the plus side is also that the lzma decompression code itself is smaller and doesn't have such a ugly "interface".
Christian Leber
Signature · 1 line
http://www.nosoftwarepatents.com
Quoted 3 lines · Willy Tarreau
I recently discovered p7zip which comes with the LZMA compression algorithm, which is somewhat better than gzip and bzip2 on most datasets, [...]
Hmmm.
Citeseer has never heard of that algorithm, top 10 google hits for "LZMA compression algorithm" are completely uninformative. Does anyone actually know, what this algorithm is doing?
Jörn
Signature · 4 lines
Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. -- Brian W. Kernighan
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Jörn Engel wrote:
| On Mon, 6 June 2005 00:35:28 +0200, Willy Tarreau wrote: | |>I recently discovered p7zip which comes with the LZMA compression algorithm, |>which is somewhat better than gzip and bzip2 on most datasets, [...] | | | Hmmm. | | Citeseer has never heard of that algorithm, top 10 google hits for | "LZMA compression algorithm" are completely uninformative. Does | anyone actually know, what this algorithm is doing? | | Jörn |
Dear Jörn
i just googled it up, and the first link ive found is:
http://martinus.geekisp.com/rublog.cgi/Projects/LZMA
might be that one...
Greetings
Oliver Leitner
Technical Staff
http://www.shells.at
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (MingW32)
iD8DBQFCo+VdWvEVE8MtwbgRAoKEAJ4pbKdAYMCd1yJnZzCdp8EfZqfCZACglVOR Q9bUv3wMpj1xCkdZS3eAiDg=
=fD7R
-----END PGP SIGNATURE-----
Quoted 10 lines · J?rn Engel
Quoted 3 lines · Willy Tarreau
I recently discovered p7zip which comes with the LZMA compression algorithm, which is somewhat better than gzip and bzip2 on most datasets, [...]
Hmmm.
Citeseer has never heard of that algorithm, top 10 google hits for "LZMA compression algorithm" are completely uninformative. Does anyone actually know, what this algorithm is doing?
It's described here :
http://en.wikipedia.org/wiki/LZMA
implemented here :
http://martinus.geekisp.com/rublog.cgi/Projects/LZMA
and here :
http://www.7-zip.org/sdk.html
Willy
Quoted 6 lines · Willy Tarreau
Quoted 4 lines · J?rn Engel
Citeseer has never heard of that algorithm, top 10 google hits for "LZMA compression algorithm" are completely uninformative. Does anyone actually know, what this algorithm is doing?
Quoted 8 lines
It's described here :
http://en.wikipedia.org/wiki/LZMAimplemented here :
http://martinus.geekisp.com/rublog.cgi/Projects/LZMAand here :
http://www.7-zip.org/sdk.html
Thanks, but I already saw all three of those before I posted my reply. Guess the only way to tell is by reading the source...
Jörn
Signature · 4 lines
Correctness comes second. Features come third. Performance comes last. Maintainability is needed for all of them.
Quoted 3 lines · J?rn Engel
Citeseer has never heard of that algorithm, top 10 google hits for "LZMA compression algorithm" are completely uninformative. Does anyone actually know, what this algorithm is doing?
Doesn't google have a searching howto or something?
First hit:
http://www.7-zip.org/sdk.html
Christian Leber
Signature · 1 line
http://www.nosoftwarepatents.com