Add support for ZSTD-compressed kernel and initramfs
A message that is not in the archive (sent off-list or not yet received): [email protected]
Kees Cook
Reviewed-byQuoted 96 lines · Nick Terrell
Hi all,
This patch set adds support for a ZSTD-compressed kernel, ramdisk, and initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs are supported on all architectures. The ZSTD-compressed kernel is only hooked up to x86 in this patch set.
Zstandard requires slightly more memory during the kernel decompression on x86 (192 KB vs 64 KB), and the memory usage is independent of the window size.
Zstandard requires memory proprortional to the window size used during compression for decompressing the ramdisk image, since streaming mode is used. Newer versions of zstd (1.3.2+) list the window size of a file with `zstd -lv <file>'. The absolute maximum amount of memory required is just over 8 MB, but it can be controlled at compression time.
This patch set has been boot tested with buildroot and QEMU based off of linux-5.6-rc6.
On i386 and
x86_64I have tested the following configurations:
* zstd compressed kernel and a separate zstd compressed initramfs
* zstd compressed kernel and a built-in zstd compressed initramfs
* gzip compressed kernel and a separate gzip compressed initramfs
* gzip compressed kernel and a built-in gzip compressed initramfsOn arm and aarch64 I tested the same configurations, except that the kernel is always gzip compressed.
Facebook has been using v1 of these patches on
x86_64devices for more than 6 months. When we switched from a xz compressed initramfs to a zstd compressed initramfs decompression time shrunk from 12 seconds to 3 seconds. When we switched from a xz compressed kernel to a zstd compressed kernel we saved 2 seconds of boot time.Facebook has been using v2 of these patches on aarch64 devices for a few weeks. When we switched from an lzma compressed initramfs to a zstd compressed initramfs decompression time shrunk from 27 seconds to 8 seconds.
The zstd compressed kernel is smaller than the gzip compressed kernel but larger than the xz or lzma compressed kernels, and it decompresses faster than everything except lz4. See the table below for the measurement of an
x86_64kernel ordered by compressed size:algo size
xz 6,509,792
lzma 6,856,576
zstd 7,399,157
gzip 8,522,527
bzip 8,629,603
lzo 9,808,035
lz4 10,705,570
none 32,565,672v1 -> v2:
- Rebase
- usr/Makefile and init/Kconfig were changed so the patches were updated
- No functional changes except to rebase
- Split the patches up into smaller chunksv2 -> v3:
- Add *.zst to the .gitignore in patch 8
- Style nits in patch 3
- Rename the PREBOOT macro toZSTD_PREBOOTandXXH_PREBOOTin patches 1 through 3Best,
Nick TerrellAdam Borowski (1):
.gitignore: add ZSTD-compressed filesNick Terrell (7):
lib: prepare zstd for preboot environment
lib: prepare xxhash for preboot environment
lib: add zstd support to decompress
init: add support for zstd compressed kernel
usr: add support for zstd compressed initramfs
x86: bumpZO_z_extra_bytesmargin for zstd
x86: Add support for ZSTD compressed kernel.gitignore | 1 + Documentation/x86/boot.rst | 6 +- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/Makefile | 5 +- arch/x86/boot/compressed/misc.c | 4 + arch/x86/boot/header.S | 8 +- arch/x86/include/asm/boot.h | 6 +- include/linux/decompress/unzstd.h | 11 + init/Kconfig | 15 +- lib/Kconfig | 4 + lib/Makefile | 1 + lib/decompress.c | 5 + lib/decompress_unzstd.c | 338 ++++++++++++++++++++++++++++++ lib/xxhash.c | 21 +- lib/zstd/decompress.c | 2 +
Quoted 8 lines
lib/zstd/fse_decompress.c | 9 +- lib/zstd/zstd_internal.h | 14 +- scripts/Makefile.lib | 15 ++ usr/Kconfig | 20 ++ usr/Makefile | 1 + 20 files changed, 460 insertions(+), 27 deletions(-) create mode 100644 include/linux/decompress/unzstd.h create mode 100644 lib/decompress_unzstd.c
This looks good to me! Please consider the series:
Thanks!
-Kees
Signature · 1 line
Kees Cook