Skip to contentSkip to conversation

Add support for ZSTD-compressed kernel

PATCH RESEND New Adam Borowski 4 participants 16 messages Mar 22, 2018 – Jun 10, 2019 3 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 3 lines · Nick Terrell

This patch set adds support for a ZSTD-compressed kernel and ramdisk images in the kernel boot process. It only integrates the support with x86, though the first patch is generic to all architectures.

I'm running this patch set since October on amd64 armhf arm64, no explosions -- besides the obvious when it's not applied yet I forget to reconfigure initramfs-tools. Which is getting tedious, so let's merge this already. :)

I've tested initrd on all of these archs; here's a debug patch to check if you're actually using it.

As for compressing the kernel itself: the second patch works as submitted (ie, x86 only). Porting it to other architectures is straightforward, but eg. on arm, most boards use u-boot which insists on decompressing the kernel by itself instead of passing control. EFI/grub should work, but I haven't tested that yet.

Meow!

Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄⠀⠀⠀⠀ A master species delegates.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 3 lines · René Rebe

Should this currently just work without any arch change on e.g. ppc64, sparc64 et al.? I could do a test build and boot if that is of any value, ...

Initrd: no reason it wouldn't work, although for anything related to the boot process testing is a very good idea. If you test, you'd really want to apply that printk patch I posted, it's too easy to accidentally get a silent fallback to gzip or uncompressed. I wonder, perhaps such a printk could be permanently added, at a low message priority?

Kernel itself: needs per-arch porting, but it's not advertised in kconfig outside x86 so that's not a problem.

If you are knowledgeful about bootloaders on ppc64, sparc64, etc, such information would be great.

Meow!

Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄⠀⠀⠀⠀ A master species delegates.
#4 lore raw

Hi,

Quoted 10 lines · Adam Borowski
Quoted 3 lines · René Rebe

Should this currently just work without any arch change on e.g. ppc64, sparc64 et al.? I could do a test build and boot if that is of any value, ...

Initrd: no reason it wouldn't work, although for anything related to the boot process testing is a very good idea. If you test, you'd really want to apply that printk patch I posted, it's too easy to accidentally get a silent fallback to gzip or uncompressed. I wonder, perhaps such a printk could be permanently added, at a low message priority?

zstd initrd worked on sparc64/sun4u - Ultra 30:

	https://www.youtube.com/watch?v=10q2OxHAzQ4

Any chance we could get this initrd zstd support upstream?

Quoted 5 lines

Kernel itself: needs per-arch porting, but it's not advertised in kconfig outside x86 so that's not a problem.

If you are knowledgeful about bootloaders on ppc64, sparc64, etc, such information would be great.

Signature · 2 lines
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
 http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
#5 lore raw
In reply to a message not in the archive
Quoted 4 lines · Nick Terrell

This patch set adds support for a ZSTD-compressed kernel and ramdisk images in the kernel boot process. It only integrates the support with x86, though the first patch is generic to all architectures.

Hi all,

Is there anything blocking this from getting merged?
Please let me know if there is anything I can do to help move this patch forward.

Thanks,
Nick

#6 lore raw

Hi Nick,

Quoted 9 lines · Nick Terrell
Quoted 4 lines · Nick Terrell

This patch set adds support for a ZSTD-compressed kernel and ramdisk images in the kernel boot process. It only integrates the support with x86, though the first patch is generic to all architectures.

Hi all,

Is there anything blocking this from getting merged?
Please let me know if there is anything I can do to help move this patch forward.

works fine on my side, initrd test booted on i486, x86-64, ppc, ppc64 and sparc64. For testing on mips64 I would first need to re-base the Octane^1 patch-set ;-)

Greetings,
René

^1) https://www.youtube.com/watch?v=AU_RV8uoTIo

Signature · 2 lines
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
 http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
#7 lore raw

Hey,

is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk

René

Quoted 27 lines · René Rebe

Hi Nick,

Quoted 9 lines · Nick Terrell
Quoted 4 lines · Nick Terrell

This patch set adds support for a ZSTD-compressed kernel and ramdisk images in the kernel boot process. It only integrates the support with x86, though the first patch is generic to all architectures.

Hi all,

Is there anything blocking this from getting merged?
Please let me know if there is anything I can do to help move this patch forward.

works fine on my side, initrd test booted on i486, x86-64, ppc, ppc64 and sparc64. For testing on mips64 I would first need to re-base the Octane^1 patch-set ;-)

Greetings,
René

^1) https://www.youtube.com/watch?v=AU_RV8uoTIo

Signature · 2 lines
ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
Signature · 4 lines
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
 DE Legal: Amtsgericht Berlin (Charlottenburg) HRB 105123B, Tax-ID#: DE251602478
 Managing Director: René Rebe
 http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
#8 lore raw
Quoted 3 lines · René Rebe

Hey,

is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk

Can you remove some other compressor for the kernel image if you merge this?

The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.

Surely zstd is universally better than some existing compressor the kernel already uses for itself for a particular sweet spot? If it isn't it's likely not needed.

-Andi

#9 lore raw

Hi,

Quoted 9 lines · Andi Kleen
Quoted 3 lines · René Rebe

Hey,

is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk

Can you remove some other compressor for the kernel image if you merge this?

The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.

Well, we (t2 formerly known as rocklinux) only changed our pkg archive compression from bzip2 to zstd once in 20 years ,-)

Quoted 3 lines

Surely zstd is universally better than some existing compressor the kernel already uses for itself for a particular sweet spot? If it isn't it's likely not needed.

Well, IMHO this is not that much code, but if you want to remove something I guess bzip2 or LZMA may be candidates.

	http://zstd.net
Signature · 2 lines
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
 http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
#10 lore raw
In reply to Andi Kleen
Quoted 8 lines · Andi Kleen
Quoted 5 lines · René Rebe

Hey,

is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc

Can you remove some other compressor for the kernel image if you merge this?

Funny you say this. I want to post this after the merge window is over:
https://github.com/kilobyte/linux/commits/nobz2-next-20180731

Sorry for a tree atop next, but there are conflicts that needed to be adressed. The patchset is also undertested (works for me, but not even compile-tested on archs I don't have a toolchain for at hand).

Quoted 2 lines

The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.

The above patchset drops just bzip2. It is the only one that's strictly beaten in every way (ratio, time, memory usage), there are also no other uses of bzip2 anywhere in the kernel so we'd get to drop its code
completely: 900 lines of Linus' happiness.

Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), but those are used elsewhere thus there's hardly any gain. If you want them gone, please say so -- I'll include their droppage.

Quoted 3 lines

Surely zstd is universally better than some existing compressor the kernel already uses for itself for a particular sweet spot? If it isn't it's likely not needed.

zstd is better in the balanced niche: unless you want extreme speed (lz4) or best compression (xz), zstd handily beats algorithms in the middle.

Meow!

Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]
#11 lore raw
Quoted 5 lines · Adam Borowski
Quoted 2 lines

The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.

The above patchset drops just bzip2. It is the only one that's strictly beaten in every way (ratio, time, memory usage), there are also no other

Does time include build time? I've been reverting back to gzip recently because I care very much about that.

Quoted 2 lines

uses of bzip2 anywhere in the kernel so we'd get to drop its code
completely: 900 lines of Linus' happiness.

Great!

Quoted 4 lines

Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), but those are used elsewhere thus there's hardly any gain. If you want them gone, please say so -- I'll include their droppage.

Yes would be good to remove the kernel image support for those too just to simplify the config process, even if it doesn't save much code.

-Andi

#12 lore raw
Quoted 9 lines · Andi Kleen
Quoted 5 lines · Adam Borowski
Quoted 2 lines

The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.

The above patchset drops just bzip2. It is the only one that's strictly beaten in every way (ratio, time, memory usage), there are also no other

Does time include build time? I've been reverting back to gzip recently because I care very much about that.

Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while ago), here's copypasta of a random 16824672 byte executable, in userspace, with default level setting:

comp decomp size
xz 8.038s 0.356s 4320292
bz2 2.265s 0.730s 5234516
zst 0.274s 0.102s 5657626
gz 0.880s 0.152s 6515505
Z 0.499s 0.133s 8932459
lzo 0.100s 0.095s 9198874

As you can see, zstd's compression time is drastically better than gzip, while ratio is better. The default level is very low (-3 on -1..-22 scale) but you can crank it up for stronger compression.

The defaults fit your use case.

Quoted 11 lines
Quoted 2 lines

uses of bzip2 anywhere in the kernel so we'd get to drop its code
completely: 900 lines of Linus' happiness.

Great!

Quoted 3 lines

Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), but those are used elsewhere thus there's hardly any gain. If you want them gone, please say so -- I'll include their droppage.

Yes would be good to remove the kernel image support for those too just to simplify the config process, even if it doesn't save much code.

There's one caveat: fast choices are quite new:
* lz4 userspace tools are not even in current Debian stable (just unstable)
* uncompressed kernel got in only this merge window
* zstd has userspace tools in Debian stable but is not merged into the kernel yet
(other dists are probably similar)

Thus, it might be a good idea to keep lzo for a while longer.

Bare lzma can probably go -- xz filters are nice for binaries of archs it knows (disabled otherwise), and lack of header requires hacks to find out the payload's size.

So it's up to you guys: do you want me to drop lzo and/or lzma? We can also drop them just for vmlinuz but not initrd.

Meow!

Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]
#13 lore raw
Quoted 48 lines · Adam Borowski
Quoted 9 lines · Andi Kleen
Quoted 5 lines · Adam Borowski
Quoted 2 lines

The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.

The above patchset drops just bzip2. It is the only one that's strictly beaten in every way (ratio, time, memory usage), there are also no other

Does time include build time? I've been reverting back to gzip recently because I care very much about that.

Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while ago), here's copypasta of a random 16824672 byte executable, in userspace, with default level setting:

comp decomp size
xz 8.038s 0.356s 4320292
bz2 2.265s 0.730s 5234516
zst 0.274s 0.102s 5657626
gz 0.880s 0.152s 6515505
Z 0.499s 0.133s 8932459
lzo 0.100s 0.095s 9198874

As you can see, zstd's compression time is drastically better than gzip, while ratio is better. The default level is very low (-3 on -1..-22 scale) but you can crank it up for stronger compression.

The defaults fit your use case.

Quoted 11 lines
Quoted 2 lines

uses of bzip2 anywhere in the kernel so we'd get to drop its code
completely: 900 lines of Linus' happiness.

Great!

Quoted 3 lines

Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), but those are used elsewhere thus there's hardly any gain. If you want them gone, please say so -- I'll include their droppage.

Yes would be good to remove the kernel image support for those too just to simplify the config process, even if it doesn't save much code.

There's one caveat: fast choices are quite new:
* lz4 userspace tools are not even in current Debian stable (just unstable)
* uncompressed kernel got in only this merge window
* zstd has userspace tools in Debian stable but is not merged into the kernel yet
(other dists are probably similar)

Quoted 5 lines

Thus, it might be a good idea to keep lzo for a while longer.

Bare lzma can probably go -- xz filters are nice for binaries of archs it knows (disabled otherwise), and lack of header requires hacks to find out the payload's size.

Personally, I'd be very happy to see LZMA go. It is a custom implementation that doesn't use the lib/xz/ library. When I was implementing decompress_zstd.c I fuzzed all of the kernel decompressors, and unlzma() will crash on invalid input. There is no reason, other than not breaking compatibility, to use LZMA over XZ.

Quoted 10 lines

So it's up to you guys: do you want me to drop lzo and/or lzma? We can also drop them just for vmlinuz but not initrd.

Meow!

Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#15 lore raw
Quoted 10 lines · René Rebe

Hey there,

just wanted to check about the status of zstd-compressed kernel. It works great for us, and the initrd part I tested on PPC, PPC64, SPARC64, HPPA, MIPS64 and probably other random things that I may forgot about in the meantime.

Would be great if something like this could be mainlined.

We'd love to get this mainlined as well!

We're using these patches internally as well. We're seeing an improvement on an Intel Atom N3710, where boot time is reduced by one second over using an xz compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel, but zstd is likely a better trade off, because it compresses much better and still has excellent decompression speed.

Since its been nearly a year since I sent these out, I will take some time to rebase and retest the patches in case anything changed, and then then resend the patches in the next weeks.

Nick

Quoted 1 line

[...]

#16 lore raw
Quoted 11 lines · Nick Terrell

We'd love to get this mainlined as well!

We're using these patches internally as well. We're seeing an improvement on an Intel Atom N3710, where boot time is reduced by one second over using an xz compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel, but zstd is likely a better trade off, because it compresses much better and still has excellent decompression speed.

Since its been nearly a year since I sent these out, I will take some time to rebase and retest the patches in case anything changed, and then then resend the patches in the next weeks.

Hi!
After the ping, I intended to resend the patch-set (with removals included) after I return from miniDebconf Hamburg, but you 1. are the author of the non-trivial part, 2. you have a better test machinery, and 3. I have a deeply seated preference for effort to be done by people who are not me.

A rebased and working version is at https://github.com/kilobyte/linux/tree/nobz2-v3
but there are no real improvements beyond rebases, a typo fix, and Paul Burton's ACK for mips.

There's an unaddressed comment by Ingo Molnar
https://lore.kernel.org/lkml/[email protected]/
for your part of the code.

So what do you suggest?

Meow!

Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ Latin:   meow 4 characters, 4 columns,  4 bytes
⣾⠁⢠⠒⠀⣿⡁ Greek:   μεου 4 characters, 4 columns,  8 bytes
⢿⡄⠘⠷⠚⠋  Runes:   ᛗᛖᛟᚹ 4 characters, 4 columns, 12 bytes
⠈⠳⣄⠀⠀⠀⠀ Chinese: 喵   1 character,  2 columns,  3 bytes <-- best!
#17 lore raw
Quoted 17 lines · Adam Borowski
Quoted 11 lines · Nick Terrell

We'd love to get this mainlined as well!

We're using these patches internally as well. We're seeing an improvement on an Intel Atom N3710, where boot time is reduced by one second over using an xz compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel, but zstd is likely a better trade off, because it compresses much better and still has excellent decompression speed.

Since its been nearly a year since I sent these out, I will take some time to rebase and retest the patches in case anything changed, and then then resend the patches in the next weeks.

Quoted 5 lines

Hi!
After the ping, I intended to resend the patch-set (with removals included) after I return from miniDebconf Hamburg, but you 1. are the author of the non-trivial part, 2. you have a better test machinery, and 3. I have a deeply seated preference for effort to be done by people who are not me.

If its okay with you I will resend the patch set with your removals included by the end of next week at the latest. I'll aim for this week.

-Nick

Quoted 17 lines

A rebased and working version is at https://github.com/kilobyte/linux/tree/nobz2-v3
but there are no real improvements beyond rebases, a typo fix, and Paul Burton's ACK for mips.

There's an unaddressed comment by Ingo Molnar
https://lore.kernel.org/lkml/[email protected]/
for your part of the code.

So what do you suggest?

Meow!

Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ Latin:   meow 4 characters, 4 columns,  4 bytes
⣾⠁⢠⠒⠀⣿⡁ Greek:   μεου 4 characters, 4 columns,  8 bytes
⢿⡄⠘⠷⠚⠋  Runes:   ᛗᛖᛟᚹ 4 characters, 4 columns, 12 bytes
⠈⠳⣄⠀⠀⠀⠀ Chinese: 喵   1 character,  2 columns,  3 bytes <-- best!
#18 lore raw

PATCH 1/1 .gitignore: add ZSTD-compressed files

In reply to Nick Terrell

For now, that's arch/x86/boot/compressed/vmlinux.bin.zst but probably more will come, thus let's be consistent with all other compressors.

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)
.gitignore +1 -0
index 97ba6b79834c..0d09cf1c053c 100644
@@ -42,6 +42,7 @@
 *.tab.[ch]
 *.tar
 *.xz
+*.zst
 Module.symvers
 modules.builtin
 
Signature · 1 line
2.18.0