Add support for ZSTD-compressed kernel
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Nick Terrell
This patch set adds support for a ZSTD-compressed kernel and ramdisk images in the kernel boot process. It only integrates the support with x86, though the first patch is generic to all architectures.
I'm running this patch set since October on amd64 armhf arm64, no explosions -- besides the obvious when it's not applied yet I forget to reconfigure initramfs-tools. Which is getting tedious, so let's merge this already. :)
I've tested initrd on all of these archs; here's a debug patch to check if you're actually using it.
As for compressing the kernel itself: the second patch works as submitted (ie, x86 only). Porting it to other architectures is straightforward, but eg. on arm, most boards use u-boot which insists on decompressing the kernel by itself instead of passing control. EFI/grub should work, but I haven't tested that yet.
Meow!
Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can. ⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener. ⠈⠳⣄⠀⠀⠀⠀ A master species delegates.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · René Rebe
Should this currently just work without any arch change on e.g. ppc64, sparc64 et al.? I could do a test build and boot if that is of any value, ...
Initrd: no reason it wouldn't work, although for anything related to the boot process testing is a very good idea. If you test, you'd really want to apply that printk patch I posted, it's too easy to accidentally get a silent fallback to gzip or uncompressed. I wonder, perhaps such a printk could be permanently added, at a low message priority?
Kernel itself: needs per-arch porting, but it's not advertised in kconfig outside x86 so that's not a problem.
If you are knowledgeful about bootloaders on ppc64, sparc64, etc, such information would be great.
Meow!
Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can. ⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener. ⠈⠳⣄⠀⠀⠀⠀ A master species delegates.
Hi,
Quoted 10 lines · Adam Borowski
Quoted 3 lines · René Rebe
Should this currently just work without any arch change on e.g. ppc64, sparc64 et al.? I could do a test build and boot if that is of any value, ...
Initrd: no reason it wouldn't work, although for anything related to the boot process testing is a very good idea. If you test, you'd really want to apply that printk patch I posted, it's too easy to accidentally get a silent fallback to gzip or uncompressed. I wonder, perhaps such a printk could be permanently added, at a low message priority?
zstd initrd worked on sparc64/sun4u - Ultra 30:
https://www.youtube.com/watch?v=10q2OxHAzQ4
Any chance we could get this initrd zstd support upstream?
Quoted 5 lines
Kernel itself: needs per-arch porting, but it's not advertised in kconfig outside x86 so that's not a problem.
If you are knowledgeful about bootloaders on ppc64, sparc64, etc, such information would be great.
Signature · 2 lines
ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
Quoted 4 lines · Nick Terrell
This patch set adds support for a ZSTD-compressed kernel and ramdisk images in the kernel boot process. It only integrates the support with x86, though the first patch is generic to all architectures.
Hi all,
Is there anything blocking this from getting merged?
Please let me know if there is anything I can do to help move this patch forward.
Thanks,
Nick
Hi Nick,
Quoted 9 lines · Nick Terrell
Quoted 4 lines · Nick Terrell
This patch set adds support for a ZSTD-compressed kernel and ramdisk images in the kernel boot process. It only integrates the support with x86, though the first patch is generic to all architectures.
Hi all,
Is there anything blocking this from getting merged?
Please let me know if there is anything I can do to help move this patch forward.
works fine on my side, initrd test booted on i486, x86-64, ppc, ppc64 and sparc64. For testing on mips64 I would first need to re-base the Octane^1 patch-set ;-)
Greetings,
René
^1) https://www.youtube.com/watch?v=AU_RV8uoTIo
Signature · 2 lines
ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
Hey,
is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk
René
Quoted 27 lines · René Rebe
Hi Nick,
Quoted 9 lines · Nick Terrell
Quoted 4 lines · Nick Terrell
This patch set adds support for a ZSTD-compressed kernel and ramdisk images in the kernel boot process. It only integrates the support with x86, though the first patch is generic to all architectures.
Hi all,
Is there anything blocking this from getting merged?
Please let me know if there is anything I can do to help move this patch forward.works fine on my side, initrd test booted on i486, x86-64, ppc, ppc64 and sparc64. For testing on mips64 I would first need to re-base the Octane^1 patch-set ;-)
Greetings,
René^1) https://www.youtube.com/watch?v=AU_RV8uoTIo
Signature · 2 lines
ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
Signature · 4 lines
ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin DE Legal: Amtsgericht Berlin (Charlottenburg) HRB 105123B, Tax-ID#: DE251602478 Managing Director: René Rebe http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
Quoted 3 lines · René Rebe
Hey,
is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk
Can you remove some other compressor for the kernel image if you merge this?
The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.
Surely zstd is universally better than some existing compressor the kernel already uses for itself for a particular sweet spot? If it isn't it's likely not needed.
-Andi
Hi,
Quoted 9 lines · Andi Kleen
Quoted 3 lines · René Rebe
Hey,
is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk
Can you remove some other compressor for the kernel image if you merge this?
The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.
Well, we (t2 formerly known as rocklinux) only changed our pkg archive compression from bzip2 to zstd once in 20 years ,-)
Quoted 3 lines
Surely zstd is universally better than some existing compressor the kernel already uses for itself for a particular sweet spot? If it isn't it's likely not needed.
Well, IMHO this is not that much code, but if you want to remove something I guess bzip2 or LZMA may be candidates.
http://zstd.net
Signature · 2 lines
ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
Quoted 8 lines · Andi Kleen
Quoted 5 lines · René Rebe
Hey,
is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc
Can you remove some other compressor for the kernel image if you merge this?
Funny you say this. I want to post this after the merge window is over:
https://github.com/kilobyte/linux/commits/nobz2-next-20180731
Sorry for a tree atop next, but there are conflicts that needed to be adressed. The patchset is also undertested (works for me, but not even compile-tested on archs I don't have a toolchain for at hand).
Quoted 2 lines
The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.
The above patchset drops just bzip2. It is the only one that's strictly beaten in every way (ratio, time, memory usage), there are also no other uses of bzip2 anywhere in the kernel so we'd get to drop its code
completely: 900 lines of Linus' happiness.
Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), but those are used elsewhere thus there's hardly any gain. If you want them gone, please say so -- I'll include their droppage.
Quoted 3 lines
Surely zstd is universally better than some existing compressor the kernel already uses for itself for a particular sweet spot? If it isn't it's likely not needed.
zstd is better in the balanced niche: unless you want extreme speed (lz4) or best compression (xz), zstd handily beats algorithms in the middle.
Meow!
Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition: ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17] ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37] ⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]
Quoted 5 lines · Adam Borowski
Quoted 2 lines
The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.
The above patchset drops just bzip2. It is the only one that's strictly beaten in every way (ratio, time, memory usage), there are also no other
Does time include build time? I've been reverting back to gzip recently because I care very much about that.
Quoted 2 lines
uses of bzip2 anywhere in the kernel so we'd get to drop its code
completely: 900 lines of Linus' happiness.
Great!
Quoted 4 lines
Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), but those are used elsewhere thus there's hardly any gain. If you want them gone, please say so -- I'll include their droppage.
Yes would be good to remove the kernel image support for those too just to simplify the config process, even if it doesn't save much code.
-Andi
Quoted 9 lines · Andi Kleen
Quoted 5 lines · Adam Borowski
Quoted 2 lines
The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.
The above patchset drops just bzip2. It is the only one that's strictly beaten in every way (ratio, time, memory usage), there are also no other
Does time include build time? I've been reverting back to gzip recently because I care very much about that.
Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while ago), here's copypasta of a random 16824672 byte executable, in userspace, with default level setting:
comp decomp size
xz 8.038s 0.356s 4320292
bz2 2.265s 0.730s 5234516
zst 0.274s 0.102s 5657626
gz 0.880s 0.152s 6515505
Z 0.499s 0.133s 8932459
lzo 0.100s 0.095s 9198874
As you can see, zstd's compression time is drastically better than gzip, while ratio is better. The default level is very low (-3 on -1..-22 scale) but you can crank it up for stronger compression.
The defaults fit your use case.
Quoted 11 lines
Quoted 2 lines
uses of bzip2 anywhere in the kernel so we'd get to drop its code
completely: 900 lines of Linus' happiness.Great!
Quoted 3 lines
Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), but those are used elsewhere thus there's hardly any gain. If you want them gone, please say so -- I'll include their droppage.
Yes would be good to remove the kernel image support for those too just to simplify the config process, even if it doesn't save much code.
There's one caveat: fast choices are quite new:
* lz4 userspace tools are not even in current Debian stable (just unstable)
* uncompressed kernel got in only this merge window
* zstd has userspace tools in Debian stable but is not merged into the kernel yet
(other dists are probably similar)
Thus, it might be a good idea to keep lzo for a while longer.
Bare lzma can probably go -- xz filters are nice for binaries of archs it knows (disabled otherwise), and lack of header requires hacks to find out the payload's size.
So it's up to you guys: do you want me to drop lzo and/or lzma? We can also drop them just for vmlinuz but not initrd.
Meow!
Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition: ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17] ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37] ⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]
Quoted 48 lines · Adam Borowski
Quoted 9 lines · Andi Kleen
Quoted 5 lines · Adam Borowski
Quoted 2 lines
The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted.
The above patchset drops just bzip2. It is the only one that's strictly beaten in every way (ratio, time, memory usage), there are also no other
Does time include build time? I've been reverting back to gzip recently because I care very much about that.
Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while ago), here's copypasta of a random 16824672 byte executable, in userspace, with default level setting:
comp decomp size
xz 8.038s 0.356s 4320292
bz2 2.265s 0.730s 5234516
zst 0.274s 0.102s 5657626
gz 0.880s 0.152s 6515505
Z 0.499s 0.133s 8932459
lzo 0.100s 0.095s 9198874As you can see, zstd's compression time is drastically better than gzip, while ratio is better. The default level is very low (-3 on -1..-22 scale) but you can crank it up for stronger compression.
The defaults fit your use case.
Quoted 11 lines
Quoted 2 lines
uses of bzip2 anywhere in the kernel so we'd get to drop its code
completely: 900 lines of Linus' happiness.Great!
Quoted 3 lines
Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), but those are used elsewhere thus there's hardly any gain. If you want them gone, please say so -- I'll include their droppage.
Yes would be good to remove the kernel image support for those too just to simplify the config process, even if it doesn't save much code.
There's one caveat: fast choices are quite new:
* lz4 userspace tools are not even in current Debian stable (just unstable)
* uncompressed kernel got in only this merge window
* zstd has userspace tools in Debian stable but is not merged into the kernel yet
(other dists are probably similar)
Quoted 5 lines
Thus, it might be a good idea to keep lzo for a while longer.
Bare lzma can probably go -- xz filters are nice for binaries of archs it knows (disabled otherwise), and lack of header requires hacks to find out the payload's size.
Personally, I'd be very happy to see LZMA go. It is a custom implementation that doesn't use the lib/xz/ library. When I was implementing decompress_zstd.c I fuzzed all of the kernel decompressors, and unlzma() will crash on invalid input. There is no reason, other than not breaking compatibility, to use LZMA over XZ.
Quoted 10 lines
So it's up to you guys: do you want me to drop lzo and/or lzma? We can also drop them just for vmlinuz but not initrd.
Meow!
Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition: ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17] ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37] ⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · René Rebe
Hey there,
just wanted to check about the status of zstd-compressed kernel. It works great for us, and the initrd part I tested on PPC, PPC64, SPARC64, HPPA, MIPS64 and probably other random things that I may forgot about in the meantime.
Would be great if something like this could be mainlined.
We'd love to get this mainlined as well!
We're using these patches internally as well. We're seeing an improvement on an Intel Atom N3710, where boot time is reduced by one second over using an xz compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel, but zstd is likely a better trade off, because it compresses much better and still has excellent decompression speed.
Since its been nearly a year since I sent these out, I will take some time to rebase and retest the patches in case anything changed, and then then resend the patches in the next weeks.
Nick
Quoted 1 line
[...]
Quoted 11 lines · Nick Terrell
We'd love to get this mainlined as well!
We're using these patches internally as well. We're seeing an improvement on an Intel Atom N3710, where boot time is reduced by one second over using an xz compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel, but zstd is likely a better trade off, because it compresses much better and still has excellent decompression speed.
Since its been nearly a year since I sent these out, I will take some time to rebase and retest the patches in case anything changed, and then then resend the patches in the next weeks.
Hi!
After the ping, I intended to resend the patch-set (with removals included) after I return from miniDebconf Hamburg, but you 1. are the author of the non-trivial part, 2. you have a better test machinery, and 3. I have a deeply seated preference for effort to be done by people who are not me.
A rebased and working version is at https://github.com/kilobyte/linux/tree/nobz2-v3
but there are no real improvements beyond rebases, a typo fix, and Paul Burton's ACK for mips.
There's an unaddressed comment by Ingo Molnar
https://lore.kernel.org/lkml/[email protected]/
for your part of the code.
So what do you suggest?
Meow!
Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ Latin: meow 4 characters, 4 columns, 4 bytes ⣾⠁⢠⠒⠀⣿⡁ Greek: μεου 4 characters, 4 columns, 8 bytes ⢿⡄⠘⠷⠚⠋ Runes: ᛗᛖᛟᚹ 4 characters, 4 columns, 12 bytes ⠈⠳⣄⠀⠀⠀⠀ Chinese: 喵 1 character, 2 columns, 3 bytes <-- best!
Quoted 17 lines · Adam Borowski
Quoted 11 lines · Nick Terrell
We'd love to get this mainlined as well!
We're using these patches internally as well. We're seeing an improvement on an Intel Atom N3710, where boot time is reduced by one second over using an xz compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel, but zstd is likely a better trade off, because it compresses much better and still has excellent decompression speed.
Since its been nearly a year since I sent these out, I will take some time to rebase and retest the patches in case anything changed, and then then resend the patches in the next weeks.
Quoted 5 lines
Hi!
After the ping, I intended to resend the patch-set (with removals included) after I return from miniDebconf Hamburg, but you 1. are the author of the non-trivial part, 2. you have a better test machinery, and 3. I have a deeply seated preference for effort to be done by people who are not me.
If its okay with you I will resend the patch set with your removals included by the end of next week at the latest. I'll aim for this week.
-Nick
Quoted 17 lines
A rebased and working version is at https://github.com/kilobyte/linux/tree/nobz2-v3
but there are no real improvements beyond rebases, a typo fix, and Paul Burton's ACK for mips.There's an unaddressed comment by Ingo Molnar
https://lore.kernel.org/lkml/[email protected]/
for your part of the code.So what do you suggest?
Meow!
Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ Latin: meow 4 characters, 4 columns, 4 bytes ⣾⠁⢠⠒⠀⣿⡁ Greek: μεου 4 characters, 4 columns, 8 bytes ⢿⡄⠘⠷⠚⠋ Runes: ᛗᛖᛟᚹ 4 characters, 4 columns, 12 bytes ⠈⠳⣄⠀⠀⠀⠀ Chinese: 喵 1 character, 2 columns, 3 bytes <-- best!
PATCH 1/1 .gitignore: add ZSTD-compressed files
In reply to Nick TerrellFor now, that's arch/x86/boot/compressed/vmlinux.bin.zst but probably more will come, thus let's be consistent with all other compressors.
--- .gitignore | 1 + 1 file changed, 1 insertion(+)
.gitignore +1 -0
@@ -42,6 +42,7 @@ *.tab.[ch] *.tar *.xz +*.zst Module.symvers modules.builtin
Signature · 1 line
2.18.0