PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 26 lines · Feng Tang
Hi Nick,
0day has found some kernel decomprssion failure case since 5.9-rc1 (
X86_32build), and it could be related with ZSTD code, though initially we bisected to some other commits.The error messages are:
early console in setup code Wrong EFI loader signature. early console in extract_kernel input_data: 0x046f50b4 input_len: 0x01ebbeb6 output: 0x01000000 output_len: 0x04fc535c kernel_total_size: 0x055f5000 needed_size: 0x055f5000 Decompressing Linux... ZSTD-compressed data is corruptThis could be reproduced by compiling the kernel with attached config, and use QEMU to boot it.
Quoted 9 lines
We suspect it could be related with the kernel size, as we only see it on big kernel, and some more info are:
* If we remove a lot of kernel config to build a much smaller kernel, it will boot fine
* If we change the zstd algorithm from zstd22 to zstd19, the kernel will boot fine with below patch
Hi,
Recently, Debian has updated initramfs-tools/initramfs-tools-core packages with ZSTD support:
[ /usr/sbin/mkinitramfs ]
case "${compress}" in
gzip) # If we're doing a reproducible build, use gzip -n
if [ -n "${SOURCE_DATE_EPOCH}" ]; then
compress="gzip -n"
# Otherwise, substitute pigz if it's available elif command -v pigz >/dev/null; then
compress=pigz
fi
;;
lz4) compress="lz4 -9 -l" ;;
zstd) compress="zstd -q -19 -T0" ;;
xz) compress="xz --check=crc32"
# If we're not doing a reproducible build, enable multithreading
test -z "${SOURCE_DATE_EPOCH}" && compress="$compress --threads=0"
;;
bzip2|lzma|lzop)
# no parameters needed
;;
*) echo "W: Unknown compression command ${compress}" >&2 ;;
esacAs you can see it uses compression-level 19 for ZSTD. Maybe it is good to change from 22 to 19 in the Linux-kernel sources?
Thanks.
Regards,
- Sedat -
Quoted 18 lines
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index 3962f59..8fe71ba 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE - $(call if_changed,zstd22) + $(call if_changed,zstd)Please let me know if you need more info, and sorry for the late report as we just tracked down to this point.
Thanks,
Feng
Quoted 48 lines · Feng Tang
Hi Nick,
0day has found some kernel decomprssion failure case since 5.9-rc1 (
X86_32build), and it could be related with ZSTD code, though initially we bisected to some other commits.The error messages are:
early console in setup code Wrong EFI loader signature. early console in extract_kernel input_data: 0x046f50b4 input_len: 0x01ebbeb6 output: 0x01000000 output_len: 0x04fc535c kernel_total_size: 0x055f5000 needed_size: 0x055f5000 Decompressing Linux... ZSTD-compressed data is corruptThis could be reproduced by compiling the kernel with attached config, and use QEMU to boot it.
We suspect it could be related with the kernel size, as we only see it on big kernel, and some more info are:
* If we remove a lot of kernel config to build a much smaller kernel, it will boot fine
* If we change the zstd algorithm from zstd22 to zstd19, the kernel will boot fine with below patch
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index 3962f59..8fe71ba 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE - $(call if_changed,zstd22) + $(call if_changed,zstd)
Quoted 2 lines
Please let me know if you need more info, and sorry for the late report as we just tracked down to this point.
Thanks for the report, I will look into it today.
Best,
Nick
Quoted 6 lines
Thanks,
Feng<zstd_x86_32.config>
Quoted 52 lines · Nick Terrell
Quoted 49 lines · Feng Tang
Hi Nick,
0day has found some kernel decomprssion failure case since 5.9-rc1 (
X86_32build), and it could be related with ZSTD code, though initially we bisected to some other commits.The error messages are:
early console in setup code Wrong EFI loader signature. early console in extract_kernel input_data: 0x046f50b4 input_len: 0x01ebbeb6 output: 0x01000000 output_len: 0x04fc535c kernel_total_size: 0x055f5000 needed_size: 0x055f5000 Decompressing Linux... ZSTD-compressed data is corruptThis could be reproduced by compiling the kernel with attached config, and use QEMU to boot it.
We suspect it could be related with the kernel size, as we only see it on big kernel, and some more info are:
* If we remove a lot of kernel config to build a much smaller kernel, it will boot fine
* If we change the zstd algorithm from zstd22 to zstd19, the kernel will boot fine with below patch
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index 3962f59..8fe71ba 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE - $(call if_changed,zstd22) + $(call if_changed,zstd)
Quoted 4 lines
Quoted 2 lines
Please let me know if you need more info, and sorry for the late report as we just tracked down to this point.
Thanks for the report, I will look into it today.
I’ve successfully reproduced, and found the issue. It turns out that this patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his fix corresponds to this upstream commit [1].
Can we get Petr's patch merged into v5.9?
This bug only happens when the window size is > 8 MB. A non-kernel workaround would be to compress the kernel level 19 instead of level 22, which uses an 8 MB window size, instead of a 128 MB window size.
The reason it only shows up for large kernels, is that the code is only buggy when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug.
Best,
Nick
[0] https://lkml.org/lkml/2020/9/14/94
[1] https://github.com/facebook/zstd/commit/8a5c0c98ae5a7884694589d7a69bc99011add94d
Quoted 9 lines
Best,
NickQuoted 6 lines
Thanks,
Feng<zstd_x86_32.config>
Feng Tang
Tested-byQuoted 34 lines · Nick Terrell
Quoted 34 lines · Nick Terrell
Quoted 30 lines · Feng Tang
Hi Nick,
0day has found some kernel decomprssion failure case since 5.9-rc1 (
X86_32build), and it could be related with ZSTD code, though initially we bisected to some other commits.The error messages are:
Decompressing Linux...ZSTD-compressed data is corrupt
This could be reproduced by compiling the kernel with attached config, and use QEMU to boot it.
We suspect it could be related with the kernel size, as we only see it on big kernel, and some more info are:
* If we remove a lot of kernel config to build a much smaller kernel, it will boot fine
* If we change the zstd algorithm from zstd22 to zstd19, the kernel will boot fine with below patch
Please let me know if you need more info, and sorry for the late report as we just tracked down to this point.
Quoted 7 lines
Quoted 1 line
Thanks for the report, I will look into it today.
CC Petr MalatI’ve successfully reproduced, and found the issue. It turns out that this patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his fix corresponds to this upstream commit [1].
Glad to know there is already a fix.
Quoted 6 lines
Can we get Petr's patch merged into v5.9?
This bug only happens when the window size is > 8 MB. A non-kernel workaround would be to compress the kernel level 19 instead of level 22, which uses an 8 MB window size, instead of a 128 MB window size.
Quoted 7 lines
The reason it only shows up for large kernels, is that the code is only buggy when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug.
Best,
Nick
With this patch, all the failed cases on my side could boot fine.
Thanks,
Feng
Sedat Dilek
Tested-byQuoted 55 lines · Feng Tang
Quoted 34 lines · Nick Terrell
Quoted 34 lines · Nick Terrell
Quoted 30 lines · Feng Tang
Hi Nick,
0day has found some kernel decomprssion failure case since 5.9-rc1 (
X86_32build), and it could be related with ZSTD code, though initially we bisected to some other commits.The error messages are:
Decompressing Linux...ZSTD-compressed data is corrupt
This could be reproduced by compiling the kernel with attached config, and use QEMU to boot it.
We suspect it could be related with the kernel size, as we only see it on big kernel, and some more info are:
* If we remove a lot of kernel config to build a much smaller kernel, it will boot fine
* If we change the zstd algorithm from zstd22 to zstd19, the kernel will boot fine with below patch
Please let me know if you need more info, and sorry for the late report as we just tracked down to this point.
Quoted 7 lines
Quoted 1 line
Thanks for the report, I will look into it today.
CC Petr MalatI’ve successfully reproduced, and found the issue. It turns out that this patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his fix corresponds to this upstream commit [1].
Glad to know there is already a fix.
Quoted 9 lines
Can we get Petr's patch merged into v5.9?
This bug only happens when the window size is > 8 MB. A non-kernel workaround would be to compress the kernel level 19 instead of level 22, which uses an 8 MB window size, instead of a 128 MB window size.
The reason it only shows up for large kernels, is that the code is only buggy when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug.
Quoted 9 lines
Quoted 4 lines
Best,
NickWith this patch, all the failed cases on my side could boot fine.
Tested-by Feng Tang <[email protected]>
I applied this patch to see if it is OK with x86 64bit - Yes, it is.
Feel free to add my:
- Sedat -
Quoted 6 lines
Thanks,
Feng