Skip to contentSkip to conversation

PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1

DISCUSSION Sedat Dilek 3 participants 5 messages Sep 28, 2020 – Oct 3, 2020 1 message not in the archive Kbuild

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 26 lines · Feng Tang

Hi Nick,

0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32 build), and it could be related with ZSTD code, though initially we bisected to some other commits.

The error messages are:

        early console in setup code
        Wrong EFI loader signature.
        early console in extract_kernel
        input_data: 0x046f50b4
        input_len: 0x01ebbeb6
        output: 0x01000000
        output_len: 0x04fc535c
        kernel_total_size: 0x055f5000
        needed_size: 0x055f5000

        Decompressing Linux...

        ZSTD-compressed data is corrupt

This could be reproduced by compiling the kernel with attached config, and use QEMU to boot it.

Quoted 9 lines

We suspect it could be related with the kernel size, as we only see it on big kernel, and some more info are:

* If we remove a lot of kernel config to build a much smaller kernel, it will boot fine

* If we change the zstd algorithm from zstd22 to zstd19, the kernel will boot fine with below patch

Hi,

Recently, Debian has updated initramfs-tools/initramfs-tools-core packages with ZSTD support:

[ /usr/sbin/mkinitramfs ]

case "${compress}" in
gzip) # If we're doing a reproducible build, use gzip -n
if [ -n "${SOURCE_DATE_EPOCH}" ]; then
compress="gzip -n"
# Otherwise, substitute pigz if it's available elif command -v pigz >/dev/null; then
compress=pigz
fi
;;

lz4)    compress="lz4 -9 -l" ;;
zstd)   compress="zstd -q -19 -T0" ;;
xz)     compress="xz --check=crc32"
       # If we're not doing a reproducible build, enable multithreading
       test -z "${SOURCE_DATE_EPOCH}" && compress="$compress --threads=0"
       ;;
bzip2|lzma|lzop)
       # no parameters needed
       ;;
*)      echo "W: Unknown compression command ${compress}" >&2 ;;
esac

As you can see it uses compression-level 19 for ZSTD. Maybe it is good to change from 22 to 19 in the Linux-kernel sources?

Thanks.

Regards,
- Sedat -

Quoted 18 lines
        diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
        index 3962f59..8fe71ba 100644
        --- a/arch/x86/boot/compressed/Makefile
        +++ b/arch/x86/boot/compressed/Makefile
        @@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
         $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE
        -       $(call if_changed,zstd22)
        +       $(call if_changed,zstd)

Please let me know if you need more info, and sorry for the late report as we just tracked down to this point.

Thanks,
Feng

#2 lore raw
In reply to a message not in the archive
Quoted 48 lines · Feng Tang

Hi Nick,

0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32 build), and it could be related with ZSTD code, though initially we bisected to some other commits.

The error messages are:

	early console in setup code
	Wrong EFI loader signature.
	early console in extract_kernel
	input_data: 0x046f50b4
	input_len: 0x01ebbeb6
	output: 0x01000000
	output_len: 0x04fc535c
	kernel_total_size: 0x055f5000
	needed_size: 0x055f5000
	
	Decompressing Linux...
	
	ZSTD-compressed data is corrupt

This could be reproduced by compiling the kernel with attached config, and use QEMU to boot it.

We suspect it could be related with the kernel size, as we only see it on big kernel, and some more info are:

* If we remove a lot of kernel config to build a much smaller kernel, it will boot fine

* If we change the zstd algorithm from zstd22 to zstd19, the kernel will boot fine with below patch

	diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
	index 3962f59..8fe71ba 100644
	--- a/arch/x86/boot/compressed/Makefile
	+++ b/arch/x86/boot/compressed/Makefile
	@@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
	 $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE
	-	$(call if_changed,zstd22)
	+	$(call if_changed,zstd)
Quoted 2 lines

Please let me know if you need more info, and sorry for the late report as we just tracked down to this point.

Thanks for the report, I will look into it today.

Best,
Nick

Quoted 6 lines

Thanks,
Feng

<zstd_x86_32.config>

#3 lore raw
Quoted 52 lines · Nick Terrell
Quoted 49 lines · Feng Tang

Hi Nick,

0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32 build), and it could be related with ZSTD code, though initially we bisected to some other commits.

The error messages are:

	early console in setup code
	Wrong EFI loader signature.
	early console in extract_kernel
	input_data: 0x046f50b4
	input_len: 0x01ebbeb6
	output: 0x01000000
	output_len: 0x04fc535c
	kernel_total_size: 0x055f5000
	needed_size: 0x055f5000
	
	Decompressing Linux...
	
	ZSTD-compressed data is corrupt

This could be reproduced by compiling the kernel with attached config, and use QEMU to boot it.

We suspect it could be related with the kernel size, as we only see it on big kernel, and some more info are:

* If we remove a lot of kernel config to build a much smaller kernel, it will boot fine

* If we change the zstd algorithm from zstd22 to zstd19, the kernel will boot fine with below patch

	diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
	index 3962f59..8fe71ba 100644
	--- a/arch/x86/boot/compressed/Makefile
	+++ b/arch/x86/boot/compressed/Makefile
	@@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
	 $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE
	-	$(call if_changed,zstd22)
	+	$(call if_changed,zstd)
Quoted 4 lines
Quoted 2 lines

Please let me know if you need more info, and sorry for the late report as we just tracked down to this point.

Thanks for the report, I will look into it today.

CC Petr Malat

I’ve successfully reproduced, and found the issue. It turns out that this patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his fix corresponds to this upstream commit [1].

Can we get Petr's patch merged into v5.9?

This bug only happens when the window size is > 8 MB. A non-kernel workaround would be to compress the kernel level 19 instead of level 22, which uses an 8 MB window size, instead of a 128 MB window size.

The reason it only shows up for large kernels, is that the code is only buggy when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug.

Best,
Nick

[0] https://lkml.org/lkml/2020/9/14/94
[1] https://github.com/facebook/zstd/commit/8a5c0c98ae5a7884694589d7a69bc99011add94d

Quoted 9 lines

Best,
Nick

Quoted 6 lines

Thanks,
Feng

<zstd_x86_32.config>

Tested-by
#4 lore raw
Quoted 34 lines · Nick Terrell
Quoted 34 lines · Nick Terrell
Quoted 30 lines · Feng Tang

Hi Nick,

0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32 build), and it could be related with ZSTD code, though initially we bisected to some other commits.

The error messages are:
Decompressing Linux...

ZSTD-compressed data is corrupt

This could be reproduced by compiling the kernel with attached config, and use QEMU to boot it.

We suspect it could be related with the kernel size, as we only see it on big kernel, and some more info are:

* If we remove a lot of kernel config to build a much smaller kernel, it will boot fine

* If we change the zstd algorithm from zstd22 to zstd19, the kernel will boot fine with below patch

Please let me know if you need more info, and sorry for the late report as we just tracked down to this point.

Quoted 7 lines
Quoted 1 line

Thanks for the report, I will look into it today.

CC Petr Malat

I’ve successfully reproduced, and found the issue. It turns out that this patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his fix corresponds to this upstream commit [1].

Glad to know there is already a fix.

Quoted 6 lines

Can we get Petr's patch merged into v5.9?

This bug only happens when the window size is > 8 MB. A non-kernel workaround would be to compress the kernel level 19 instead of level 22, which uses an 8 MB window size, instead of a 128 MB window size.

Quoted 7 lines

The reason it only shows up for large kernels, is that the code is only buggy when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug.

Best,
Nick

[0] https://lkml.org/lkml/2020/9/14/94

With this patch, all the failed cases on my side could boot fine.

Thanks,
Feng

Quoted 1 line

[1] https://github.com/facebook/zstd/commit/8a5c0c98ae5a7884694589d7a69bc99011add94d

author Tested-by
#5 lore raw
Quoted 55 lines · Feng Tang
Quoted 34 lines · Nick Terrell
Quoted 34 lines · Nick Terrell
Quoted 30 lines · Feng Tang

Hi Nick,

0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32 build), and it could be related with ZSTD code, though initially we bisected to some other commits.

The error messages are:
Decompressing Linux...

ZSTD-compressed data is corrupt

This could be reproduced by compiling the kernel with attached config, and use QEMU to boot it.

We suspect it could be related with the kernel size, as we only see it on big kernel, and some more info are:

* If we remove a lot of kernel config to build a much smaller kernel, it will boot fine

* If we change the zstd algorithm from zstd22 to zstd19, the kernel will boot fine with below patch

Please let me know if you need more info, and sorry for the late report as we just tracked down to this point.

Quoted 7 lines
Quoted 1 line

Thanks for the report, I will look into it today.

CC Petr Malat

I’ve successfully reproduced, and found the issue. It turns out that this patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his fix corresponds to this upstream commit [1].

Glad to know there is already a fix.

Quoted 9 lines

Can we get Petr's patch merged into v5.9?

This bug only happens when the window size is > 8 MB. A non-kernel workaround would be to compress the kernel level 19 instead of level 22, which uses an 8 MB window size, instead of a 128 MB window size.

The reason it only shows up for large kernels, is that the code is only buggy when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug.

Quoted 9 lines
Quoted 4 lines

Best,
Nick

[0] https://lkml.org/lkml/2020/9/14/94

With this patch, all the failed cases on my side could boot fine.

I applied this patch to see if it is OK with x86 64bit - Yes, it is.

Feel free to add my:

- Sedat -

Quoted 6 lines

Thanks,
Feng

Quoted 1 line

[1] https://github.com/facebook/zstd/commit/8a5c0c98ae5a7884694589d7a69bc99011add94d