Illegal operation with clang + CONFIG_MARCH_Z13
Hi all,
Recently, Fedora moved their baseline from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13:
https://src.fedoraproject.org/rpms/kernel/c/aff6e8acdaa437e9f06ef4166ca2209071223f8d
Unfortunately, this causes clang built kernels to have an "illegal operation" panic when booting in QEMU. This is trivially reproducible on mainline with the versions of clang that s390 supports.
# Switch from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13
$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig menuconfig bzImage
$ qemu-system-s390x \ -M s390-ccw-virtio \ -kernel arch/s390/boot/bzImage \ -display none \ -initrd .../boot-utils/images/s390/rootfs.cpio \ -m 512m \ -nodefaults \ -no-reboot \ -serial mon:stdio ... [ 0.558817] Linux version 6.0.0-rc5-00017-gd1221cea11fc ([email protected]) (clang version 14.0.5 (Fedora 14.0.5-7.fc38), GNU ld version 2.38-4.fc37) #1 SMP Tue Sep 13 08:35:38 MST 2022 ... [ 1.675787] illegal operation: 0001 ilc:3 [#1] SMP [ 1.675888] Modules linked in: [ 1.676044] CPU: 0 PID: 59 Comm: modprobe Not tainted 6.0.0-rc5-00017-gd1221cea11fc #1 [ 1.676134] Hardware name: QEMU 8561 QEMU (KVM/Linux) [ 1.676202] Krnl PSW : 0704d00180000000 0000000000579fbc (load_elf_binary+0x31c/0x11c0) [ 1.676459] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [ 1.676489] Krnl GPRS: 0000000000000001 00000000044f0000 0000000000000000 0000000000000000 [ 1.676506] 8000000000000080 0000000000000000 0000000000000000 00000000035fa900 [ 1.676522] 0000000000000001 000000000353ce00 0000000000000001 00000000044693c0 [ 1.676538] 000000000276fa00 000000000446a000 0000000000579f9e 00000380002ebc78 [ 1.677128] Krnl Code: 0000000000579fac: a7f405b8 brc 15,000000000057ab1c [ 1.677128] 0000000000579fb0: e31003400004 lg %r1,832 [ 1.677128] #0000000000579fb6: e3001780003b lzrf %r0,1920(%r1) [ 1.677128] >0000000000579fbc: 50001780 st %r0,1920(%r1) [ 1.677128] 0000000000579fc0: e31003400004 lg %r1,832 [ 1.677128] 0000000000579fc6: c40800aa99a9 lgrl %r0,0000000001acd318 [ 1.677128] 0000000000579fcc: e3001d400124 stg %r0,7488(%r1) [ 1.677128] 0000000000579fd2: e31003400004 lg %r1,832 [ 1.677343] Call Trace: [ 1.677564] [<0000000000579fbc>] load_elf_binary+0x31c/0x11c0 [ 1.677635] ([<0000000000579f9e>] load_elf_binary+0x2fe/0x11c0) [ 1.677652] [<00000000004e1ae6>] bprm_execve+0x4f6/0x7b0 [ 1.677671] [<00000000004e1256>] kernel_execve+0x3b6/0x3d0 [ 1.677687] [<000000000017e1c8>] call_usermodehelper_exec_async+0x158/0x1d0 [ 1.677706] [<0000000000103a9a>] __ret_from_fork+0x3a/0x60 [ 1.677719] [<0000000000f8562a>] ret_from_fork+0xa/0x40 [ 1.677748] Last Breaking-Event-Address: [ 1.677757] [<0000000000579fa2>] load_elf_binary+0x302/0x11c0 [ 1.678012] Kernel panic - not syncing: Fatal exception: panic_on_oops
The rootfs image is available in our boot-utils repo [1], should it be necessary for reproducing this issue. I do not see any problems with booting a CONFIG_MARCH_Z13 kernel compiled with GCC 12.2.1 from Fedora but it is possible that clang and GCC generate different code. I am not sure if this is a compiler problem or an emulation one, which is where I was looking for some input from you all.
[1]: https://github.com/ClangBuiltLinux/boot-utils
Cheers,
Nathan
Christian Borntraeger
reviewerAm 13.09.22 um 19:34 schrieb Nathan Chancellor:
Quoted 24 lines
Hi all,
Recently, Fedora moved their baseline from
CONFIG_MARCH_ZEC12toCONFIG_MARCH_Z13:https://src.fedoraproject.org/rpms/kernel/c/aff6e8acdaa437e9f06ef4166ca2209071223f8d
Unfortunately, this causes clang built kernels to have an "illegal operation" panic when booting in QEMU. This is trivially reproducible on mainline with the versions of clang that s390 supports.
# Switch from
CONFIG_MARCH_ZEC12toCONFIG_MARCH_Z13$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig menuconfig bzImage$ qemu-system-s390x \(quoted diff) +0
-8-M s390-ccw-virtio \ -kernel arch/s390/boot/bzImage \ -display none \ -initrd .../boot-utils/images/s390/rootfs.cpio \ -m 512m \ -nodefaults \ -no-reboot \ -serial mon:stdio
Quoted 6 lines
[ 0.558817] Linux version 6.0.0-rc5-00017-gd1221cea11fc ([email protected]) (clang version 14.0.5 (Fedora 14.0.5-7.fc38), GNU ld version 2.38-4.fc37) #1 SMP Tue Sep 13 08:35:38 MST 2022 ... [ 1.675787] illegal operation: 0001 ilc:3 [#1] SMP [ 1.675888] Modules linked in: [ 1.676044] CPU: 0 PID: 59 Comm: modprobe Not tainted 6.0.0-rc5-00017-gd1221cea11fc #1 [ 1.676134] Hardware name: QEMU 8561 QEMU (KVM/Linux)
What qemu version is that and what command line did you use (any cpu model?)
Quoted 9 lines
[ 1.676202] Krnl PSW : 0704d00180000000 0000000000579fbc (load_elf_binary+0x31c/0x11c0) [ 1.676459] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [ 1.676489] Krnl GPRS: 0000000000000001 00000000044f0000 0000000000000000 0000000000000000 [ 1.676506] 8000000000000080 0000000000000000 0000000000000000 00000000035fa900 [ 1.676522] 0000000000000001 000000000353ce00 0000000000000001 00000000044693c0 [ 1.676538] 000000000276fa00 000000000446a000 0000000000579f9e 00000380002ebc78 [ 1.677128] Krnl Code: 0000000000579fac: a7f405b8 brc 15,000000000057ab1c [ 1.677128] 0000000000579fb0: e31003400004 lg %r1,832 [ 1.677128] #0000000000579fb6: e3001780003b lzrf %r0,1920(%r1)
QEMU does support lzrf as far asI can tell as soon as you have S390_FEAT_STFLE_53 which is part of the qemu_V4_1 cpu model.
Quoted 28 lines
[ 1.677128] >0000000000579fbc: 50001780 st %r0,1920(%r1) [ 1.677128] 0000000000579fc0: e31003400004 lg %r1,832 [ 1.677128] 0000000000579fc6: c40800aa99a9 lgrl %r0,0000000001acd318 [ 1.677128] 0000000000579fcc: e3001d400124 stg %r0,7488(%r1) [ 1.677128] 0000000000579fd2: e31003400004 lg %r1,832 [ 1.677343] Call Trace: [ 1.677564] [<0000000000579fbc>] load_elf_binary+0x31c/0x11c0 [ 1.677635] ([<0000000000579f9e>] load_elf_binary+0x2fe/0x11c0) [ 1.677652] [<00000000004e1ae6>] bprm_execve+0x4f6/0x7b0 [ 1.677671] [<00000000004e1256>] kernel_execve+0x3b6/0x3d0 [ 1.677687] [<000000000017e1c8>] call_usermodehelper_exec_async+0x158/0x1d0 [ 1.677706] [<0000000000103a9a>] __ret_from_fork+0x3a/0x60 [ 1.677719] [<0000000000f8562a>] ret_from_fork+0xa/0x40 [ 1.677748] Last Breaking-Event-Address: [ 1.677757] [<0000000000579fa2>] load_elf_binary+0x302/0x11c0 [ 1.678012] Kernel panic - not syncing: Fatal exception: panic_on_oopsThe rootfs image is available in our boot-utils repo [1], should it be necessary for reproducing this issue. I do not see any problems with booting a
CONFIG_MARCH_Z13kernel compiled with GCC 12.2.1 from Fedora but it is possible that clang and GCC generate different code. I am not sure if this is a compiler problem or an emulation one, which is where I was looking for some input from you all.[1]: https://github.com/ClangBuiltLinux/boot-utils
Cheers,
Nathan
Quoted 25 lines · Christian Borntraeger
Am 13.09.22 um 19:34 schrieb Nathan Chancellor:
Quoted 24 lines
Hi all,
Recently, Fedora moved their baseline from
CONFIG_MARCH_ZEC12toCONFIG_MARCH_Z13:https://src.fedoraproject.org/rpms/kernel/c/aff6e8acdaa437e9f06ef4166ca2209071223f8d
Unfortunately, this causes clang built kernels to have an "illegal operation" panic when booting in QEMU. This is trivially reproducible on mainline with the versions of clang that s390 supports.
# Switch from
CONFIG_MARCH_ZEC12toCONFIG_MARCH_Z13$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig menuconfig bzImage$ qemu-system-s390x \(quoted diff) +0
-8-M s390-ccw-virtio \ -kernel arch/s390/boot/bzImage \ -display none \ -initrd .../boot-utils/images/s390/rootfs.cpio \ -m 512m \ -nodefaults \ -no-reboot \ -serial mon:stdio
Quoted 8 lines
Quoted 6 lines
[ 0.558817] Linux version 6.0.0-rc5-00017-gd1221cea11fc ([email protected]) (clang version 14.0.5 (Fedora 14.0.5-7.fc38), GNU ld version 2.38-4.fc37) #1 SMP Tue Sep 13 08:35:38 MST 2022 ... [ 1.675787] illegal operation: 0001 ilc:3 [#1] SMP [ 1.675888] Modules linked in: [ 1.676044] CPU: 0 PID: 59 Comm: modprobe Not tainted 6.0.0-rc5-00017-gd1221cea11fc #1 [ 1.676134] Hardware name: QEMU 8561 QEMU (KVM/Linux)What qemu version is that and what command line did you use (any cpu model?)
I was able to reproduce this with QEMU 6.0.0 through 7.1.0, which are the versions that can boot a clang built kernel. I don't think we specify a cpu model, you can see the full command above. If there is something we should be doing differently, please let us know!
Quoted 41 lines
Quoted 9 lines
[ 1.676202] Krnl PSW : 0704d00180000000 0000000000579fbc (load_elf_binary+0x31c/0x11c0) [ 1.676459] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [ 1.676489] Krnl GPRS: 0000000000000001 00000000044f0000 0000000000000000 0000000000000000 [ 1.676506] 8000000000000080 0000000000000000 0000000000000000 00000000035fa900 [ 1.676522] 0000000000000001 000000000353ce00 0000000000000001 00000000044693c0 [ 1.676538] 000000000276fa00 000000000446a000 0000000000579f9e 00000380002ebc78 [ 1.677128] Krnl Code: 0000000000579fac: a7f405b8 brc 15,000000000057ab1c [ 1.677128] 0000000000579fb0: e31003400004 lg %r1,832 [ 1.677128] #0000000000579fb6: e3001780003b lzrf %r0,1920(%r1)QEMU does support lzrf as far asI can tell as soon as you have
S390_FEAT_STFLE_53which is part of theqemu_V4_1cpu model.Quoted 28 lines
[ 1.677128] >0000000000579fbc: 50001780 st %r0,1920(%r1) [ 1.677128] 0000000000579fc0: e31003400004 lg %r1,832 [ 1.677128] 0000000000579fc6: c40800aa99a9 lgrl %r0,0000000001acd318 [ 1.677128] 0000000000579fcc: e3001d400124 stg %r0,7488(%r1) [ 1.677128] 0000000000579fd2: e31003400004 lg %r1,832 [ 1.677343] Call Trace: [ 1.677564] [<0000000000579fbc>] load_elf_binary+0x31c/0x11c0 [ 1.677635] ([<0000000000579f9e>] load_elf_binary+0x2fe/0x11c0) [ 1.677652] [<00000000004e1ae6>] bprm_execve+0x4f6/0x7b0 [ 1.677671] [<00000000004e1256>] kernel_execve+0x3b6/0x3d0 [ 1.677687] [<000000000017e1c8>] call_usermodehelper_exec_async+0x158/0x1d0 [ 1.677706] [<0000000000103a9a>] __ret_from_fork+0x3a/0x60 [ 1.677719] [<0000000000f8562a>] ret_from_fork+0xa/0x40 [ 1.677748] Last Breaking-Event-Address: [ 1.677757] [<0000000000579fa2>] load_elf_binary+0x302/0x11c0 [ 1.678012] Kernel panic - not syncing: Fatal exception: panic_on_oopsThe rootfs image is available in our boot-utils repo [1], should it be necessary for reproducing this issue. I do not see any problems with booting a
CONFIG_MARCH_Z13kernel compiled with GCC 12.2.1 from Fedora but it is possible that clang and GCC generate different code. I am not sure if this is a compiler problem or an emulation one, which is where I was looking for some input from you all.[1]: https://github.com/ClangBuiltLinux/boot-utils
Cheers,
Nathan
Christian Borntraeger
reviewerAm 14.09.22 um 11:00 schrieb Nathan Chancellor:
Quoted 35 lines · Christian Borntraeger
Quoted 25 lines
Am 13.09.22 um 19:34 schrieb Nathan Chancellor:
Quoted 24 lines
Hi all,
Recently, Fedora moved their baseline from
CONFIG_MARCH_ZEC12toCONFIG_MARCH_Z13:https://src.fedoraproject.org/rpms/kernel/c/aff6e8acdaa437e9f06ef4166ca2209071223f8d
Unfortunately, this causes clang built kernels to have an "illegal operation" panic when booting in QEMU. This is trivially reproducible on mainline with the versions of clang that s390 supports.
# Switch from
CONFIG_MARCH_ZEC12toCONFIG_MARCH_Z13$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig menuconfig bzImage$ qemu-system-s390x \(quoted diff) +0
-8-M s390-ccw-virtio \ -kernel arch/s390/boot/bzImage \ -display none \ -initrd .../boot-utils/images/s390/rootfs.cpio \ -m 512m \ -nodefaults \ -no-reboot \ -serial mon:stdioQuoted 8 lines
Quoted 6 lines
[ 0.558817] Linux version 6.0.0-rc5-00017-gd1221cea11fc ([email protected]) (clang version 14.0.5 (Fedora 14.0.5-7.fc38), GNU ld version 2.38-4.fc37) #1 SMP Tue Sep 13 08:35:38 MST 2022 ... [ 1.675787] illegal operation: 0001 ilc:3 [#1] SMP [ 1.675888] Modules linked in: [ 1.676044] CPU: 0 PID: 59 Comm: modprobe Not tainted 6.0.0-rc5-00017-gd1221cea11fc #1 [ 1.676134] Hardware name: QEMU 8561 QEMU (KVM/Linux)What qemu version is that and what command line did you use (any cpu model?)
Quoted 23 lines
I was able to reproduce this with QEMU 6.0.0 through 7.1.0, which are the versions that can boot a clang built kernel. I don't think we specify a cpu model, you can see the full command above. If there is something we should be doing differently, please let us know!
Quoted 18 lines
Quoted 9 lines
[ 1.676202] Krnl PSW : 0704d00180000000 0000000000579fbc (load_elf_binary+0x31c/0x11c0) [ 1.676459] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [ 1.676489] Krnl GPRS: 0000000000000001 00000000044f0000 0000000000000000 0000000000000000 [ 1.676506] 8000000000000080 0000000000000000 0000000000000000 00000000035fa900 [ 1.676522] 0000000000000001 000000000353ce00 0000000000000001 00000000044693c0 [ 1.676538] 000000000276fa00 000000000446a000 0000000000579f9e 00000380002ebc78 [ 1.677128] Krnl Code: 0000000000579fac: a7f405b8 brc 15,000000000057ab1c [ 1.677128] 0000000000579fb0: e31003400004 lg %r1,832 [ 1.677128] #0000000000579fb6: e3001780003b lzrf %r0,1920(%r1)QEMU does support lzrf as far asI can tell as soon as you have
S390_FEAT_STFLE_53which is part of theqemu_V4_1cpu model.Quoted 5 lines
[ 1.677128] >0000000000579fbc: 50001780 st %r0,1920(%r1) [ 1.677128] 0000000000579fc0: e31003400004 lg %r1,832 [ 1.677128] 0000000000579fc6: c40800aa99a9 lgrl %r0,0000000001acd318 [ 1.677128] 0000000000579fcc: e3001d400124 stg %r0,7488(%r1) [ 1.677128] 0000000000579fd2: e31003400004 lg %r1,832
Can you check if the following qemu patch helps?
target/s390x/tcg/insn-data.def +1 -1
@@ -466,7 +466,7 @@ C(0xe39f, LAT, RXY_a, LAT, 0, m2_32u, r1, 0, lat, 0) C(0xe385, LGAT, RXY_a, LAT, 0, a2, r1, 0, lgat, 0) /* LOAD AND ZERO RIGHTMOST BYTE */ - C(0xe3eb, LZRF, RXY_a, LZRB, 0, m2_32u, new, r1_32, lzrb, 0) + C(0xe33b, LZRF, RXY_a, LZRB, 0, m2_32u, new, r1_32, lzrb, 0) C(0xe32a, LZRG, RXY_a, LZRB, 0, m2_64, r1, 0, lzrb, 0) /* LOAD LOGICAL AND ZERO RIGHTMOST BYTE */ C(0xe33a, LLZRGF, RXY_a, LZRB, 0, m2_32u, r1, 0, lzrb, 0)
Nathan Chancellor
Tested-byQuoted 54 lines · Christian Borntraeger
Am 14.09.22 um 11:00 schrieb Nathan Chancellor:
Quoted 53 lines · Christian Borntraeger
Quoted 25 lines
Am 13.09.22 um 19:34 schrieb Nathan Chancellor:
Quoted 24 lines
Hi all,
Recently, Fedora moved their baseline from
CONFIG_MARCH_ZEC12toCONFIG_MARCH_Z13:https://src.fedoraproject.org/rpms/kernel/c/aff6e8acdaa437e9f06ef4166ca2209071223f8d
Unfortunately, this causes clang built kernels to have an "illegal operation" panic when booting in QEMU. This is trivially reproducible on mainline with the versions of clang that s390 supports.
# Switch from
CONFIG_MARCH_ZEC12toCONFIG_MARCH_Z13$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig menuconfig bzImage$ qemu-system-s390x \(quoted diff) +0
-8-M s390-ccw-virtio \ -kernel arch/s390/boot/bzImage \ -display none \ -initrd .../boot-utils/images/s390/rootfs.cpio \ -m 512m \ -nodefaults \ -no-reboot \ -serial mon:stdioQuoted 8 lines
Quoted 6 lines
[ 0.558817] Linux version 6.0.0-rc5-00017-gd1221cea11fc ([email protected]) (clang version 14.0.5 (Fedora 14.0.5-7.fc38), GNU ld version 2.38-4.fc37) #1 SMP Tue Sep 13 08:35:38 MST 2022 ... [ 1.675787] illegal operation: 0001 ilc:3 [#1] SMP [ 1.675888] Modules linked in: [ 1.676044] CPU: 0 PID: 59 Comm: modprobe Not tainted 6.0.0-rc5-00017-gd1221cea11fc #1 [ 1.676134] Hardware name: QEMU 8561 QEMU (KVM/Linux)What qemu version is that and what command line did you use (any cpu model?)
I was able to reproduce this with QEMU 6.0.0 through 7.1.0, which are the versions that can boot a clang built kernel. I don't think we specify a cpu model, you can see the full command above. If there is something we should be doing differently, please let us know!
Quoted 13 lines
Quoted 9 lines
[ 1.676202] Krnl PSW : 0704d00180000000 0000000000579fbc (load_elf_binary+0x31c/0x11c0) [ 1.676459] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [ 1.676489] Krnl GPRS: 0000000000000001 00000000044f0000 0000000000000000 0000000000000000 [ 1.676506] 8000000000000080 0000000000000000 0000000000000000 00000000035fa900 [ 1.676522] 0000000000000001 000000000353ce00 0000000000000001 00000000044693c0 [ 1.676538] 000000000276fa00 000000000446a000 0000000000579f9e 00000380002ebc78 [ 1.677128] Krnl Code: 0000000000579fac: a7f405b8 brc 15,000000000057ab1c [ 1.677128] 0000000000579fb0: e31003400004 lg %r1,832 [ 1.677128] #0000000000579fb6: e3001780003b lzrf %r0,1920(%r1)QEMU does support lzrf as far asI can tell as soon as you have
S390_FEAT_STFLE_53which is part of theqemu_V4_1cpu model.
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
[ 1.677128] >0000000000579fbc: 50001780 st %r0,1920(%r1) [ 1.677128] 0000000000579fc0: e31003400004 lg %r1,832 [ 1.677128] 0000000000579fc6: c40800aa99a9 lgrl %r0,0000000001acd318 [ 1.677128] 0000000000579fcc: e3001d400124 stg %r0,7488(%r1) [ 1.677128] 0000000000579fd2: e31003400004 lg %r1,832Can you check if the following qemu patch helps?
Yes, that appears to work for me! If you send it along formally, feel free to include:
Quoted 14 lines
target/s390x/tcg/insn-data.def +1
-1@@ -466,7 +466,7 @@ C(0xe39f, LAT, RXY_a, LAT, 0, m2_32u, r1, 0, lat, 0) C(0xe385, LGAT, RXY_a, LAT, 0, a2, r1, 0, lgat, 0) /* LOAD AND ZERO RIGHTMOST BYTE */ - C(0xe3eb, LZRF, RXY_a, LZRB, 0, m2_32u, new, r1_32, lzrb, 0) + C(0xe33b, LZRF, RXY_a, LZRB, 0, m2_32u, new, r1_32, lzrb, 0) C(0xe32a, LZRG, RXY_a, LZRB, 0, m2_64, r1, 0, lzrb, 0) /* LOAD LOGICAL AND ZERO RIGHTMOST BYTE */ C(0xe33a, LLZRGF, RXY_a, LZRB, 0, m2_32u, r1, 0, lzrb, 0)