Skip to contentSkip to conversation

Illegal operation with clang + CONFIG_MARCH_Z13

DISCUSSION Nathan Chancellor 2 participants 5 messages Sep 13, 2022 – Sep 14, 2022 s390

lore
Share & tools
lore
#0 lore raw

Hi all,

Recently, Fedora moved their baseline from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13:

https://src.fedoraproject.org/rpms/kernel/c/aff6e8acdaa437e9f06ef4166ca2209071223f8d

Unfortunately, this causes clang built kernels to have an "illegal operation" panic when booting in QEMU. This is trivially reproducible on mainline with the versions of clang that s390 supports.

# Switch from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13

$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig menuconfig bzImage
$ qemu-system-s390x \
-M s390-ccw-virtio \
-kernel arch/s390/boot/bzImage \
-display none \
-initrd .../boot-utils/images/s390/rootfs.cpio \
-m 512m \
-nodefaults \
-no-reboot \
-serial mon:stdio
...
[    0.558817] Linux version 6.0.0-rc5-00017-gd1221cea11fc ([email protected]) (clang version 14.0.5 (Fedora 14.0.5-7.fc38), GNU ld version 2.38-4.fc37) #1 SMP Tue Sep 13 08:35:38 MST 2022
...
[    1.675787] illegal operation: 0001 ilc:3 [#1] SMP
[    1.675888] Modules linked in:
[    1.676044] CPU: 0 PID: 59 Comm: modprobe Not tainted 6.0.0-rc5-00017-gd1221cea11fc #1
[    1.676134] Hardware name: QEMU 8561 QEMU (KVM/Linux)
[    1.676202] Krnl PSW : 0704d00180000000 0000000000579fbc (load_elf_binary+0x31c/0x11c0)
[    1.676459]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
[    1.676489] Krnl GPRS: 0000000000000001 00000000044f0000 0000000000000000 0000000000000000
[    1.676506]            8000000000000080 0000000000000000 0000000000000000 00000000035fa900
[    1.676522]            0000000000000001 000000000353ce00 0000000000000001 00000000044693c0
[    1.676538]            000000000276fa00 000000000446a000 0000000000579f9e 00000380002ebc78
[    1.677128] Krnl Code: 0000000000579fac: a7f405b8            brc     15,000000000057ab1c
[    1.677128]            0000000000579fb0: e31003400004        lg      %r1,832
[    1.677128]           #0000000000579fb6: e3001780003b        lzrf    %r0,1920(%r1)
[    1.677128]           >0000000000579fbc: 50001780            st      %r0,1920(%r1)
[    1.677128]            0000000000579fc0: e31003400004        lg      %r1,832
[    1.677128]            0000000000579fc6: c40800aa99a9        lgrl    %r0,0000000001acd318
[    1.677128]            0000000000579fcc: e3001d400124        stg     %r0,7488(%r1)
[    1.677128]            0000000000579fd2: e31003400004        lg      %r1,832
[    1.677343] Call Trace:
[    1.677564]  [<0000000000579fbc>] load_elf_binary+0x31c/0x11c0
[    1.677635] ([<0000000000579f9e>] load_elf_binary+0x2fe/0x11c0)
[    1.677652]  [<00000000004e1ae6>] bprm_execve+0x4f6/0x7b0
[    1.677671]  [<00000000004e1256>] kernel_execve+0x3b6/0x3d0
[    1.677687]  [<000000000017e1c8>] call_usermodehelper_exec_async+0x158/0x1d0
[    1.677706]  [<0000000000103a9a>] __ret_from_fork+0x3a/0x60
[    1.677719]  [<0000000000f8562a>] ret_from_fork+0xa/0x40
[    1.677748] Last Breaking-Event-Address:
[    1.677757]  [<0000000000579fa2>] load_elf_binary+0x302/0x11c0
[    1.678012] Kernel panic - not syncing: Fatal exception: panic_on_oops

The rootfs image is available in our boot-utils repo [1], should it be necessary for reproducing this issue. I do not see any problems with booting a CONFIG_MARCH_Z13 kernel compiled with GCC 12.2.1 from Fedora but it is possible that clang and GCC generate different code. I am not sure if this is a compiler problem or an emulation one, which is where I was looking for some input from you all.

[1]: https://github.com/ClangBuiltLinux/boot-utils

Cheers,
Nathan

#1 lore raw

Am 13.09.22 um 19:34 schrieb Nathan Chancellor:

Quoted 24 lines

Hi all,

Recently, Fedora moved their baseline from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13:

https://src.fedoraproject.org/rpms/kernel/c/aff6e8acdaa437e9f06ef4166ca2209071223f8d

Unfortunately, this causes clang built kernels to have an "illegal operation" panic when booting in QEMU. This is trivially reproducible on mainline with the versions of clang that s390 supports.

# Switch from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13

$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig menuconfig bzImage
$ qemu-system-s390x \
(quoted diff) +0 -8
-M s390-ccw-virtio \
-kernel arch/s390/boot/bzImage \
-display none \
-initrd .../boot-utils/images/s390/rootfs.cpio \
-m 512m \
-nodefaults \
-no-reboot \
-serial mon:stdio
...
Quoted 6 lines
[    0.558817] Linux version 6.0.0-rc5-00017-gd1221cea11fc ([email protected]) (clang version 14.0.5 (Fedora 14.0.5-7.fc38), GNU ld version 2.38-4.fc37) #1 SMP Tue Sep 13 08:35:38 MST 2022
...
[    1.675787] illegal operation: 0001 ilc:3 [#1] SMP
[    1.675888] Modules linked in:
[    1.676044] CPU: 0 PID: 59 Comm: modprobe Not tainted 6.0.0-rc5-00017-gd1221cea11fc #1
[    1.676134] Hardware name: QEMU 8561 QEMU (KVM/Linux)

What qemu version is that and what command line did you use (any cpu model?)

Quoted 9 lines
[    1.676202] Krnl PSW : 0704d00180000000 0000000000579fbc (load_elf_binary+0x31c/0x11c0)
[    1.676459]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
[    1.676489] Krnl GPRS: 0000000000000001 00000000044f0000 0000000000000000 0000000000000000
[    1.676506]            8000000000000080 0000000000000000 0000000000000000 00000000035fa900
[    1.676522]            0000000000000001 000000000353ce00 0000000000000001 00000000044693c0
[    1.676538]            000000000276fa00 000000000446a000 0000000000579f9e 00000380002ebc78
[    1.677128] Krnl Code: 0000000000579fac: a7f405b8            brc     15,000000000057ab1c
[    1.677128]            0000000000579fb0: e31003400004        lg      %r1,832
[    1.677128]           #0000000000579fb6: e3001780003b        lzrf    %r0,1920(%r1)

QEMU does support lzrf as far asI can tell as soon as you have S390_FEAT_STFLE_53 which is part of the qemu_V4_1 cpu model.

Quoted 28 lines
[    1.677128]           >0000000000579fbc: 50001780            st      %r0,1920(%r1)
[    1.677128]            0000000000579fc0: e31003400004        lg      %r1,832
[    1.677128]            0000000000579fc6: c40800aa99a9        lgrl    %r0,0000000001acd318
[    1.677128]            0000000000579fcc: e3001d400124        stg     %r0,7488(%r1)
[    1.677128]            0000000000579fd2: e31003400004        lg      %r1,832
[    1.677343] Call Trace:
[    1.677564]  [<0000000000579fbc>] load_elf_binary+0x31c/0x11c0
[    1.677635] ([<0000000000579f9e>] load_elf_binary+0x2fe/0x11c0)
[    1.677652]  [<00000000004e1ae6>] bprm_execve+0x4f6/0x7b0
[    1.677671]  [<00000000004e1256>] kernel_execve+0x3b6/0x3d0
[    1.677687]  [<000000000017e1c8>] call_usermodehelper_exec_async+0x158/0x1d0
[    1.677706]  [<0000000000103a9a>] __ret_from_fork+0x3a/0x60
[    1.677719]  [<0000000000f8562a>] ret_from_fork+0xa/0x40
[    1.677748] Last Breaking-Event-Address:
[    1.677757]  [<0000000000579fa2>] load_elf_binary+0x302/0x11c0
[    1.678012] Kernel panic - not syncing: Fatal exception: panic_on_oops

The rootfs image is available in our boot-utils repo [1], should it be necessary for reproducing this issue. I do not see any problems with booting a CONFIG_MARCH_Z13 kernel compiled with GCC 12.2.1 from Fedora but it is possible that clang and GCC generate different code. I am not sure if this is a compiler problem or an emulation one, which is where I was looking for some input from you all.

[1]: https://github.com/ClangBuiltLinux/boot-utils

Cheers,
Nathan

#2 lore raw
Quoted 25 lines · Christian Borntraeger

Am 13.09.22 um 19:34 schrieb Nathan Chancellor:

Quoted 24 lines

Hi all,

Recently, Fedora moved their baseline from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13:

https://src.fedoraproject.org/rpms/kernel/c/aff6e8acdaa437e9f06ef4166ca2209071223f8d

Unfortunately, this causes clang built kernels to have an "illegal operation" panic when booting in QEMU. This is trivially reproducible on mainline with the versions of clang that s390 supports.

# Switch from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13

$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig menuconfig bzImage
$ qemu-system-s390x \
(quoted diff) +0 -8
-M s390-ccw-virtio \
-kernel arch/s390/boot/bzImage \
-display none \
-initrd .../boot-utils/images/s390/rootfs.cpio \
-m 512m \
-nodefaults \
-no-reboot \
-serial mon:stdio
...
Quoted 8 lines
Quoted 6 lines
[    0.558817] Linux version 6.0.0-rc5-00017-gd1221cea11fc ([email protected]) (clang version 14.0.5 (Fedora 14.0.5-7.fc38), GNU ld version 2.38-4.fc37) #1 SMP Tue Sep 13 08:35:38 MST 2022
...
[    1.675787] illegal operation: 0001 ilc:3 [#1] SMP
[    1.675888] Modules linked in:
[    1.676044] CPU: 0 PID: 59 Comm: modprobe Not tainted 6.0.0-rc5-00017-gd1221cea11fc #1
[    1.676134] Hardware name: QEMU 8561 QEMU (KVM/Linux)

What qemu version is that and what command line did you use (any cpu model?)

I was able to reproduce this with QEMU 6.0.0 through 7.1.0, which are the versions that can boot a clang built kernel. I don't think we specify a cpu model, you can see the full command above. If there is something we should be doing differently, please let us know!

Quoted 41 lines
Quoted 9 lines
[    1.676202] Krnl PSW : 0704d00180000000 0000000000579fbc (load_elf_binary+0x31c/0x11c0)
[    1.676459]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
[    1.676489] Krnl GPRS: 0000000000000001 00000000044f0000 0000000000000000 0000000000000000
[    1.676506]            8000000000000080 0000000000000000 0000000000000000 00000000035fa900
[    1.676522]            0000000000000001 000000000353ce00 0000000000000001 00000000044693c0
[    1.676538]            000000000276fa00 000000000446a000 0000000000579f9e 00000380002ebc78
[    1.677128] Krnl Code: 0000000000579fac: a7f405b8            brc     15,000000000057ab1c
[    1.677128]            0000000000579fb0: e31003400004        lg      %r1,832
[    1.677128]           #0000000000579fb6: e3001780003b        lzrf    %r0,1920(%r1)

QEMU does support lzrf as far asI can tell as soon as you have S390_FEAT_STFLE_53 which is part of the qemu_V4_1 cpu model.

Quoted 28 lines
[    1.677128]           >0000000000579fbc: 50001780            st      %r0,1920(%r1)
[    1.677128]            0000000000579fc0: e31003400004        lg      %r1,832
[    1.677128]            0000000000579fc6: c40800aa99a9        lgrl    %r0,0000000001acd318
[    1.677128]            0000000000579fcc: e3001d400124        stg     %r0,7488(%r1)
[    1.677128]            0000000000579fd2: e31003400004        lg      %r1,832
[    1.677343] Call Trace:
[    1.677564]  [<0000000000579fbc>] load_elf_binary+0x31c/0x11c0
[    1.677635] ([<0000000000579f9e>] load_elf_binary+0x2fe/0x11c0)
[    1.677652]  [<00000000004e1ae6>] bprm_execve+0x4f6/0x7b0
[    1.677671]  [<00000000004e1256>] kernel_execve+0x3b6/0x3d0
[    1.677687]  [<000000000017e1c8>] call_usermodehelper_exec_async+0x158/0x1d0
[    1.677706]  [<0000000000103a9a>] __ret_from_fork+0x3a/0x60
[    1.677719]  [<0000000000f8562a>] ret_from_fork+0xa/0x40
[    1.677748] Last Breaking-Event-Address:
[    1.677757]  [<0000000000579fa2>] load_elf_binary+0x302/0x11c0
[    1.678012] Kernel panic - not syncing: Fatal exception: panic_on_oops

The rootfs image is available in our boot-utils repo [1], should it be necessary for reproducing this issue. I do not see any problems with booting a CONFIG_MARCH_Z13 kernel compiled with GCC 12.2.1 from Fedora but it is possible that clang and GCC generate different code. I am not sure if this is a compiler problem or an emulation one, which is where I was looking for some input from you all.

[1]: https://github.com/ClangBuiltLinux/boot-utils

Cheers,
Nathan

#3 lore raw

Am 14.09.22 um 11:00 schrieb Nathan Chancellor:

Quoted 35 lines · Christian Borntraeger
Quoted 25 lines

Am 13.09.22 um 19:34 schrieb Nathan Chancellor:

Quoted 24 lines

Hi all,

Recently, Fedora moved their baseline from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13:

https://src.fedoraproject.org/rpms/kernel/c/aff6e8acdaa437e9f06ef4166ca2209071223f8d

Unfortunately, this causes clang built kernels to have an "illegal operation" panic when booting in QEMU. This is trivially reproducible on mainline with the versions of clang that s390 supports.

# Switch from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13

$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig menuconfig bzImage
$ qemu-system-s390x \
(quoted diff) +0 -8
-M s390-ccw-virtio \
-kernel arch/s390/boot/bzImage \
-display none \
-initrd .../boot-utils/images/s390/rootfs.cpio \
-m 512m \
-nodefaults \
-no-reboot \
-serial mon:stdio
...
Quoted 8 lines
Quoted 6 lines
[    0.558817] Linux version 6.0.0-rc5-00017-gd1221cea11fc ([email protected]) (clang version 14.0.5 (Fedora 14.0.5-7.fc38), GNU ld version 2.38-4.fc37) #1 SMP Tue Sep 13 08:35:38 MST 2022
...
[    1.675787] illegal operation: 0001 ilc:3 [#1] SMP
[    1.675888] Modules linked in:
[    1.676044] CPU: 0 PID: 59 Comm: modprobe Not tainted 6.0.0-rc5-00017-gd1221cea11fc #1
[    1.676134] Hardware name: QEMU 8561 QEMU (KVM/Linux)

What qemu version is that and what command line did you use (any cpu model?)

Quoted 23 lines

I was able to reproduce this with QEMU 6.0.0 through 7.1.0, which are the versions that can boot a clang built kernel. I don't think we specify a cpu model, you can see the full command above. If there is something we should be doing differently, please let us know!

Quoted 18 lines
Quoted 9 lines
[    1.676202] Krnl PSW : 0704d00180000000 0000000000579fbc (load_elf_binary+0x31c/0x11c0)
[    1.676459]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
[    1.676489] Krnl GPRS: 0000000000000001 00000000044f0000 0000000000000000 0000000000000000
[    1.676506]            8000000000000080 0000000000000000 0000000000000000 00000000035fa900
[    1.676522]            0000000000000001 000000000353ce00 0000000000000001 00000000044693c0
[    1.676538]            000000000276fa00 000000000446a000 0000000000579f9e 00000380002ebc78
[    1.677128] Krnl Code: 0000000000579fac: a7f405b8            brc     15,000000000057ab1c
[    1.677128]            0000000000579fb0: e31003400004        lg      %r1,832
[    1.677128]           #0000000000579fb6: e3001780003b        lzrf    %r0,1920(%r1)

QEMU does support lzrf as far asI can tell as soon as you have S390_FEAT_STFLE_53 which is part of the qemu_V4_1 cpu model.

Quoted 5 lines
[    1.677128]           >0000000000579fbc: 50001780            st      %r0,1920(%r1)
[    1.677128]            0000000000579fc0: e31003400004        lg      %r1,832
[    1.677128]            0000000000579fc6: c40800aa99a9        lgrl    %r0,0000000001acd318
[    1.677128]            0000000000579fcc: e3001d400124        stg     %r0,7488(%r1)
[    1.677128]            0000000000579fd2: e31003400004        lg      %r1,832

Can you check if the following qemu patch helps?

target/s390x/tcg/insn-data.def +1 -1
index 5e448bb2c488..6bb479340a38 100644
@@ -466,7 +466,7 @@
     C(0xe39f, LAT,     RXY_a, LAT, 0, m2_32u, r1, 0, lat, 0)
     C(0xe385, LGAT,    RXY_a, LAT, 0, a2, r1, 0, lgat, 0)
 /* LOAD AND ZERO RIGHTMOST BYTE */
-    C(0xe3eb, LZRF,    RXY_a, LZRB, 0, m2_32u, new, r1_32, lzrb, 0)
+    C(0xe33b, LZRF,    RXY_a, LZRB, 0, m2_32u, new, r1_32, lzrb, 0)
     C(0xe32a, LZRG,    RXY_a, LZRB, 0, m2_64, r1, 0, lzrb, 0)
 /* LOAD LOGICAL AND ZERO RIGHTMOST BYTE */
     C(0xe33a, LLZRGF,  RXY_a, LZRB, 0, m2_32u, r1, 0, lzrb, 0)
author Tested-by
#4 lore raw
Quoted 54 lines · Christian Borntraeger

Am 14.09.22 um 11:00 schrieb Nathan Chancellor:

Quoted 53 lines · Christian Borntraeger
Quoted 25 lines

Am 13.09.22 um 19:34 schrieb Nathan Chancellor:

Quoted 24 lines

Hi all,

Recently, Fedora moved their baseline from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13:

https://src.fedoraproject.org/rpms/kernel/c/aff6e8acdaa437e9f06ef4166ca2209071223f8d

Unfortunately, this causes clang built kernels to have an "illegal operation" panic when booting in QEMU. This is trivially reproducible on mainline with the versions of clang that s390 supports.

# Switch from CONFIG_MARCH_ZEC12 to CONFIG_MARCH_Z13

$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig menuconfig bzImage
$ qemu-system-s390x \
(quoted diff) +0 -8
-M s390-ccw-virtio \
-kernel arch/s390/boot/bzImage \
-display none \
-initrd .../boot-utils/images/s390/rootfs.cpio \
-m 512m \
-nodefaults \
-no-reboot \
-serial mon:stdio
...
Quoted 8 lines
Quoted 6 lines
[    0.558817] Linux version 6.0.0-rc5-00017-gd1221cea11fc ([email protected]) (clang version 14.0.5 (Fedora 14.0.5-7.fc38), GNU ld version 2.38-4.fc37) #1 SMP Tue Sep 13 08:35:38 MST 2022
...
[    1.675787] illegal operation: 0001 ilc:3 [#1] SMP
[    1.675888] Modules linked in:
[    1.676044] CPU: 0 PID: 59 Comm: modprobe Not tainted 6.0.0-rc5-00017-gd1221cea11fc #1
[    1.676134] Hardware name: QEMU 8561 QEMU (KVM/Linux)

What qemu version is that and what command line did you use (any cpu model?)

I was able to reproduce this with QEMU 6.0.0 through 7.1.0, which are the versions that can boot a clang built kernel. I don't think we specify a cpu model, you can see the full command above. If there is something we should be doing differently, please let us know!

Quoted 13 lines
Quoted 9 lines
[    1.676202] Krnl PSW : 0704d00180000000 0000000000579fbc (load_elf_binary+0x31c/0x11c0)
[    1.676459]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
[    1.676489] Krnl GPRS: 0000000000000001 00000000044f0000 0000000000000000 0000000000000000
[    1.676506]            8000000000000080 0000000000000000 0000000000000000 00000000035fa900
[    1.676522]            0000000000000001 000000000353ce00 0000000000000001 00000000044693c0
[    1.676538]            000000000276fa00 000000000446a000 0000000000579f9e 00000380002ebc78
[    1.677128] Krnl Code: 0000000000579fac: a7f405b8            brc     15,000000000057ab1c
[    1.677128]            0000000000579fb0: e31003400004        lg      %r1,832
[    1.677128]           #0000000000579fb6: e3001780003b        lzrf    %r0,1920(%r1)

QEMU does support lzrf as far asI can tell as soon as you have S390_FEAT_STFLE_53 which is part of the qemu_V4_1 cpu model.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
[    1.677128]           >0000000000579fbc: 50001780            st      %r0,1920(%r1)
[    1.677128]            0000000000579fc0: e31003400004        lg      %r1,832
[    1.677128]            0000000000579fc6: c40800aa99a9        lgrl    %r0,0000000001acd318
[    1.677128]            0000000000579fcc: e3001d400124        stg     %r0,7488(%r1)
[    1.677128]            0000000000579fd2: e31003400004        lg      %r1,832

Can you check if the following qemu patch helps?

Yes, that appears to work for me! If you send it along formally, feel free to include:

Quoted 14 lines
target/s390x/tcg/insn-data.def +1 -1
index 5e448bb2c488..6bb479340a38 100644
@@ -466,7 +466,7 @@
     C(0xe39f, LAT,     RXY_a, LAT, 0, m2_32u, r1, 0, lat, 0)
     C(0xe385, LGAT,    RXY_a, LAT, 0, a2, r1, 0, lgat, 0)
 /* LOAD AND ZERO RIGHTMOST BYTE */
-    C(0xe3eb, LZRF,    RXY_a, LZRB, 0, m2_32u, new, r1_32, lzrb, 0)
+    C(0xe33b, LZRF,    RXY_a, LZRB, 0, m2_32u, new, r1_32, lzrb, 0)
     C(0xe32a, LZRG,    RXY_a, LZRB, 0, m2_64, r1, 0, lzrb, 0)
 /* LOAD LOGICAL AND ZERO RIGHTMOST BYTE */
     C(0xe33a, LLZRGF,  RXY_a, LZRB, 0, m2_32u, r1, 0, lzrb, 0)