s390x/tcg: Fix RISBHG
PATCH 1/1 s390x/tcg: Fix RISBHG
RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.
[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 0000200180000000 0000000000017a1e
R:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3
GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0
0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0
00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8
0000000000000009 0000000000000002 0000000000000008 000000000000bce0One example of a buggy instruction is:
17dde: ec 1e 00 9f 20 5d risbhg %r1,%r14,0,159,32
With %r14 = 0x9 and %r1 = 0x7 should result in %r1 = 0x900000007, however, results in %r1 = 0.
Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.
With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.
---
This BUG was a nightmare to debug and the code a nightmare to understand.
To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]
--- target/s390x/translate.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-)
target/s390x/translate.c +8 -10
@@ -3815,22 +3815,23 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) pmask =0xffffffff00000000ull ;break ;case 0x51 :/* risblg */ - i3 &=31 ; - i4 &=31 ; + i3 = (i3 &31 ) +32 ; + i4 = (i4 &31 ) +32 ; pmask =0x00000000ffffffffull ;break ;default :g_assert_not_reached (); } -/* MASK is the set of bits to be inserted from R2. -Take care for I3/I4 wraparound. */ - mask = pmask >> i3; +/* MASK is the set of bits to be inserted from R2. */ if (i3 <= i4) { - mask ^= pmask >> i4 >>1 ; +/* [0...i3---i4...63] */ + mask = (-1ull >> i3) & (-1ull << (63 - i4)); }else { - mask |= ~(pmask >> i4 >>1 ); +/* [0---i4...i3---63] */ + mask = (-1ull >> i3) | (-1ull << (63 - i4)); } +/* For RISBLG/RISBHG, the wrapping is limited to the high/low doubleword. */ mask &= pmask;/* IMASK is the set of bits to be kept from R1. In the case of the high/low
@@ -3843,9 +3844,6 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) len = i4 - i3 +1 ; pos =63 - i4; rot = i5 &63 ; -if (s->fields.op2 ==0x5d ) { - pos +=32 ; - }/* In some cases we can implement this with extract. */ if (imask ==0 && pos ==0 && len >0 && len <= rot) {
Signature · 1 line
2.29.2
Quoted 39 lines
Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:
Quoted 35 lines · David Hildenbrand
RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.
[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 00002001800000000000000000017a1eR:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3 GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0 0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0 00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8 0000000000000009 0000000000000002 0000000000000008 000000000000bce0One example of a buggy instruction is:
17dde: ec 1e 00 9f 20 5d risbhg %r1,%r14,0,159,32With %r14 =
0x9and %r1 =0x7should result in %r1 =0x900000007, however, results in %r1 = 0.Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.
With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.
Fixes 2d6a869833d9 ("target-s390: Implement RISBG")Reported-by Nick Desaulniers <[email protected]>Cc Guenter Roeck <[email protected]>Cc Christian Borntraeger <[email protected]>Signed-off-by David Hildenbrand <[email protected]>---
Quoted 8 lines
Quoted 5 lines
This BUG was a nightmare to debug and the code a nightmare to understand.
To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]In that case, a huge thank you!!! for this work! ++beers_owed.
:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.
Quoted 23 lines
Quoted 23 lines
---
target/s390x/translate.c| 18 ++++++++----------1 file changed, 8 insertions(+), 10 deletions(-)target/s390x/translate.c +2
-2@@ -3815,22 +3815,23 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) pmask =0xffffffff00000000ull ;break ;case 0x51 :/* risblg */ - i3 &=31 ; - i4 &=31 ; + i3 = (i3 &31 ) +32 ; + i4 = (i4 &31 ) +32 ; pmask =0x00000000ffffffffull ;break ;default :g_assert_not_reached (); }
Quoted 16 lines
target/s390x/translate.c static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) · line 3833
Quoted 13 lines
target/s390x/translate.c +5
-5-/* MASK is the set of bits to be inserted from R2. -Take care for I3/I4 wraparound. */ - mask = pmask >> i3; +/* MASK is the set of bits to be inserted from R2. */ if (i3 <= i4) { - mask ^= pmask >> i4 >>1 ; +/* [0...i3---i4...63] */ + mask = (-1ull >> i3) & (-1ull << (63 - i4)); }else { - mask |= ~(pmask >> i4 >>1 ); +/* [0---i4...i3---63] */ + mask = (-1ull >> i3) | (-1ull << (63 - i4)); }The expression evaluated looks the same to me for both sides of the conditional, but the comments differ. Intentional?
& vs |, so the result differs.
Cheers!
Quoted 33 lines · David Hildenbrand
RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.
[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 00002001800000000000000000017a1eR:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3 GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0 0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0 00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8 0000000000000009 0000000000000002 0000000000000008 000000000000bce0One example of a buggy instruction is:
17dde: ec 1e 00 9f 20 5d risbhg %r1,%r14,0,159,32With %r14 =
0x9and %r1 =0x7should result in %r1 =0x900000007, however, results in %r1 = 0.Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.
With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.
Fixes 2d6a869833d9 ("target-s390: Implement RISBG")Reported-by Nick Desaulniers <[email protected]>Cc Guenter Roeck <[email protected]>Cc Christian Borntraeger <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Quoted 7 lines
---
This BUG was a nightmare to debug and the code a nightmare to understand.
To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]
In that case, a huge thank you!!! for this work! ++beers_owed.
Quoted 26 lines
--- target/s390x/translate.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-)target/s390x/translate.c +2
-5@@ -3815,22 +3815,23 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) pmask =0xffffffff00000000ull ;break ;case 0x51 :/* risblg */ - i3 &=31 ; - i4 &=31 ; + i3 = (i3 &31 ) +32 ; + i4 = (i4 &31 ) +32 ; pmask =0x00000000ffffffffull ;break ;default :g_assert_not_reached (); } -/* MASK is the set of bits to be inserted from R2. -Take care for I3/I4 wraparound. */ - mask = pmask >> i3;
target/s390x/translate.c static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) · line 3833
Quoted 10 lines
target/s390x/translate.c +5
-2+/* MASK is the set of bits to be inserted from R2. */ if (i3 <= i4) { - mask ^= pmask >> i4 >>1 ; +/* [0...i3---i4...63] */ + mask = (-1ull >> i3) & (-1ull << (63 - i4)); }else { - mask |= ~(pmask >> i4 >>1 ); +/* [0---i4...i3---63] */ + mask = (-1ull >> i3) | (-1ull << (63 - i4)); }
The expression evaluated looks the same to me for both sides of the conditional, but the comments differ. Intentional?
Quoted 17 lines
target/s390x/translate.c +1
-4+/* For RISBLG/RISBHG, the wrapping is limited to the high/low doubleword. */ mask &= pmask;/* IMASK is the set of bits to be kept from R1. In the case of the high/low @@ -3843,9 +3844,6 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) len = i4 - i3 +1 ; pos =63 - i4; rot = i5 &63 ; -if (s->fields.op2 ==0x5d ) { - pos +=32 ; - }/* In some cases we can implement this with extract. */ if (imask ==0 && pos ==0 && len >0 && len <= rot) { --2.29.2
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 42 lines · David Hildenbrand
Quoted 42 lines
Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:
Quoted 37 lines · David Hildenbrand
RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.
[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 00002001800000000000000000017a1eR:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3 GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0 0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0 00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8 0000000000000009 0000000000000002 0000000000000008 000000000000bce0One example of a buggy instruction is:
17dde: ec 1e 00 9f 20 5d risbhg %r1,%r14,0,159,32With %r14 =
0x9and %r1 =0x7should result in %r1 =0x900000007, however, results in %r1 = 0.Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.
With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.
Fixes 2d6a869833d9 ("target-s390: Implement RISBG")Reported-by Nick Desaulniers <[email protected]>Cc Guenter Roeck <[email protected]>Cc Christian Borntraeger <[email protected]>Signed-off-by David Hildenbrand <[email protected]>---
This BUG was a nightmare to debug and the code a nightmare to understand.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]In that case, a huge thank you!!! for this work! ++beers_owed.
:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.
Hmm...so I don't think clang can build a Linux kernel image with CONFIG_MARCH_Z13=y just yet; just defconfig. Otherwise looks like clang barfs on some of the inline asm constraints.
It looks like with your patch applied we get further into the boot! I'm not seeing any output with:
$ /android0/qemu/build/qemu-system-s390x -cpu qemu -append 'conmode=sclp console=ttyS0' -display none -initrd /<path/to>/boot-utils/images/s390/rootfs.cpio -kernel arch/s390/boot/bzImage -m 512m -nodefaults -serial mon:stdio
(Based on a quick skim through
https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.ludd/ludd_r_lmtkernelparameter.html).
Do I have all of those right?
If I attach GDB to QEMU running that kernel image, I was able to view the print banner once via lx-dmesg gdb macro in the kernel, but it seems on subsequent runs control flow gets diverted unexpected post entry to start_kernel() always to s390_base_pgm_handler ...errr..at least when I try to single step in GDB. Tried with linux-5.10.y, mainline, and linux-next.
qemu: 470dd6bd360782f5137f7e3376af6a44658eb1d3 + your patch
llvm: 106e66f3f555c8f887e82c5f04c3e77bdaf345e8 linux-5.10.y: d1988041d19dc8b532579bdbb7c4a978391c0011
linux: 71c061d2443814de15e177489d5cc00a4a253ef3
linux-next: f87684f6470f5f02bd47d4afb900366e5d2f31b6
(gdb) hbreak setup_arch
Hardware assisted breakpoint 1 at 0x142229e: filearch/s390/kernel/setup.c, line 1091.
(gdb) c
Continuing.
Program received signal SIGTRAP, Trace/breakpoint trap. 0x00000000014222a0 in setup_arch (cmdline_p=0x11d7ed8) at arch/s390/kernel/setup.c:1091
1091 if (MACHINE_IS_VM)
(gdb) lx-dmesg
[ 0.376351] Linux version 5.11.0-rc2-00157-ga2885c701c30
([email protected]) (Nick Desaulniers clang version 12.0.0 ([email protected]:llvm/llvm-project.git e75fec2b238f0e26cfb7645f2208baebe3440d41), GNU ld (GNU Binutils for Debian) 2.35.1) #81 SMP Thu Jan 7 17:57:34 PST 2021
Quoted 41 lines
Quoted 24 lines
Quoted 24 lines
---
target/s390x/translate.c| 18 ++++++++----------1 file changed, 8 insertions(+), 10 deletions(-)target/s390x/translate.c +2
-2@@ -3815,22 +3815,23 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) pmask =0xffffffff00000000ull ;break ;case 0x51 :/* risblg */ - i3 &=31 ; - i4 &=31 ; + i3 = (i3 &31 ) +32 ; + i4 = (i4 &31 ) +32 ; pmask =0x00000000ffffffffull ;break ;default :g_assert_not_reached (); }Quoted 16 lines
target/s390x/translate.c static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) · line 3833
Quoted 13 lines
target/s390x/translate.c +5
-5-/* MASK is the set of bits to be inserted from R2. -Take care for I3/I4 wraparound. */ - mask = pmask >> i3; +/* MASK is the set of bits to be inserted from R2. */ if (i3 <= i4) { - mask ^= pmask >> i4 >>1 ; +/* [0...i3---i4...63] */ + mask = (-1ull >> i3) & (-1ull << (63 - i4)); }else { - mask |= ~(pmask >> i4 >>1 ); +/* [0---i4...i3---63] */ + mask = (-1ull >> i3) | (-1ull << (63 - i4)); }The expression evaluated looks the same to me for both sides of the conditional, but the comments differ. Intentional?
Quoted 1 line
& vs |, so the result differs.
D'oh, I can never spot single character differences!
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 47 lines · Nick Desaulniers
Quoted 46 lines · David Hildenbrand
Quoted 46 lines
Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:
Quoted 40 lines · David Hildenbrand
RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.
[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 00002001800000000000000000017a1eR:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3 GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0 0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0 00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8 0000000000000009 0000000000000002 0000000000000008 000000000000bce0One example of a buggy instruction is:
17dde: ec 1e 00 9f 20 5d risbhg %r1,%r14,0,159,32With %r14 =
0x9and %r1 =0x7should result in %r1 =0x900000007, however, results in %r1 = 0.Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.
With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.
Fixes 2d6a869833d9 ("target-s390: Implement RISBG")Reported-by Nick Desaulniers <[email protected]>Cc Guenter Roeck <[email protected]>Cc Christian Borntraeger <[email protected]>Signed-off-by David Hildenbrand <[email protected]>---
This BUG was a nightmare to debug and the code a nightmare to understand.
To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
In that case, a huge thank you!!! for this work! ++beers_owed.
:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.
Hmm...so I don't think clang can build a Linux kernel image with
CONFIG_MARCH_Z13=yjust yet; just defconfig. Otherwise looks like clang barfs on some of the inline asm constraints.
Ah, right. I overwrote my manual config by a temporary defconfig :)
So, I'm on x86-64 F33.
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
LLVM version 11.0.0
I cannot directly use "LLVM=1" for cross-compilation, as I keep getting "error: unknown emulation: elf64_s390" from ld.lld and "error: invalid output format: 'elf64-s390'" from llvm-objcopy. I assume that's fixed in llvm12?
1. I patch around it (strange, I remember CC= .. used to work, but it no longer does)
---
index e30cf02da8b8..89c57062ed5d 100644
Makefile +7 -7
@@ -427,13 +427,13 @@ KBUILD_HOSTLDLIBS := $(HOST_LFS_LIBS) $(HOSTLDLIBS) CPP = $(CC) -E ifneq ($(LLVM),) CC = clang -LD = ld.lld -AR = llvm-ar -NM = llvm-nm -OBJCOPY = llvm-objcopy -OBJDUMP = llvm-objdump -READELF = llvm-readelf -STRIP = llvm-strip +LD = $(CROSS_COMPILE)ld +AR = $(CROSS_COMPILE)ar +NM = $(CROSS_COMPILE)nm +OBJCOPY = $(CROSS_COMPILE)objcopy +OBJDUMP = $(CROSS_COMPILE)objdump +READELF = $(CROSS_COMPILE)readelf +STRIP = $(CROSS_COMPILE)strip else CC = $(CROSS_COMPILE)gcc LD = $(CROSS_COMPILE)ld
---
2. Compile using clang
Using latest linux-next (1c925d2030afd354a02c23500386e620e662622b) + above patch
---
#!/bin/bash
export ARCH=s390;
export CROSS_COMPILE=s390x-linux-gnu-
export LLVM=1
make distclean
make defconfig
# Make F32 initrd boot without inserting modules./scripts/config -e CONFIG_SCSI_ISCSI_ATTRS./scripts/config -e CONFIG_ISCSI_TCP
make -j40 > /dev/null
---
3. Run it via QEMU. I boot a full Fedora 32 using the cloud-image + initrd from Fedora 32 (tried to stick to your cmdline where possible)
./build/qemu-system-s390x \
-m 512M \
-cpu qemu \
-display none \
-nodefaults \
-kernel ../linux-cross/arch/s390/boot/bzImage \
-append "root=/dev/vda1 conmode=sclp console=ttyS0" \
-initrd ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \
-hda ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \
-serial mon:stdio
KASLR disabled: CPU has no PRNG
[ 0.408769] Linux version 5.11.0-rc2-next-20210108-dirty (dhildenb@desktop) (clang version 11.0.0 (Fedora 11.0.0-2.fc33), GNU ld version 2.35.1-1.fc33) #1 SMP Fri Jan 8 10:23:01 CET 2021 [ 0.410266] setup: Linux is running under KVM in 64-bit mode [ 0.415840] setup: The maximum memory size is 512MB [ 0.417278] cpu: 1 configured CPUs, 0 standby CPUs
...
Fedora 32 (Cloud Edition)
Kernel 5.11.0-rc2-next-20210108-dirty on an s390x (ttysclp0)
atomic-00 login:
Quoted 36 lines
It looks like with your patch applied we get further into the boot! I'm not seeing any output with:
$ /android0/qemu/build/qemu-system-s390x -cpu qemu -append 'conmode=sclp console=ttyS0' -display none -initrd /<path/to>/boot-utils/images/s390/rootfs.cpio -kernel arch/s390/boot/bzImage -m 512m -nodefaults -serial mon:stdio(Based on a quick skim through
https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.ludd/ludd_r_lmtkernelparameter.html).
Do I have all of those right?If I attach GDB to QEMU running that kernel image, I was able to view the print banner once via
lx-dmesggdb macro in the kernel, but it seems on subsequent runs control flow gets diverted unexpected post entry tostart_kernel()always tos390_base_pgm_handler...errr..at least when I try to single step in GDB. Tried with linux-5.10.y, mainline, and linux-next.qemu:
470dd6bd360782f5137f7e3376af6a44658eb1d3+ your patch
llvm:106e66f3f555c8f887e82c5f04c3e77bdaf345e8linux-5.10.y:d1988041d19dc8b532579bdbb7c4a978391c0011
linux:71c061d2443814de15e177489d5cc00a4a253ef3
linux-next:f87684f6470f5f02bd47d4afb900366e5d2f31b6(gdb) hbreak
setup_arch
Hardware assisted breakpoint 1 at0x142229e: filearch/s390/kernel/setup.c, line 1091.
(gdb) c
Continuing.Program received signal SIGTRAP, Trace/breakpoint trap.
0x00000000014222a0insetup_arch(cmdline_p=0x11d7ed8) atarch/s390/kernel/setup.c:1091
1091 if (MACHINE_IS_VM)
(gdb) lx-dmesg
Quoted 5 lines
[ 0.376351] Linux version 5.11.0-rc2-00157-ga2885c701c30([email protected]) (Nick Desaulniers clang version 12.0.0 ([email protected]:llvm/llvm-project.git
e75fec2b238f0e26cfb7645f2208baebe3440d41), GNU ld (GNU Binutils for Debian) 2.35.1) #81 SMP Thu Jan 7 17:57:34 PST 2021
So you're using llvm 12. Maybe that makes a difference. Or we have an issue with our arm64 backend. Or using ld.lld and friends make a difference. Guess I'd have to custom-compile llvm12 (gah) ... maybe I can find some rpms somewhere.
Signature · 3 lines
Thanks, David / dhildenb
Quoted 130 lines · David Hildenbrand
Quoted 47 lines · Nick Desaulniers
Quoted 46 lines · David Hildenbrand
Quoted 46 lines
Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:
Quoted 40 lines · David Hildenbrand
RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.
[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 00002001800000000000000000017a1eR:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3 GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0 0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0 00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8 0000000000000009 0000000000000002 0000000000000008 000000000000bce0One example of a buggy instruction is:
17dde: ec 1e 00 9f 20 5d risbhg %r1,%r14,0,159,32With %r14 =
0x9and %r1 =0x7should result in %r1 =0x900000007, however, results in %r1 = 0.Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.
With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.
Fixes 2d6a869833d9 ("target-s390: Implement RISBG")Reported-by Nick Desaulniers <[email protected]>Cc Guenter Roeck <[email protected]>Cc Christian Borntraeger <[email protected]>Signed-off-by David Hildenbrand <[email protected]>---
This BUG was a nightmare to debug and the code a nightmare to understand.
To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
In that case, a huge thank you!!! for this work! ++beers_owed.
:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.
Hmm...so I don't think clang can build a Linux kernel image with
CONFIG_MARCH_Z13=yjust yet; just defconfig. Otherwise looks like clang barfs on some of the inline asm constraints.Ah, right. I overwrote my manual config by a temporary defconfig :)
So, I'm on x86-64 F33.
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
LLVM version 11.0.0I cannot directly use "LLVM=1" for cross-compilation, as I keep getting "error: unknown emulation:
elf64_s390" from ld.lld and "error: invalid output format: 'elf64-s390'" from llvm-objcopy. I assume that's fixed in llvm12?1. I patch around it (strange, I remember CC= .. used to work, but it no longer does)
---
index
e30cf02da8b8..89c57062ed5d100644Makefile +7
-8@@ -427,13 +427,13 @@ KBUILD_HOSTLDLIBS := $(HOST_LFS_LIBS) $(HOSTLDLIBS) CPP = $(CC) -E ifneq ($(LLVM),) CC = clang -LD = ld.lld -AR = llvm-ar -NM = llvm-nm -OBJCOPY = llvm-objcopy -OBJDUMP = llvm-objdump -READELF = llvm-readelf -STRIP = llvm-strip +LD = $(CROSS_COMPILE)ld +AR = $(CROSS_COMPILE)ar +NM = $(CROSS_COMPILE)nm +OBJCOPY = $(CROSS_COMPILE)objcopy +OBJDUMP = $(CROSS_COMPILE)objdump +READELF = $(CROSS_COMPILE)readelf +STRIP = $(CROSS_COMPILE)strip else CC = $(CROSS_COMPILE)gcc LD = $(CROSS_COMPILE)ld ---2. Compile using clang
Using latest linux-next (
1c925d2030afd354a02c23500386e620e662622b) + above patch---
#!/bin/bash
export ARCH=s390;
export CROSS_COMPILE=s390x-linux-gnu-
export LLVM=1
make distclean
make defconfig# Make F32 initrd boot without inserting modules
./scripts/config-eCONFIG_SCSI_ISCSI_ATTRS./scripts/config-eCONFIG_ISCSI_TCPmake -j40 >
/dev/null---
3. Run it via QEMU. I boot a full Fedora 32 using the cloud-image + initrd from Fedora 32 (tried to stick to your cmdline where possible)
./build/qemu-system-s390x \
Makefile
Quoted 8 lines
Makefile +0
-8-m 512M \ -cpu qemu \ -display none \ -nodefaults \ -kernel ../linux-cross/arch/s390/boot/bzImage \ -append "root=/dev/vda1 conmode=sclp console=ttyS0" \ -initrd ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \ -hda ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \
^ -hda ../Fedora-Cloud-Base-32-1.6.x86_64.qcow2 \
Signature · 3 lines
Thanks, David / dhildenb
Quoted 130 lines · David Hildenbrand
Quoted 129 lines · David Hildenbrand
Quoted 47 lines · Nick Desaulniers
Quoted 46 lines · David Hildenbrand
Quoted 46 lines
Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:
Quoted 40 lines · David Hildenbrand
RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.
[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 00002001800000000000000000017a1eR:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3 GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0 0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0 00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8 0000000000000009 0000000000000002 0000000000000008 000000000000bce0One example of a buggy instruction is:
17dde: ec 1e 00 9f 20 5d risbhg %r1,%r14,0,159,32With %r14 =
0x9and %r1 =0x7should result in %r1 =0x900000007, however, results in %r1 = 0.Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.
With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.
Fixes 2d6a869833d9 ("target-s390: Implement RISBG")Reported-by Nick Desaulniers <[email protected]>Cc Guenter Roeck <[email protected]>Cc Christian Borntraeger <[email protected]>Signed-off-by David Hildenbrand <[email protected]>---
This BUG was a nightmare to debug and the code a nightmare to understand.
To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
In that case, a huge thank you!!! for this work! ++beers_owed.
:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.
Hmm...so I don't think clang can build a Linux kernel image with
CONFIG_MARCH_Z13=yjust yet; just defconfig. Otherwise looks like clang barfs on some of the inline asm constraints.Ah, right. I overwrote my manual config by a temporary defconfig :)
So, I'm on x86-64 F33.
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
LLVM version 11.0.0I cannot directly use "LLVM=1" for cross-compilation, as I keep getting "error: unknown emulation:
elf64_s390" from ld.lld and "error: invalid output format: 'elf64-s390'" from llvm-objcopy. I assume that's fixed in llvm12?1. I patch around it (strange, I remember CC= .. used to work, but it no longer does)
---
index
e30cf02da8b8..89c57062ed5d100644Makefile +7
-8@@ -427,13 +427,13 @@ KBUILD_HOSTLDLIBS := $(HOST_LFS_LIBS) $(HOSTLDLIBS) CPP = $(CC) -E ifneq ($(LLVM),) CC = clang -LD = ld.lld -AR = llvm-ar -NM = llvm-nm -OBJCOPY = llvm-objcopy -OBJDUMP = llvm-objdump -READELF = llvm-readelf -STRIP = llvm-strip +LD = $(CROSS_COMPILE)ld +AR = $(CROSS_COMPILE)ar +NM = $(CROSS_COMPILE)nm +OBJCOPY = $(CROSS_COMPILE)objcopy +OBJDUMP = $(CROSS_COMPILE)objdump +READELF = $(CROSS_COMPILE)readelf +STRIP = $(CROSS_COMPILE)strip else CC = $(CROSS_COMPILE)gcc LD = $(CROSS_COMPILE)ld ---2. Compile using clang
Using latest linux-next (
1c925d2030afd354a02c23500386e620e662622b) + above patch---
#!/bin/bash
export ARCH=s390;
export CROSS_COMPILE=s390x-linux-gnu-
export LLVM=1
make distclean
make defconfig# Make F32 initrd boot without inserting modules
./scripts/config-eCONFIG_SCSI_ISCSI_ATTRS./scripts/config-eCONFIG_ISCSI_TCPmake -j40 >
/dev/null---
3. Run it via QEMU. I boot a full Fedora 32 using the cloud-image + initrd from Fedora 32 (tried to stick to your cmdline where possible)
Quoted 13 lines
Makefile
Quoted 9 lines
./build/qemu-system-s390x \
Makefile +0
-8-m 512M \ -cpu qemu \ -display none \ -nodefaults \ -kernel ../linux-cross/arch/s390/boot/bzImage \ -append "root=/dev/vda1 conmode=sclp console=ttyS0" \ -initrd ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \ -hda ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \^ -hda ../Fedora-Cloud-Base-32-1.6.x86_64.qcow2 \
Using homebrew llvm-12/clang-12 results in a kernel that is indeed stuck is in an endless program exception loop, seeming to try printing a backtrace over and over again. Yet, no output besides
"KASLR disabled: CPU has no PRNG"
Kernel runs under KVM, so more debugging to do to get clang-12 builds running as well.
Signature · 3 lines
Thanks, David / dhildenb
Quoted 64 lines · David Hildenbrand
Quoted 47 lines · Nick Desaulniers
Quoted 46 lines · David Hildenbrand
Quoted 46 lines
Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:
Quoted 40 lines · David Hildenbrand
RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.
[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 00002001800000000000000000017a1eR:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3 GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0 0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0 00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8 0000000000000009 0000000000000002 0000000000000008 000000000000bce0One example of a buggy instruction is:
17dde: ec 1e 00 9f 20 5d risbhg %r1,%r14,0,159,32With %r14 =
0x9and %r1 =0x7should result in %r1 =0x900000007, however, results in %r1 = 0.Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.
With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.
Fixes 2d6a869833d9 ("target-s390: Implement RISBG")Reported-by Nick Desaulniers <[email protected]>Cc Guenter Roeck <[email protected]>Cc Christian Borntraeger <[email protected]>Signed-off-by David Hildenbrand <[email protected]>---
This BUG was a nightmare to debug and the code a nightmare to understand.
To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
In that case, a huge thank you!!! for this work! ++beers_owed.
:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.
Hmm...so I don't think clang can build a Linux kernel image with
CONFIG_MARCH_Z13=yjust yet; just defconfig. Otherwise looks like clang barfs on some of the inline asm constraints.Ah, right. I overwrote my manual config by a temporary defconfig :)
So, I'm on x86-64 F33.
Quoted 7 lines
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
LLVM version 11.0.0I cannot directly use "LLVM=1" for cross-compilation, as I keep getting "error: unknown emulation:
elf64_s390" from ld.lld and "error: invalid output format: 'elf64-s390'" from llvm-objcopy. I assume that's fixed in llvm12?
Right, I suspect that even if ld.lld understood that emulation mode target, it would still fail due to lack of big endian support. We've been building with simply CC=clang for s390 linux kernels.
Via: https://www.kernel.org/doc/html/latest/kbuild/llvm.html#llvm-utilities
we usually start with make CC=clang then work our way up to make LLVM=1. So you shouldn't need the below patching, just use CC=clang.
Quoted 22 lines
1. I patch around it (strange, I remember CC= .. used to work, but it no longer does)
---
index
e30cf02da8b8..89c57062ed5d100644Makefile +2
-7@@ -427,13 +427,13 @@ KBUILD_HOSTLDLIBS := $(HOST_LFS_LIBS) $(HOSTLDLIBS) CPP = $(CC) -E ifneq ($(LLVM),) CC = clang -LD = ld.lld -AR = llvm-ar -NM = llvm-nm -OBJCOPY = llvm-objcopy -OBJDUMP = llvm-objdump -READELF = llvm-readelf -STRIP = llvm-strip +LD = $(CROSS_COMPILE)ld +AR = $(CROSS_COMPILE)ar
Makefile KBUILD_HOSTLDLIBS := $(HOST_LFS_LIBS) $(HOSTLDLIBS)
Quoted 10 lines
Makefile +5
-1+NM = $(CROSS_COMPILE)nm +OBJCOPY = $(CROSS_COMPILE)objcopy +OBJDUMP = $(CROSS_COMPILE)objdump +READELF = $(CROSS_COMPILE)readelf +STRIP = $(CROSS_COMPILE)strip else CC = $(CROSS_COMPILE)gcc LD = $(CROSS_COMPILE)ld ---
Pulling from your github branch, everything looks good; buildroot support looks good. I'll wire this up to our CI so that we can help report regressions!
Signature · 2 lines
Thanks, ~Nick Desaulniers