Skip to contentSkip to conversation

s390x/tcg: Fix RISBHG

PATCH Superseded David Hildenbrand 3 participants 8 messages Jan 7, 2021 – Jan 8, 2021 Kbuild

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 s390x/tcg: Fix RISBHG

RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.

[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 0000200180000000 0000000000017a1e

         R:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3
   GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0
         0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0
         00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8
         0000000000000009 0000000000000002 0000000000000008 000000000000bce0

One example of a buggy instruction is:

    17dde:       ec 1e 00 9f 20 5d       risbhg  %r1,%r14,0,159,32

With %r14 = 0x9 and %r1 = 0x7 should result in %r1 = 0x900000007, however, results in %r1 = 0.

Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.

With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.

Fixes 2d6a869833d9 ("target-s390: Implement RISBG")
Cc Guenter Roeck <[email protected]>
Cc Christian Borntraeger <[email protected]>

---

This BUG was a nightmare to debug and the code a nightmare to understand.

To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]

---
 target/s390x/translate.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
target/s390x/translate.c +8 -10
index 3d5c0d6106..39e33eeb67 100644
@@ -3815,22 +3815,23 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o)
         pmask = 0xffffffff00000000ull;
         break;
     case 0x51: /* risblg */
-        i3 &= 31;
-        i4 &= 31;
+        i3 = (i3 & 31) + 32;
+        i4 = (i4 & 31) + 32;
         pmask = 0x00000000ffffffffull;
         break;
     default:
         g_assert_not_reached();
     }
 
-    /* MASK is the set of bits to be inserted from R2.
-       Take care for I3/I4 wraparound.  */
-    mask = pmask >> i3;
+    /* MASK is the set of bits to be inserted from R2. */
     if (i3 <= i4) {
-        mask ^= pmask >> i4 >> 1;
+        /* [0...i3---i4...63] */
+        mask = (-1ull >> i3) & (-1ull << (63 - i4));
     } else {
-        mask |= ~(pmask >> i4 >> 1);
+        /* [0---i4...i3---63] */
+        mask = (-1ull >> i3) | (-1ull << (63 - i4));
     }
+    /* For RISBLG/RISBHG, the wrapping is limited to the high/low doubleword. */
     mask &= pmask;
 
     /* IMASK is the set of bits to be kept from R1.  In the case of the high/low
@@ -3843,9 +3844,6 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o)
     len = i4 - i3 + 1;
     pos = 63 - i4;
     rot = i5 & 63;
-    if (s->fields.op2 == 0x5d) {
-        pos += 32;
-    }
 
     /* In some cases we can implement this with extract.  */
     if (imask == 0 && pos == 0 && len > 0 && len <= rot) {
Signature · 1 line
2.29.2
#2 lore raw
In reply to Nick Desaulniers via
Quoted 39 lines

Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:

Quoted 35 lines · David Hildenbrand

RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.

[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 0000200180000000 0000000000017a1e

        R:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3
  GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0
        0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0
        00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8
        0000000000000009 0000000000000002 0000000000000008 000000000000bce0

One example of a buggy instruction is:

   17dde:       ec 1e 00 9f 20 5d       risbhg  %r1,%r14,0,159,32

With %r14 = 0x9 and %r1 = 0x7 should result in %r1 = 0x900000007, however, results in %r1 = 0.

Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.

With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.

Fixes 2d6a869833d9 ("target-s390: Implement RISBG")
Cc Guenter Roeck <[email protected]>
Cc Christian Borntraeger <[email protected]>

---

Quoted 8 lines
Quoted 5 lines

This BUG was a nightmare to debug and the code a nightmare to understand.

To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]

In that case, a huge thank you!!! for this work! ++beers_owed.

:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.

Quoted 23 lines
Quoted 23 lines

---
target/s390x/translate.c | 18 ++++++++----------

1 file changed, 8 insertions(+), 10 deletions(-)
target/s390x/translate.c +2 -2
index 3d5c0d6106..39e33eeb67 100644
@@ -3815,22 +3815,23 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o)
        pmask = 0xffffffff00000000ull;
        break;
    case 0x51: /* risblg */
-        i3 &= 31;
-        i4 &= 31;
+        i3 = (i3 & 31) + 32;
+        i4 = (i4 & 31) + 32;
        pmask = 0x00000000ffffffffull;
        break;
    default:
        g_assert_not_reached();
    }
 
Quoted 16 lines

target/s390x/translate.c static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) · line 3833

Quoted 13 lines
target/s390x/translate.c +5 -5
-    /* MASK is the set of bits to be inserted from R2.
-       Take care for I3/I4 wraparound.  */
-    mask = pmask >> i3;
+    /* MASK is the set of bits to be inserted from R2. */
    if (i3 <= i4) {
-        mask ^= pmask >> i4 >> 1;
+        /* [0...i3---i4...63] */
+        mask = (-1ull >> i3) & (-1ull << (63 - i4));
    } else {
-        mask |= ~(pmask >> i4 >> 1);
+        /* [0---i4...i3---63] */
+        mask = (-1ull >> i3) | (-1ull << (63 - i4));
    }

The expression evaluated looks the same to me for both sides of the conditional, but the comments differ. Intentional?

& vs |, so the result differs.

Cheers!

#1 lore raw
In reply to David Hildenbrand
Quoted 33 lines · David Hildenbrand

RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.

[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 0000200180000000 0000000000017a1e

         R:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3
   GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0
         0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0
         00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8
         0000000000000009 0000000000000002 0000000000000008 000000000000bce0

One example of a buggy instruction is:

    17dde:       ec 1e 00 9f 20 5d       risbhg  %r1,%r14,0,159,32

With %r14 = 0x9 and %r1 = 0x7 should result in %r1 = 0x900000007, however, results in %r1 = 0.

Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.

With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.

Fixes 2d6a869833d9 ("target-s390: Implement RISBG")
Cc Guenter Roeck <[email protected]>
Cc Christian Borntraeger <[email protected]>
Quoted 7 lines

---

This BUG was a nightmare to debug and the code a nightmare to understand.

To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]

In that case, a huge thank you!!! for this work! ++beers_owed.

Quoted 26 lines
---
 target/s390x/translate.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
target/s390x/translate.c +2 -5
index 3d5c0d6106..39e33eeb67 100644
@@ -3815,22 +3815,23 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o)
         pmask = 0xffffffff00000000ull;
         break;
     case 0x51: /* risblg */
-        i3 &= 31;
-        i4 &= 31;
+        i3 = (i3 & 31) + 32;
+        i4 = (i4 & 31) + 32;
         pmask = 0x00000000ffffffffull;
         break;
     default:
         g_assert_not_reached();
     }
 
-    /* MASK is the set of bits to be inserted from R2.
-       Take care for I3/I4 wraparound.  */
-    mask = pmask >> i3;

target/s390x/translate.c static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) · line 3833

Quoted 10 lines
target/s390x/translate.c +5 -2
+    /* MASK is the set of bits to be inserted from R2. */
     if (i3 <= i4) {
-        mask ^= pmask >> i4 >> 1;
+        /* [0...i3---i4...63] */
+        mask = (-1ull >> i3) & (-1ull << (63 - i4));
     } else {
-        mask |= ~(pmask >> i4 >> 1);
+        /* [0---i4...i3---63] */
+        mask = (-1ull >> i3) | (-1ull << (63 - i4));
     }

The expression evaluated looks the same to me for both sides of the conditional, but the comments differ. Intentional?

Quoted 17 lines
target/s390x/translate.c +1 -4
+    /* For RISBLG/RISBHG, the wrapping is limited to the high/low doubleword. */
     mask &= pmask;
 
     /* IMASK is the set of bits to be kept from R1.  In the case of the high/low
@@ -3843,9 +3844,6 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o)
     len = i4 - i3 + 1;
     pos = 63 - i4;
     rot = i5 & 63;
-    if (s->fields.op2 == 0x5d) {
-        pos += 32;
-    }
 
     /* In some cases we can implement this with extract.  */
     if (imask == 0 && pos == 0 && len > 0 && len <= rot) {
--

2.29.2

Signature · 2 lines
Thanks,
~Nick Desaulniers
#3 lore raw
In reply to David Hildenbrand
Quoted 42 lines · David Hildenbrand
Quoted 42 lines

Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:

Quoted 37 lines · David Hildenbrand

RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.

[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 0000200180000000 0000000000017a1e

        R:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3
  GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0
        0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0
        00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8
        0000000000000009 0000000000000002 0000000000000008 000000000000bce0

One example of a buggy instruction is:

   17dde:       ec 1e 00 9f 20 5d       risbhg  %r1,%r14,0,159,32

With %r14 = 0x9 and %r1 = 0x7 should result in %r1 = 0x900000007, however, results in %r1 = 0.

Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.

With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.

Fixes 2d6a869833d9 ("target-s390: Implement RISBG")
Cc Guenter Roeck <[email protected]>
Cc Christian Borntraeger <[email protected]>

---

This BUG was a nightmare to debug and the code a nightmare to understand.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]

In that case, a huge thank you!!! for this work! ++beers_owed.

:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.

Hmm...so I don't think clang can build a Linux kernel image with CONFIG_MARCH_Z13=y just yet; just defconfig. Otherwise looks like clang barfs on some of the inline asm constraints.

It looks like with your patch applied we get further into the boot! I'm not seeing any output with:

$ /android0/qemu/build/qemu-system-s390x -cpu qemu -append
'conmode=sclp console=ttyS0' -display none -initrd
/<path/to>/boot-utils/images/s390/rootfs.cpio -kernel
arch/s390/boot/bzImage -m 512m -nodefaults -serial mon:stdio

(Based on a quick skim through
https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.ludd/ludd_r_lmtkernelparameter.html).
Do I have all of those right?

If I attach GDB to QEMU running that kernel image, I was able to view the print banner once via lx-dmesg gdb macro in the kernel, but it seems on subsequent runs control flow gets diverted unexpected post entry to start_kernel() always to s390_base_pgm_handler ...errr..at least when I try to single step in GDB. Tried with linux-5.10.y, mainline, and linux-next.

qemu: 470dd6bd360782f5137f7e3376af6a44658eb1d3 + your patch
llvm: 106e66f3f555c8f887e82c5f04c3e77bdaf345e8 linux-5.10.y: d1988041d19dc8b532579bdbb7c4a978391c0011
linux: 71c061d2443814de15e177489d5cc00a4a253ef3
linux-next: f87684f6470f5f02bd47d4afb900366e5d2f31b6

(gdb) hbreak setup_arch
Hardware assisted breakpoint 1 at 0x142229e: file
arch/s390/kernel/setup.c, line 1091.
(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap. 0x00000000014222a0 in setup_arch (cmdline_p=0x11d7ed8) at arch/s390/kernel/setup.c:1091
1091 if (MACHINE_IS_VM)
(gdb) lx-dmesg

[    0.376351] Linux version 5.11.0-rc2-00157-ga2885c701c30

([email protected]) (Nick Desaulniers clang version 12.0.0 ([email protected]:llvm/llvm-project.git e75fec2b238f0e26cfb7645f2208baebe3440d41), GNU ld (GNU Binutils for Debian) 2.35.1) #81 SMP Thu Jan 7 17:57:34 PST 2021

Quoted 41 lines
Quoted 24 lines
Quoted 24 lines

---
target/s390x/translate.c | 18 ++++++++----------

1 file changed, 8 insertions(+), 10 deletions(-)
target/s390x/translate.c +2 -2
index 3d5c0d6106..39e33eeb67 100644
@@ -3815,22 +3815,23 @@ static DisasJumpType op_risbg(DisasContext *s, DisasOps *o)
        pmask = 0xffffffff00000000ull;
        break;
    case 0x51: /* risblg */
-        i3 &= 31;
-        i4 &= 31;
+        i3 = (i3 & 31) + 32;
+        i4 = (i4 & 31) + 32;
        pmask = 0x00000000ffffffffull;
        break;
    default:
        g_assert_not_reached();
    }
 
Quoted 16 lines

target/s390x/translate.c static DisasJumpType op_risbg(DisasContext *s, DisasOps *o) · line 3833

Quoted 13 lines
target/s390x/translate.c +5 -5
-    /* MASK is the set of bits to be inserted from R2.
-       Take care for I3/I4 wraparound.  */
-    mask = pmask >> i3;
+    /* MASK is the set of bits to be inserted from R2. */
    if (i3 <= i4) {
-        mask ^= pmask >> i4 >> 1;
+        /* [0...i3---i4...63] */
+        mask = (-1ull >> i3) & (-1ull << (63 - i4));
    } else {
-        mask |= ~(pmask >> i4 >> 1);
+        /* [0---i4...i3---63] */
+        mask = (-1ull >> i3) | (-1ull << (63 - i4));
    }

The expression evaluated looks the same to me for both sides of the conditional, but the comments differ. Intentional?

Quoted 1 line

& vs |, so the result differs.

D'oh, I can never spot single character differences!

Signature · 2 lines
Thanks,
~Nick Desaulniers
#4 lore raw
Quoted 47 lines · Nick Desaulniers
Quoted 46 lines · David Hildenbrand
Quoted 46 lines

Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:

Quoted 40 lines · David Hildenbrand

RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.

[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 0000200180000000 0000000000017a1e

        R:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3
  GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0
        0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0
        00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8
        0000000000000009 0000000000000002 0000000000000008 000000000000bce0

One example of a buggy instruction is:

   17dde:       ec 1e 00 9f 20 5d       risbhg  %r1,%r14,0,159,32

With %r14 = 0x9 and %r1 = 0x7 should result in %r1 = 0x900000007, however, results in %r1 = 0.

Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.

With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.

Fixes 2d6a869833d9 ("target-s390: Implement RISBG")
Cc Guenter Roeck <[email protected]>
Cc Christian Borntraeger <[email protected]>

---

This BUG was a nightmare to debug and the code a nightmare to understand.

To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

In that case, a huge thank you!!! for this work! ++beers_owed.

:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.

Hmm...so I don't think clang can build a Linux kernel image with CONFIG_MARCH_Z13=y just yet; just defconfig. Otherwise looks like clang barfs on some of the inline asm constraints.

Ah, right. I overwrote my manual config by a temporary defconfig :)

So, I'm on x86-64 F33.

clang version 11.0.0 (Fedora 11.0.0-2.fc33)
LLVM version 11.0.0

I cannot directly use "LLVM=1" for cross-compilation, as I keep getting "error: unknown emulation: elf64_s390" from ld.lld and "error: invalid output format: 'elf64-s390'" from llvm-objcopy. I assume that's fixed in llvm12?

1. I patch around it (strange, I remember CC= .. used to work, but it no longer does)

---

index e30cf02da8b8..89c57062ed5d 100644

Makefile +7 -7
@@ -427,13 +427,13 @@ KBUILD_HOSTLDLIBS   := $(HOST_LFS_LIBS) $(HOSTLDLIBS)
 CPP            = $(CC) -E
 ifneq ($(LLVM),)
 CC             = clang
-LD             = ld.lld
-AR             = llvm-ar
-NM             = llvm-nm
-OBJCOPY                = llvm-objcopy
-OBJDUMP                = llvm-objdump
-READELF                = llvm-readelf
-STRIP          = llvm-strip
+LD             = $(CROSS_COMPILE)ld
+AR             = $(CROSS_COMPILE)ar
+NM             = $(CROSS_COMPILE)nm
+OBJCOPY                = $(CROSS_COMPILE)objcopy
+OBJDUMP                = $(CROSS_COMPILE)objdump
+READELF                = $(CROSS_COMPILE)readelf
+STRIP          = $(CROSS_COMPILE)strip
 else
 CC             = $(CROSS_COMPILE)gcc
 LD             = $(CROSS_COMPILE)ld

---

2. Compile using clang

Using latest linux-next (1c925d2030afd354a02c23500386e620e662622b) + above patch

---

#!/bin/bash
export ARCH=s390;
export CROSS_COMPILE=s390x-linux-gnu-
export LLVM=1
make distclean
make defconfig

# Make F32 initrd boot without inserting modules
./scripts/config -e CONFIG_SCSI_ISCSI_ATTRS
./scripts/config -e CONFIG_ISCSI_TCP

make -j40 > /dev/null

---

3. Run it via QEMU. I boot a full Fedora 32 using the cloud-image + initrd from Fedora 32 (tried to stick to your cmdline where possible)

./build/qemu-system-s390x \
-m 512M \
-cpu qemu \
-display none \
-nodefaults \
-kernel ../linux-cross/arch/s390/boot/bzImage \
-append "root=/dev/vda1 conmode=sclp console=ttyS0" \
-initrd ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \
-hda ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \
-serial mon:stdio

KASLR disabled: CPU has no PRNG

[    0.408769] Linux version 5.11.0-rc2-next-20210108-dirty
(dhildenb@desktop) (clang version 11.0.0 (Fedora 11.0.0-2.fc33), GNU ld
version 2.35.1-1.fc33) #1 SMP Fri Jan 8 10:23:01 CET 2021
[    0.410266] setup: Linux is running under KVM in 64-bit mode
[    0.415840] setup: The maximum memory size is 512MB
[    0.417278] cpu: 1 configured CPUs, 0 standby CPUs

...

Fedora 32 (Cloud Edition)
Kernel 5.11.0-rc2-next-20210108-dirty on an s390x (ttysclp0)

atomic-00 login:

Quoted 36 lines

It looks like with your patch applied we get further into the boot! I'm not seeing any output with:

$ /android0/qemu/build/qemu-system-s390x -cpu qemu -append
'conmode=sclp console=ttyS0' -display none -initrd
/<path/to>/boot-utils/images/s390/rootfs.cpio -kernel
arch/s390/boot/bzImage -m 512m -nodefaults -serial mon:stdio

(Based on a quick skim through
https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.ludd/ludd_r_lmtkernelparameter.html).
Do I have all of those right?

If I attach GDB to QEMU running that kernel image, I was able to view the print banner once via lx-dmesg gdb macro in the kernel, but it seems on subsequent runs control flow gets diverted unexpected post entry to start_kernel() always to s390_base_pgm_handler ...errr..at least when I try to single step in GDB. Tried with linux-5.10.y, mainline, and linux-next.

qemu: 470dd6bd360782f5137f7e3376af6a44658eb1d3 + your patch
llvm: 106e66f3f555c8f887e82c5f04c3e77bdaf345e8 linux-5.10.y: d1988041d19dc8b532579bdbb7c4a978391c0011
linux: 71c061d2443814de15e177489d5cc00a4a253ef3
linux-next: f87684f6470f5f02bd47d4afb900366e5d2f31b6

(gdb) hbreak setup_arch
Hardware assisted breakpoint 1 at 0x142229e: file
arch/s390/kernel/setup.c, line 1091.
(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap. 0x00000000014222a0 in setup_arch (cmdline_p=0x11d7ed8) at arch/s390/kernel/setup.c:1091
1091 if (MACHINE_IS_VM)
(gdb) lx-dmesg

Quoted 5 lines
[    0.376351] Linux version 5.11.0-rc2-00157-ga2885c701c30

([email protected]) (Nick Desaulniers clang version 12.0.0 ([email protected]:llvm/llvm-project.git e75fec2b238f0e26cfb7645f2208baebe3440d41), GNU ld (GNU Binutils for Debian) 2.35.1) #81 SMP Thu Jan 7 17:57:34 PST 2021

So you're using llvm 12. Maybe that makes a difference. Or we have an issue with our arm64 backend. Or using ld.lld and friends make a difference. Guess I'd have to custom-compile llvm12 (gah) ... maybe I can find some rpms somewhere.

Signature · 3 lines
Thanks,

David / dhildenb
#5 lore raw
Quoted 130 lines · David Hildenbrand
Quoted 47 lines · Nick Desaulniers
Quoted 46 lines · David Hildenbrand
Quoted 46 lines

Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:

Quoted 40 lines · David Hildenbrand

RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.

[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 0000200180000000 0000000000017a1e

        R:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3
  GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0
        0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0
        00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8
        0000000000000009 0000000000000002 0000000000000008 000000000000bce0

One example of a buggy instruction is:

   17dde:       ec 1e 00 9f 20 5d       risbhg  %r1,%r14,0,159,32

With %r14 = 0x9 and %r1 = 0x7 should result in %r1 = 0x900000007, however, results in %r1 = 0.

Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.

With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.

Fixes 2d6a869833d9 ("target-s390: Implement RISBG")
Cc Guenter Roeck <[email protected]>
Cc Christian Borntraeger <[email protected]>

---

This BUG was a nightmare to debug and the code a nightmare to understand.

To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

In that case, a huge thank you!!! for this work! ++beers_owed.

:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.

Hmm...so I don't think clang can build a Linux kernel image with CONFIG_MARCH_Z13=y just yet; just defconfig. Otherwise looks like clang barfs on some of the inline asm constraints.

Ah, right. I overwrote my manual config by a temporary defconfig :)

So, I'm on x86-64 F33.

clang version 11.0.0 (Fedora 11.0.0-2.fc33)
LLVM version 11.0.0

I cannot directly use "LLVM=1" for cross-compilation, as I keep getting "error: unknown emulation: elf64_s390" from ld.lld and "error: invalid output format: 'elf64-s390'" from llvm-objcopy. I assume that's fixed in llvm12?

1. I patch around it (strange, I remember CC= .. used to work, but it no longer does)

---

index e30cf02da8b8..89c57062ed5d 100644

Makefile +7 -8
@@ -427,13 +427,13 @@ KBUILD_HOSTLDLIBS   := $(HOST_LFS_LIBS) $(HOSTLDLIBS)
 CPP            = $(CC) -E
 ifneq ($(LLVM),)
 CC             = clang
-LD             = ld.lld
-AR             = llvm-ar
-NM             = llvm-nm
-OBJCOPY                = llvm-objcopy
-OBJDUMP                = llvm-objdump
-READELF                = llvm-readelf
-STRIP          = llvm-strip
+LD             = $(CROSS_COMPILE)ld
+AR             = $(CROSS_COMPILE)ar
+NM             = $(CROSS_COMPILE)nm
+OBJCOPY                = $(CROSS_COMPILE)objcopy
+OBJDUMP                = $(CROSS_COMPILE)objdump
+READELF                = $(CROSS_COMPILE)readelf
+STRIP          = $(CROSS_COMPILE)strip
 else
 CC             = $(CROSS_COMPILE)gcc
 LD             = $(CROSS_COMPILE)ld
 
---

2. Compile using clang

Using latest linux-next (1c925d2030afd354a02c23500386e620e662622b) + above patch

---

#!/bin/bash
export ARCH=s390;
export CROSS_COMPILE=s390x-linux-gnu-
export LLVM=1
make distclean
make defconfig

# Make F32 initrd boot without inserting modules
./scripts/config -e CONFIG_SCSI_ISCSI_ATTRS
./scripts/config -e CONFIG_ISCSI_TCP

make -j40 > /dev/null

---

3. Run it via QEMU. I boot a full Fedora 32 using the cloud-image + initrd from Fedora 32 (tried to stick to your cmdline where possible)

./build/qemu-system-s390x \

Makefile

Quoted 8 lines
Makefile +0 -8
-m 512M \
-cpu qemu \
-display none \
-nodefaults \
-kernel ../linux-cross/arch/s390/boot/bzImage \
-append "root=/dev/vda1 conmode=sclp console=ttyS0" \
-initrd ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \
-hda ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \

^ -hda ../Fedora-Cloud-Base-32-1.6.x86_64.qcow2 \

Signature · 3 lines
Thanks,

David / dhildenb
#6 lore raw
Quoted 130 lines · David Hildenbrand
Quoted 129 lines · David Hildenbrand
Quoted 47 lines · Nick Desaulniers
Quoted 46 lines · David Hildenbrand
Quoted 46 lines

Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:

Quoted 40 lines · David Hildenbrand

RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.

[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 0000200180000000 0000000000017a1e

        R:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3
  GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0
        0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0
        00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8
        0000000000000009 0000000000000002 0000000000000008 000000000000bce0

One example of a buggy instruction is:

   17dde:       ec 1e 00 9f 20 5d       risbhg  %r1,%r14,0,159,32

With %r14 = 0x9 and %r1 = 0x7 should result in %r1 = 0x900000007, however, results in %r1 = 0.

Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.

With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.

Fixes 2d6a869833d9 ("target-s390: Implement RISBG")
Cc Guenter Roeck <[email protected]>
Cc Christian Borntraeger <[email protected]>

---

This BUG was a nightmare to debug and the code a nightmare to understand.

To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

In that case, a huge thank you!!! for this work! ++beers_owed.

:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.

Hmm...so I don't think clang can build a Linux kernel image with CONFIG_MARCH_Z13=y just yet; just defconfig. Otherwise looks like clang barfs on some of the inline asm constraints.

Ah, right. I overwrote my manual config by a temporary defconfig :)

So, I'm on x86-64 F33.

clang version 11.0.0 (Fedora 11.0.0-2.fc33)
LLVM version 11.0.0

I cannot directly use "LLVM=1" for cross-compilation, as I keep getting "error: unknown emulation: elf64_s390" from ld.lld and "error: invalid output format: 'elf64-s390'" from llvm-objcopy. I assume that's fixed in llvm12?

1. I patch around it (strange, I remember CC= .. used to work, but it no longer does)

---

index e30cf02da8b8..89c57062ed5d 100644

Makefile +7 -8
@@ -427,13 +427,13 @@ KBUILD_HOSTLDLIBS   := $(HOST_LFS_LIBS) $(HOSTLDLIBS)
 CPP            = $(CC) -E
 ifneq ($(LLVM),)
 CC             = clang
-LD             = ld.lld
-AR             = llvm-ar
-NM             = llvm-nm
-OBJCOPY                = llvm-objcopy
-OBJDUMP                = llvm-objdump
-READELF                = llvm-readelf
-STRIP          = llvm-strip
+LD             = $(CROSS_COMPILE)ld
+AR             = $(CROSS_COMPILE)ar
+NM             = $(CROSS_COMPILE)nm
+OBJCOPY                = $(CROSS_COMPILE)objcopy
+OBJDUMP                = $(CROSS_COMPILE)objdump
+READELF                = $(CROSS_COMPILE)readelf
+STRIP          = $(CROSS_COMPILE)strip
 else
 CC             = $(CROSS_COMPILE)gcc
 LD             = $(CROSS_COMPILE)ld
 
---

2. Compile using clang

Using latest linux-next (1c925d2030afd354a02c23500386e620e662622b) + above patch

---

#!/bin/bash
export ARCH=s390;
export CROSS_COMPILE=s390x-linux-gnu-
export LLVM=1
make distclean
make defconfig

# Make F32 initrd boot without inserting modules
./scripts/config -e CONFIG_SCSI_ISCSI_ATTRS
./scripts/config -e CONFIG_ISCSI_TCP

make -j40 > /dev/null

---

3. Run it via QEMU. I boot a full Fedora 32 using the cloud-image + initrd from Fedora 32 (tried to stick to your cmdline where possible)

Quoted 13 lines

Makefile

Quoted 9 lines

./build/qemu-system-s390x \

Makefile +0 -8
-m 512M \
-cpu qemu \
-display none \
-nodefaults \
-kernel ../linux-cross/arch/s390/boot/bzImage \
-append "root=/dev/vda1 conmode=sclp console=ttyS0" \
-initrd ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \
-hda ../Fedora-Cloud-Base-32-1.6.x86_64-initrd.img \

^ -hda ../Fedora-Cloud-Base-32-1.6.x86_64.qcow2 \

Using homebrew llvm-12/clang-12 results in a kernel that is indeed stuck is in an endless program exception loop, seeming to try printing a backtrace over and over again. Yet, no output besides

"KASLR disabled: CPU has no PRNG"

Kernel runs under KVM, so more debugging to do to get clang-12 builds running as well.

Signature · 3 lines
Thanks,

David / dhildenb
#7 lore raw
In reply to David Hildenbrand
Quoted 64 lines · David Hildenbrand
Quoted 47 lines · Nick Desaulniers
Quoted 46 lines · David Hildenbrand
Quoted 46 lines

Am 08.01.2021 um 00:21 schrieb Nick Desaulniers <[email protected]>:

Quoted 40 lines · David Hildenbrand

RISBHG is broken and currently hinders clang builds of upstream kernels from booting: the kernel crashes early, while decompressing the image.

[...]
Kernel fault: interruption code 0005 ilc:2
Kernel random base: 0000000000000000
PSW : 0000200180000000 0000000000017a1e

        R:0 T:0 IO:0 EX:0 Key:0 M:0 W:0 P:0 AS:0 CC:2 PM:0 RI:0 EA:3
  GPRS: 0000000000000001 0000000c00000000 00000003fffffff4 00000000fffffff0
        0000000000000000 00000000fffffff4 000000000000000c 00000000fffffff0
        00000000fffffffc 0000000000000000 00000000fffffff8 00000000008e25a8
        0000000000000009 0000000000000002 0000000000000008 000000000000bce0

One example of a buggy instruction is:

   17dde:       ec 1e 00 9f 20 5d       risbhg  %r1,%r14,0,159,32

With %r14 = 0x9 and %r1 = 0x7 should result in %r1 = 0x900000007, however, results in %r1 = 0.

Let's interpret values of i3/i4 as documented in the PoP and make computation of "mask" only based on i3 and i4 and use "pmask" only at the very end to make sure wrapping is only applied to the high/low doubleword.

With this patch, I can successfully boot a v5.10 kernel built with clang, and gcc builds keep on working.

Fixes 2d6a869833d9 ("target-s390: Implement RISBG")
Cc Guenter Roeck <[email protected]>
Cc Christian Borntraeger <[email protected]>

---

This BUG was a nightmare to debug and the code a nightmare to understand.

To make clang/gcc builds boot, the following fix is required as well on top of current master: "[PATCH] target/s390x: Fix ALGSI"
https://lkml.kernel.org/r/[email protected]

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

In that case, a huge thank you!!! for this work! ++beers_owed.

:) a kernel build for z13 should work with the (default) „-cpu qemu“ cpu type.

Hmm...so I don't think clang can build a Linux kernel image with CONFIG_MARCH_Z13=y just yet; just defconfig. Otherwise looks like clang barfs on some of the inline asm constraints.

Ah, right. I overwrote my manual config by a temporary defconfig :)

So, I'm on x86-64 F33.

Quoted 7 lines

clang version 11.0.0 (Fedora 11.0.0-2.fc33)
LLVM version 11.0.0

I cannot directly use "LLVM=1" for cross-compilation, as I keep getting "error: unknown emulation: elf64_s390" from ld.lld and "error: invalid output format: 'elf64-s390'" from llvm-objcopy. I assume that's fixed in llvm12?

Right, I suspect that even if ld.lld understood that emulation mode target, it would still fail due to lack of big endian support. We've been building with simply CC=clang for s390 linux kernels.
Via: https://www.kernel.org/doc/html/latest/kbuild/llvm.html#llvm-utilities
we usually start with make CC=clang then work our way up to make LLVM=1. So you shouldn't need the below patching, just use CC=clang.

Quoted 22 lines

1. I patch around it (strange, I remember CC= .. used to work, but it no longer does)

---

index e30cf02da8b8..89c57062ed5d 100644

Makefile +2 -7
@@ -427,13 +427,13 @@ KBUILD_HOSTLDLIBS   := $(HOST_LFS_LIBS) $(HOSTLDLIBS)
 CPP            = $(CC) -E
 ifneq ($(LLVM),)
 CC             = clang
-LD             = ld.lld
-AR             = llvm-ar
-NM             = llvm-nm
-OBJCOPY                = llvm-objcopy
-OBJDUMP                = llvm-objdump
-READELF                = llvm-readelf
-STRIP          = llvm-strip
+LD             = $(CROSS_COMPILE)ld
+AR             = $(CROSS_COMPILE)ar

Makefile KBUILD_HOSTLDLIBS := $(HOST_LFS_LIBS) $(HOSTLDLIBS)

Quoted 10 lines
Makefile +5 -1
+NM             = $(CROSS_COMPILE)nm
+OBJCOPY                = $(CROSS_COMPILE)objcopy
+OBJDUMP                = $(CROSS_COMPILE)objdump
+READELF                = $(CROSS_COMPILE)readelf
+STRIP          = $(CROSS_COMPILE)strip
 else
 CC             = $(CROSS_COMPILE)gcc
 LD             = $(CROSS_COMPILE)ld
 
---

Pulling from your github branch, everything looks good; buildroot support looks good. I'll wire this up to our CI so that we can help report regressions!

Signature · 2 lines
Thanks,
~Nick Desaulniers