Skip to contentSkip to conversation

hw/s390x/pci: Fix endianness issue

RFC In review Philippe Mathieu-Daudé 6 participants 15 messages Nov 17, 2020

lore
Share & tools
lore
#0 lore raw

RFC 1/1 hw/s390x/pci: Fix endianness issue

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
 hw/s390x/s390-pci-inst.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +1 -1
index 58cd041d17f..cfb54b4d8ec 100644
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
         ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
         S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));
         if (!group) {
             /* We do not allow access to unknown groups */
             /* The group must have been obtained with a vfio device */
Signature · 1 line
2.26.2
#1 lore raw
Quoted 21 lines · On Tue, 17 Nov 2020 13:01:15 +0100 Philippe Mathieu-Daudé <[email protected]>…

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
 hw/s390x/s390-pci-inst.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 311

Quoted 8 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
         ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
         S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));
         if (!group) {
             /* We do not allow access to unknown groups */
             /* The group must have been obtained with a vfio device */

Hmm... when I tested this change on top of 28dc86a0729, it worked; when tested on top of my s390-fixes branch, I get

operand exception: 0015 ilc:3 [#1] SMP

CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.8.0-rc3+ #174
Hardware name: QEMU 2964 QEMU (KVM/Linux)
Krnl PSW : 0704d00180000000 00000000001727be (zpci_mod_fc+0x2e/0xf8)
           R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
Krnl GPRS: 000003d080fb8040 8002000000000004 8002000000000004 000003e00000bb30
           000003e00000bb2f 0000000040ffffff 000000003efa8804 0000000000000002
           000003e00000bb2f 8002000000000004 0000000000000000 000000003f1b4400
           000000003f41c000 0000000000000000 000003e00000ba88 000003e00000ba20
Krnl Code: 00000000001727ae: e3e0f0980024	stg	%r14,152(%r15)
           00000000001727b4: b9040084		lgr	%r8,%r4
          #00000000001727b8: e310300000d0	mpcifc	%r1,0(%r3)
          >00000000001727be: b22200b0		ipm	%r11
           00000000001727c2: 88b0001c		srl	%r11,28
           00000000001727c6: 42b0f0a7		stc	%r11,167(%r15)
           00000000001727ca: eb110018000c	srlg	%r1,%r1,24
           00000000001727d0: 42104000		stc	%r1,0(%r4)
Call Trace:
 [<00000000001727be>] zpci_mod_fc+0x2e/0xf8 
 [<000000000016d364>] zpci_register_ioat+0x64/0x88 
 [<0000000000170532>] zpci_dma_init_device+0x13a/0x1b8 
 [<000000000016e008>] zpci_enable_device+0x48/0x70 
 [<000000000016e1b2>] zpci_create_device+0x14a/0x170 
 [<0000000000171224>] clp_add_pci_device+0x504/0x528 
 [<0000000000170c46>] clp_list_pci+0xf6/0x1d0 
 [<00000000001712f4>] clp_scan_pci_devices+0x44/0x68 
 [<00000000011e859e>] pci_base_init+0x12e/0x1a0 
 [<00000000011ddf92>] do_one_initcall+0x132/0x2e0 
 [<00000000011de3cc>] kernel_init_freeable+0x28c/0x308 
 [<0000000000cfc822>] kernel_init+0x22/0x150 
 [<0000000000d08f60>] ret_from_fork+0x24/0x28 

INFO: lockdep is turned off.
Last Breaking-Event-Address:
[<000000000016d35e>] zpci_register_ioat+0x5e/0x88

(Yeah, I know it's not the newest guest kernel.)

This patch probably uncovers further endianness problems in the following zpci patches. Looking.

#2 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
 hw/s390x/s390-pci-inst.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
         ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
         S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

#3 lore raw
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
 hw/s390x/s390-pci-inst.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
         ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
         S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 3 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

Now than I had lunch, your comment is obvious...

IIUC we should use ldl_le_p() here and fix all the other uses.

Quoted 4 lines

thanks
-- PMM

#4 lore raw
Quoted 35 lines · On Tue, 17 Nov 2020 14:12:00 +0100 Philippe Mathieu-Daudé <[email protected]>…
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
 hw/s390x/s390-pci-inst.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
         ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
         S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));  

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 8 lines
Quoted 3 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

Now than I had lunch, your comment is obvious...

IIUC we should use ldl_le_p() here and fix all the other uses.

I'm not sure whether it shouldn't be be -- waiting for the IBM folks to comment on the expected layout.

Quoted 6 lines
Quoted 5 lines

thanks
-- PMM

#5 lore raw
In reply to Peter Maydell
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
  hw/s390x/s390-pci-inst.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
          ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
          S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 7 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

Hi, I think we better modify the structure here, g should be a byte.

Connie, can you please try this if it resolves the issue?

hw/s390x/s390-pci-inst.h +2 -1
index fa3bf8b5aa..641d19c815 100644
@@ -146,7 +146,8 @@ typedef struct ClpReqQueryPciGrp {
     uint32_t fmt;
     uint64_t reserved1;
 #define CLP_REQ_QPCIG_MASK_PFGID 0xff
-    uint32_t g;
+    uint32_t g0 :24;
+    uint32_t g  :8;
     uint32_t reserved2;
     uint64_t reserved3;
 } QEMU_PACKED ClpReqQueryPciGrp;
Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#6 lore raw
Quoted 53 lines · On Tue, 17 Nov 2020 14:23:57 +0100 Pierre Morel <[email protected]> wrote:
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
  hw/s390x/s390-pci-inst.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
          ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
          S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));  

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 7 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

Hi, I think we better modify the structure here, g should be a byte.

Connie, can you please try this if it resolves the issue?

hw/s390x/s390-pci-inst.h +0 -0
index fa3bf8b5aa..641d19c815 100644

hw/s390x/s390-pci-inst.h typedef struct ClpReqQueryPciGrp { · line 153

Quoted 9 lines
hw/s390x/s390-pci-inst.h +2 -1
@@ -146,7 +146,8 @@ typedef struct ClpReqQueryPciGrp {
      uint32_t fmt;
      uint64_t reserved1;
  #define CLP_REQ_QPCIG_MASK_PFGID 0xff
-    uint32_t g;
+    uint32_t g0 :24;
+    uint32_t g  :8;
      uint32_t reserved2;
      uint64_t reserved3;
  } QEMU_PACKED ClpReqQueryPciGrp;

No, same crash... I fear there are more things broken wrt endianness.

#7 lore raw
Quoted 59 lines · Cornelia Huck
Quoted 56 lines · On Tue, 17 Nov 2020 14:23:57 +0100 Pierre Morel <[email protected]> wrote:
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
   hw/s390x/s390-pci-inst.c | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
           ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
           S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 7 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

Hi, I think we better modify the structure here, g should be a byte.

Connie, can you please try this if it resolves the issue?

hw/s390x/s390-pci-inst.h +0 -1
index fa3bf8b5aa..641d19c815 100644
@@ -146,7 +146,8 @@ typedef struct ClpReqQueryPciGrp {
       uint32_t fmt;
       uint64_t reserved1;
   #define CLP_REQ_QPCIG_MASK_PFGID 0xff
-    uint32_t g;
Quoted 9 lines

hw/s390x/s390-pci-inst.h typedef struct ClpReqQueryPciGrp { · line 153

Quoted 6 lines
hw/s390x/s390-pci-inst.h +2 -0
+    uint32_t g0 :24;
+    uint32_t g  :8;
       uint32_t reserved2;
       uint64_t reserved3;
   } QEMU_PACKED ClpReqQueryPciGrp;

No, same crash... I fear there are more things broken wrt endianness.

Sorry, just getting online now, looking at the code.... Are the 2 memcpy calls added in 9670ee75 and 28dc86a0 the issue? Won't they just present the Q PCI FN / Q PCI FN GRP results in host endianness?

#8 lore raw
Quoted 70 lines · On Tue, 17 Nov 2020 09:02:37 -0500 Matthew Rosato <[email protected]>…
Quoted 59 lines · Cornelia Huck
Quoted 56 lines · On Tue, 17 Nov 2020 14:23:57 +0100 Pierre Morel <[email protected]> wrote:
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
   hw/s390x/s390-pci-inst.c | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
           ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
           S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));  

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 7 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

Hi, I think we better modify the structure here, g should be a byte.

Connie, can you please try this if it resolves the issue?

hw/s390x/s390-pci-inst.h +0 -1
index fa3bf8b5aa..641d19c815 100644
@@ -146,7 +146,8 @@ typedef struct ClpReqQueryPciGrp {
       uint32_t fmt;
       uint64_t reserved1;
   #define CLP_REQ_QPCIG_MASK_PFGID 0xff
-    uint32_t g;
Quoted 9 lines

hw/s390x/s390-pci-inst.h typedef struct ClpReqQueryPciGrp { · line 153

Quoted 6 lines
hw/s390x/s390-pci-inst.h +2 -0
+    uint32_t g0 :24;
+    uint32_t g  :8;
       uint32_t reserved2;
       uint64_t reserved3;
   } QEMU_PACKED ClpReqQueryPciGrp;

No, same crash... I fear there are more things broken wrt endianness.

Quoted 4 lines

Sorry, just getting online now, looking at the code.... Are the 2 memcpy calls added in 9670ee75 and 28dc86a0 the issue? Won't they just present the Q PCI FN / Q PCI FN GRP results in host endianness?

I just re-added some st?_p operations in set_pbdev_info and that fixes at least the crash I was seeing with Phil's patch applied. Still, no pci functions get detected, so that's not enough. Those memcpy calls look like a possible culprit.

#9 lore raw
Quoted 78 lines · Cornelia Huck
Quoted 70 lines · On Tue, 17 Nov 2020 09:02:37 -0500 Matthew Rosato <[email protected]>…
Quoted 59 lines · Cornelia Huck
Quoted 56 lines · On Tue, 17 Nov 2020 14:23:57 +0100 Pierre Morel <[email protected]> wrote:
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
    hw/s390x/s390-pci-inst.c | 2 +-
    1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
            ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
            S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 7 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

Hi, I think we better modify the structure here, g should be a byte.

Connie, can you please try this if it resolves the issue?

hw/s390x/s390-pci-inst.h +0 -1
index fa3bf8b5aa..641d19c815 100644
@@ -146,7 +146,8 @@ typedef struct ClpReqQueryPciGrp {
        uint32_t fmt;
        uint64_t reserved1;
    #define CLP_REQ_QPCIG_MASK_PFGID 0xff
-    uint32_t g;
Quoted 9 lines

hw/s390x/s390-pci-inst.h typedef struct ClpReqQueryPciGrp { · line 153

Quoted 6 lines
hw/s390x/s390-pci-inst.h +2 -0
+    uint32_t g0 :24;
+    uint32_t g  :8;
        uint32_t reserved2;
        uint64_t reserved3;
    } QEMU_PACKED ClpReqQueryPciGrp;

No, same crash... I fear there are more things broken wrt endianness.

Quoted 4 lines

Sorry, just getting online now, looking at the code.... Are the 2 memcpy calls added in 9670ee75 and 28dc86a0 the issue? Won't they just present the Q PCI FN / Q PCI FN GRP results in host endianness?

Quoted 5 lines

I just re-added some st?_p operations in set_pbdev_info and that fixes at least the crash I was seeing with Phil's patch applied. Still, no pci functions get detected, so that's not enough. Those memcpy calls look like a possible culprit.

OK, so if everything in set_pbdev_info and s390_pci_init_default_group() is handled with st?_p operations, then the memcpy should be OK...

Pierre was on to something with his recommendation, as the group id is only 1B of the 'g' field (see CLP_REQ_QPCIG_MASK_PFGID) - the other bits just happen to be unused.

Did you include his change with your st?_p changes to set_pbdev_info (sorry, I don't have this environment set up but clearly need to do so for future testing)

#10 lore raw
Quoted 88 lines · On Tue, 17 Nov 2020 09:34:41 -0500 Matthew Rosato <[email protected]>…
Quoted 78 lines · Cornelia Huck
Quoted 70 lines · On Tue, 17 Nov 2020 09:02:37 -0500 Matthew Rosato <[email protected]>…
Quoted 59 lines · Cornelia Huck
Quoted 56 lines · On Tue, 17 Nov 2020 14:23:57 +0100 Pierre Morel <[email protected]> wrote:
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
    hw/s390x/s390-pci-inst.c | 2 +-
    1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
            ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
            S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));  

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 7 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

Hi, I think we better modify the structure here, g should be a byte.

Connie, can you please try this if it resolves the issue?

hw/s390x/s390-pci-inst.h +0 -1
index fa3bf8b5aa..641d19c815 100644
@@ -146,7 +146,8 @@ typedef struct ClpReqQueryPciGrp {
        uint32_t fmt;
        uint64_t reserved1;
    #define CLP_REQ_QPCIG_MASK_PFGID 0xff
-    uint32_t g;
Quoted 9 lines

hw/s390x/s390-pci-inst.h typedef struct ClpReqQueryPciGrp { · line 153

Quoted 6 lines
hw/s390x/s390-pci-inst.h +2 -0
+    uint32_t g0 :24;
+    uint32_t g  :8;
        uint32_t reserved2;
        uint64_t reserved3;
    } QEMU_PACKED ClpReqQueryPciGrp;

No, same crash... I fear there are more things broken wrt endianness.

Quoted 4 lines

Sorry, just getting online now, looking at the code.... Are the 2 memcpy calls added in 9670ee75 and 28dc86a0 the issue? Won't they just present the Q PCI FN / Q PCI FN GRP results in host endianness?

Quoted 5 lines

I just re-added some st?_p operations in set_pbdev_info and that fixes at least the crash I was seeing with Phil's patch applied. Still, no pci functions get detected, so that's not enough. Those memcpy calls look like a possible culprit.

OK, so if everything in set_pbdev_info and s390_pci_init_default_group() is handled with st?_p operations, then the memcpy should be OK...

Quoted 7 lines

Pierre was on to something with his recommendation, as the group id is only 1B of the 'g' field (see CLP_REQ_QPCIG_MASK_PFGID) - the other bits just happen to be unused.

Did you include his change with your st?_p changes to set_pbdev_info (sorry, I don't have this environment set up but clearly need to do so for future testing)

I tried in conjunction with Phil's patch (otherwise, I don't even get to the part where it crashes.) Do we need to apply that mask somewhere? It is hard to guess if you don't know what the structure is supposed to look like :)

#11 lore raw
Quoted 95 lines · Cornelia Huck
Quoted 92 lines · On Tue, 17 Nov 2020 09:34:41 -0500 Matthew Rosato <[email protected]>…
Quoted 78 lines · Cornelia Huck
Quoted 70 lines · On Tue, 17 Nov 2020 09:02:37 -0500 Matthew Rosato <[email protected]>…
Quoted 59 lines · Cornelia Huck
Quoted 56 lines · On Tue, 17 Nov 2020 14:23:57 +0100 Pierre Morel <[email protected]> wrote:
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
     hw/s390x/s390-pci-inst.c | 2 +-
     1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
             ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
             S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 7 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

Hi, I think we better modify the structure here, g should be a byte.

Connie, can you please try this if it resolves the issue?

hw/s390x/s390-pci-inst.h +0 -1
index fa3bf8b5aa..641d19c815 100644
@@ -146,7 +146,8 @@ typedef struct ClpReqQueryPciGrp {
         uint32_t fmt;
         uint64_t reserved1;
     #define CLP_REQ_QPCIG_MASK_PFGID 0xff
-    uint32_t g;
Quoted 9 lines

hw/s390x/s390-pci-inst.h typedef struct ClpReqQueryPciGrp { · line 153

Quoted 6 lines
hw/s390x/s390-pci-inst.h +2 -0
+    uint32_t g0 :24;
+    uint32_t g  :8;
         uint32_t reserved2;
         uint64_t reserved3;
     } QEMU_PACKED ClpReqQueryPciGrp;

No, same crash... I fear there are more things broken wrt endianness.

Quoted 4 lines

Sorry, just getting online now, looking at the code.... Are the 2 memcpy calls added in 9670ee75 and 28dc86a0 the issue? Won't they just present the Q PCI FN / Q PCI FN GRP results in host endianness?

Quoted 5 lines

I just re-added some st?_p operations in set_pbdev_info and that fixes at least the crash I was seeing with Phil's patch applied. Still, no pci functions get detected, so that's not enough. Those memcpy calls look like a possible culprit.

OK, so if everything in set_pbdev_info and s390_pci_init_default_group() is handled with st?_p operations, then the memcpy should be OK...

Pierre was on to something with his recommendation, as the group id is only 1B of the 'g' field (see CLP_REQ_QPCIG_MASK_PFGID) - the other bits just happen to be unused.

Quoted 9 lines
Quoted 3 lines

Did you include his change with your st?_p changes to set_pbdev_info (sorry, I don't have this environment set up but clearly need to do so for future testing)

I tried in conjunction with Phil's patch (otherwise, I don't even get to the part where it crashes.) Do we need to apply that mask somewhere? It is hard to guess if you don't know what the structure is supposed to look like :)

OK, I just got the issue reproduced here (no PCI without Phil's patch, crash with Phil's patch); Let me investigate and I will get back.

#12 lore raw
Quoted 100 lines · On Tue, 17 Nov 2020 11:01:31 -0500 Matthew Rosato <[email protected]>…
Quoted 99 lines · Cornelia Huck
Quoted 95 lines · On Tue, 17 Nov 2020 09:34:41 -0500 Matthew Rosato <[email protected]>…
Quoted 78 lines · Cornelia Huck
Quoted 70 lines · On Tue, 17 Nov 2020 09:02:37 -0500 Matthew Rosato <[email protected]>…
Quoted 59 lines · Cornelia Huck
Quoted 56 lines · On Tue, 17 Nov 2020 14:23:57 +0100 Pierre Morel <[email protected]> wrote:
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
     hw/s390x/s390-pci-inst.c | 2 +-
     1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
             ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
             S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));  

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 7 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

Hi, I think we better modify the structure here, g should be a byte.

Connie, can you please try this if it resolves the issue?

hw/s390x/s390-pci-inst.h +0 -1
index fa3bf8b5aa..641d19c815 100644
@@ -146,7 +146,8 @@ typedef struct ClpReqQueryPciGrp {
         uint32_t fmt;
         uint64_t reserved1;
     #define CLP_REQ_QPCIG_MASK_PFGID 0xff
-    uint32_t g;
Quoted 9 lines

hw/s390x/s390-pci-inst.h typedef struct ClpReqQueryPciGrp { · line 153

Quoted 6 lines
hw/s390x/s390-pci-inst.h +2 -0
+    uint32_t g0 :24;
+    uint32_t g  :8;
         uint32_t reserved2;
         uint64_t reserved3;
     } QEMU_PACKED ClpReqQueryPciGrp;

No, same crash... I fear there are more things broken wrt endianness.

Quoted 4 lines

Sorry, just getting online now, looking at the code.... Are the 2 memcpy calls added in 9670ee75 and 28dc86a0 the issue? Won't they just present the Q PCI FN / Q PCI FN GRP results in host endianness?

Quoted 5 lines

I just re-added some st?_p operations in set_pbdev_info and that fixes at least the crash I was seeing with Phil's patch applied. Still, no pci functions get detected, so that's not enough. Those memcpy calls look like a possible culprit.

OK, so if everything in set_pbdev_info and s390_pci_init_default_group() is handled with st?_p operations, then the memcpy should be OK...

Pierre was on to something with his recommendation, as the group id is only 1B of the 'g' field (see CLP_REQ_QPCIG_MASK_PFGID) - the other bits just happen to be unused.

Did you include his change with your st?_p changes to set_pbdev_info (sorry, I don't have this environment set up but clearly need to do so for future testing)

Quoted 8 lines
Quoted 5 lines

I tried in conjunction with Phil's patch (otherwise, I don't even get to the part where it crashes.) Do we need to apply that mask somewhere? It is hard to guess if you don't know what the structure is supposed to look like :)

OK, I just got the issue reproduced here (no PCI without Phil's patch, crash with Phil's patch); Let me investigate and I will get back.

I think I may have something (there were some more fields that needed care). Let me check whether it works on s390x-on-s390x as well, then I'll polish it and post.

(Still hoping to get that included; I have two other fixes queued.)

#13 lore raw
In reply to Matthew Rosato
Quoted 87 lines · Matthew Rosato
Quoted 80 lines · Cornelia Huck
Quoted 72 lines · On Tue, 17 Nov 2020 09:02:37 -0500 Matthew Rosato <[email protected]>…
Quoted 62 lines · Cornelia Huck

On Tue, 17 Nov 2020 14:23:57 +0100
Pierre Morel <[email protected]> wrote:

Quoted 59 lines · Peter Maydell
Quoted 36 lines
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested
---
    hw/s390x/s390-pci-inst.c | 2 +-

    1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, · line 309

Quoted 6 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2,
 uintptr_t ra)
             ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
             S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 7 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

Hi, I think we better modify the structure here, g should be a byte.

Connie, can you please try this if it resolves the issue?

hw/s390x/s390-pci-inst.h +1 -1
index fa3bf8b5aa..641d19c815 100644
@@ -146,7 +146,8 @@ typedef struct ClpReqQueryPciGrp {
         uint32_t fmt;
         uint64_t reserved1;
     #define CLP_REQ_QPCIG_MASK_PFGID 0xff
-    uint32_t g;
+    uint32_t g0 :24;
Quoted 8 lines
Quoted 5 lines
hw/s390x/s390-pci-inst.h +1 -0
+    uint32_t g  :8;
         uint32_t reserved2;
         uint64_t reserved3;
     } QEMU_PACKED ClpReqQueryPciGrp;

No, same crash... I fear there are more things broken wrt endianness.

Quoted 4 lines

Sorry, just getting online now, looking at the code....  Are the 2 memcpy calls added in 9670ee75 and 28dc86a0 the issue?  Won't they just present the Q PCI FN / Q PCI FN GRP results in host endianness?

Quoted 5 lines

I just re-added some st?_p operations in set_pbdev_info and that fixes at least the crash I was seeing with Phil's patch applied. Still, no pci functions get detected, so that's not enough. Those memcpy calls look like a possible culprit.

Quoted 8 lines

OK, so if everything in set_pbdev_info and s390_pci_init_default_group() is handled with st?_p operations, then the memcpy should be OK...

Pierre was on to something with his recommendation, as the group id is only 1B of the 'g' field (see CLP_REQ_QPCIG_MASK_PFGID) - the other bits just happen to be unused.

Did you include his change with your st?_p changes to set_pbdev_info

As Peter also already wrote: Bitfields are not endianess safe either. You'd need to replace the g0:24 with "uint8_t g0[3]" to get it working that way.

Thomas

#14 lore raw
In reply to Pierre Morel
Quoted 55 lines · Pierre Morel
Quoted 34 lines · Peter Maydell
Quoted 22 lines · Philippe Mathieu-Daudé

Fix an endianness issue reported by Cornelia:

Quoted 4 lines

s390x tcg guest on x86, virtio-pci devices are not detected. The relevant feature bits are visible to the guest. Same breakage with different guest kernels.
KVM guests and s390x tcg guests on s390x are fine.

Fixes 28dc86a0729 ("s390x/pci: use a PCI Group structure")

---
RFC because review-only patch, untested

---
  hw/s390x/s390-pci-inst.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-inst.c +0 -0
index 58cd041d17f..cfb54b4d8ec 100644

hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 308

Quoted 5 lines
hw/s390x/s390-pci-inst.c +1 -1
@@ -305,7 +305,7 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
          ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
          S390PCIGroup *group;
 
-        group = s390_group_find(reqgrp->g);
+        group = s390_group_find(ldl_p(&reqgrp->g));

'g' in the ClpReqQueryPciGrp struct is a uint32_t, so adding the ldl_p() will have no effect unless (a) the structure is not 4-aligned and (b) the host will fault on unaligned accesses, which isn't the case for x86 hosts.

Quoted 7 lines

Q: is this struct really in host order, or should we be using ldl_le_p() or ldl_be_p() and friends here and elsewhere?

thanks
-- PMM

Hi, I think we better modify the structure here, g should be a byte.

Connie, can you please try this if it resolves the issue?

hw/s390x/s390-pci-inst.h +0 -0
index fa3bf8b5aa..641d19c815 100644

hw/s390x/s390-pci-inst.h typedef struct ClpReqQueryPciGrp { · line 151

Quoted 7 lines
hw/s390x/s390-pci-inst.h +2 -1
@@ -146,7 +146,8 @@ typedef struct ClpReqQueryPciGrp {
      uint32_t fmt;
      uint64_t reserved1;
  #define CLP_REQ_QPCIG_MASK_PFGID 0xff
-    uint32_t g;
+    uint32_t g0 :24;
+    uint32_t g  :8;
      uint32_t reserved2;

Bitfields in a struct intending to be a representation of a guest in-memory structure? Almost certainly the wrong thing because the order of g and g0 will differ depending on host endianness...

hw/s390x/s390-pci-inst.h typedef struct ClpReqQueryPciGrp { · line 153

Quoted 2 lines
hw/s390x/s390-pci-inst.h +0 -0
      uint64_t reserved3;
  } QEMU_PACKED ClpReqQueryPciGrp;

thanks
-- PMM