Skip to contentSkip to conversation

s390x: Fix intermittent errors with IPL from PCI devices on s390x

PATCH Applied [email protected] 4 participants 15 messages Jun 30 – Jul 7

lore
Share & tools
lore
#0 lore raw

v2 cover s390x: Fix intermittent errors with IPL from PCI devices on s390x

From: Jared Rossi <[email protected]>

When booting from a PCI device on s390x there may be intermittent failures caused by mismatched byte swaps. Refactor the byte swapping to ensure that it is properly synchronized.

Correct how the queue notification offset is calculated so that it may be used for multi-queue configurations in the future.

Handle some switch cases that were not caught for PCI devices.

Changes v1 -> v2:
- Split addition of missing break into its own patch
- Add PCI case to generic is_viritio_supported() wapper
- Switch the order of patches formerly 3 and 4
- Use virtio_is_supported() instead of checking cutype when writing IPLB location to cover both CCW and PCI virtio devices

Jared Rossi (3):
pc-bios/s390-ccw/virtio.c: Fix missing break for PCI notifications pc-bios/s390-ccw: Verify virtio support when booting from virtio PCI

    device on s390x

pc-bios/s390-ccw: write IPLB location for non-net virtio devices

Zhuoying Cai (3):
pc-bios/s390-ccw: Refactor byte swapping pc-bios/s390-ccw: Add per-queue notification offset for multi-queue
virtio configurations
s390x: Enable boot menu for virtio pci device

 hw/s390x/ipl.c                |  1 +
 pc-bios/s390-ccw/main.c       |  5 ++-
 pc-bios/s390-ccw/virtio-pci.c | 57 ++++++++++++++++++++++++++---------
 pc-bios/s390-ccw/virtio-pci.h |  5 ++-
 pc-bios/s390-ccw/virtio.c     | 29 ++++++++++--------
 pc-bios/s390-ccw/virtio.h     |  2 ++
 6 files changed, 69 insertions(+), 30 deletions(-)
Signature · 1 line
2.54.0
#1 lore raw

PATCH v2 1/6 pc-bios/s390-ccw: Refactor byte swapping

From: Zhuoying Cai <[email protected]>

Introduce local variables to cache the byte-swapped values eliminating some redundant byte swap operations. Additionally, do byte swap when polling to avoid a special case where endianness is preserved.

---
 pc-bios/s390-ccw/virtio.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
pc-bios/s390-ccw/virtio.c +12 -12
index 390b55c7b9..a448dc96e2 100644
@@ -153,14 +153,14 @@ static void vr_bswap_descriptor(VRingDesc *desc)
 
 void vring_send_buf(VRing *vr, void *p, int len, int flags)
 {
-    if (!be_ipl()) {
-        vr->avail->idx = bswap16(vr->avail->idx);
-    }
+    uint16_t avail_idx;
+
+    avail_idx = be_ipl() ? vr->avail->idx : bswap16(vr->avail->idx);
 
     /* For follow-up chains we need to keep the first entry point */
     if (!(flags & VRING_HIDDEN_IS_CHAIN)) {
-        vr->avail->ring[vr->avail->idx % vr->num] = be_ipl() ? vr->next_idx :
-                                                               bswap16(vr->next_idx);
+        vr->avail->ring[avail_idx % vr->num] = be_ipl() ? vr->next_idx :
+                                                          bswap16(vr->next_idx);
     }
 
     vr->desc[vr->next_idx].addr = (unsigned long)p;
@@ -177,23 +177,23 @@ void vring_send_buf(VRing *vr, void *p, int len, int flags)
 
     /* Chains only have a single ID */
     if (!(flags & VRING_DESC_F_NEXT)) {
-        vr->avail->idx++;
-    }
-
-    if (!be_ipl()) {
-        vr->avail->idx = bswap16(vr->avail->idx);
+        avail_idx++;
+        vr->avail->idx = be_ipl() ? avail_idx : bswap16(avail_idx);
     }
 }
 
 int vr_poll(VRing *vr)
 {
-    if (vr->used->idx == vr->used_idx) {
+    uint16_t used_idx;
+
+    used_idx = be_ipl() ? vr->used->idx : bswap16(vr->used->idx);
+    if (used_idx == vr->used_idx) {
         vring_notify(vr);
         yield();
         return 0;
     }
 
-    vr->used_idx = vr->used->idx; /* Endianness is preserved */
+    vr->used_idx = used_idx;
     vr->next_idx = 0;
     vr->desc[0].len = 0;
     vr->desc[0].flags = 0;
Signature · 1 line
2.54.0
#2 lore raw

PATCH v2 2/6 pc-bios/s390-ccw/virtio.c: Fix missing break for PCI notifications

In reply to [email protected]

From: Jared Rossi <[email protected]>

Add a break after calling the PCI specific notification function instead of falling through to the default case.

---
 pc-bios/s390-ccw/virtio.c | 1 +
 1 file changed, 1 insertion(+)
pc-bios/s390-ccw/virtio.c +1 -0
index a448dc96e2..7883871033 100644
@@ -115,6 +115,7 @@ bool vring_notify(VRing *vr)
         break;
     case S390_IPL_TYPE_PCI:
         vr->cookie = virtio_pci_notify(vr->id);
+        break;
     default:
         return 1;
     }
Signature · 1 line
2.54.0
Reviewed-by on v2 2/6
#3 lore raw

Re: [PATCH v2 2/6] pc-bios/s390-ccw/virtio.c: Fix missing break for PCI notifications

Quoted 5 lines · On Tue, 2026-06-30 at 10:19 -0400, [email protected] wrote:

From: Jared Rossi <[email protected]>

Add a break after calling the PCI specific notification function instead of falling through to the default case.

Fixes d72fb5e6b2 ("pc-bios/s390-ccw: Add support for virtio-blk-pci IPL")
Quoted 5 lines
---
 pc-bios/s390-ccw/virtio.c | 1 +
 1 file changed, 1 insertion(+)
Reviewed-by on v2 2/6
#4 lore raw

Re: [PATCH v2 2/6] pc-bios/s390-ccw/virtio.c: Fix missing break for PCI notifications

In reply to [email protected]
Quoted 6 lines · On 6/30/26 10:19 AM, [email protected] wrote:

From: Jared Rossi <[email protected]>

Add a break after calling the PCI specific notification function instead of falling through to the default case.

With fixes tag as pointed out by Eric:

#5 lore raw

PATCH v2 3/6 pc-bios/s390-ccw: Add per-queue notification offset for multi-queue virtio configurations

In reply to [email protected]

From: Zhuoying Cai <[email protected]>

The initial support for virtio-blk-pci IPL devices used a single virt-queue, but other device types require multiple queues, and for PCI device types this also requires a per-queue notification offset.

Add a PCI notify field to the VRing struct so that each queue has a unique notify offset as defined in the virtio spec.

---
 pc-bios/s390-ccw/virtio-pci.c | 38 +++++++++++++++++++++--------------
 pc-bios/s390-ccw/virtio-pci.h |  2 +-
 pc-bios/s390-ccw/virtio.c     |  2 +-
 pc-bios/s390-ccw/virtio.h     |  1 +
 4 files changed, 26 insertions(+), 17 deletions(-)
pc-bios/s390-ccw/virtio-pci.c +23 -15
index 53bdb52e76..2c83ec4f13 100644
@@ -21,7 +21,6 @@ VirtioPciCap c_cap; /* Common capabilities  */
 VirtioPciCap d_cap; /* Device capabilities  */
 VirtioPciCap n_cap; /* Notify capabilities  */
 uint32_t notify_mult;
-uint16_t q_notify_offset;
 
 static int virtio_pci_set_status(uint8_t status)
 {
@@ -74,10 +73,10 @@ int virtio_pci_reset(VDev *vdev)
     return 0;
 }
 
-long virtio_pci_notify(int vq_id)
+long virtio_pci_notify(VRing *vr)
 {
-    uint32_t offset = n_cap.off + notify_mult * q_notify_offset;
-    return vpci_bswap16_write(offset, n_cap.bar, (uint16_t) vq_id);
+    uint32_t offset = n_cap.off + notify_mult * vr->pci_notify;
+    return vpci_bswap16_write(offset, n_cap.bar, (uint16_t) vr->id);
 }
 
 /*
@@ -301,8 +300,7 @@ static int virtio_pci_read_pci_cap_config(void)
     }
 
     rc = vpci_read_bswap32(pos + VPCI_N_CAP_MULT, PCI_CFGBAR, &notify_mult);
-    if (rc || vpci_read_bswap16(c_cap.off + VPCI_C_OFFSET_Q_NOFF, c_cap.bar,
-                                &q_notify_offset)) {
+    if (rc) {
         puts("Failed to read notification queue configuration");
         return -EIO;
     }
@@ -332,7 +330,6 @@ int virtio_pci_setup(VDev *vdev)
     VRing *vr;
     int rc;
     uint8_t status;
-    uint16_t vq_size;
     int i = 0;
 
     vdev->guessed_disk_nature = VIRTIO_GDN_NONE;
@@ -380,28 +377,39 @@ int virtio_pci_setup(VDev *vdev)
         return -EIO;
     }
 
-    if (vpci_read_bswap16(VPCI_C_OFFSET_Q_SIZE, c_cap.bar, &vq_size)) {
-        puts("Failed to read virt-queue configuration");
-        return -EIO;
-    }
-
     /* Configure virt-queues for pci */
     for (i = 0; i < vdev->nr_vqs; i++) {
+        uint16_t vq_size;
+        uint16_t vq_notify;
         VqInfo info = {
             .queue = (unsigned long long) virtio_get_ring_area(i),
             .align = KVM_S390_VIRTIO_RING_ALIGN,
             .index = i,
-            .num = vq_size,
+            .num = 0,
         };
 
         vr = &vdev->vrings[i];
-        vring_init(vr, &info);
 
-        if (vpci_set_selected_vq(vr->id)) {
+        if (vpci_set_selected_vq(i)) {
             puts("Failed to set selected virt-queue");
             return -EIO;
         }
 
+        if (vpci_read_bswap16(c_cap.off + VPCI_C_OFFSET_Q_SIZE, c_cap.bar, &vq_size)) {
+            printf("Failed to read virt-queue %d size\n", i);
+            return -EIO;
+        }
+
+        info.num = vq_size;
+
+        if (vpci_read_bswap16(c_cap.off + VPCI_C_OFFSET_Q_NOFF, c_cap.bar, &vq_notify)) {
+            printf("Failed to read virt-queue %d notify offset\n", i);
+            return -EIO;
+        }
+
+        vr->pci_notify = vq_notify;
+        vring_init(vr, &info);
+
         rc = set_pci_vq_addr(VPCI_C_OFFSET_Q_DESCLO, vr->desc);
         rc |= set_pci_vq_addr(VPCI_C_OFFSET_Q_AVAILLO, vr->avail);
         rc |= set_pci_vq_addr(VPCI_C_OFFSET_Q_USEDLO, vr->used);
pc-bios/s390-ccw/virtio-pci.h +1 -1
index 90d07cb9a7..2494df1619 100644
@@ -64,7 +64,7 @@ typedef struct VirtioPciCap  VirtioPciCap;
 
 void virtio_pci_id2type(VDev *vdev, uint16_t device_id);
 int virtio_pci_reset(VDev *vdev);
-long virtio_pci_notify(int vq_id);
+long virtio_pci_notify(VRing *vr);
 int virtio_pci_setup(VDev *vdev);
 int virtio_pci_setup_device(void);
 
pc-bios/s390-ccw/virtio.c +1 -1
index 7883871033..df04479aa6 100644
@@ -114,7 +114,7 @@ bool vring_notify(VRing *vr)
         vr->cookie = virtio_ccw_notify(vdev.schid, vr->id, vr->cookie);
         break;
     case S390_IPL_TYPE_PCI:
-        vr->cookie = virtio_pci_notify(vr->id);
+        vr->cookie = virtio_pci_notify(vr);
         break;
     default:
         return 1;
pc-bios/s390-ccw/virtio.h +1 -0
index d32a4830ca..75ae5bdbc2 100644
@@ -107,6 +107,7 @@ struct VRing {
     VRingUsed *used;
     long cookie;
     int id;
+    uint16_t pci_notify;
 };
 typedef struct VRing VRing;
 
Signature · 1 line
2.54.0
Reviewed-by on v2 3/6
#6 lore raw

Re: [PATCH v2 3/6] pc-bios/s390-ccw: Add per-queue notification offset for multi-queue virtio configurations

Quoted 9 lines · On Tue, 2026-06-30 at 10:19 -0400, [email protected] wrote:

From: Zhuoying Cai <[email protected]>

The initial support for virtio-blk-pci IPL devices used a single virt-queue, but other device types require multiple queues, and for PCI device types this also requires a per-queue notification offset.

Add a PCI notify field to the VRing struct so that each queue has a unique notify offset as defined in the virtio spec.

Quoted 8 lines
---
 pc-bios/s390-ccw/virtio-pci.c | 38 +++++++++++++++++++++--------------
 pc-bios/s390-ccw/virtio-pci.h |  2 +-
 pc-bios/s390-ccw/virtio.c     |  2 +-
 pc-bios/s390-ccw/virtio.h     |  1 +
 4 files changed, 26 insertions(+), 17 deletions(-)

Thank you for pulling out the bugfix from this patch. This still looks good.

#7 lore raw

PATCH v2 4/6 pc-bios/s390-ccw: Verify virtio support when booting from virtio PCI device on s390x

In reply to [email protected]

From: Jared Rossi <[email protected]>

The virtio specification requires that each PCI device has a vendor ID of 0x1af4. Verify this value before continuing with boot process.

---
 pc-bios/s390-ccw/main.c       |  1 +
 pc-bios/s390-ccw/virtio-pci.c | 19 +++++++++++++++++++
 pc-bios/s390-ccw/virtio-pci.h |  3 +++
 pc-bios/s390-ccw/virtio.c     |  2 ++
 pc-bios/s390-ccw/virtio.h     |  1 +
 5 files changed, 26 insertions(+)
pc-bios/s390-ccw/main.c +1 -0
index 26287cfd81..fb47d29bb1 100644
@@ -163,6 +163,7 @@ static bool find_fid(uint32_t fid)
     }
 
     vdev->pci_fh = entry.fh;
+    vdev->vendor_id = entry.vendor_id;
     virtio_pci_id2type(vdev, entry.device_id);
 
     return vdev->dev_type != 0;
pc-bios/s390-ccw/virtio-pci.c +19 -0
index 2c83ec4f13..f501252c81 100644
@@ -325,6 +325,20 @@ static int enable_pci_bus_master(void)
     return 0;
 }
 
+bool virtio_pci_is_supported(VDev *vdev)
+{
+    if (vdev->vendor_id == PCI_VENDOR_VIRTIO) {
+        switch (vdev->dev_type) {
+        case VIRTIO_ID_BLOCK:
+            return true;
+        default:
+            return false;
+        }
+    }
+
+    return false;
+}
+
 int virtio_pci_setup(VDev *vdev)
 {
     VRing *vr;
@@ -335,6 +349,11 @@ int virtio_pci_setup(VDev *vdev)
     vdev->guessed_disk_nature = VIRTIO_GDN_NONE;
     vdev->cmd_vr_idx = 0;
 
+    if (!virtio_pci_is_supported(vdev)) {
+        puts("Virtio PCI unsupported for this device ID");
+        return -ENODEV;
+    }
+
     if (virtio_pci_read_pci_cap_config()) {
         puts("Invalid virtio PCI capabilities");
         return -EIO;
pc-bios/s390-ccw/virtio-pci.h +3 -0
index 2494df1619..33b683bd92 100644
@@ -56,6 +56,8 @@
 
 #define VIRTIO_F_VERSION_1          1   /* Feature bit 32 */
 
+#define PCI_VENDOR_VIRTIO           0x1af4
+
 struct VirtioPciCap {
     uint8_t bar;     /* Which PCIAS it's in */
     uint32_t off;    /* Offset within bar */
@@ -65,6 +67,7 @@ typedef struct VirtioPciCap  VirtioPciCap;
 void virtio_pci_id2type(VDev *vdev, uint16_t device_id);
 int virtio_pci_reset(VDev *vdev);
 long virtio_pci_notify(VRing *vr);
+bool virtio_pci_is_supported(VDev *vdev);
 int virtio_pci_setup(VDev *vdev);
 int virtio_pci_setup_device(void);
 
pc-bios/s390-ccw/virtio.c +2 -0
index df04479aa6..a0d249db24 100644
@@ -247,6 +247,8 @@ bool virtio_is_supported(VDev *vdev)
     case S390_IPL_TYPE_QEMU_SCSI:
     case S390_IPL_TYPE_CCW:
         return virtio_ccw_is_supported(vdev);
+    case S390_IPL_TYPE_PCI:
+        return virtio_pci_is_supported(vdev);
     default:
         return false;
     }
pc-bios/s390-ccw/virtio.h +1 -0
index 75ae5bdbc2..aa307025e0 100644
@@ -259,6 +259,7 @@ struct VDev {
     bool scsi_device_selected;
     ScsiDevice selected_scsi_device;
     uint32_t pci_fh;
+    uint16_t vendor_id;
     uint32_t max_transfer;
     uint32_t guest_features[2];
 };
Signature · 1 line
2.54.0
Reviewed-by on v2 4/6
#8 lore raw

Re: [PATCH v2 4/6] pc-bios/s390-ccw: Verify virtio support when booting from virtio PCI device on s390x

Quoted 5 lines · On Tue, 2026-06-30 at 10:19 -0400, [email protected] wrote:

From: Jared Rossi <[email protected]>

The virtio specification requires that each PCI device has a vendor ID of 0x1af4. Verify this value before continuing with boot process.

Quoted 8 lines
---
 pc-bios/s390-ccw/main.c       |  1 +
 pc-bios/s390-ccw/virtio-pci.c | 19 +++++++++++++++++++
 pc-bios/s390-ccw/virtio-pci.h |  3 +++
 pc-bios/s390-ccw/virtio.c     |  2 ++
 pc-bios/s390-ccw/virtio.h     |  1 +
 5 files changed, 26 insertions(+)

Looks nice.

#9 lore raw

PATCH v2 5/6 pc-bios/s390-ccw: write IPLB location for non-net virtio devices

In reply to [email protected]

From: Jared Rossi <[email protected]>

When IPL type is PCI, cutype is unknown, so the IPLB location does not get written to lowcore. Instead of checking cutype, check if the device is virtio generically, which covers both CCW and PCI variants. Net devices are excluded.

---
 pc-bios/s390-ccw/main.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
pc-bios/s390-ccw/main.c +2 -1
index fb47d29bb1..9c147ac5fa 100644
@@ -52,7 +52,8 @@ void write_subsystem_identification(void)
 
 void write_iplb_location(void)
 {
-    if (cutype == CU_TYPE_VIRTIO && virtio_get_device_type() != VIRTIO_ID_NET) {
+    if (virtio_is_supported(virtio_get_device()) &&
+        virtio_get_device_type() != VIRTIO_ID_NET) {
         lowcore->ptr_iplb = ptr2u32(&iplb);
     }
 }
Signature · 1 line
2.54.0
Reviewed-by on v2 5/6
#10 lore raw

Re: [PATCH v2 5/6] pc-bios/s390-ccw: write IPLB location for non-net virtio devices

Quoted 11 lines · On Tue, 2026-06-30 at 10:19 -0400, [email protected] wrote:

From: Jared Rossi <[email protected]>

When IPL type is PCI, cutype is unknown, so the IPLB location does not get written to lowcore. Instead of checking cutype, check if the device is virtio generically, which covers both CCW and PCI variants. Net devices are excluded.

---
 pc-bios/s390-ccw/main.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
#11 lore raw

PATCH v2 6/6 s390x: Enable boot menu for virtio pci device

In reply to [email protected]

From: Zhuoying Cai <[email protected]>

Add S390_IPL_TYPE_PCI to the boot menu handling logic to enable interactive boot menu support for virtio PCI devices on s390x.

---
 hw/s390x/ipl.c          | 1 +
 pc-bios/s390-ccw/main.c | 1 +
 2 files changed, 2 insertions(+)
hw/s390x/ipl.c +1 -0
index 4cca21c621..d5fdc3ea0f 100644
@@ -315,6 +315,7 @@ static void s390_ipl_set_boot_menu(S390IPLState *ipl)
             return;
         }
         break;
+    case S390_IPL_TYPE_PCI:
     case S390_IPL_TYPE_QEMU_SCSI:
         break;
     default:
pc-bios/s390-ccw/main.c +1 -0
index 9c147ac5fa..8bc6e8eaa3 100644
@@ -185,6 +185,7 @@ static void menu_setup(VDev *vdev)
 
     switch (vdev->ipl_type) {
     case S390_IPL_TYPE_CCW:
+    case S390_IPL_TYPE_PCI:
     case S390_IPL_TYPE_QEMU_SCSI:
         menu_set_parms(qipl.qipl_flags & BOOT_MENU_FLAG_MASK,
                        qipl.boot_menu_timeout);
Signature · 1 line
2.54.0
Tested-by on the v2 series
#13 lore raw
In reply to [email protected]
Quoted 10 lines · On 6/30/26 10:19 AM, [email protected] wrote:

From: Jared Rossi <[email protected]>

When booting from a PCI device on s390x there may be intermittent failures caused by mismatched byte swaps. Refactor the byte swapping to ensure that it is properly synchronized.

Correct how the queue notification offset is calculated so that it may be used for multi-queue configurations in the future.

Handle some switch cases that were not caught for PCI devices.

For the series:

Worth nothing that, with this series applied (and after generating a new s390-ccw.img) I am no longer able to reproduce the error reported in:

https://gitlab.com/qemu-project/qemu/-/work_items/3350

Thanks,
Matt

#14 lore raw
In reply to [email protected]
Quoted 30 lines · On Tue, Jun 30 2026, [email protected] wrote:

From: Jared Rossi <[email protected]>

When booting from a PCI device on s390x there may be intermittent failures caused by mismatched byte swaps. Refactor the byte swapping to ensure that it is properly synchronized.

Correct how the queue notification offset is calculated so that it may be used for multi-queue configurations in the future.

Handle some switch cases that were not caught for PCI devices.

Changes v1 -> v2:
- Split addition of missing break into its own patch
- Add PCI case to generic is_viritio_supported() wapper
- Switch the order of patches formerly 3 and 4
- Use virtio_is_supported() instead of checking cutype when writing IPLB location to cover both CCW and PCI virtio devices

Jared Rossi (3):
pc-bios/s390-ccw/virtio.c: Fix missing break for PCI notifications pc-bios/s390-ccw: Verify virtio support when booting from virtio PCI

    device on s390x

pc-bios/s390-ccw: write IPLB location for non-net virtio devices

Zhuoying Cai (3):
pc-bios/s390-ccw: Refactor byte swapping pc-bios/s390-ccw: Add per-queue notification offset for multi-queue
virtio configurations
s390x: Enable boot menu for virtio pci device

Quoted 8 lines
 hw/s390x/ipl.c                |  1 +
 pc-bios/s390-ccw/main.c       |  5 ++-
 pc-bios/s390-ccw/virtio-pci.c | 57 ++++++++++++++++++++++++++---------
 pc-bios/s390-ccw/virtio-pci.h |  5 ++-
 pc-bios/s390-ccw/virtio.c     | 29 ++++++++++--------
 pc-bios/s390-ccw/virtio.h     |  2 ++
 6 files changed, 69 insertions(+), 30 deletions(-)

Thanks, applied.