s390x: Fix intermittent errors with IPL from PCI devices on s390x
v2 cover s390x: Fix intermittent errors with IPL from PCI devices on s390x
From: Jared Rossi <[email protected]>
When booting from a PCI device on s390x there may be intermittent failures caused by mismatched byte swaps. Refactor the byte swapping to ensure that it is properly synchronized.
Correct how the queue notification offset is calculated so that it may be used for multi-queue configurations in the future.
Handle some switch cases that were not caught for PCI devices.
Changes v1 -> v2:
- Split addition of missing break into its own patch
- Add PCI case to generic is_viritio_supported() wapper
- Switch the order of patches formerly 3 and 4
- Use virtio_is_supported() instead of checking cutype when writing IPLB location to cover both CCW and PCI virtio devices
Jared Rossi (3):
pc-bios/s390-ccw/virtio.c: Fix missing break for PCI notifications pc-bios/s390-ccw: Verify virtio support when booting from virtio PCI
device on s390x
pc-bios/s390-ccw: write IPLB location for non-net virtio devices
Zhuoying Cai (3):
pc-bios/s390-ccw: Refactor byte swapping pc-bios/s390-ccw: Add per-queue notification offset for multi-queue
virtio configurations
s390x: Enable boot menu for virtio pci device
hw/s390x/ipl.c | 1 + pc-bios/s390-ccw/main.c | 5 ++- pc-bios/s390-ccw/virtio-pci.c | 57 ++++++++++++++++++++++++++--------- pc-bios/s390-ccw/virtio-pci.h | 5 ++- pc-bios/s390-ccw/virtio.c | 29 ++++++++++-------- pc-bios/s390-ccw/virtio.h | 2 ++ 6 files changed, 69 insertions(+), 30 deletions(-)
Signature · 1 line
2.54.0
PATCH v2 1/6 pc-bios/s390-ccw: Refactor byte swapping
From: Zhuoying Cai <[email protected]>
Introduce local variables to cache the byte-swapped values eliminating some redundant byte swap operations. Additionally, do byte swap when polling to avoid a special case where endianness is preserved.
--- pc-bios/s390-ccw/virtio.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
pc-bios/s390-ccw/virtio.c +12 -12
@@ -153,14 +153,14 @@ static void vr_bswap_descriptor(VRingDesc *desc)void vring_send_buf (VRing *vr,void *p,int len,int flags) { -if (!be_ipl ()) { - vr->avail->idx =bswap16 (vr->avail->idx); - } +uint16_t avail_idx; + + avail_idx =be_ipl () ? vr->avail->idx :bswap16 (vr->avail->idx);/* For follow-up chains we need to keep the first entry point */ if (!(flags &VRING_HIDDEN_IS_CHAIN )) { - vr->avail->ring[vr->avail->idx % vr->num] =be_ipl () ? vr->next_idx : -bswap16 (vr->next_idx); + vr->avail->ring[avail_idx % vr->num] =be_ipl () ? vr->next_idx : +bswap16 (vr->next_idx); } vr->desc[vr->next_idx].addr = (unsigned long )p;
@@ -177,23 +177,23 @@ void vring_send_buf(VRing *vr, void *p, int len, int flags)/* Chains only have a single ID */ if (!(flags &VRING_DESC_F_NEXT )) { - vr->avail->idx++; - } - -if (!be_ipl ()) { - vr->avail->idx =bswap16 (vr->avail->idx); + avail_idx++; + vr->avail->idx =be_ipl () ? avail_idx :bswap16 (avail_idx); } }int vr_poll (VRing *vr) { -if (vr->used->idx == vr->used_idx) { +uint16_t used_idx; + + used_idx =be_ipl () ? vr->used->idx :bswap16 (vr->used->idx); +if (used_idx == vr->used_idx) {vring_notify (vr);yield ();return 0 ; } - vr->used_idx = vr->used->idx;/* Endianness is preserved */ + vr->used_idx = used_idx; vr->next_idx =0 ; vr->desc[0 ].len =0 ; vr->desc[0 ].flags =0 ;
Signature · 1 line
2.54.0
PATCH v2 2/6 pc-bios/s390-ccw/virtio.c: Fix missing break for PCI notifications
In reply to [email protected]From: Jared Rossi <[email protected]>
Add a break after calling the PCI specific notification function instead of falling through to the default case.
--- pc-bios/s390-ccw/virtio.c | 1 + 1 file changed, 1 insertion(+)
pc-bios/s390-ccw/virtio.c +1 -0
@@ -115,6 +115,7 @@ bool vring_notify(VRing *vr)break ;case S390_IPL_TYPE_PCI : vr->cookie =virtio_pci_notify (vr->id); +break ;default :return 1 ; }
Signature · 1 line
2.54.0
Re: [PATCH v2 2/6] pc-bios/s390-ccw/virtio.c: Fix missing break for PCI notifications
Quoted 5 lines · On Tue, 2026-06-30 at 10:19 -0400, [email protected] wrote:
From: Jared Rossi <[email protected]>
Add a break after calling the PCI specific notification function instead of falling through to the default case.
Quoted 5 lines
Signed-off-by Jared Rossi <[email protected]>--- pc-bios/s390-ccw/virtio.c | 1 + 1 file changed, 1 insertion(+)
Re: [PATCH v2 2/6] pc-bios/s390-ccw/virtio.c: Fix missing break for PCI notifications
In reply to [email protected]Quoted 6 lines · On 6/30/26 10:19 AM, [email protected] wrote:
From: Jared Rossi <[email protected]>
Add a break after calling the PCI specific notification function instead of falling through to the default case.
Signed-off-by Jared Rossi <[email protected]>
With fixes tag as pointed out by Eric:
PATCH v2 3/6 pc-bios/s390-ccw: Add per-queue notification offset for multi-queue virtio configurations
In reply to [email protected]From: Zhuoying Cai <[email protected]>
The initial support for virtio-blk-pci IPL devices used a single virt-queue, but other device types require multiple queues, and for PCI device types this also requires a per-queue notification offset.
Add a PCI notify field to the VRing struct so that each queue has a unique notify offset as defined in the virtio spec.
--- pc-bios/s390-ccw/virtio-pci.c | 38 +++++++++++++++++++++-------------- pc-bios/s390-ccw/virtio-pci.h | 2 +- pc-bios/s390-ccw/virtio.c | 2 +- pc-bios/s390-ccw/virtio.h | 1 + 4 files changed, 26 insertions(+), 17 deletions(-)
pc-bios/s390-ccw/virtio-pci.c +23 -15
@@ -21,7 +21,6 @@ VirtioPciCap c_cap; /* Common capabilities */VirtioPciCap d_cap;/* Device capabilities */ VirtioPciCap n_cap;/* Notify capabilities */ uint32_t notify_mult; -uint16_t q_notify_offset;static int virtio_pci_set_status (uint8_t status) {
@@ -74,10 +73,10 @@ int virtio_pci_reset(VDev *vdev)return 0 ; } -long virtio_pci_notify (int vq_id) +long virtio_pci_notify (VRing *vr) { -uint32_t offset = n_cap.off + notify_mult * q_notify_offset; -return vpci_bswap16_write (offset, n_cap.bar, (uint16_t ) vq_id); +uint32_t offset = n_cap.off + notify_mult * vr->pci_notify; +return vpci_bswap16_write (offset, n_cap.bar, (uint16_t ) vr->id); }/*
@@ -301,8 +300,7 @@ static int virtio_pci_read_pci_cap_config(void) } rc =vpci_read_bswap32 (pos +VPCI_N_CAP_MULT ,PCI_CFGBAR , ¬ify_mult); -if (rc ||vpci_read_bswap16 (c_cap.off +VPCI_C_OFFSET_Q_NOFF , c_cap.bar, - &q_notify_offset)) { +if (rc) {puts ("Failed to read notification queue configuration" );return -EIO ; }
@@ -332,7 +330,6 @@ int virtio_pci_setup(VDev *vdev)VRing *vr;int rc;uint8_t status; -uint16_t vq_size;int i =0 ; vdev->guessed_disk_nature =VIRTIO_GDN_NONE ;
@@ -380,28 +377,39 @@ int virtio_pci_setup(VDev *vdev)return -EIO ; } -if (vpci_read_bswap16 (VPCI_C_OFFSET_Q_SIZE , c_cap.bar, &vq_size)) { -puts ("Failed to read virt-queue configuration" ); -return -EIO ; - } -/* Configure virt-queues for pci */ for (i =0 ; i < vdev->nr_vqs; i++) { +uint16_t vq_size; +uint16_t vq_notify;VqInfo info = { .queue = (unsigned long long )virtio_get_ring_area (i), .align =KVM_S390_VIRTIO_RING_ALIGN , .index = i, - .num = vq_size, + .num =0 , }; vr = &vdev->vrings[i]; -vring_init (vr, &info); -if (vpci_set_selected_vq (vr->id)) { +if (vpci_set_selected_vq (i)) {puts ("Failed to set selected virt-queue" );return -EIO ; } +if (vpci_read_bswap16 (c_cap.off +VPCI_C_OFFSET_Q_SIZE , c_cap.bar, &vq_size)) { +printf ("Failed to read virt-queue %d size\n" , i); +return -EIO ; + } + + info.num = vq_size; + +if (vpci_read_bswap16 (c_cap.off +VPCI_C_OFFSET_Q_NOFF , c_cap.bar, &vq_notify)) { +printf ("Failed to read virt-queue %d notify offset\n" , i); +return -EIO ; + } + + vr->pci_notify = vq_notify; +vring_init (vr, &info); + rc =set_pci_vq_addr (VPCI_C_OFFSET_Q_DESCLO , vr->desc); rc |=set_pci_vq_addr (VPCI_C_OFFSET_Q_AVAILLO , vr->avail); rc |=set_pci_vq_addr (VPCI_C_OFFSET_Q_USEDLO , vr->used);
pc-bios/s390-ccw/virtio-pci.h +1 -1
@@ -64,7 +64,7 @@ typedef struct VirtioPciCap VirtioPciCap;void virtio_pci_id2type (VDev *vdev,uint16_t device_id);int virtio_pci_reset (VDev *vdev); -long virtio_pci_notify (int vq_id); +long virtio_pci_notify (VRing *vr);int virtio_pci_setup (VDev *vdev);int virtio_pci_setup_device (void );
pc-bios/s390-ccw/virtio.c +1 -1
@@ -114,7 +114,7 @@ bool vring_notify(VRing *vr) vr->cookie =virtio_ccw_notify (vdev.schid, vr->id, vr->cookie);break ;case S390_IPL_TYPE_PCI : - vr->cookie =virtio_pci_notify (vr->id); + vr->cookie =virtio_pci_notify (vr);break ;default :return 1 ;
pc-bios/s390-ccw/virtio.h +1 -0
@@ -107,6 +107,7 @@ struct VRing {VRingUsed *used;long cookie;int id; +uint16_t pci_notify; };typedef struct VRing VRing ;
Signature · 1 line
2.54.0
Re: [PATCH v2 3/6] pc-bios/s390-ccw: Add per-queue notification offset for multi-queue virtio configurations
Quoted 9 lines · On Tue, 2026-06-30 at 10:19 -0400, [email protected] wrote:
From: Zhuoying Cai <[email protected]>
The initial support for virtio-blk-pci IPL devices used a single virt-queue, but other device types require multiple queues, and for PCI device types this also requires a per-queue notification offset.
Add a PCI notify field to the VRing struct so that each queue has a unique notify offset as defined in the virtio spec.
Quoted 8 lines
Signed-off-by Zhuoying Cai <[email protected]>Signed-off-by Jared Rossi <[email protected]>--- pc-bios/s390-ccw/virtio-pci.c | 38 +++++++++++++++++++++-------------- pc-bios/s390-ccw/virtio-pci.h | 2 +- pc-bios/s390-ccw/virtio.c | 2 +- pc-bios/s390-ccw/virtio.h | 1 + 4 files changed, 26 insertions(+), 17 deletions(-)
Thank you for pulling out the bugfix from this patch. This still looks good.
PATCH v2 4/6 pc-bios/s390-ccw: Verify virtio support when booting from virtio PCI device on s390x
In reply to [email protected]From: Jared Rossi <[email protected]>
The virtio specification requires that each PCI device has a vendor ID of 0x1af4. Verify this value before continuing with boot process.
--- pc-bios/s390-ccw/main.c | 1 + pc-bios/s390-ccw/virtio-pci.c | 19 +++++++++++++++++++ pc-bios/s390-ccw/virtio-pci.h | 3 +++ pc-bios/s390-ccw/virtio.c | 2 ++ pc-bios/s390-ccw/virtio.h | 1 + 5 files changed, 26 insertions(+)
pc-bios/s390-ccw/main.c +1 -0
@@ -163,6 +163,7 @@ static bool find_fid(uint32_t fid) } vdev->pci_fh = entry.fh; + vdev->vendor_id = entry.vendor_id;virtio_pci_id2type (vdev, entry.device_id);return vdev->dev_type !=0 ;
pc-bios/s390-ccw/virtio-pci.c +19 -0
@@ -325,6 +325,20 @@ static int enable_pci_bus_master(void)return 0 ; } +bool virtio_pci_is_supported (VDev *vdev) +{ +if (vdev->vendor_id ==PCI_VENDOR_VIRTIO ) { +switch (vdev->dev_type) { +case VIRTIO_ID_BLOCK : +return true; +default : +return false; + } + } + +return false; +} +int virtio_pci_setup (VDev *vdev) {VRing *vr;
@@ -335,6 +349,11 @@ int virtio_pci_setup(VDev *vdev) vdev->guessed_disk_nature =VIRTIO_GDN_NONE ; vdev->cmd_vr_idx =0 ; +if (!virtio_pci_is_supported (vdev)) { +puts ("Virtio PCI unsupported for this device ID" ); +return -ENODEV ; + } +if (virtio_pci_read_pci_cap_config ()) {puts ("Invalid virtio PCI capabilities" );return -EIO ;
pc-bios/s390-ccw/virtio-pci.h +3 -0
@@ -56,6 +56,8 @@#define VIRTIO_F_VERSION_1 1/* Feature bit 32 */ +#define PCI_VENDOR_VIRTIO 0x1af4 +struct VirtioPciCap {uint8_t bar;/* Which PCIAS it's in */ uint32_t off;/* Offset within bar */
@@ -65,6 +67,7 @@ typedef struct VirtioPciCap VirtioPciCap;void virtio_pci_id2type (VDev *vdev,uint16_t device_id);int virtio_pci_reset (VDev *vdev);long virtio_pci_notify (VRing *vr); +bool virtio_pci_is_supported (VDev *vdev);int virtio_pci_setup (VDev *vdev);int virtio_pci_setup_device (void );
pc-bios/s390-ccw/virtio.c +2 -0
@@ -247,6 +247,8 @@ bool virtio_is_supported(VDev *vdev)case S390_IPL_TYPE_QEMU_SCSI :case S390_IPL_TYPE_CCW :return virtio_ccw_is_supported (vdev); +case S390_IPL_TYPE_PCI : +return virtio_pci_is_supported (vdev);default :return false; }
pc-bios/s390-ccw/virtio.h +1 -0
@@ -259,6 +259,7 @@ struct VDev {bool scsi_device_selected;ScsiDevice selected_scsi_device;uint32_t pci_fh; +uint16_t vendor_id;uint32_t max_transfer;uint32_t guest_features[2 ]; };
Signature · 1 line
2.54.0
Re: [PATCH v2 4/6] pc-bios/s390-ccw: Verify virtio support when booting from virtio PCI device on s390x
Quoted 5 lines · On Tue, 2026-06-30 at 10:19 -0400, [email protected] wrote:
From: Jared Rossi <[email protected]>
The virtio specification requires that each PCI device has a vendor ID of
0x1af4. Verify this value before continuing with boot process.
Quoted 8 lines
Signed-off-by Jared Rossi <[email protected]>--- pc-bios/s390-ccw/main.c | 1 + pc-bios/s390-ccw/virtio-pci.c | 19 +++++++++++++++++++ pc-bios/s390-ccw/virtio-pci.h | 3 +++ pc-bios/s390-ccw/virtio.c | 2 ++ pc-bios/s390-ccw/virtio.h | 1 + 5 files changed, 26 insertions(+)
Looks nice.
PATCH v2 5/6 pc-bios/s390-ccw: write IPLB location for non-net virtio devices
In reply to [email protected]From: Jared Rossi <[email protected]>
When IPL type is PCI, cutype is unknown, so the IPLB location does not get written to lowcore. Instead of checking cutype, check if the device is virtio generically, which covers both CCW and PCI variants. Net devices are excluded.
--- pc-bios/s390-ccw/main.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
pc-bios/s390-ccw/main.c +2 -1
@@ -52,7 +52,8 @@ void write_subsystem_identification(void)void write_iplb_location (void ) { -if (cutype ==CU_TYPE_VIRTIO &&virtio_get_device_type () !=VIRTIO_ID_NET ) { +if (virtio_is_supported (virtio_get_device ()) && +virtio_get_device_type () !=VIRTIO_ID_NET ) { lowcore->ptr_iplb =ptr2u32 (&iplb); } }
Signature · 1 line
2.54.0
Re: [PATCH v2 5/6] pc-bios/s390-ccw: write IPLB location for non-net virtio devices
Quoted 11 lines · On Tue, 2026-06-30 at 10:19 -0400, [email protected] wrote:
From: Jared Rossi <[email protected]>
When IPL type is PCI, cutype is unknown, so the IPLB location does not get written to lowcore. Instead of checking cutype, check if the device is virtio generically, which covers both CCW and PCI variants. Net devices are excluded.
Signed-off-by Jared Rossi <[email protected]>--- pc-bios/s390-ccw/main.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
PATCH v2 6/6 s390x: Enable boot menu for virtio pci device
In reply to [email protected]From: Zhuoying Cai <[email protected]>
Add S390_IPL_TYPE_PCI to the boot menu handling logic to enable interactive boot menu support for virtio PCI devices on s390x.
--- hw/s390x/ipl.c | 1 + pc-bios/s390-ccw/main.c | 1 + 2 files changed, 2 insertions(+)
hw/s390x/ipl.c +1 -0
@@ -315,6 +315,7 @@ static void s390_ipl_set_boot_menu(S390IPLState *ipl)return ; }break ; +case S390_IPL_TYPE_PCI :case S390_IPL_TYPE_QEMU_SCSI :break ;default :
pc-bios/s390-ccw/main.c +1 -0
@@ -185,6 +185,7 @@ static void menu_setup(VDev *vdev)switch (vdev->ipl_type) {case S390_IPL_TYPE_CCW : +case S390_IPL_TYPE_PCI :case S390_IPL_TYPE_QEMU_SCSI :menu_set_parms (qipl.qipl_flags &BOOT_MENU_FLAG_MASK , qipl.boot_menu_timeout);
Signature · 1 line
2.54.0
Re: [PATCH v2 6/6] s390x: Enable boot menu for virtio pci device
Quoted 4 lines · On 6/30/26 10:19 AM, [email protected] wrote:
From: Zhuoying Cai <[email protected]>
Add
S390_IPL_TYPE_PCIto the boot menu handling logic to enable interactive boot menu support for virtio PCI devices on s390x.
Thanks, that message was bugging me!
Quoted 10 lines · On 6/30/26 10:19 AM, [email protected] wrote:
From: Jared Rossi <[email protected]>
When booting from a PCI device on s390x there may be intermittent failures caused by mismatched byte swaps. Refactor the byte swapping to ensure that it is properly synchronized.
Correct how the queue notification offset is calculated so that it may be used for multi-queue configurations in the future.
Handle some switch cases that were not caught for PCI devices.
For the series:
Worth nothing that, with this series applied (and after generating a new s390-ccw.img) I am no longer able to reproduce the error reported in:
https://gitlab.com/qemu-project/qemu/-/work_items/3350
Thanks,
Matt
Cornelia Huck
AppliedQuoted 30 lines · On Tue, Jun 30 2026, [email protected] wrote:
From: Jared Rossi <[email protected]>
When booting from a PCI device on s390x there may be intermittent failures caused by mismatched byte swaps. Refactor the byte swapping to ensure that it is properly synchronized.
Correct how the queue notification offset is calculated so that it may be used for multi-queue configurations in the future.
Handle some switch cases that were not caught for PCI devices.
Changes v1 -> v2:
- Split addition of missing break into its own patch
- Add PCI case to genericis_viritio_supported()wapper
- Switch the order of patches formerly 3 and 4
- Usevirtio_is_supported()instead of checking cutype when writing IPLB location to cover both CCW and PCI virtio devicesJared Rossi (3):
pc-bios/s390-ccw/virtio.c: Fix missing break for PCI notifications pc-bios/s390-ccw: Verify virtio support when booting from virtio PCIdevice on s390xpc-bios/s390-ccw: write IPLB location for non-net virtio devices
Zhuoying Cai (3):
pc-bios/s390-ccw: Refactor byte swapping pc-bios/s390-ccw: Add per-queue notification offset for multi-queue
virtio configurations
s390x: Enable boot menu for virtio pci device
Quoted 8 lines
hw/s390x/ipl.c | 1 + pc-bios/s390-ccw/main.c | 5 ++- pc-bios/s390-ccw/virtio-pci.c | 57 ++++++++++++++++++++++++++--------- pc-bios/s390-ccw/virtio-pci.h | 5 ++- pc-bios/s390-ccw/virtio.c | 29 ++++++++++-------- pc-bios/s390-ccw/virtio.h | 2 ++ 6 files changed, 69 insertions(+), 30 deletions(-)
Thanks, applied.