s390x/pci: fix ISM reset
cover s390x/pci: fix ISM reset
Commit ef1535901a0 (re-)introduced an issue where passthrough ISM devices on s390x would enter an error state after reboot. This was previously fixed by 03451953c79e, using device reset callbacks, however the change in ef1535901a0 effectively triggers a cold reset of the pci bus before the device reset callbacks are triggered.
To resolve this, this series proposes to remove the use of the reset callback for ISM cleanup and instead trigger ISM reset from subsystem_reset before triggering bus resets. This has to happen before the bus resets because the reset of s390-pcihost will trigger reset of the PCI bus followed by the s390-pci bus, and the former will trigger vfio-pci reset / the aperture-wide unmap that ISM gets upset about.
/s390-pcihost (s390-pcihost)
/pci.0 (PCI)
/s390-pcibus.0 (s390-pcibus)While fixing this, it was also noted that kernel warnings could be seen that indicate a guest ISC reference count error. That's because in some reset cases we were not bothering to disable AIF, but would again re-enable it after the reset (causing the reference count to grow erroneously). This was a base issue that went unnoticed because the kernel previously did not detect and issue a warning for this scenario.
Matthew Rosato (3):
s390x/pci: avoid double enable/disable of aif
s390x/pci: refresh fh before disabling aif
s390x/pci: drive ISM reset from subsystem reset
hw/s390x/s390-pci-bus.c | 26 ++++++++++++++++--------- hw/s390x/s390-pci-kvm.c | 34 +++++++++++++++++++++++++++++++-- hw/s390x/s390-virtio-ccw.c | 2 ++ include/hw/s390x/s390-pci-bus.h | 2 ++ 4 files changed, 53 insertions(+), 11 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/3 s390x/pci: avoid double enable/disable of aif
Use a flag to keep track of whether AIF is currently enabled. This can be used to avoid enabling/disabling AIF multiple times as well as to determine whether or not it should be disabled during reset processing.
--- hw/s390x/s390-pci-kvm.c | 25 +++++++++++++++++++++++-- include/hw/s390x/s390-pci-bus.h | 1 + 2 files changed, 24 insertions(+), 2 deletions(-)
hw/s390x/s390-pci-kvm.c +23 -2
@@ -27,6 +27,7 @@ bool s390_pci_kvm_interp_allowed(void)int s390_pci_kvm_aif_enable (S390PCIBusDevice *pbdev,ZpciFib *fib,bool assist) { +int rc;struct kvm_s390_zpci_op args = { .fh = pbdev->fh, .op =KVM_S390_ZPCIOP_REG_AEN ,
@@ -38,15 +39,35 @@ int s390_pci_kvm_aif_enable(S390PCIBusDevice *pbdev, ZpciFib *fib, bool assist) .u.reg_aen.flags = (assist) ? 0 : KVM_S390_ZPCIOP_REGAEN_HOST }; -return kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args); +if (pbdev->aif) { +return -EINVAL ; + } + + rc =kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args); +if (rc ==0 ) { + pbdev->aif = true; + } + +return rc; }int s390_pci_kvm_aif_disable (S390PCIBusDevice *pbdev) { +int rc; +struct kvm_s390_zpci_op args = { .fh = pbdev->fh, .op =KVM_S390_ZPCIOP_DEREG_AEN }; -return kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args); +if (!pbdev->aif) { +return -EINVAL ; + } + + rc =kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args); +if (rc ==0 ) { + pbev->aif = false; + } + +return rc; }
include/hw/s390x/s390-pci-bus.h +1 -0
@@ -361,6 +361,7 @@ struct S390PCIBusDevice {bool unplug_requested;bool interp;bool forwarding_assist; +bool aif;QTAILQ_ENTRY (S390PCIBusDevice) link; };
Signature · 1 line
2.43.0
Re: [PATCH 1/3] s390x/pci: avoid double enable/disable of aif
Quoted 56 lines · Matthew Rosato
Use a flag to keep track of whether AIF is currently enabled. This can be
used to avoid enabling/disabling AIF multiple times as well as to determine
whether or not it should be disabled during reset processing.Fixes d0bc7091c2 ("s390x/pci: enable adapter event notification forinterpreted devices")
Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-kvm.c | 25 +++++++++++++++++++++++--include/hw/s390x/s390-pci-bus.h| 1 +2 files changed, 24 insertions(+), 2 deletions(-)hw/s390x/s390-pci-kvm.c +0
-0@@ -27,6 +27,7 @@ bool s390_pci_kvm_interp_allowed(void)
ints390_pci_kvm_aif_enable(S390PCIBusDevice *pbdev, ZpciFib *fib, bool assist)
{
+ int rc;
structkvm_s390_zpci_opargs = {
.fh =pbdev->fh,
.op =KVM_S390_ZPCIOP_REG_AEN,hw/s390x/s390-pci-kvm.c +0
-0@@ -38,15 +39,35 @@ int s390_pci_kvm_aif_enable(S390PCIBusDevice *pbdev, ZpciFib *fib, bool assist).u.reg_aen.flags = (assist) ? 0 :
KVM_S390_ZPCIOP_REGAEN_HOST
};
- returnkvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); + if (pbdev->aif) {
+ return-EINVAL;
+ }hw/s390x/s390-pci-kvm.c +7
-0+ + rc = kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); + if (rc == 0) { + pbdev->aif = true; + } + + return rc;}
ints390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev){
+ int rc;
+
structkvm_s390_zpci_opargs = {
.fh =pbdev->fh,
.op =KVM_S390_ZPCIOP_DEREG_AEN
};
Quoted 8 lines
- return
kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); + if (!pbdev->aif) {
+ return-EINVAL;
+ }
+
+ rc =kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); + if (rc == 0) {
+pbev->aif= false;
s/pbev/pbdev/
You fix this in patch 2. :)
With that fixed:
Quoted 17 lines
+ }
+
+ return rc;
}include/hw/s390x/s390- renamed from include/hw/s390x/s390-pci-bus.h +0
-0
pci-bus.h
indexb1bdbeaeb5..435e788867100644include/hw/s390x/s390-pci-bus.h +1
-0@@ -361,6 +361,7 @@ struct S390PCIBusDevice { bool unplug_requested; bool interp; bool forwarding_assist; + bool aif; QTAILQ_ENTRY(S390PCIBusDevice) link; };
Re: [PATCH 1/3] s390x/pci: avoid double enable/disable of aif
Quoted 10 lines
Quoted 9 lines
- return
kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); + if (!pbdev->aif) {
+ return-EINVAL;
+ }
+
+ rc =kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); + if (rc == 0) {
+pbev->aif= false;
Quoted 4 lines
s/pbev/pbdev/
You fix this in patch 2. :)
Oops, thanks for catching that.
Re: [PATCH 1/3] s390x/pci: avoid double enable/disable of aif
In reply to Matthew RosatoQuoted 3 lines · Matthew Rosato
Use a flag to keep track of whether AIF is currently enabled. This can be used to avoid enabling/disabling AIF multiple times as well as to determine whether or not it should be disabled during reset processing.
Why don't we disable AIF always at reset ? Doesn't KVM handle multiple calls to KVM_S390_ZPCIOP_DEREG_AEN cleanly ? Just asking, I am no expert there.
Thanks,
C.
Quoted 70 lines
Fixes d0bc7091c2 ("s390x/pci: enable adapter event notification for interpreted devices")Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-kvm.c | 25 +++++++++++++++++++++++-- include/hw/s390x/s390-pci-bus.h | 1 + 2 files changed, 24 insertions(+), 2 deletions(-)hw/s390x/s390-pci-kvm.c +23
-2@@ -27,6 +27,7 @@ bool s390_pci_kvm_interp_allowed(void)int s390_pci_kvm_aif_enable (S390PCIBusDevice *pbdev,ZpciFib *fib,bool assist) { +int rc;struct kvm_s390_zpci_op args = { .fh = pbdev->fh, .op =KVM_S390_ZPCIOP_REG_AEN ,@@ -38,15 +39,35 @@ int s390_pci_kvm_aif_enable(S390PCIBusDevice *pbdev, ZpciFib *fib, bool assist) .u.reg_aen.flags = (assist) ? 0 : KVM_S390_ZPCIOP_REGAEN_HOST }; -return kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args); +if (pbdev->aif) { +return -EINVAL ; + } + + rc =kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args); +if (rc ==0 ) { + pbdev->aif = true; + } + +return rc; }int s390_pci_kvm_aif_disable (S390PCIBusDevice *pbdev) { +int rc; +struct kvm_s390_zpci_op args = { .fh = pbdev->fh, .op =KVM_S390_ZPCIOP_DEREG_AEN }; -return kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args); +if (!pbdev->aif) { +return -EINVAL ; + } + + rc =kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args); +if (rc ==0 ) { + pbev->aif = false; + } + +return rc; }include/hw/s390x/s390-pci-bus.h +1
-0@@ -361,6 +361,7 @@ struct S390PCIBusDevice {bool unplug_requested;bool interp;bool forwarding_assist; +bool aif;QTAILQ_ENTRY (S390PCIBusDevice) link; };
Re: [PATCH 1/3] s390x/pci: avoid double enable/disable of aif
Quoted 8 lines · Cédric Le Goater
Quoted 3 lines · Matthew Rosato
Use a flag to keep track of whether AIF is currently enabled. This can be used to avoid enabling/disabling AIF multiple times as well as to determine whether or not it should be disabled during reset processing.
Why don't we disable AIF always at reset ? Doesn't KVM handle multiple calls to
KVM_S390_ZPCIOP_DEREG_AENcleanly ? Just asking, I am no expert there.
This may be some amount of defensive programming on my part :) Really, we're more concerned about enabling AIF twice without disabling AIF in between, and if we attempt to do so we should fail immediately rather than try messing with the hostdev.
The kernel warning you were seeing was exactly because we got the guest ISC users count wrong due to a mismatch between the number of enables and disables; in a sense, that warning is the KVM cleanup handling things (by disabling the gisc that AIF was using but also spitting out the warning that we got the gisc usage count wrong).
I suspect you're right in that there's room to improve the KVM code so that we can catch this earlier rather than continuing to increment/decrement the guest ISC count and waiting to catch it at the end of the VM lifecycle (the GISC count is per-VM, but we should only ever have 1 instance per-device but are allowing >1 per-device). I'll have a look at that separately from this series but IMHO QEMU should also try and 'behave' and be aware of what it previously enabled/disabled; if QEMU is trying to enable AIF multiple times then QEMU is doing something wrong.
PATCH 2/3 s390x/pci: refresh fh before disabling aif
In reply to Matthew RosatoTypically we refresh the host fh during CLP enable, however it's possible that the device goes through multiple reset events before the guest performs another CLP enable. Let's handle this for now by refreshing the host handle from vfio before disabling aif.
--- hw/s390x/s390-pci-kvm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
hw/s390x/s390-pci-kvm.c +10 -1
@@ -18,6 +18,7 @@#include "hw/s390x/s390-pci-bus.h" #include "hw/s390x/s390-pci-kvm.h" #include "hw/s390x/s390-pci-inst.h" +#include "hw/s390x/s390-pci-vfio.h" #include "cpu_models.h" bool s390_pci_kvm_interp_allowed (void )
@@ -64,9 +65,17 @@ int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev)return -EINVAL ; } +/* +* The device may have already been reset but we still want to relinquish +* the guest ISC, so always be sure to use an up-to-date host fh. +*/ +if (!s390_pci_get_host_fh (pbdev, &args.fh)) { +return -EPERM ; + } + rc =kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args);if (rc ==0 ) { - pbev->aif = false; + pbdev->aif = false; }return rc;
Signature · 1 line
2.43.0
Re: [PATCH 2/3] s390x/pci: refresh fh before disabling aif
Quoted 7 lines · Matthew Rosato
Typically we refresh the host fh during CLP enable, however it's possible
that the device goes through multiple reset events before the guest performs another CLP enable. Let's handle this for now by refreshing the
host handle from vfio before disabling aif.
Quoted 7 lines
Fixes 03451953c7 ("s390x/pci: reset ISM passthrough devices onshutdown and system reset")
Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-kvm.c | 11 ++++++++++-1 file changed, 10 insertions(+), 1 deletion(-)
Re: [PATCH 2/3] s390x/pci: refresh fh before disabling aif
In reply to Matthew RosatoHello Matthew,
Quoted 31 lines · Matthew Rosato
Typically we refresh the host fh during CLP enable, however it's possible that the device goes through multiple reset events before the guest performs another CLP enable. Let's handle this for now by refreshing the host handle from vfio before disabling aif.
Fixes 03451953c7 ("s390x/pci: reset ISM passthrough devices on shutdown and system reset")Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-kvm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)hw/s390x/s390-pci-kvm.c +4
-0@@ -18,6 +18,7 @@#include "hw/s390x/s390-pci-bus.h" #include "hw/s390x/s390-pci-kvm.h" #include "hw/s390x/s390-pci-inst.h" +#include "hw/s390x/s390-pci-vfio.h" #include "cpu_models.h" bool s390_pci_kvm_interp_allowed (void )@@ -64,9 +65,17 @@ int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev)return -EINVAL ; } +/* +* The device may have already been reset but we still want to relinquish +* the guest ISC, so always be sure to use an up-to-date host fh.
hw/s390x/s390-pci-kvm.c int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev) · line 79
Quoted 10 lines
hw/s390x/s390-pci-kvm.c +6
-1+*/ +if (!s390_pci_get_host_fh (pbdev, &args.fh)) { +return -EPERM ; + } + rc =kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args);if (rc ==0 ) { - pbev->aif = false; + pbdev->aif = false; }
This belongs to patch 1.
Thanks,
C.
Quoted 2 lines
hw/s390x/s390-pci-kvm.c +0
-0return rc;
Re: [PATCH 2/3] s390x/pci: refresh fh before disabling aif
Quoted 36 lines · Cédric Le Goater
Hello Matthew,
Quoted 33 lines · Matthew Rosato
Typically we refresh the host fh during CLP enable, however it's possible that the device goes through multiple reset events before the guest performs another CLP enable. Let's handle this for now by refreshing the host handle from vfio before disabling aif.
Fixes 03451953c7 ("s390x/pci: reset ISM passthrough devices on shutdown and system reset")Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-kvm.c| 11 ++++++++++-1 file changed, 10 insertions(+), 1 deletion(-)hw/s390x/s390-pci-kvm.c +1
-0@@ -18,6 +18,7 @@ #include "hw/s390x/s390-pci-bus.h" #include "hw/s390x/s390-pci-kvm.h" #include "hw/s390x/s390-pci-inst.h" +#include "hw/s390x/s390-pci-vfio.h" #include "cpu_models.h" bool s390_pci_kvm_interp_allowed(void)@@ -64,9 +65,17 @@ int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev) return -EINVAL; }+ /*
+ * The device may have already been reset but we still want to relinquish + * the guest ISC, so always be sure to use an up-to-date host fh. + */
+ if (!s390_pci_get_host_fh(pbdev, &args.fh)) {
+ return-EPERM;
+ }
Quoted 10 lines
Quoted 6 lines
+
rc =kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); if (rc == 0) {
-pbev->aif= false;
+pbdev->aif= false;
}This belongs to patch 1.
Thanks for pointing that out, will fix.
Re: [PATCH 2/3] s390x/pci: refresh fh before disabling aif
In reply to Matthew RosatoQuoted 25 lines · Matthew Rosato
Typically we refresh the host fh during CLP enable, however it's possible that the device goes through multiple reset events before the guest performs another CLP enable. Let's handle this for now by refreshing the host handle from vfio before disabling aif.
Fixes 03451953c7 ("s390x/pci: reset ISM passthrough devices on shutdown and system reset")Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-kvm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)hw/s390x/s390-pci-kvm.c +1
-0@@ -18,6 +18,7 @@#include "hw/s390x/s390-pci-bus.h" #include "hw/s390x/s390-pci-kvm.h" #include "hw/s390x/s390-pci-inst.h" +#include "hw/s390x/s390-pci-vfio.h" #include "cpu_models.h" bool s390_pci_kvm_interp_allowed (void )
hw/s390x/s390-pci-kvm.c int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev) · line 74
Quoted 10 lines
hw/s390x/s390-pci-kvm.c +7
-0@@ -64,9 +65,17 @@ int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev)return -EINVAL ; } +/* +* The device may have already been reset but we still want to relinquish +* the guest ISC, so always be sure to use an up-to-date host fh. +*/ +if (!s390_pci_get_host_fh (pbdev, &args.fh)) { +return -EPERM ; + }
The callers of s390_pci_kvm_aif_disable() all test the original host function with :
if (pbdev->interp && (pbdev->fh & FH_MASK_ENABLE))
This change possibly fetches a new one. Shouldn't we move the test also in s390_pci_kvm_aif_disable() ?
Thanks,
C.
Quoted 8 lines
hw/s390x/s390-pci-kvm.c +2
-1+ rc =kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args);if (rc ==0 ) { - pbev->aif = false; + pbdev->aif = false; }return rc;
Re: [PATCH 2/3] s390x/pci: refresh fh before disabling aif
Quoted 38 lines · Cédric Le Goater
Quoted 26 lines · Matthew Rosato
Typically we refresh the host fh during CLP enable, however it's possible that the device goes through multiple reset events before the guest performs another CLP enable. Let's handle this for now by refreshing the host handle from vfio before disabling aif.
Fixes 03451953c7 ("s390x/pci: reset ISM passthrough devices on shutdown and system reset")Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-kvm.c| 11 ++++++++++-1 file changed, 10 insertions(+), 1 deletion(-)hw/s390x/s390-pci-kvm.c +1
-0@@ -18,6 +18,7 @@ #include "hw/s390x/s390-pci-bus.h" #include "hw/s390x/s390-pci-kvm.h" #include "hw/s390x/s390-pci-inst.h" +#include "hw/s390x/s390-pci-vfio.h" #include "cpu_models.h" bool s390_pci_kvm_interp_allowed(void)@@ -64,9 +65,17 @@ int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev) return -EINVAL; }Quoted 7 lines
+ /*
+ * The device may have already been reset but we still want to relinquish + * the guest ISC, so always be sure to use an up-to-date host fh. + */
+ if (!s390_pci_get_host_fh(pbdev, &args.fh)) {
+ return-EPERM;
+ }The callers of
s390_pci_kvm_aif_disable()all test the original host function with :
Quoted 6 lines
if (
pbdev->interp&& (pbdev->fh&FH_MASK_ENABLE))This change possibly fetches a new one. Shouldn't we move the test also in
s390_pci_kvm_aif_disable()?
Those codepaths are actually testing the enablement bit of the guest-visible function handle, not the host function handle. Basically asking "was the guest using this device". These handles (host and guest) are sync'd during guest CLP enable (necessary for interpretation to work) but will become disjoint once we reset the device until the next guest CLP ENABLE. They can also become disjoint once the guest does a CLP DISABLE. What this change is doing is basically making sure we disable AIF on the hostdev, but does NOT replace pbdev->fh. The guest-visible function handle will get sync'd again when the guest does a new CLP ENABLE after reset.
PATCH 3/3 s390x/pci: drive ISM reset from subsystem reset
In reply to Matthew RosatoISM devices are sensitive to manipulation of the IOMMU, so the ISM device needs to be reset before the vfio-pci device is reset (triggering a full UNMAP). In order to ensure this occurs, trigger ISM device resets from subsystem_reset before triggering the PCI bus reset (which will also trigger vfio-pci reset). This only needs to be done for ISM devices which were enabled for use by the guest.
Further, ensure that AIF is disabled as part of the reset event.
--- hw/s390x/s390-pci-bus.c | 26 +++++++++++++++++--------- hw/s390x/s390-virtio-ccw.c | 2 ++ include/hw/s390x/s390-pci-bus.h | 1 + 3 files changed, 20 insertions(+), 9 deletions(-)
hw/s390x/s390-pci-bus.c +17 -9
@@ -151,20 +151,12 @@ static void s390_pci_shutdown_notifier(Notifier *n, void *opaque)pci_device_reset (pbdev->pdev); } -static void s390_pci_reset_cb (void *opaque) -{ -S390PCIBusDevice *pbdev = opaque; - -pci_device_reset (pbdev->pdev); -} -static void s390_pci_perform_unplug (S390PCIBusDevice *pbdev) {HotplugHandler *hotplug_ctrl;if (pbdev->pft ==ZPCI_PFT_ISM ) {notifier_remove (&pbdev->shutdown_notifier); -qemu_unregister_reset (s390_pci_reset_cb, pbdev); }/* Unplug the PCI device */
@@ -1132,7 +1124,6 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,if (pbdev->pft ==ZPCI_PFT_ISM ) { pbdev->shutdown_notifier.notify = s390_pci_shutdown_notifier;qemu_register_shutdown_notifier (&pbdev->shutdown_notifier); -qemu_register_reset (s390_pci_reset_cb, pbdev); } }else { pbdev->fh |=FH_SHM_EMUL ;
@@ -1279,6 +1270,23 @@ static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev,pci_default_write_config (pdev,PCI_SUBORDINATE_BUS , s->bus_no,1 ); } +void s390_pci_ism_reset (void ) +{ +S390pciState *s =s390_get_phb (); + +S390PCIBusDevice *pbdev, *next; + +/* Trigger reset event for each passthrough ISM device currently in-use */ + QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next) { +if (pbdev->interp && pbdev->pft ==ZPCI_PFT_ISM && + pbdev->fh &FH_MASK_ENABLE ) { +s390_pci_kvm_aif_disable (pbdev); + +pci_device_reset (pbdev->pdev); + } + } +} +static void s390_pcihost_reset (DeviceState *dev) {S390pciState *s =S390_PCI_HOST_BRIDGE (dev);
hw/s390x/s390-virtio-ccw.c +2 -0
@@ -118,6 +118,8 @@ static void subsystem_reset(void)DeviceState *dev;int i; +s390_pci_ism_reset (); +for (i =0 ; i <ARRAY_SIZE (reset_dev_types); i++) { dev =DEVICE (object_resolve_path_type ("" , reset_dev_types[i],NULL ));if (dev) {
include/hw/s390x/s390-pci-bus.h +1 -0
@@ -401,5 +401,6 @@ S390PCIBusDevice *s390_pci_find_dev_by_target(S390pciState *s, const char *target);S390PCIBusDevice *s390_pci_find_next_avail_dev (S390pciState *s,S390PCIBusDevice *pbdev); +void s390_pci_ism_reset (void );#endif
Signature · 1 line
2.43.0
Re: [PATCH 3/3] s390x/pci: drive ISM reset from subsystem reset
Quoted 73 lines · Matthew Rosato
ISM devices are sensitive to manipulation of the IOMMU, so the ISM device
needs to be reset before the vfio-pci device is reset (triggering a full
UNMAP). In order to ensure this occurs, trigger ISM device resets fromsubsystem_resetbefore triggering the PCI bus reset (which will also trigger vfio-pci reset). This only needs to be done for ISM devices which were enabled for use by the guest.
Further, ensure that AIF is disabled as part of the reset event.Fixes ef1535901a ("s390x: do a subsystem reset before the unprotecton reboot")
Fixes 03451953c7 ("s390x/pci: reset ISM passthrough devices onshutdown and system reset")
Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-bus.c | 26 +++++++++++++++++--------- hw/s390x/s390-virtio-ccw.c | 2 ++include/hw/s390x/s390-pci-bus.h| 1 +3 files changed, 20 insertions(+), 9 deletions(-)hw/s390x/s390-pci-bus.c +0
-0@@ -151,20 +151,12 @@ static void s390_pci_shutdown_notifier(Notifier *n, void *opaque) pci_device_reset(pbdev->pdev); }hw/s390x/s390-pci-bus.c +0
-7-static void s390_pci_reset_cb (void *opaque) -{ - S390PCIBusDevice *pbdev = opaque; - - pci_device_reset(pbdev->pdev); -} -static void
s390_pci_perform_unplug(S390PCIBusDevice *pbdev){
HotplugHandler*hotplug_ctrl;
if (pbdev->pft==ZPCI_PFT_ISM) {notifier_remove(&pbdev->shutdown_notifier); -qemu_unregister_reset(s390_pci_reset_cb, pbdev); }
/* Unplug the PCI device */hw/s390x/s390-pci-bus.c +0
-0@@ -1132,7 +1124,6 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, if (pbdev->pft == ZPCI_PFT_ISM) {
pbdev->shutdown_notifier.notify=s390_pci_shutdown_notifier; qemu_register_shutdown_notifier(&pbdev-Quoted 1 line
shutdown_notifier);-
qemu_register_reset(s390_pci_reset_cb, pbdev); }
} else {
pbdev->fh|=FH_SHM_EMUL;hw/s390x/s390-pci-bus.c +0
-0@@ -1279,6 +1270,23 @@ static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev, pci_default_write_config(pdev, PCI_SUBORDINATE_BUS, s->bus_no,1 ); }hw/s390x/s390-pci-bus.c +7
-0+void s390_pci_ism_reset (void ) +{ + S390pciState *s = s390_get_phb(); + + S390PCIBusDevice *pbdev, *next; + + /* Trigger reset event for each passthrough ISM device currently
Quoted 7 lines
in-use */
+QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next){ + if (pbdev->interp&&pbdev->pft==ZPCI_PFT_ISM&& +pbdev->fh&FH_MASK_ENABLE) {
+s390_pci_kvm_aif_disable(pbdev);
+
+pci_device_reset(pbdev->pdev);
Do we care about the loss of a reset for ISM devices in a !interpretation case? (I seem to think such a configuration is not possible today, and so we don't care, but could use a reminder.)
Quoted 35 lines
+ }
+ }hw/s390x/s390-pci-bus.c +2
-0+} +static void
s390_pcihost_reset(DeviceState *dev)
{
S390pciState *s =S390_PCI_HOST_BRIDGE(dev);hw/s390x/s390-virtio-ccw.c +0
-0@@ -118,6 +118,8 @@ static void subsystem_reset(void) DeviceState *dev; int i;
+s390_pci_ism_reset();
+
for (i = 0; i <ARRAY_SIZE(reset_dev_types); i++) { dev =DEVICE(object_resolve_path_type("", reset_dev_types[i], NULL));
if (dev) {include/hw/s390x/s390- renamed from include/hw/s390x/s390-pci-bus.h +0
-0
pci-bus.h
index435e788867..2c43ea123f100644include/hw/s390x/s390-pci-bus.h +0
-0@@ -401,5 +401,6 @@ S390PCIBusDevice *s390_pci_find_dev_by_target (S390pciState *s, const char *target); S390PCIBusDevice *s390_pci_find_next_avail_dev(S390pciState *s,S390PCIBusDevice *pbdev);
+voids390_pci_ism_reset(void);
#endif
Re: [PATCH 3/3] s390x/pci: drive ISM reset from subsystem reset
Quoted 13 lines · Eric Farman
Quoted 11 lines · Matthew Rosato
hw/s390x/s390-pci-bus.c +7
-0+void s390_pci_ism_reset (void ) +{ + S390pciState *s = s390_get_phb(); + + S390PCIBusDevice *pbdev, *next; + + /* Trigger reset event for each passthrough ISM device currentlyin-use */
+QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next){ + if (pbdev->interp&&pbdev->pft==ZPCI_PFT_ISM&&
Quoted 9 lines
Quoted 4 lines
+
pbdev->fh&FH_MASK_ENABLE) {
+s390_pci_kvm_aif_disable(pbdev);
+
+pci_device_reset(pbdev->pdev);Do we care about the loss of a reset for ISM devices in a !interpretation case? (I seem to think such a configuration is not possible today, and so we don't care, but could use a reminder.)
ISM passthrough is currently only allowed when interpretation is enabled. So the check is redundant today but allows us to re-evaluate the need if we ever add support for ISM without interpretation.
Re: [PATCH 3/3] s390x/pci: drive ISM reset from subsystem reset
In reply to Matthew RosatoAdding Alex,
Quoted 61 lines · Matthew Rosato
ISM devices are sensitive to manipulation of the IOMMU, so the ISM device needs to be reset before the vfio-pci device is reset (triggering a full UNMAP). In order to ensure this occurs, trigger ISM device resets from
subsystem_resetbefore triggering the PCI bus reset (which will also trigger vfio-pci reset). This only needs to be done for ISM devices which were enabled for use by the guest.
Further, ensure that AIF is disabled as part of the reset event.Fixes ef1535901a ("s390x: do a subsystem reset before the unprotect on reboot")Fixes 03451953c7 ("s390x/pci: reset ISM passthrough devices on shutdown and system reset")Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 26 +++++++++++++++++--------- hw/s390x/s390-virtio-ccw.c | 2 ++ include/hw/s390x/s390-pci-bus.h | 1 + 3 files changed, 20 insertions(+), 9 deletions(-)hw/s390x/s390-pci-bus.c +6
-9@@ -151,20 +151,12 @@ static void s390_pci_shutdown_notifier(Notifier *n, void *opaque)pci_device_reset (pbdev->pdev); } -static void s390_pci_reset_cb (void *opaque) -{ -S390PCIBusDevice *pbdev = opaque; - -pci_device_reset (pbdev->pdev); -} -static void s390_pci_perform_unplug (S390PCIBusDevice *pbdev) {HotplugHandler *hotplug_ctrl;if (pbdev->pft ==ZPCI_PFT_ISM ) {notifier_remove (&pbdev->shutdown_notifier); -qemu_unregister_reset (s390_pci_reset_cb, pbdev); }/* Unplug the PCI device */ @@ -1132,7 +1124,6 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,if (pbdev->pft ==ZPCI_PFT_ISM ) { pbdev->shutdown_notifier.notify = s390_pci_shutdown_notifier;qemu_register_shutdown_notifier (&pbdev->shutdown_notifier); -qemu_register_reset (s390_pci_reset_cb, pbdev); } }else { pbdev->fh |=FH_SHM_EMUL ;@@ -1279,6 +1270,23 @@ static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev,pci_default_write_config (pdev,PCI_SUBORDINATE_BUS , s->bus_no,1 ); } +void s390_pci_ism_reset (void ) +{ +S390pciState *s =s390_get_phb (); + +S390PCIBusDevice *pbdev, *next; +
hw/s390x/s390-pci-bus.c static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev, · line 1288
Quoted 10 lines
hw/s390x/s390-pci-bus.c +10
-0+/* Trigger reset event for each passthrough ISM device currently in-use */ + QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next) { +if (pbdev->interp && pbdev->pft ==ZPCI_PFT_ISM && + pbdev->fh &FH_MASK_ENABLE ) { +s390_pci_kvm_aif_disable (pbdev); + +pci_device_reset (pbdev->pdev); + } + } +}
Could we instead define a VFIOPCIDevice::resetfn handler for these ISM devices (1014:04ed) ? This would be cleaner if possible.
If so, as a prerequisite, we would need to introduce in a little VFIO helper to define custom reset handlers.
Thanks,
C.
Quoted 28 lines
hw/s390x/s390-pci-bus.c +1
-0+static void s390_pcihost_reset (DeviceState *dev) {S390pciState *s =S390_PCI_HOST_BRIDGE (dev);hw/s390x/s390-virtio-ccw.c +2
-0@@ -118,6 +118,8 @@ static void subsystem_reset(void)DeviceState *dev;int i; +s390_pci_ism_reset (); +for (i =0 ; i <ARRAY_SIZE (reset_dev_types); i++) { dev =DEVICE (object_resolve_path_type ("" , reset_dev_types[i],NULL ));if (dev) {include/hw/s390x/s390-pci-bus.h +1
-0@@ -401,5 +401,6 @@ S390PCIBusDevice *s390_pci_find_dev_by_target(S390pciState *s, const char *target);S390PCIBusDevice *s390_pci_find_next_avail_dev (S390pciState *s,S390PCIBusDevice *pbdev); +void s390_pci_ism_reset (void );#endif
Re: [PATCH 3/3] s390x/pci: drive ISM reset from subsystem reset
Quoted 75 lines · Cédric Le Goater
Adding Alex,
Quoted 57 lines · Matthew Rosato
ISM devices are sensitive to manipulation of the IOMMU, so the ISM device needs to be reset before the vfio-pci device is reset (triggering a full UNMAP). In order to ensure this occurs, trigger ISM device resets from
subsystem_resetbefore triggering the PCI bus reset (which will also trigger vfio-pci reset). This only needs to be done for ISM devices which were enabled for use by the guest.
Further, ensure that AIF is disabled as part of the reset event.Fixes ef1535901a ("s390x: do a subsystem reset before the unprotect on reboot")Fixes 03451953c7 ("s390x/pci: reset ISM passthrough devices on shutdown and system reset")Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-bus.c | 26 +++++++++++++++++--------- hw/s390x/s390-virtio-ccw.c | 2 ++include/hw/s390x/s390-pci-bus.h| 1 +3 files changed, 20 insertions(+), 9 deletions(-)hw/s390x/s390-pci-bus.c +5
-8@@ -151,20 +151,12 @@ static void s390_pci_shutdown_notifier(Notifier *n, void *opaque) pci_device_reset(pbdev->pdev); } -static void s390_pci_reset_cb(void *opaque) -{ - S390PCIBusDevice *pbdev = opaque; - - pci_device_reset(pbdev->pdev); -} - static void s390_pci_perform_unplug(S390PCIBusDevice *pbdev) { HotplugHandler *hotplug_ctrl; if (pbdev->pft == ZPCI_PFT_ISM) { notifier_remove(&pbdev->shutdown_notifier); - qemu_unregister_reset(s390_pci_reset_cb, pbdev); } /* Unplug the PCI device */@@ -1132,7 +1124,6 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, if (pbdev->pft == ZPCI_PFT_ISM) { pbdev->shutdown_notifier.notify = s390_pci_shutdown_notifier; qemu_register_shutdown_notifier(&pbdev->shutdown_notifier); - qemu_register_reset(s390_pci_reset_cb, pbdev); } } else { pbdev->fh |= FH_SHM_EMUL;@@ -1279,6 +1270,23 @@ static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev, pci_default_write_config(pdev, PCI_SUBORDINATE_BUS, s->bus_no, 1); } +void s390_pci_ism_reset(void) +{ + S390pciState *s = s390_get_phb(); + + S390PCIBusDevice *pbdev, *next; +hw/s390x/s390-pci-bus.c static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev, · line 1287
Quoted 10 lines
hw/s390x/s390-pci-bus.c +10
-0+ /* Trigger reset event for each passthrough ISM device currently in-use */ + QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next) { + if (pbdev->interp && pbdev->pft == ZPCI_PFT_ISM && + pbdev->fh & FH_MASK_ENABLE) { + s390_pci_kvm_aif_disable(pbdev); + + pci_device_reset(pbdev->pdev); + } + } +}Could we instead define a
VFIOPCIDevice::resetfnhandler for these ISM devices (1014:04ed) ? This would be cleaner if possible.
Quoted 7 lines
If so, as a prerequisite, we would need to introduce in a little VFIO helper to define custom reset handlers.
Thanks,
C.
Oh interesting, I had not noticed that. This may well work -- resetfn is currently setup via vfio_setup_resetfn_quirk but it would probably be easier to have a helper that takes the vdev and a function pointer so that we can provide a platform-specific reset handler (rather than having hw/vfio/pci-quirks.c worry about CONFIG_S390 etc). I'll have to play around with this.
Re: [PATCH 3/3] s390x/pci: drive ISM reset from subsystem reset
Quoted 79 lines · Matthew Rosato
Quoted 78 lines · Cédric Le Goater
Adding Alex,
Quoted 57 lines · Matthew Rosato
ISM devices are sensitive to manipulation of the IOMMU, so the ISM device needs to be reset before the vfio-pci device is reset (triggering a full UNMAP). In order to ensure this occurs, trigger ISM device resets from
subsystem_resetbefore triggering the PCI bus reset (which will also trigger vfio-pci reset). This only needs to be done for ISM devices which were enabled for use by the guest.
Further, ensure that AIF is disabled as part of the reset event.Fixes ef1535901a ("s390x: do a subsystem reset before the unprotect on reboot")Fixes 03451953c7 ("s390x/pci: reset ISM passthrough devices on shutdown and system reset")Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-bus.c | 26 +++++++++++++++++--------- hw/s390x/s390-virtio-ccw.c | 2 ++include/hw/s390x/s390-pci-bus.h| 1 +3 files changed, 20 insertions(+), 9 deletions(-)hw/s390x/s390-pci-bus.c +5
-8@@ -151,20 +151,12 @@ static void s390_pci_shutdown_notifier(Notifier *n, void *opaque) pci_device_reset(pbdev->pdev); } -static void s390_pci_reset_cb(void *opaque) -{ - S390PCIBusDevice *pbdev = opaque; - - pci_device_reset(pbdev->pdev); -} - static void s390_pci_perform_unplug(S390PCIBusDevice *pbdev) { HotplugHandler *hotplug_ctrl; if (pbdev->pft == ZPCI_PFT_ISM) { notifier_remove(&pbdev->shutdown_notifier); - qemu_unregister_reset(s390_pci_reset_cb, pbdev); } /* Unplug the PCI device */@@ -1132,7 +1124,6 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, if (pbdev->pft == ZPCI_PFT_ISM) { pbdev->shutdown_notifier.notify = s390_pci_shutdown_notifier; qemu_register_shutdown_notifier(&pbdev->shutdown_notifier); - qemu_register_reset(s390_pci_reset_cb, pbdev); } } else { pbdev->fh |= FH_SHM_EMUL;@@ -1279,6 +1270,23 @@ static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev, pci_default_write_config(pdev, PCI_SUBORDINATE_BUS, s->bus_no, 1); } +void s390_pci_ism_reset(void) +{ + S390pciState *s = s390_get_phb(); + + S390PCIBusDevice *pbdev, *next; +hw/s390x/s390-pci-bus.c static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev, · line 1287
Quoted 10 lines
hw/s390x/s390-pci-bus.c +10
-0+ /* Trigger reset event for each passthrough ISM device currently in-use */ + QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next) { + if (pbdev->interp && pbdev->pft == ZPCI_PFT_ISM && + pbdev->fh & FH_MASK_ENABLE) { + s390_pci_kvm_aif_disable(pbdev); + + pci_device_reset(pbdev->pdev); + } + } +}Could we instead define a
VFIOPCIDevice::resetfnhandler for these ISM devices (1014:04ed) ? This would be cleaner if possible.If so, as a prerequisite, we would need to introduce in a little VFIO helper to define custom reset handlers.
Quoted 8 lines
Quoted 4 lines
Thanks,
C.
Oh interesting, I had not noticed that. This may well work -- resetfn is currently setup via
vfio_setup_resetfn_quirkbut it would probably be easier to have a helper that takes the vdev and a function pointer so that we can provide a platform-specific reset handler (rather than havinghw/vfio/pci-quirks.cworry aboutCONFIG_S390etc). I'll have to play around with this.
Hmm, it was a good idea but I don't think this will work. I tried to hack something together today but I'm definitely seeing paths where the vfio_listener_region_del happens before the call to vfio_pci_reset (which would ultimately trigger the new custom resetfn).
Perhaps we should stick with the call from subsystem_reset -- it will ensure that the ISM cleanup happens after guest CPUs are stopped but before vfio does its cleanup.
Re: [PATCH 3/3] s390x/pci: drive ISM reset from subsystem reset
Quoted 82 lines · Matthew Rosato
Quoted 81 lines · Matthew Rosato
Quoted 80 lines · Cédric Le Goater
Adding Alex,
Quoted 57 lines · Matthew Rosato
ISM devices are sensitive to manipulation of the IOMMU, so the ISM device needs to be reset before the vfio-pci device is reset (triggering a full UNMAP). In order to ensure this occurs, trigger ISM device resets from
subsystem_resetbefore triggering the PCI bus reset (which will also trigger vfio-pci reset). This only needs to be done for ISM devices which were enabled for use by the guest.
Further, ensure that AIF is disabled as part of the reset event.Fixes ef1535901a ("s390x: do a subsystem reset before the unprotect on reboot")Fixes 03451953c7 ("s390x/pci: reset ISM passthrough devices on shutdown and system reset")Reported-by Cédric Le Goater <[email protected]>Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 26 +++++++++++++++++--------- hw/s390x/s390-virtio-ccw.c | 2 ++ include/hw/s390x/s390-pci-bus.h | 1 + 3 files changed, 20 insertions(+), 9 deletions(-)hw/s390x/s390-pci-bus.c +5
-8@@ -151,20 +151,12 @@ static void s390_pci_shutdown_notifier(Notifier *n, void *opaque) pci_device_reset(pbdev->pdev); } -static void s390_pci_reset_cb(void *opaque) -{ - S390PCIBusDevice *pbdev = opaque; - - pci_device_reset(pbdev->pdev); -} - static void s390_pci_perform_unplug(S390PCIBusDevice *pbdev) { HotplugHandler *hotplug_ctrl; if (pbdev->pft == ZPCI_PFT_ISM) { notifier_remove(&pbdev->shutdown_notifier); - qemu_unregister_reset(s390_pci_reset_cb, pbdev); } /* Unplug the PCI device */@@ -1132,7 +1124,6 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, if (pbdev->pft == ZPCI_PFT_ISM) { pbdev->shutdown_notifier.notify = s390_pci_shutdown_notifier; qemu_register_shutdown_notifier(&pbdev->shutdown_notifier); - qemu_register_reset(s390_pci_reset_cb, pbdev); } } else { pbdev->fh |= FH_SHM_EMUL;@@ -1279,6 +1270,23 @@ static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev, pci_default_write_config(pdev, PCI_SUBORDINATE_BUS, s->bus_no, 1); } +void s390_pci_ism_reset(void) +{ + S390pciState *s = s390_get_phb(); + + S390PCIBusDevice *pbdev, *next; +hw/s390x/s390-pci-bus.c static void s390_pci_enumerate_bridge(PCIBus *bus, PCIDevice *pdev, · line 1287
Quoted 10 lines
hw/s390x/s390-pci-bus.c +10
-0+ /* Trigger reset event for each passthrough ISM device currently in-use */ + QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next) { + if (pbdev->interp && pbdev->pft == ZPCI_PFT_ISM && + pbdev->fh & FH_MASK_ENABLE) { + s390_pci_kvm_aif_disable(pbdev); + + pci_device_reset(pbdev->pdev); + } + } +}Could we instead define a
VFIOPCIDevice::resetfnhandler for these ISM devices (1014:04ed) ? This would be cleaner if possible.If so, as a prerequisite, we would need to introduce in a little VFIO helper to define custom reset handlers.
Thanks,
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
C.
Oh interesting, I had not noticed that. This may well work -- resetfn is currently setup via
vfio_setup_resetfn_quirkbut it would probably be easier to have a helper that takes the vdev and a function pointer so that we can provide a platform-specific reset handler (rather than havinghw/vfio/pci-quirks.cworry aboutCONFIG_S390etc). I'll have to play around with this.Hmm, it was a good idea but I don't think this will work. I tried to hack something together today but I'm definitely seeing paths where the
vfio_listener_region_delhappens before the call tovfio_pci_reset(which would ultimately trigger the new custom resetfn).
OK.
Quoted 1 line
Perhaps we should stick with the call from
subsystem_reset-- it will ensure that the ISM cleanup happens after guest CPUs are stopped but before vfio does its cleanup.
Let's keep the subsystem_reset() method then. Please add a comment on the reset ordering.
Thanks,
C.
Quoted 1 line
17.01.2024 01:31, Matthew Rosato:
Quoted 17 lines
Commit
ef1535901a0(re-)introduced an issue where passthrough ISM devices on s390x would enter an error state after reboot. This was previously fixed by03451953c79e, using device reset callbacks, however the change inef1535901a0effectively triggers a cold reset of the pci bus before the device reset callbacks are triggered.To resolve this, this series proposes to remove the use of the reset callback for ISM cleanup and instead trigger ISM reset from
subsystem_resetbefore triggering bus resets. This has to happen before the bus resets because the reset of s390-pcihost will trigger reset of the PCI bus followed by the s390-pci bus, and the former will trigger vfio-pci reset / the aperture-wide unmap that ISM gets upset about./s390-pcihost (s390-pcihost) /pci.0 (PCI) /s390-pcibus.0 (s390-pcibus)
Quoted 6 lines
While fixing this, it was also noted that kernel warnings could be seen that indicate a guest ISC reference count error. That's because in some reset cases we were not bothering to disable AIF, but would again re-enable it after the reset (causing the reference count to grow erroneously). This was a base issue that went unnoticed because the kernel previously did not detect and issue a warning for this scenario.
Is it a -stable material, or not worth picking up for stable?
Thanks,
/mjt
Quoted 23 lines · Michael Tokarev
17.01.2024 01:31, Matthew Rosato:
Quoted 6 lines
Commit
ef1535901a0(re-)introduced an issue where passthrough ISM devices on s390x would enter an error state after reboot. This was previously fixed by03451953c79e, using device reset callbacks, however the change inef1535901a0effectively triggers a cold reset of the pci bus before the device reset callbacks are triggered.Quoted 15 lines
To resolve this, this series proposes to remove the use of the reset callback for ISM cleanup and instead trigger ISM reset from
subsystem_resetbefore triggering bus resets. This has to happen before the bus resets because the reset of s390-pcihost will trigger reset of the PCI bus followed by the s390-pci bus, and the former will trigger vfio-pci reset / the aperture-wide unmap that ISM gets upset about.
/s390-pcihost (s390-pcihost)
/pci.0 (PCI)
/s390-pcibus.0 (s390-pcibus)
While fixing this, it was also noted that kernel warnings could be seen that indicate a guest ISC reference count error. That's because in some reset cases we were not bothering to disable AIF, but would again re-enable it after the reset (causing the reference count to grow erroneously). This was a base issue that went unnoticed because the kernel previously did not detect and issue a warning for this scenario.
Quoted 1 line
Is it a -stable material, or not worth picking up for stable?
It's definitely stable material, but IIUC there will be a v2 with some minor fixes.
Thomas
18.01.2024 10:19, Thomas Huth:
Quoted 3 lines
Quoted 1 line
Is it a -stable material, or not worth picking up for stable?
It's definitely stable material, but IIUC there will be a v2 with some minor fixes.
Yeah, I figured there will be v2.
Just to remind, - please add Cc: qemu-stable@ when appropriate (or mark it any other way, or just forward it qemu-stable@, whatever, - just so it wont get lost). There's no need to do that for this patchset, as I already noticed this one :)
Thank you for the comments!
/mjt