Skip to contentSkip to conversation

pcie: Add hotplug detect state register to cmask

PATCH In review Leonardo Bras 4 participants 15 messages Jul 6, 2023 – Jul 10, 2023

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 pcie: Add hotplug detect state register to cmask

When trying to migrate a machine type pc-q35-6.0 or lower, with this cmdline options,

-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \
-device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1

the following bug happens after all ram pages were sent:

qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argument

This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }

In this scenario, hotplug_handler_plug() calls pcie_cap_slot_plug_cb(), which sets dev->config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI hotplug for the guest. After a while the guest will deal with this hotplug and qemu will clear the above bit.

Then, during migration, get_pci_config_device() will compare the configs of both the freshly created device and the one that is being received via migration, which will differ due to the PCI_EXP_SLTSTA_PDS bit and cause the bug to reproduce.

To avoid this fake incompatibility, there are tree fields in PCIDevice that can help:

- wmask: Used to implement R/W bytes, and
- w1cmask: Used to implement RW1C(Write 1 to Clear) bytes
- cmask: Used to enable config checks on load.

According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.

So, clear PCI_EXP_SLTSTA_PDS bit on cmask, so the fake incompatibility on get_pci_config_device() does not abort the migration.

---
 hw/pci/pcie.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/pci/pcie.c +4 -0
index b8c24cf45f..cae56bf1c8 100644
@@ -659,6 +659,10 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)
     pci_word_test_and_set_mask(dev->w1cmask + pos + PCI_EXP_SLTSTA,
                                PCI_EXP_HP_EV_SUPPORTED);
 
+    /* Avoid migration abortion when this device hot-removed by guest */
+    pci_word_test_and_clear_mask(dev->cmask + pos + PCI_EXP_SLTSTA,
+                                 PCI_EXP_SLTSTA_PDS);
+
     dev->exp.hpev_notified = false;
 
     qbus_set_hotplug_handler(BUS(pci_bridge_get_sec_bus(PCI_BRIDGE(dev))),
Signature · 1 line
2.41.0
Reviewed-by on v2 1/1
#1 lore raw
Quoted 57 lines · Leonardo Bras

When trying to migrate a machine type pc-q35-6.0 or lower, with this cmdline options,

(quoted diff) +0 -2
-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \
-device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1

the following bug happens after all ram pages were sent:

qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argument

This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }

In this scenario, hotplug_handler_plug() calls pcie_cap_slot_plug_cb(), which sets dev->config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI hotplug for the guest. After a while the guest will deal with this hotplug and qemu will clear the above bit.

Then, during migration, get_pci_config_device() will compare the configs of both the freshly created device and the one that is being received via migration, which will differ due to the PCI_EXP_SLTSTA_PDS bit and cause the bug to reproduce.

To avoid this fake incompatibility, there are tree fields in PCIDevice that can help:

- wmask: Used to implement R/W bytes, and
- w1cmask: Used to implement RW1C(Write 1 to Clear) bytes
- cmask: Used to enable config checks on load.

According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.

So, clear PCI_EXP_SLTSTA_PDS bit on cmask, so the fake incompatibility on get_pci_config_device() does not abort the migration.

---
 hw/pci/pcie.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/pci/pcie.c +0 -0
index b8c24cf45f..cae56bf1c8 100644

hw/pci/pcie.c void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s) · line 663

Quoted 5 lines
hw/pci/pcie.c +1 -0
@@ -659,6 +659,10 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)
     pci_word_test_and_set_mask(dev->w1cmask + pos + PCI_EXP_SLTSTA,
                                PCI_EXP_HP_EV_SUPPORTED);
 
+    /* Avoid migration abortion when this device hot-removed by guest
 */

I would have included here the text in the commit:

According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.

and

This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }

so if we ever remove the machine type pc-q35-6.0, we can drop it.

Yes, I know that we don't drop machine types, but we should at some point.

Quoted 6 lines
hw/pci/pcie.c +3 -0
+    pci_word_test_and_clear_mask(dev->cmask + pos + PCI_EXP_SLTSTA,
+                                 PCI_EXP_SLTSTA_PDS);
+
     dev->exp.hpev_notified = false;
 

qbus_set_hotplug_handler(BUS(pci_bridge_get_sec_bus(PCI_BRIDGE(dev))),

I agree that this is (at least) a step on the right direction.

I wmould had expected to have to need some check related to the value of:

{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }

But I will not claim _any_ understanding of the PCI specification.

So:

about that it fixes the migration bug.

#2 lore raw
Quoted 72 lines · Juan Quintela
Quoted 57 lines · Leonardo Bras

When trying to migrate a machine type pc-q35-6.0 or lower, with this cmdline options,

(quoted diff) +0 -2
-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \
-device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1

the following bug happens after all ram pages were sent:

qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argument

This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }

In this scenario, hotplug_handler_plug() calls pcie_cap_slot_plug_cb(), which sets dev->config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI hotplug for the guest. After a while the guest will deal with this hotplug and qemu will clear the above bit.

Then, during migration, get_pci_config_device() will compare the configs of both the freshly created device and the one that is being received via migration, which will differ due to the PCI_EXP_SLTSTA_PDS bit and cause the bug to reproduce.

To avoid this fake incompatibility, there are tree fields in PCIDevice that can help:

- wmask: Used to implement R/W bytes, and
- w1cmask: Used to implement RW1C(Write 1 to Clear) bytes
- cmask: Used to enable config checks on load.

According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.

So, clear PCI_EXP_SLTSTA_PDS bit on cmask, so the fake incompatibility on get_pci_config_device() does not abort the migration.

---
 hw/pci/pcie.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/pci/pcie.c +0 -0
index b8c24cf45f..cae56bf1c8 100644

hw/pci/pcie.c void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s) · line 663

Quoted 5 lines
hw/pci/pcie.c +1 -0
@@ -659,6 +659,10 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)
     pci_word_test_and_set_mask(dev->w1cmask + pos + PCI_EXP_SLTSTA,
                                PCI_EXP_HP_EV_SUPPORTED);
 
+    /* Avoid migration abortion when this device hot-removed by guest
 */

I would have included here the text in the commit:

According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.

Quoted 6 lines

and

This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }

so if we ever remove the machine type pc-q35-6.0, we can drop it.

It makes sense adding this to the code.

In the remove machine-type case, IIUC we would have to drop every machine-type older than pc-q35-6.0.
Also, we would not support migration to any machine without ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, right?

Quoted 9 lines

Yes, I know that we don't drop machine types, but we should at some point.

Quoted 5 lines
hw/pci/pcie.c +3 -0
+    pci_word_test_and_clear_mask(dev->cmask + pos + PCI_EXP_SLTSTA,
+                                 PCI_EXP_SLTSTA_PDS);
+
     dev->exp.hpev_notified = false;
 
Quoted 9 lines
Quoted 1 line

qbus_set_hotplug_handler(BUS(pci_bridge_get_sec_bus(PCI_BRIDGE(dev))),

I agree that this is (at least) a step on the right direction.

I wmould had expected to have to need some check related to the value of:

{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }

This bug affects versions older than qemu 6.0. If we add this, we would have some extra work backporting this to older versions (if necessary) because the 'property' did not exist back then.

Quoted 8 lines

But I will not claim _any_ understanding of the PCI specification.

So:

about that it fixes the migration bug.

Thanks Juan!
Leo

#3 lore raw
In reply to Leonardo Bras
Quoted 14 lines · Leonardo Bras

When trying to migrate a machine type pc-q35-6.0 or lower, with this cmdline options,

(quoted diff) +0 -2
-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \
-device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1

the following bug happens after all ram pages were sent:

qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argument

Quoted 7 lines

This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }

In this scenario, hotplug_handler_plug() calls pcie_cap_slot_plug_cb(), which sets dev->config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI hotplug for the guest. After a while the guest will deal with this hotplug and qemu will clear the above bit.

Do you mean that the bit will be cleared after this point for the whole lifecycle of the VM, as long as the pcie topology doesn't change again?

"This bit indicates the presence of an adapter in the slot"

IIUC the adapter in the slot is there, why it's cleared rather than set?

Quoted 13 lines

Then, during migration, get_pci_config_device() will compare the configs of both the freshly created device and the one that is being received via migration, which will differ due to the PCI_EXP_SLTSTA_PDS bit and cause the bug to reproduce.

To avoid this fake incompatibility, there are tree fields in PCIDevice that can help:

- wmask: Used to implement R/W bytes, and
- w1cmask: Used to implement RW1C(Write 1 to Clear) bytes
- cmask: Used to enable config checks on load.

Quoted 7 lines

According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.

So, clear PCI_EXP_SLTSTA_PDS bit on cmask, so the fake incompatibility on get_pci_config_device() does not abort the migration.

Yes, using cmask makes more sense to me, but we'd need some pci developer to ack it at last I guess, anyway.

Quoted 3 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

Thanks,

Quoted 22 lines
---
 hw/pci/pcie.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/pci/pcie.c +4 -0
index b8c24cf45f..cae56bf1c8 100644
@@ -659,6 +659,10 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)
     pci_word_test_and_set_mask(dev->w1cmask + pos + PCI_EXP_SLTSTA,
                                PCI_EXP_HP_EV_SUPPORTED);
 
+    /* Avoid migration abortion when this device hot-removed by guest */
+    pci_word_test_and_clear_mask(dev->cmask + pos + PCI_EXP_SLTSTA,
+                                 PCI_EXP_SLTSTA_PDS);
+
     dev->exp.hpev_notified = false;
 
     qbus_set_hotplug_handler(BUS(pci_bridge_get_sec_bus(PCI_BRIDGE(dev))),
Signature · 1 line
2.41.0
Signature · 1 line
Peter Xu
#4 lore raw
Quoted 27 lines · Peter Xu
Quoted 14 lines · Leonardo Bras

When trying to migrate a machine type pc-q35-6.0 or lower, with this cmdline options,

(quoted diff) +0 -2
-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \
-device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1

the following bug happens after all ram pages were sent:

qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argument

Quoted 7 lines

This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }

In this scenario, hotplug_handler_plug() calls pcie_cap_slot_plug_cb(), which sets dev->config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI hotplug for the guest. After a while the guest will deal with this hotplug and qemu will clear the above bit.

Do you mean that the bit will be cleared after this point for the whole lifecycle of the VM, as long as the pcie topology doesn't change again?

Quoted 3 lines

"This bit indicates the presence of an adapter in the slot"

IIUC the adapter in the slot is there, why it's cleared rather than set?

Fort some reason the guest is powering down the device, and we have in qemu:

/*
* If the slot is populated, power indicator is off and power
* controller is off, it is safe to detach the devices. *
* Note: don't detach if condition was already true:
* this is a work around for guests that overwrite
* control of powered off slots before powering them on. */
if ((sltsta & PCI_EXP_SLTSTA_PDS) && pcie_sltctl_powered_off(val) &&
!pcie_sltctl_powered_off(old_slt_ctl))
{
pcie_cap_slot_do_unplug(dev); // clears PCI_EXP_SLTSTA_PDS
}

Quoted 19 lines
Quoted 19 lines

Then, during migration, get_pci_config_device() will compare the configs of both the freshly created device and the one that is being received via migration, which will differ due to the PCI_EXP_SLTSTA_PDS bit and cause the bug to reproduce.

To avoid this fake incompatibility, there are tree fields in PCIDevice that can help:

- wmask: Used to implement R/W bytes, and
- w1cmask: Used to implement RW1C(Write 1 to Clear) bytes
- cmask: Used to enable config checks on load.

According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.

Quoted 5 lines
Quoted 2 lines

So, clear PCI_EXP_SLTSTA_PDS bit on cmask, so the fake incompatibility on get_pci_config_device() does not abort the migration.

Yes, using cmask makes more sense to me, but we'd need some pci developer to ack it at last I guess, anyway.

Agree! I am waiting for Michael's opinion on this.

Quoted 9 lines
Quoted 4 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.

Quoted 2 lines

Thanks,

Thank you!
Leo

Quoted 27 lines
Quoted 21 lines
---
 hw/pci/pcie.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/pci/pcie.c +4 -1
index b8c24cf45f..cae56bf1c8 100644
@@ -659,6 +659,10 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)
     pci_word_test_and_set_mask(dev->w1cmask + pos + PCI_EXP_SLTSTA,
                                PCI_EXP_HP_EV_SUPPORTED);
 
+    /* Avoid migration abortion when this device hot-removed by guest */
+    pci_word_test_and_clear_mask(dev->cmask + pos + PCI_EXP_SLTSTA,
+                                 PCI_EXP_SLTSTA_PDS);
+
     dev->exp.hpev_notified = false;
 
     qbus_set_hotplug_handler(BUS(pci_bridge_get_sec_bus(PCI_BRIDGE(dev))),
--
Quoted 2 lines

2.41.0

--
Peter Xu

#5 lore raw
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 4 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.

I see, thanks!

But then do you know why it's never hit before? I assume it means this bug has been there for a long time.

Signature · 1 line
Peter Xu
#6 lore raw
Quoted 7 lines · Peter Xu
Quoted 5 lines · Leonardo Bras Soares Passos
Quoted 4 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

Quoted 7 lines
Quoted 2 lines

I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.

I see, thanks!

But then do you know why it's never hit before? I assume it means this bug has been there for a long time.

Oh, I totally missed updating the commit msg on this:

---
In this scenario, hotplug_handler_plug() calls pcie_cap_slot_plug_cb(), which sets dev->config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI
- hotplug for the guest. After a while the guest will deal with this hotplug
- and qemu will clear the above bit.
+ hotplug for the guest. After a while, if the guest powers down the device
+ qemu will clear the above bit.
---

The whole idea is that the guest powers down the device, which causes qemu to hot-remove it, and clear PCI_EXP_SLTSTA_PDS.

---
/*
* If the slot is populated, power indicator is off and power
* controller is off, it is safe to detach the devices. *
* Note: don't detach if condition was already true:
* this is a work around for guests that overwrite
* control of powered off slots before powering them on. */
if ((sltsta & PCI_EXP_SLTSTA_PDS) && pcie_sltctl_powered_off(val) &&
!pcie_sltctl_powered_off(old_slt_ctl))
{
pcie_cap_slot_do_unplug(dev); // clear PCI_EXP_SLTSTA_PDS
}
---

Since the bit is different on source & target qemu, the migration is aborted.

Quoted 4 lines

--
Peter Xu

Thanks for reviewing Peter!
I will send a v3 with the updated commit msg and add the comments suggested by Juan in the source code.

Thanks!
Leo

#7 lore raw
In reply to Peter Xu
Quoted 6 lines · Peter Xu
Quoted 5 lines · Leonardo Bras Soares Passos
Quoted 4 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

Quoted 7 lines
Quoted 2 lines

I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.

I see, thanks!

But then do you know why it's never hit before? I assume it means this bug has been there for a long time.

It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.

Signature · 1 line
MST
#8 lore raw
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 4 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.

Quoted 8 lines
Quoted 4 lines

I see, thanks!

But then do you know why it's never hit before? I assume it means this bug has been there for a long time.

It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.

Michael, do you mean that Leo's patch is wrong?

I just got understood why it got cleared - I think Leo didn't mention that the device was actually offlined before migration, IIUC that's why the PDS bit got cleared, if PDS was trying to describe that of the slot.

According to:

    /* Used to enable checks on load. Note that writable bits are
     * never checked even if set in cmask. */
    uint8_t *cmask;

It does sound reasonable to me to have PDS cleared when device offlined. Since hypervisor doesn't really know what the condition the slot presence bit would be when migrating, it seems we should just clear the bit in cmask.

So with the last reply from Leo, the patch looks all right to me. It's just that as Leo mentioned, we should mention the offline process if that's the case, because that's definitely an important step to reproduce the issue.

Thanks,

Signature · 1 line
Peter Xu
#9 lore raw
Quoted 13 lines · Peter Xu
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 4 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.

I see, thanks!

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

But then do you know why it's never hit before? I assume it means this bug has been there for a long time.

It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.

Michael, do you mean that Leo's patch is wrong?

I mean his patch is exactly right. cmask was designed with this kind of use case in mind.
Will queue.

Quoted 15 lines

I just got understood why it got cleared - I think Leo didn't mention that the device was actually offlined before migration, IIUC that's why the PDS bit got cleared, if PDS was trying to describe that of the slot.

According to:

    /* Used to enable checks on load. Note that writable bits are
     * never checked even if set in cmask. */
    uint8_t *cmask;

It does sound reasonable to me to have PDS cleared when device offlined. Since hypervisor doesn't really know what the condition the slot presence bit would be when migrating, it seems we should just clear the bit in cmask.

Quoted 5 lines

So with the last reply from Leo, the patch looks all right to me. It's just that as Leo mentioned, we should mention the offline process if that's the case, because that's definitely an important step to reproduce the issue.

Thanks,

If you want to suggest more text to the commit log, for the benefit of backporters, that is fine by me.

Quoted 2 lines
Signature · 1 line
Peter Xu
#10 lore raw
Quoted 3 lines · Michael S. Tsirkin

I mean his patch is exactly right. cmask was designed with this kind of use case in mind.
Will queue.

That's great news.

Quoted 2 lines

If you want to suggest more text to the commit log, for the benefit of backporters, that is fine by me.

If you're fine with it I've no issue; since we're reaching soft-freeze I'd think it better if it can be merged on time (or just slightly enhance the commit message when merge).

Thanks!

Signature · 1 line
Peter Xu
#11 lore raw
In reply to Michael S. Tsirkin
Quoted 22 lines · Michael S. Tsirkin
Quoted 20 lines · Peter Xu
Quoted 18 lines · Michael S. Tsirkin
Quoted 6 lines · Peter Xu
Quoted 5 lines · Leonardo Bras Soares Passos
Quoted 4 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

Quoted 7 lines
Quoted 2 lines

I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.

I see, thanks!

But then do you know why it's never hit before? I assume it means this bug has been there for a long time.

It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.

Quoted 6 lines
Quoted 1 line

Michael, do you mean that Leo's patch is wrong?

I mean his patch is exactly right. cmask was designed with this kind of use case in mind.
Will queue.

Thanks Michael!

Any chance this will get in on time for v8.1 ?

Quoted 28 lines
Quoted 16 lines

I just got understood why it got cleared - I think Leo didn't mention that the device was actually offlined before migration, IIUC that's why the PDS bit got cleared, if PDS was trying to describe that of the slot.

According to:

    /* Used to enable checks on load. Note that writable bits are
     * never checked even if set in cmask. */
    uint8_t *cmask;

It does sound reasonable to me to have PDS cleared when device offlined. Since hypervisor doesn't really know what the condition the slot presence bit would be when migrating, it seems we should just clear the bit in cmask.

Quoted 5 lines

So with the last reply from Leo, the patch looks all right to me. It's just that as Leo mentioned, we should mention the offline process if that's the case, because that's definitely an important step to reproduce the issue.

Thanks,

If you want to suggest more text to the commit log, for the benefit of backporters, that is fine by me.

Quoted 2 lines

--
Peter Xu

#12 lore raw
Quoted 26 lines · Leonardo Bras Soares Passos
Quoted 25 lines · Michael S. Tsirkin
Quoted 21 lines · Peter Xu
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 4 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.

Quoted 8 lines
Quoted 4 lines

I see, thanks!

But then do you know why it's never hit before? I assume it means this bug has been there for a long time.

It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.

Michael, do you mean that Leo's patch is wrong?

Quoted 7 lines
Quoted 3 lines

I mean his patch is exactly right. cmask was designed with this kind of use case in mind.
Will queue.

Thanks Michael!

Any chance this will get in on time for v8.1 ?

Yes, working on pull request now.

Quoted 28 lines
Quoted 28 lines
Quoted 16 lines

I just got understood why it got cleared - I think Leo didn't mention that the device was actually offlined before migration, IIUC that's why the PDS bit got cleared, if PDS was trying to describe that of the slot.

According to:

    /* Used to enable checks on load. Note that writable bits are
     * never checked even if set in cmask. */
    uint8_t *cmask;

It does sound reasonable to me to have PDS cleared when device offlined. Since hypervisor doesn't really know what the condition the slot presence bit would be when migrating, it seems we should just clear the bit in cmask.

Quoted 5 lines

So with the last reply from Leo, the patch looks all right to me. It's just that as Leo mentioned, we should mention the offline process if that's the case, because that's definitely an important step to reproduce the issue.

Thanks,

If you want to suggest more text to the commit log, for the benefit of backporters, that is fine by me.

Quoted 2 lines

--
Peter Xu

#13 lore raw
Quoted 32 lines · Michael S. Tsirkin
Quoted 30 lines · Leonardo Bras Soares Passos
Quoted 28 lines · Michael S. Tsirkin
Quoted 13 lines · Peter Xu
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 4 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.

I see, thanks!

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

But then do you know why it's never hit before? I assume it means this bug has been there for a long time.

It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.

Michael, do you mean that Leo's patch is wrong?

I mean his patch is exactly right. cmask was designed with this kind of use case in mind.
Will queue.

Quoted 5 lines
Quoted 3 lines

Thanks Michael!

Any chance this will get in on time for v8.1 ?

Yes, working on pull request now.

Thanks!

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
Quoted 18 lines

I just got understood why it got cleared - I think Leo didn't mention that the device was actually offlined before migration, IIUC that's why the PDS bit got cleared, if PDS was trying to describe that of the slot.

According to:

    /* Used to enable checks on load. Note that writable bits are
     * never checked even if set in cmask. */
    uint8_t *cmask;

It does sound reasonable to me to have PDS cleared when device offlined. Since hypervisor doesn't really know what the condition the slot presence bit would be when migrating, it seems we should just clear the bit in cmask.

Quoted 5 lines

So with the last reply from Leo, the patch looks all right to me. It's just that as Leo mentioned, we should mention the offline process if that's the case, because that's definitely an important step to reproduce the issue.

Thanks,

If you want to suggest more text to the commit log, for the benefit of backporters, that is fine by me.

Quoted 2 lines

--
Peter Xu

#14 lore raw
In reply to Peter Xu
Quoted 7 lines · Peter Xu
Quoted 5 lines · Leonardo Bras Soares Passos
Quoted 4 lines

I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?

For example, is this broken on 6.0 binaries too with pc-q35-6.0?

Quoted 7 lines
Quoted 2 lines

I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.

I see, thanks!

But then do you know why it's never hit before? I assume it means this bug has been there for a long time.

Even longer than expected:

I did some testing looking for the bug:

qemu v5.0.1 reproduces
qemu v4.0.1 reproduces
qemu v3.0.1 reproduces
qemu v2.12.1 reproduces

I decided to stop testing at this point, because it required python2 for building qemu, and it is far enough (5 years).

Seems to be a very old bug, that just hasn't bothered anyone until now.

Quoted 4 lines

--
Peter Xu

Thanks!
Leo