pcie: Add hotplug detect state register to cmask
PATCH v2 1/1 pcie: Add hotplug detect state register to cmask
When trying to migrate a machine type pc-q35-6.0 or lower, with this cmdline options,
-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \
-device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1
the following bug happens after all ram pages were sent:
qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argument
This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }
In this scenario, hotplug_handler_plug() calls pcie_cap_slot_plug_cb(), which sets dev->config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI hotplug for the guest. After a while the guest will deal with this hotplug and qemu will clear the above bit.
Then, during migration, get_pci_config_device() will compare the configs of both the freshly created device and the one that is being received via migration, which will differ due to the PCI_EXP_SLTSTA_PDS bit and cause the bug to reproduce.
To avoid this fake incompatibility, there are tree fields in PCIDevice that can help:
- wmask: Used to implement R/W bytes, and
- w1cmask: Used to implement RW1C(Write 1 to Clear) bytes
- cmask: Used to enable config checks on load.
According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.
So, clear PCI_EXP_SLTSTA_PDS bit on cmask, so the fake incompatibility on get_pci_config_device() does not abort the migration.
--- hw/pci/pcie.c | 4 ++++ 1 file changed, 4 insertions(+)
hw/pci/pcie.c +4 -0
@@ -659,6 +659,10 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)pci_word_test_and_set_mask (dev->w1cmask + pos +PCI_EXP_SLTSTA ,PCI_EXP_HP_EV_SUPPORTED ); +/* Avoid migration abortion when this device hot-removed by guest */ +pci_word_test_and_clear_mask (dev->cmask + pos +PCI_EXP_SLTSTA , +PCI_EXP_SLTSTA_PDS ); + dev->exp.hpev_notified = false; qbus_set_hotplug_handler(BUS(pci_bridge_get_sec_bus(PCI_BRIDGE(dev))),
Signature · 1 line
2.41.0
Quoted 57 lines · Leonardo Bras
When trying to migrate a machine type pc-q35-6.0 or lower, with this cmdline options,
(quoted diff) +0
-2-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \ -device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1the following bug happens after all ram pages were sent:
qemu-kvm:
get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance0x0of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argumentThis happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC",ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }In this scenario,
hotplug_handler_plug()callspcie_cap_slot_plug_cb(), which setsdev->configbyte0x6ewith bitPCI_EXP_SLTSTA_PDSto signal PCI hotplug for the guest. After a while the guest will deal with this hotplug and qemu will clear the above bit.Then, during migration,
get_pci_config_device()will compare the configs of both the freshly created device and the one that is being received via migration, which will differ due to thePCI_EXP_SLTSTA_PDSbit and cause the bug to reproduce.To avoid this fake incompatibility, there are tree fields in PCIDevice that can help:
- wmask: Used to implement R/W bytes, and
- w1cmask: Used to implement RW1C(Write 1 to Clear) bytes
- cmask: Used to enable config checks on load.According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.
So, clear
PCI_EXP_SLTSTA_PDSbit on cmask, so the fake incompatibility onget_pci_config_device()does not abort the migration.Signed-off-by Leonardo Bras <[email protected]>--- hw/pci/pcie.c | 4 ++++ 1 file changed, 4 insertions(+)hw/pci/pcie.c +0
-0
hw/pci/pcie.c void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s) · line 663
Quoted 5 lines
hw/pci/pcie.c +1
-0@@ -659,6 +659,10 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)pci_word_test_and_set_mask (dev->w1cmask + pos +PCI_EXP_SLTSTA ,PCI_EXP_HP_EV_SUPPORTED ); +/* Avoid migration abortion when this device hot-removed by guest */
I would have included here the text in the commit:
According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.
and
This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }
so if we ever remove the machine type pc-q35-6.0, we can drop it.
Yes, I know that we don't drop machine types, but we should at some point.
Quoted 6 lines
hw/pci/pcie.c +3
-0+pci_word_test_and_clear_mask (dev->cmask + pos +PCI_EXP_SLTSTA , +PCI_EXP_SLTSTA_PDS ); + dev->exp.hpev_notified = false;qbus_set_hotplug_handler(BUS(pci_bridge_get_sec_bus(PCI_BRIDGE(dev))),
I agree that this is (at least) a step on the right direction.
I wmould had expected to have to need some check related to the value of:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }
But I will not claim _any_ understanding of the PCI specification.
So:
about that it fixes the migration bug.
Quoted 72 lines · Juan Quintela
Quoted 57 lines · Leonardo Bras
When trying to migrate a machine type pc-q35-6.0 or lower, with this cmdline options,
(quoted diff) +0
-2-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \ -device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1the following bug happens after all ram pages were sent:
qemu-kvm:
get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance0x0of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argumentThis happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC",ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }In this scenario,
hotplug_handler_plug()callspcie_cap_slot_plug_cb(), which setsdev->configbyte0x6ewith bitPCI_EXP_SLTSTA_PDSto signal PCI hotplug for the guest. After a while the guest will deal with this hotplug and qemu will clear the above bit.Then, during migration,
get_pci_config_device()will compare the configs of both the freshly created device and the one that is being received via migration, which will differ due to thePCI_EXP_SLTSTA_PDSbit and cause the bug to reproduce.To avoid this fake incompatibility, there are tree fields in PCIDevice that can help:
- wmask: Used to implement R/W bytes, and
- w1cmask: Used to implement RW1C(Write 1 to Clear) bytes
- cmask: Used to enable config checks on load.According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.
So, clear
PCI_EXP_SLTSTA_PDSbit on cmask, so the fake incompatibility onget_pci_config_device()does not abort the migration.Signed-off-by Leonardo Bras <[email protected]>--- hw/pci/pcie.c | 4 ++++ 1 file changed, 4 insertions(+)hw/pci/pcie.c +0
-0hw/pci/pcie.c void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s) · line 663
Quoted 5 lines
hw/pci/pcie.c +1
-0@@ -659,6 +659,10 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)pci_word_test_and_set_mask (dev->w1cmask + pos +PCI_EXP_SLTSTA ,PCI_EXP_HP_EV_SUPPORTED ); +/* Avoid migration abortion when this device hot-removed by guest */ I would have included here the text in the commit:
According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.
Quoted 6 lines
and
This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC",ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }so if we ever remove the machine type pc-q35-6.0, we can drop it.
It makes sense adding this to the code.
In the remove machine-type case, IIUC we would have to drop every machine-type older than pc-q35-6.0.
Also, we would not support migration to any machine without ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, right?
Quoted 9 lines
Yes, I know that we don't drop machine types, but we should at some point.
Quoted 5 lines
hw/pci/pcie.c +3
-0+pci_word_test_and_clear_mask (dev->cmask + pos +PCI_EXP_SLTSTA , +PCI_EXP_SLTSTA_PDS ); + dev->exp.hpev_notified = false;
Quoted 9 lines
Quoted 1 line
qbus_set_hotplug_handler(BUS(pci_bridge_get_sec_bus(PCI_BRIDGE(dev))),
I agree that this is (at least) a step on the right direction.
I wmould had expected to have to need some check related to the value of:
{ "ICH9-LPC",
ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }
This bug affects versions older than qemu 6.0. If we add this, we would have some extra work backporting this to older versions (if necessary) because the 'property' did not exist back then.
Quoted 8 lines
But I will not claim _any_ understanding of the PCI specification.
So:
Reviewed-by Juan Quintela <[email protected]>about that it fixes the migration bug.
Thanks Juan!
Leo
Quoted 14 lines · Leonardo Bras
When trying to migrate a machine type pc-q35-6.0 or lower, with this cmdline options,
(quoted diff) +0
-2-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \ -device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1the following bug happens after all ram pages were sent:
qemu-kvm:
get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance0x0of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argument
Quoted 7 lines
This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC",ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }In this scenario,
hotplug_handler_plug()callspcie_cap_slot_plug_cb(), which setsdev->configbyte0x6ewith bitPCI_EXP_SLTSTA_PDSto signal PCI hotplug for the guest. After a while the guest will deal with this hotplug and qemu will clear the above bit.
Do you mean that the bit will be cleared after this point for the whole lifecycle of the VM, as long as the pcie topology doesn't change again?
"This bit indicates the presence of an adapter in the slot"
IIUC the adapter in the slot is there, why it's cleared rather than set?
Quoted 13 lines
Then, during migration,
get_pci_config_device()will compare the configs of both the freshly created device and the one that is being received via migration, which will differ due to thePCI_EXP_SLTSTA_PDSbit and cause the bug to reproduce.To avoid this fake incompatibility, there are tree fields in PCIDevice that can help:
- wmask: Used to implement R/W bytes, and
- w1cmask: Used to implement RW1C(Write 1 to Clear) bytes
- cmask: Used to enable config checks on load.
Quoted 7 lines
According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.
So, clear
PCI_EXP_SLTSTA_PDSbit on cmask, so the fake incompatibility onget_pci_config_device()does not abort the migration.
Yes, using cmask makes more sense to me, but we'd need some pci developer to ack it at last I guess, anyway.
Quoted 3 lines
Signed-off-by Leonardo Bras <[email protected]>
I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
Thanks,
Quoted 22 lines
--- hw/pci/pcie.c | 4 ++++ 1 file changed, 4 insertions(+)hw/pci/pcie.c +4
-0@@ -659,6 +659,10 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)pci_word_test_and_set_mask (dev->w1cmask + pos +PCI_EXP_SLTSTA ,PCI_EXP_HP_EV_SUPPORTED ); +/* Avoid migration abortion when this device hot-removed by guest */ +pci_word_test_and_clear_mask (dev->cmask + pos +PCI_EXP_SLTSTA , +PCI_EXP_SLTSTA_PDS ); + dev->exp.hpev_notified = false; qbus_set_hotplug_handler(BUS(pci_bridge_get_sec_bus(PCI_BRIDGE(dev))),Signature · 1 line
2.41.0
Signature · 1 line
Peter Xu
Quoted 27 lines · Peter Xu
Quoted 14 lines · Leonardo Bras
When trying to migrate a machine type pc-q35-6.0 or lower, with this cmdline options,
(quoted diff) +0
-2-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \ -device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1the following bug happens after all ram pages were sent:
qemu-kvm:
get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance0x0of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argumentQuoted 7 lines
This happens on pc-q35-6.0 or lower because of:
{ "ICH9-LPC",ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }In this scenario,
hotplug_handler_plug()callspcie_cap_slot_plug_cb(), which setsdev->configbyte0x6ewith bitPCI_EXP_SLTSTA_PDSto signal PCI hotplug for the guest. After a while the guest will deal with this hotplug and qemu will clear the above bit.Do you mean that the bit will be cleared after this point for the whole lifecycle of the VM, as long as the pcie topology doesn't change again?
Quoted 3 lines
"This bit indicates the presence of an adapter in the slot"
IIUC the adapter in the slot is there, why it's cleared rather than set?
Fort some reason the guest is powering down the device, and we have in qemu:
/*
* If the slot is populated, power indicator is off and power
* controller is off, it is safe to detach the devices. *
* Note: don't detach if condition was already true:
* this is a work around for guests that overwrite
* control of powered off slots before powering them on. */
if ((sltsta & PCI_EXP_SLTSTA_PDS) && pcie_sltctl_powered_off(val) &&
!pcie_sltctl_powered_off(old_slt_ctl))
{
pcie_cap_slot_do_unplug(dev); // clears PCI_EXP_SLTSTA_PDS
}
Quoted 19 lines
Quoted 19 lines
Then, during migration,
get_pci_config_device()will compare the configs of both the freshly created device and the one that is being received via migration, which will differ due to thePCI_EXP_SLTSTA_PDSbit and cause the bug to reproduce.To avoid this fake incompatibility, there are tree fields in PCIDevice that can help:
- wmask: Used to implement R/W bytes, and
- w1cmask: Used to implement RW1C(Write 1 to Clear) bytes
- cmask: Used to enable config checks on load.According to PCI Express® Base Specification Revision 5.0 Version 1.0, table 7-27 (Slot Status Register) bit 6, the "Presence Detect State" is listed as RO (read-only), so it only makes sense to make use of the cmask field.
Quoted 5 lines
Quoted 2 lines
So, clear
PCI_EXP_SLTSTA_PDSbit on cmask, so the fake incompatibility onget_pci_config_device()does not abort the migration.Yes, using cmask makes more sense to me, but we'd need some pci developer to ack it at last I guess, anyway.
Agree! I am waiting for Michael's opinion on this.
Quoted 9 lines
Quoted 4 lines
Signed-off-by Leonardo Bras <[email protected]>I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.
Quoted 2 lines
Thanks,
Thank you!
Leo
Quoted 27 lines
Quoted 21 lines
--- hw/pci/pcie.c | 4 ++++ 1 file changed, 4 insertions(+)hw/pci/pcie.c +4
-1@@ -659,6 +659,10 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)pci_word_test_and_set_mask (dev->w1cmask + pos +PCI_EXP_SLTSTA ,PCI_EXP_HP_EV_SUPPORTED ); +/* Avoid migration abortion when this device hot-removed by guest */ +pci_word_test_and_clear_mask (dev->cmask + pos +PCI_EXP_SLTSTA , +PCI_EXP_SLTSTA_PDS ); + dev->exp.hpev_notified = false; qbus_set_hotplug_handler(BUS(pci_bridge_get_sec_bus(PCI_BRIDGE(dev))), --Quoted 2 lines
2.41.0
--
Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 4 lines
I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.
I see, thanks!
But then do you know why it's never hit before? I assume it means this bug has been there for a long time.
Signature · 1 line
Peter Xu
Quoted 7 lines · Peter Xu
Quoted 5 lines · Leonardo Bras Soares Passos
Quoted 4 lines
I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
Quoted 7 lines
Quoted 2 lines
I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.
I see, thanks!
But then do you know why it's never hit before? I assume it means this bug has been there for a long time.
Oh, I totally missed updating the commit msg on this:
---
In this scenario, hotplug_handler_plug() calls pcie_cap_slot_plug_cb(), which sets dev->config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI
- hotplug for the guest. After a while the guest will deal with this hotplug
- and qemu will clear the above bit.
+ hotplug for the guest. After a while, if the guest powers down the device
+ qemu will clear the above bit.
---
The whole idea is that the guest powers down the device, which causes qemu to hot-remove it, and clear PCI_EXP_SLTSTA_PDS.
---
/*
* If the slot is populated, power indicator is off and power
* controller is off, it is safe to detach the devices. *
* Note: don't detach if condition was already true:
* this is a work around for guests that overwrite
* control of powered off slots before powering them on. */
if ((sltsta & PCI_EXP_SLTSTA_PDS) && pcie_sltctl_powered_off(val) &&
!pcie_sltctl_powered_off(old_slt_ctl))
{
pcie_cap_slot_do_unplug(dev); // clear PCI_EXP_SLTSTA_PDS
}
---
Since the bit is different on source & target qemu, the migration is aborted.
Quoted 4 lines
--
Peter Xu
Thanks for reviewing Peter!
I will send a v3 with the updated commit msg and add the comments suggested by Juan in the source code.
Thanks!
Leo
Quoted 6 lines · Peter Xu
Quoted 5 lines · Leonardo Bras Soares Passos
Quoted 4 lines
I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
Quoted 7 lines
Quoted 2 lines
I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.
I see, thanks!
But then do you know why it's never hit before? I assume it means this bug has been there for a long time.
It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.
Signature · 1 line
MST
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 4 lines
I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.
Quoted 8 lines
Quoted 4 lines
I see, thanks!
But then do you know why it's never hit before? I assume it means this bug has been there for a long time.
It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.
Michael, do you mean that Leo's patch is wrong?
I just got understood why it got cleared - I think Leo didn't mention that the device was actually offlined before migration, IIUC that's why the PDS bit got cleared, if PDS was trying to describe that of the slot.
According to:
/* Used to enable checks on load. Note that writable bits are
* never checked even if set in cmask. */
uint8_t *cmask;It does sound reasonable to me to have PDS cleared when device offlined. Since hypervisor doesn't really know what the condition the slot presence bit would be when migrating, it seems we should just clear the bit in cmask.
So with the last reply from Leo, the patch looks all right to me. It's just that as Leo mentioned, we should mention the offline process if that's the case, because that's definitely an important step to reproduce the issue.
Thanks,
Signature · 1 line
Peter Xu
Quoted 13 lines · Peter Xu
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 4 lines
I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.
I see, thanks!
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
But then do you know why it's never hit before? I assume it means this bug has been there for a long time.
It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.Michael, do you mean that Leo's patch is wrong?
I mean his patch is exactly right. cmask was designed with this kind of use case in mind.
Will queue.
Quoted 15 lines
I just got understood why it got cleared - I think Leo didn't mention that the device was actually offlined before migration, IIUC that's why the PDS bit got cleared, if PDS was trying to describe that of the slot.
According to:
/* Used to enable checks on load. Note that writable bits are * never checked even if set in cmask. */ uint8_t *cmask;It does sound reasonable to me to have PDS cleared when device offlined. Since hypervisor doesn't really know what the condition the slot presence bit would be when migrating, it seems we should just clear the bit in cmask.
Quoted 5 lines
So with the last reply from Leo, the patch looks all right to me. It's just that as Leo mentioned, we should mention the offline process if that's the case, because that's definitely an important step to reproduce the issue.
Thanks,
If you want to suggest more text to the commit log, for the benefit of backporters, that is fine by me.
Quoted 2 lines
Signature · 1 line
Peter Xu
Quoted 3 lines · Michael S. Tsirkin
I mean his patch is exactly right. cmask was designed with this kind of use case in mind.
Will queue.
That's great news.
Quoted 2 lines
If you want to suggest more text to the commit log, for the benefit of backporters, that is fine by me.
If you're fine with it I've no issue; since we're reaching soft-freeze I'd think it better if it can be merged on time (or just slightly enhance the commit message when merge).
Thanks!
Signature · 1 line
Peter Xu
Quoted 22 lines · Michael S. Tsirkin
Quoted 20 lines · Peter Xu
Quoted 18 lines · Michael S. Tsirkin
Quoted 6 lines · Peter Xu
Quoted 5 lines · Leonardo Bras Soares Passos
Quoted 4 lines
I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
Quoted 7 lines
Quoted 2 lines
I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.
I see, thanks!
But then do you know why it's never hit before? I assume it means this bug has been there for a long time.
It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.
Quoted 6 lines
Quoted 1 line
Michael, do you mean that Leo's patch is wrong?
I mean his patch is exactly right. cmask was designed with this kind of use case in mind.
Will queue.
Thanks Michael!
Any chance this will get in on time for v8.1 ?
Quoted 28 lines
Quoted 16 lines
I just got understood why it got cleared - I think Leo didn't mention that the device was actually offlined before migration, IIUC that's why the PDS bit got cleared, if PDS was trying to describe that of the slot.
According to:
/* Used to enable checks on load. Note that writable bits are * never checked even if set in cmask. */ uint8_t *cmask;It does sound reasonable to me to have PDS cleared when device offlined. Since hypervisor doesn't really know what the condition the slot presence bit would be when migrating, it seems we should just clear the bit in cmask.
Quoted 5 lines
So with the last reply from Leo, the patch looks all right to me. It's just that as Leo mentioned, we should mention the offline process if that's the case, because that's definitely an important step to reproduce the issue.
Thanks,
If you want to suggest more text to the commit log, for the benefit of backporters, that is fine by me.
Quoted 2 lines
--
Peter Xu
Quoted 26 lines · Leonardo Bras Soares Passos
Quoted 25 lines · Michael S. Tsirkin
Quoted 21 lines · Peter Xu
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 4 lines
I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.
Quoted 8 lines
Quoted 4 lines
I see, thanks!
But then do you know why it's never hit before? I assume it means this bug has been there for a long time.
It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.Michael, do you mean that Leo's patch is wrong?
Quoted 7 lines
Quoted 3 lines
I mean his patch is exactly right. cmask was designed with this kind of use case in mind.
Will queue.Thanks Michael!
Any chance this will get in on time for v8.1 ?
Yes, working on pull request now.
Quoted 28 lines
Quoted 28 lines
Quoted 16 lines
I just got understood why it got cleared - I think Leo didn't mention that the device was actually offlined before migration, IIUC that's why the PDS bit got cleared, if PDS was trying to describe that of the slot.
According to:
/* Used to enable checks on load. Note that writable bits are * never checked even if set in cmask. */ uint8_t *cmask;It does sound reasonable to me to have PDS cleared when device offlined. Since hypervisor doesn't really know what the condition the slot presence bit would be when migrating, it seems we should just clear the bit in cmask.
Quoted 5 lines
So with the last reply from Leo, the patch looks all right to me. It's just that as Leo mentioned, we should mention the offline process if that's the case, because that's definitely an important step to reproduce the issue.
Thanks,
If you want to suggest more text to the commit log, for the benefit of backporters, that is fine by me.
Quoted 2 lines
--
Peter Xu
Quoted 32 lines · Michael S. Tsirkin
Quoted 30 lines · Leonardo Bras Soares Passos
Quoted 28 lines · Michael S. Tsirkin
Quoted 13 lines · Peter Xu
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 4 lines
I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.
I see, thanks!
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
But then do you know why it's never hit before? I assume it means this bug has been there for a long time.
It's a race - you have to migrate after the bit has been set before the bit got cleared.
cmask is exactly for bits that qemu modifies itself.Michael, do you mean that Leo's patch is wrong?
I mean his patch is exactly right. cmask was designed with this kind of use case in mind.
Will queue.
Quoted 5 lines
Quoted 3 lines
Thanks Michael!
Any chance this will get in on time for v8.1 ?
Yes, working on pull request now.
Thanks!
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
Quoted 18 lines
I just got understood why it got cleared - I think Leo didn't mention that the device was actually offlined before migration, IIUC that's why the PDS bit got cleared, if PDS was trying to describe that of the slot.
According to:
/* Used to enable checks on load. Note that writable bits are * never checked even if set in cmask. */ uint8_t *cmask;It does sound reasonable to me to have PDS cleared when device offlined. Since hypervisor doesn't really know what the condition the slot presence bit would be when migrating, it seems we should just clear the bit in cmask.
Quoted 5 lines
So with the last reply from Leo, the patch looks all right to me. It's just that as Leo mentioned, we should mention the offline process if that's the case, because that's definitely an important step to reproduce the issue.
Thanks,
If you want to suggest more text to the commit log, for the benefit of backporters, that is fine by me.
Quoted 2 lines
--
Peter Xu
Quoted 7 lines · Peter Xu
Quoted 5 lines · Leonardo Bras Soares Passos
Quoted 4 lines
I asked the same question, and I still keep confused: whether there's a first bad commit? Starting from when it fails?
For example, is this broken on 6.0 binaries too with pc-q35-6.0?
Quoted 7 lines
Quoted 2 lines
I tested for qemu 6.0, and it still reproduces, but have not pursued this any further.
I see, thanks!
But then do you know why it's never hit before? I assume it means this bug has been there for a long time.
Even longer than expected:
I did some testing looking for the bug:
qemu v5.0.1 reproduces
qemu v4.0.1 reproduces
qemu v3.0.1 reproduces
qemu v2.12.1 reproduces
I decided to stop testing at this point, because it required python2 for building qemu, and it is far enough (5 years).
Seems to be a very old bug, that just hasn't bothered anyone until now.
Quoted 4 lines
--
Peter Xu
Thanks!
Leo