failover: unregister ROM on unplug
PATCH v4 1/1 failover: unregister ROM on unplug
The intend of failover is to allow to migrate a VM with a VFIO networking card without disrupting the network operation by switching to a virtio-net device during the migration.
This simple change allows to test failover with a simulated device like e1000e rather than a vfio device, even if it's useless in real life it can help to debug failover.
This is interesting to developers that want to test failover on a system with no vfio device. Moreover it simplifies host networking configuration as we can use the same bridge for virtio-net and the other failover networking device.
Without this change the migration of a system configured with failover fails with:
... -device virtio-net-pci,id=virtionet0,failover=on,... \ -device e1000,failover_pair_id=virtionet0,... \ ...
(qemu) migrate ...
Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration error while loading state for instance 0x0 of device 'ram' load of migration failed: Invalid argument
This happens because QEMU correctly unregisters the interface vmstate but not the ROM one. This patch fixes that.
---
Notes:
v4:
export and use pci_del_option_rom()
v3:
remove useless space before comma
v2:
reset has_rom to false
update commit log message
include/hw/pci/pci.h | 2 ++ hw/net/virtio-net.c | 1 + hw/pci/pci.c | 3 +-- 3 files changed, 4 insertions(+), 2 deletions(-)
include/hw/pci/pci.h +2 -0
@@ -369,6 +369,8 @@ void pci_register_vga(PCIDevice *pci_dev, MemoryRegion *mem,void pci_unregister_vga (PCIDevice *pci_dev);pcibus_t pci_get_bar_addr (PCIDevice *pci_dev,int region_num); +void pci_del_option_rom (PCIDevice *pdev); +int pci_add_capability (PCIDevice *pdev,uint8_t cap_id,uint8_t offset,uint8_t size,Error **errp);
hw/net/virtio-net.c +1 -0
@@ -3256,6 +3256,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationState *s)if (migration_in_setup (s) && !should_be_hidden) {if (failover_unplug_primary (n, dev)) {vmstate_unregister (VMSTATE_IF (dev),qdev_get_vmsd (dev), dev); +pci_del_option_rom (PCI_DEVICE (dev));qapi_event_send_unplug_primary (dev->id);qatomic_set (&n->failover_primary_hidden, true); }else {
hw/pci/pci.c +1 -2
@@ -228,7 +228,6 @@ static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);static void pci_add_option_rom (PCIDevice *pdev,bool is_default_rom,Error **); -static void pci_del_option_rom (PCIDevice *pdev);static uint16_t pci_default_sub_vendor_id =PCI_SUBVENDOR_ID_REDHAT_QUMRANET ;static uint16_t pci_default_sub_device_id =PCI_SUBDEVICE_ID_QEMU ;
@@ -2429,7 +2428,7 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,pci_register_bar (pdev,PCI_ROM_SLOT ,0 , &pdev->rom); } -static void pci_del_option_rom (PCIDevice *pdev) +void pci_del_option_rom (PCIDevice *pdev) {if (!pdev->has_rom)return ;
Signature · 1 line
2.31.1
Quoted 40 lines · Laurent Vivier
The intend of failover is to allow to migrate a VM with a VFIO networking card without disrupting the network operation by switching to a virtio-net device during the migration.
This simple change allows to test failover with a simulated device like e1000e rather than a vfio device, even if it's useless in real life it can help to debug failover.
This is interesting to developers that want to test failover on a system with no vfio device. Moreover it simplifies host networking configuration as we can use the same bridge for virtio-net and the other failover networking device.
Without this change the migration of a system configured with failover fails with:
... -device virtio-net-pci,id=virtionet0,failover=on,... \ -device e1000,failover_pair_id=virtionet0,... \ ...(qemu) migrate ...
Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration error while loading state for instance
0x0of device 'ram' load of migration failed: Invalid argumentThis happens because QEMU correctly unregisters the interface vmstate but not the ROM one. This patch fixes that.
Signed-off-by Laurent Vivier <[email protected]>---
Notes:
v4:
export and usepci_del_option_rom()v3:
remove useless space before comma
Quoted 8 lines
v2:
resethas_romto false
update commit log messageinclude/hw/pci/pci.h | 2 ++ hw/net/virtio-net.c | 1 + hw/pci/pci.c | 3 +-- 3 files changed, 4 insertions(+), 2 deletions(-)
Quoted 23 lines · Laurent Vivier
The intend of failover is to allow to migrate a VM with a VFIO networking card without disrupting the network operation by switching to a virtio-net device during the migration.
This simple change allows to test failover with a simulated device like e1000e rather than a vfio device, even if it's useless in real life it can help to debug failover.
This is interesting to developers that want to test failover on a system with no vfio device. Moreover it simplifies host networking configuration as we can use the same bridge for virtio-net and the other failover networking device.
Without this change the migration of a system configured with failover fails with:
... -device virtio-net-pci,id=virtionet0,failover=on,... \ -device e1000,failover_pair_id=virtionet0,... \ ...(qemu) migrate ...
Quoted 8 lines
Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration error while loading state for instance
0x0of device 'ram' load of migration failed: Invalid argumentThis happens because QEMU correctly unregisters the interface vmstate but not the ROM one. This patch fixes that.
Signed-off-by Laurent Vivier <[email protected]>
And does failover_replug_primary roll it all back then?
Quoted 66 lines
---
Notes:
v4:
export and usepci_del_option_rom()v3:
remove useless space before commav2:
resethas_romto false
update commit log messageinclude/hw/pci/pci.h | 2 ++ hw/net/virtio-net.c | 1 + hw/pci/pci.c | 3 +-- 3 files changed, 4 insertions(+), 2 deletions(-)include/hw/pci/pci.h +2
-0@@ -369,6 +369,8 @@ void pci_register_vga(PCIDevice *pci_dev, MemoryRegion *mem,void pci_unregister_vga (PCIDevice *pci_dev);pcibus_t pci_get_bar_addr (PCIDevice *pci_dev,int region_num); +void pci_del_option_rom (PCIDevice *pdev); +int pci_add_capability (PCIDevice *pdev,uint8_t cap_id,uint8_t offset,uint8_t size,Error **errp);hw/net/virtio-net.c +1
-0@@ -3256,6 +3256,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationState *s)if (migration_in_setup (s) && !should_be_hidden) {if (failover_unplug_primary (n, dev)) {vmstate_unregister (VMSTATE_IF (dev),qdev_get_vmsd (dev), dev); +pci_del_option_rom (PCI_DEVICE (dev));qapi_event_send_unplug_primary (dev->id);qatomic_set (&n->failover_primary_hidden, true); }else {hw/pci/pci.c +1
-2@@ -228,7 +228,6 @@ static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);static void pci_add_option_rom (PCIDevice *pdev,bool is_default_rom,Error **); -static void pci_del_option_rom (PCIDevice *pdev);static uint16_t pci_default_sub_vendor_id =PCI_SUBVENDOR_ID_REDHAT_QUMRANET ;static uint16_t pci_default_sub_device_id =PCI_SUBDEVICE_ID_QEMU ;@@ -2429,7 +2428,7 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,pci_register_bar (pdev,PCI_ROM_SLOT ,0 , &pdev->rom); } -static void pci_del_option_rom (PCIDevice *pdev) +void pci_del_option_rom (PCIDevice *pdev) {if (!pdev->has_rom)return ;Signature · 1 line
2.31.1
Quoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Laurent Vivier
The intend of failover is to allow to migrate a VM with a VFIO networking card without disrupting the network operation by switching to a virtio-net device during the migration.
This simple change allows to test failover with a simulated device like e1000e rather than a vfio device, even if it's useless in real life it can help to debug failover.
This is interesting to developers that want to test failover on a system with no vfio device. Moreover it simplifies host networking configuration as we can use the same bridge for virtio-net and the other failover networking device.
Without this change the migration of a system configured with failover fails with:
... -device virtio-net-pci,id=virtionet0,failover=on,... \ -device e1000,failover_pair_id=virtionet0,... \ ...(qemu) migrate ...
Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration error while loading state for instance
0x0of device 'ram' load of migration failed: Invalid argument
Quoted 6 lines
Quoted 4 lines
This happens because QEMU correctly unregisters the interface vmstate but not the ROM one. This patch fixes that.
Signed-off-by Laurent Vivier <[email protected]>And does
failover_replug_primaryroll it all back then?
It seems not.
To check I have started a migration, then I have canceled it, and I have started a migration to a file (migrate "exec:cat > mig") and I have analyzed the result with the script scripts/analyze-migration.py
But:
* with VFIO it's not a problem as we never migrate VFIO device itself,
* with an emulated PCI device, it's only a problem if we disable failover after having canceled a first try (because if we keep failover we acts like for VFIO and we unplug the card and don't migrate vmstates). This means we need a scenario like that to hit the bug:
- enable failover with an emulated PCI device
- migrate to another machine
- cancel/abort the migration before the end of the migration
- unplug the virtio-net device to disable the failover behavior,
- migrate the machine again with only the emulated PCI device
but as I said previously failover with emulated PCI device is only for developers and
test purpose and not to use in production...
Thanks,
Laurent
Quoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Laurent Vivier
The intend of failover is to allow to migrate a VM with a VFIO networking card without disrupting the network operation by switching to a virtio-net device during the migration.
This simple change allows to test failover with a simulated device like e1000e rather than a vfio device, even if it's useless in real life it can help to debug failover.
This is interesting to developers that want to test failover on a system with no vfio device. Moreover it simplifies host networking configuration as we can use the same bridge for virtio-net and the other failover networking device.
Without this change the migration of a system configured with failover fails with:
... -device virtio-net-pci,id=virtionet0,failover=on,... \ -device e1000,failover_pair_id=virtionet0,... \ ...(qemu) migrate ...
Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration error while loading state for instance
0x0of device 'ram' load of migration failed: Invalid argument
Quoted 6 lines
Quoted 4 lines
This happens because QEMU correctly unregisters the interface vmstate but not the ROM one. This patch fixes that.
Signed-off-by Laurent Vivier <[email protected]>And does
failover_replug_primaryroll it all back then?
In fact, I think we cannot roll it back because we don't have all the information to do the vmstate_register() again. Perhaps we can implement a vmstate_disable()/vmstate_enable() to disable the state migration without removing the information from the list?
Thanks,
Laurent
Quoted 23 lines · Laurent Vivier
The intend of failover is to allow to migrate a VM with a VFIO networking card without disrupting the network operation by switching to a virtio-net device during the migration.
This simple change allows to test failover with a simulated device like e1000e rather than a vfio device, even if it's useless in real life it can help to debug failover.
This is interesting to developers that want to test failover on a system with no vfio device. Moreover it simplifies host networking configuration as we can use the same bridge for virtio-net and the other failover networking device.
Without this change the migration of a system configured with failover fails with:
... -device virtio-net-pci,id=virtionet0,failover=on,... \ -device e1000,failover_pair_id=virtionet0,... \ ...(qemu) migrate ...
Quoted 8 lines
Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration error while loading state for instance
0x0of device 'ram' load of migration failed: Invalid argumentThis happens because QEMU correctly unregisters the interface vmstate but not the ROM one. This patch fixes that.
Signed-off-by Laurent Vivier <[email protected]>
Build fails on qemu-system-m68k:
/usr/bin/ld: libqemu-m68k-softmmu.fa.p/hw_net_virtio-net.c.o: in function `virtio_net_handle_migration_primary':
/scm/qemu/build/../hw/net/virtio-net.c:3259: undefined reference to `pci_del_option_rom'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
make[1]: *** [Makefile:154: run-ninja] Error 1
It's not pretty to poke at pci from generic virtio. Should we maybe wrap vmstate_unregister and pci_del_option_rom to allow removing all migrateable things related to the device in one go somehow?
Quoted 66 lines
---
Notes:
v4:
export and usepci_del_option_rom()v3:
remove useless space before commav2:
resethas_romto false
update commit log messageinclude/hw/pci/pci.h | 2 ++ hw/net/virtio-net.c | 1 + hw/pci/pci.c | 3 +-- 3 files changed, 4 insertions(+), 2 deletions(-)include/hw/pci/pci.h +2
-0@@ -369,6 +369,8 @@ void pci_register_vga(PCIDevice *pci_dev, MemoryRegion *mem,void pci_unregister_vga (PCIDevice *pci_dev);pcibus_t pci_get_bar_addr (PCIDevice *pci_dev,int region_num); +void pci_del_option_rom (PCIDevice *pdev); +int pci_add_capability (PCIDevice *pdev,uint8_t cap_id,uint8_t offset,uint8_t size,Error **errp);hw/net/virtio-net.c +1
-0@@ -3256,6 +3256,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationState *s)if (migration_in_setup (s) && !should_be_hidden) {if (failover_unplug_primary (n, dev)) {vmstate_unregister (VMSTATE_IF (dev),qdev_get_vmsd (dev), dev); +pci_del_option_rom (PCI_DEVICE (dev));qapi_event_send_unplug_primary (dev->id);qatomic_set (&n->failover_primary_hidden, true); }else {hw/pci/pci.c +1
-2@@ -228,7 +228,6 @@ static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);static void pci_add_option_rom (PCIDevice *pdev,bool is_default_rom,Error **); -static void pci_del_option_rom (PCIDevice *pdev);static uint16_t pci_default_sub_vendor_id =PCI_SUBVENDOR_ID_REDHAT_QUMRANET ;static uint16_t pci_default_sub_device_id =PCI_SUBDEVICE_ID_QEMU ;@@ -2429,7 +2428,7 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,pci_register_bar (pdev,PCI_ROM_SLOT ,0 , &pdev->rom); } -static void pci_del_option_rom (PCIDevice *pdev) +void pci_del_option_rom (PCIDevice *pdev) {if (!pdev->has_rom)return ;Signature · 1 line
2.31.1
Quoted 36 lines · Michael S. Tsirkin
Quoted 23 lines · Laurent Vivier
The intend of failover is to allow to migrate a VM with a VFIO networking card without disrupting the network operation by switching to a virtio-net device during the migration.
This simple change allows to test failover with a simulated device like e1000e rather than a vfio device, even if it's useless in real life it can help to debug failover.
This is interesting to developers that want to test failover on a system with no vfio device. Moreover it simplifies host networking configuration as we can use the same bridge for virtio-net and the other failover networking device.
Without this change the migration of a system configured with failover fails with:
... -device virtio-net-pci,id=virtionet0,failover=on,... \ -device e1000,failover_pair_id=virtionet0,... \ ...(qemu) migrate ...
Quoted 8 lines
Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration error while loading state for instance
0x0of device 'ram' load of migration failed: Invalid argumentThis happens because QEMU correctly unregisters the interface vmstate but not the ROM one. This patch fixes that.
Signed-off-by Laurent Vivier <[email protected]>Build fails on qemu-system-m68k:
Quoted 7 lines
/usr/bin/ld:libqemu-m68k-softmmu.fa.p/hw_net_virtio-net.c.o: in function `virtio_net_handle_migration_primary':
/scm/qemu/build/../hw/net/virtio-net.c:3259: undefined reference to `pci_del_option_rom'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
make[1]: *** [Makefile:154: run-ninja] Error 1It's not pretty to poke at pci from generic virtio.
I agree with you. The problem is failover is implemented in virtio-net while it relies on virtio-pci hotplug capability.
Quoted 3 lines
Should we maybe wrap
vmstate_unregisterandpci_del_option_romto allow removing all migrateable things related to the device in one go somehow?
I'm going to have a look to see how to do.
I have already a patch that moves all the failover PCI unplug/hotplug functions into the PCI device implementation (and removes all this stuff from virtio-net). Perhaps we can rely on that. The bonus with this moves is it allows to automatically unplug/hotplug any PCI device during a migration without the failover environment (but failover uses it).
Thanks,
Laurent
Quoted 68 lines
Quoted 67 lines
---
Notes:
v4:
export and usepci_del_option_rom()v3:
remove useless space before commav2:
resethas_romto false
update commit log messageinclude/hw/pci/pci.h | 2 ++ hw/net/virtio-net.c | 1 + hw/pci/pci.c | 3 +-- 3 files changed, 4 insertions(+), 2 deletions(-)include/hw/pci/pci.h +2
-0@@ -369,6 +369,8 @@ void pci_register_vga(PCIDevice *pci_dev, MemoryRegion *mem,void pci_unregister_vga (PCIDevice *pci_dev);pcibus_t pci_get_bar_addr (PCIDevice *pci_dev,int region_num); +void pci_del_option_rom (PCIDevice *pdev); +int pci_add_capability (PCIDevice *pdev,uint8_t cap_id,uint8_t offset,uint8_t size,Error **errp);hw/net/virtio-net.c +1
-0@@ -3256,6 +3256,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationState *s)if (migration_in_setup (s) && !should_be_hidden) {if (failover_unplug_primary (n, dev)) {vmstate_unregister (VMSTATE_IF (dev),qdev_get_vmsd (dev), dev); +pci_del_option_rom (PCI_DEVICE (dev));qapi_event_send_unplug_primary (dev->id);qatomic_set (&n->failover_primary_hidden, true); }else {hw/pci/pci.c +1
-2@@ -228,7 +228,6 @@ static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);static void pci_add_option_rom (PCIDevice *pdev,bool is_default_rom,Error **); -static void pci_del_option_rom (PCIDevice *pdev);static uint16_t pci_default_sub_vendor_id =PCI_SUBVENDOR_ID_REDHAT_QUMRANET ;static uint16_t pci_default_sub_device_id =PCI_SUBDEVICE_ID_QEMU ;@@ -2429,7 +2428,7 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,pci_register_bar (pdev,PCI_ROM_SLOT ,0 , &pdev->rom); } -static void pci_del_option_rom (PCIDevice *pdev) +void pci_del_option_rom (PCIDevice *pdev) {if (!pdev->has_rom)return ;Signature · 1 line
2.31.1
Quoted 23 lines · Laurent Vivier
The intend of failover is to allow to migrate a VM with a VFIO networking card without disrupting the network operation by switching to a virtio-net device during the migration.
This simple change allows to test failover with a simulated device like e1000e rather than a vfio device, even if it's useless in real life it can help to debug failover.
This is interesting to developers that want to test failover on a system with no vfio device. Moreover it simplifies host networking configuration as we can use the same bridge for virtio-net and the other failover networking device.
Without this change the migration of a system configured with failover fails with:
... -device virtio-net-pci,id=virtionet0,failover=on,... \ -device e1000,failover_pair_id=virtionet0,... \ ...(qemu) migrate ...
Quoted 8 lines
Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration error while loading state for instance
0x0of device 'ram' load of migration failed: Invalid argumentThis happens because QEMU correctly unregisters the interface vmstate but not the ROM one. This patch fixes that.
Signed-off-by Laurent Vivier <[email protected]>
I began to wonder about this. Why are we sending the option ROM at all? I think there's no need to do it for the primary ...
Quoted 66 lines
---
Notes:
v4:
export and usepci_del_option_rom()v3:
remove useless space before commav2:
resethas_romto false
update commit log messageinclude/hw/pci/pci.h | 2 ++ hw/net/virtio-net.c | 1 + hw/pci/pci.c | 3 +-- 3 files changed, 4 insertions(+), 2 deletions(-)include/hw/pci/pci.h +2
-0@@ -369,6 +369,8 @@ void pci_register_vga(PCIDevice *pci_dev, MemoryRegion *mem,void pci_unregister_vga (PCIDevice *pci_dev);pcibus_t pci_get_bar_addr (PCIDevice *pci_dev,int region_num); +void pci_del_option_rom (PCIDevice *pdev); +int pci_add_capability (PCIDevice *pdev,uint8_t cap_id,uint8_t offset,uint8_t size,Error **errp);hw/net/virtio-net.c +1
-0@@ -3256,6 +3256,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationState *s)if (migration_in_setup (s) && !should_be_hidden) {if (failover_unplug_primary (n, dev)) {vmstate_unregister (VMSTATE_IF (dev),qdev_get_vmsd (dev), dev); +pci_del_option_rom (PCI_DEVICE (dev));qapi_event_send_unplug_primary (dev->id);qatomic_set (&n->failover_primary_hidden, true); }else {hw/pci/pci.c +1
-2@@ -228,7 +228,6 @@ static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);static void pci_add_option_rom (PCIDevice *pdev,bool is_default_rom,Error **); -static void pci_del_option_rom (PCIDevice *pdev);static uint16_t pci_default_sub_vendor_id =PCI_SUBVENDOR_ID_REDHAT_QUMRANET ;static uint16_t pci_default_sub_device_id =PCI_SUBDEVICE_ID_QEMU ;@@ -2429,7 +2428,7 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,pci_register_bar (pdev,PCI_ROM_SLOT ,0 , &pdev->rom); } -static void pci_del_option_rom (PCIDevice *pdev) +void pci_del_option_rom (PCIDevice *pdev) {if (!pdev->has_rom)return ;Signature · 1 line
2.31.1
Quoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Laurent Vivier
The intend of failover is to allow to migrate a VM with a VFIO networking card without disrupting the network operation by switching to a virtio-net device during the migration.
This simple change allows to test failover with a simulated device like e1000e rather than a vfio device, even if it's useless in real life it can help to debug failover.
This is interesting to developers that want to test failover on a system with no vfio device. Moreover it simplifies host networking configuration as we can use the same bridge for virtio-net and the other failover networking device.
Without this change the migration of a system configured with failover fails with:
... -device virtio-net-pci,id=virtionet0,failover=on,... \ -device e1000,failover_pair_id=virtionet0,... \ ...(qemu) migrate ...
Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration error while loading state for instance
0x0of device 'ram' load of migration failed: Invalid argument
Quoted 7 lines
Quoted 4 lines
This happens because QEMU correctly unregisters the interface vmstate but not the ROM one. This patch fixes that.
Signed-off-by Laurent Vivier <[email protected]>I began to wonder about this. Why are we sending the option ROM at all? I think there's no need to do it for the primary ...
I think there is no way to check the ROM is the same on source and destination, and it has to be.
By sending the ROM:
- we can check the size is the same, otherwise the migration fails,
- after the migration, even if a different ROM was provided on the destination side, the guest executes the ROM provided by the source.
Thanks,
Laurent