vfio/pci: migration: Skip config space check for Vendor Specific Information in VSC during restore/load
PATCH v3 1/1 vfio/pci: migration: Skip config space check for Vendor Specific Information in VSC during restore/load
In case of migration, during restore operation, qemu checks config space of the pci device with the config space in the migration stream captured during save operation. In case of config space data mismatch, restore operation is failed.
config space check is done in function get_pci_config_device(). By default VSC (vendor-specific-capability) in config space is checked.
Due to qemu's config space check for VSC, live migration is broken across NVIDIA vGPU devices in situation where source and destination host driver is different. In this situation, Vendor Specific Information in VSC varies on the destination to ensure vGPU feature capabilities exposed to the guest driver are compatible with destination host.
If a vfio-pci device is migration capable and vfio-pci vendor driver is OK with volatile Vendor Specific Info in VSC then qemu should exempt config space check for Vendor Specific Info. It is vendor driver's responsibility to ensure that VSC is consistent across migration. Here consistency could mean that VSC format should be same on source and destination, however actual Vendor Specific Info may not be byte-to-byte identical.
This patch skips the check for Vendor Specific Information in VSC for VFIO-PCI device by clearing pdev->cmask[] offsets. Config space check is still enforced for 3 byte VSC header. If cmask[] is not set for an offset, then qemu skips config space check for that offset.
---
Version Historyv2->v3:
- Config space check skipped only for Vendor Specific Info in VSC, check is still enforced for 3 byte VSC header.
- Updated commit description with live migration failure scenario.v1->v2:
- Limited scope of change to vfio-pci devices instead of all pci devices.
hw/vfio/pci.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
hw/vfio/pci.c +24 -0
@@ -2132,6 +2132,27 @@ static void vfio_check_af_flr(VFIOPCIDevice *vdev, uint8_t pos) } } +static int vfio_add_vendor_specific_cap (VFIOPCIDevice *vdev,int pos, +uint8_t size,Error **errp) +{ +PCIDevice *pdev = &vdev->pdev; + + pos =pci_add_capability (pdev,PCI_CAP_ID_VNDR , pos, size, errp); +if (pos <0 ) { +return pos; + } + +/* +* Exempt config space check for Vendor Specific Information during restore/load. +* Config space check is still enforced for 3 byte VSC header. +*/ +if (size >3 ) { +memset (pdev->cmask + pos +3 ,0 , size -3 ); + } + +return pos; +} +static int vfio_add_std_cap (VFIOPCIDevice *vdev,uint8_t pos,Error **errp) {PCIDevice *pdev = &vdev->pdev;
@@ -2199,6 +2220,9 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint8_t pos, Error **errp)vfio_check_af_flr (vdev, pos); ret =pci_add_capability (pdev, cap_id, pos, size, errp);break ; +case PCI_CAP_ID_VNDR : + ret =vfio_add_vendor_specific_cap (vdev, pos, size, errp); +break ;default : ret =pci_add_capability (pdev, cap_id, pos, size, errp);break ;
Signature · 1 line
2.34.1
Quoted 35 lines · On Fri, 22 Mar 2024 12:12:10 +0530 Vinayak Kale <[email protected]> wrote:
In case of migration, during restore operation, qemu checks config space of the pci device with the config space in the migration stream captured during save operation. In case of config space data mismatch, restore operation is failed.
config space check is done in function
get_pci_config_device(). By default VSC (vendor-specific-capability) in config space is checked.Due to qemu's config space check for VSC, live migration is broken across NVIDIA vGPU devices in situation where source and destination host driver is different. In this situation, Vendor Specific Information in VSC varies on the destination to ensure vGPU feature capabilities exposed to the guest driver are compatible with destination host.
If a vfio-pci device is migration capable and vfio-pci vendor driver is OK with volatile Vendor Specific Info in VSC then qemu should exempt config space check for Vendor Specific Info. It is vendor driver's responsibility to ensure that VSC is consistent across migration. Here consistency could mean that VSC format should be same on source and destination, however actual Vendor Specific Info may not be byte-to-byte identical.
This patch skips the check for Vendor Specific Information in VSC for VFIO-PCI device by clearing pdev->cmask[] offsets. Config space check is still enforced for 3 byte VSC header. If cmask[] is not set for an offset, then qemu skips config space check for that offset.
Signed-off-by Vinayak Kale <[email protected]>---
Version Historyv2->v3:
- Config space check skipped only for Vendor Specific Info in VSC, check is still enforced for 3 byte VSC header.
- Updated commit description with live migration failure scenario.v1->v2:
- Limited scope of change to vfio-pci devices instead of all pci devices.
Quoted 2 lines
hw/vfio/pci.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
Quoted 42 lines
hw/vfio/pci.c +24
-0@@ -2132,6 +2132,27 @@ static void vfio_check_af_flr(VFIOPCIDevice *vdev, uint8_t pos) } } +static int vfio_add_vendor_specific_cap (VFIOPCIDevice *vdev,int pos, +uint8_t size,Error **errp) +{ +PCIDevice *pdev = &vdev->pdev; + + pos =pci_add_capability (pdev,PCI_CAP_ID_VNDR , pos, size, errp); +if (pos <0 ) { +return pos; + } + +/* +* Exempt config space check for Vendor Specific Information during restore/load. +* Config space check is still enforced for 3 byte VSC header. +*/ +if (size >3 ) { +memset (pdev->cmask + pos +3 ,0 , size -3 ); + } + +return pos; +} +static int vfio_add_std_cap (VFIOPCIDevice *vdev,uint8_t pos,Error **errp) {PCIDevice *pdev = &vdev->pdev;@@ -2199,6 +2220,9 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint8_t pos, Error **errp)vfio_check_af_flr (vdev, pos); ret =pci_add_capability (pdev, cap_id, pos, size, errp);break ; +case PCI_CAP_ID_VNDR : + ret =vfio_add_vendor_specific_cap (vdev, pos, size, errp); +break ;default : ret =pci_add_capability (pdev, cap_id, pos, size, errp);break ;
Michael S. Tsirkin
Reviewed-byQuoted 38 lines · Alex Williamson
Quoted 35 lines · On Fri, 22 Mar 2024 12:12:10 +0530 Vinayak Kale <[email protected]> wrote:
In case of migration, during restore operation, qemu checks config space of the pci device with the config space in the migration stream captured during save operation. In case of config space data mismatch, restore operation is failed.
config space check is done in function
get_pci_config_device(). By default VSC (vendor-specific-capability) in config space is checked.Due to qemu's config space check for VSC, live migration is broken across NVIDIA vGPU devices in situation where source and destination host driver is different. In this situation, Vendor Specific Information in VSC varies on the destination to ensure vGPU feature capabilities exposed to the guest driver are compatible with destination host.
If a vfio-pci device is migration capable and vfio-pci vendor driver is OK with volatile Vendor Specific Info in VSC then qemu should exempt config space check for Vendor Specific Info. It is vendor driver's responsibility to ensure that VSC is consistent across migration. Here consistency could mean that VSC format should be same on source and destination, however actual Vendor Specific Info may not be byte-to-byte identical.
This patch skips the check for Vendor Specific Information in VSC for VFIO-PCI device by clearing pdev->cmask[] offsets. Config space check is still enforced for 3 byte VSC header. If cmask[] is not set for an offset, then qemu skips config space check for that offset.
Signed-off-by Vinayak Kale <[email protected]>---
Version Historyv2->v3:
- Config space check skipped only for Vendor Specific Info in VSC, check is still enforced for 3 byte VSC header.
- Updated commit description with live migration failure scenario.v1->v2:
- Limited scope of change to vfio-pci devices instead of all pci devices.
Quoted 5 lines
Quoted 2 lines
hw/vfio/pci.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)Acked-by Alex Williamson <[email protected]>
A very reasonable way to do it.
Merge through the VFIO tree I presume?
Quoted 43 lines
Quoted 43 lines
hw/vfio/pci.c +24
-0@@ -2132,6 +2132,27 @@ static void vfio_check_af_flr(VFIOPCIDevice *vdev, uint8_t pos) } } +static int vfio_add_vendor_specific_cap (VFIOPCIDevice *vdev,int pos, +uint8_t size,Error **errp) +{ +PCIDevice *pdev = &vdev->pdev; + + pos =pci_add_capability (pdev,PCI_CAP_ID_VNDR , pos, size, errp); +if (pos <0 ) { +return pos; + } + +/* +* Exempt config space check for Vendor Specific Information during restore/load. +* Config space check is still enforced for 3 byte VSC header. +*/ +if (size >3 ) { +memset (pdev->cmask + pos +3 ,0 , size -3 ); + } + +return pos; +} +static int vfio_add_std_cap (VFIOPCIDevice *vdev,uint8_t pos,Error **errp) {PCIDevice *pdev = &vdev->pdev;@@ -2199,6 +2220,9 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint8_t pos, Error **errp)vfio_check_af_flr (vdev, pos); ret =pci_add_capability (pdev, cap_id, pos, size, errp);break ; +case PCI_CAP_ID_VNDR : + ret =vfio_add_vendor_specific_cap (vdev, pos, size, errp); +break ;default : ret =pci_add_capability (pdev, cap_id, pos, size, errp);break ;
Quoted 43 lines · On Wed, 27 Mar 2024 16:11:37 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 42 lines · Alex Williamson
Quoted 37 lines · On Fri, 22 Mar 2024 12:12:10 +0530 Vinayak Kale <[email protected]> wrote:
In case of migration, during restore operation, qemu checks config space of the pci device with the config space in the migration stream captured during save operation. In case of config space data mismatch, restore operation is failed.
config space check is done in function
get_pci_config_device(). By default VSC (vendor-specific-capability) in config space is checked.Due to qemu's config space check for VSC, live migration is broken across NVIDIA vGPU devices in situation where source and destination host driver is different. In this situation, Vendor Specific Information in VSC varies on the destination to ensure vGPU feature capabilities exposed to the guest driver are compatible with destination host.
If a vfio-pci device is migration capable and vfio-pci vendor driver is OK with volatile Vendor Specific Info in VSC then qemu should exempt config space check for Vendor Specific Info. It is vendor driver's responsibility to ensure that VSC is consistent across migration. Here consistency could mean that VSC format should be same on source and destination, however actual Vendor Specific Info may not be byte-to-byte identical.
This patch skips the check for Vendor Specific Information in VSC for VFIO-PCI device by clearing pdev->cmask[] offsets. Config space check is still enforced for 3 byte VSC header. If cmask[] is not set for an offset, then qemu skips config space check for that offset.
Signed-off-by Vinayak Kale <[email protected]>---
Version Historyv2->v3:
- Config space check skipped only for Vendor Specific Info in VSC, check is still enforced for 3 byte VSC header.
- Updated commit description with live migration failure scenario.v1->v2:
- Limited scope of change to vfio-pci devices instead of all pci devices.hw/vfio/pci.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
Quoted 8 lines
Quoted 1 line
Acked-by Alex Williamson <[email protected]>A very reasonable way to do it.
Reviewed-by Michael S. Tsirkin <[email protected]>Merge through the VFIO tree I presume?
Yep, Cédric said he´d grab it for 9.1. Thanks,
Alex
Quoted 43 lines
Quoted 42 lines
Quoted 42 lines
hw/vfio/pci.c +24
-0@@ -2132,6 +2132,27 @@ static void vfio_check_af_flr(VFIOPCIDevice *vdev, uint8_t pos) } } +static int vfio_add_vendor_specific_cap (VFIOPCIDevice *vdev,int pos, +uint8_t size,Error **errp) +{ +PCIDevice *pdev = &vdev->pdev; + + pos =pci_add_capability (pdev,PCI_CAP_ID_VNDR , pos, size, errp); +if (pos <0 ) { +return pos; + } + +/* +* Exempt config space check for Vendor Specific Information during restore/load. +* Config space check is still enforced for 3 byte VSC header. +*/ +if (size >3 ) { +memset (pdev->cmask + pos +3 ,0 , size -3 ); + } + +return pos; +} +static int vfio_add_std_cap (VFIOPCIDevice *vdev,uint8_t pos,Error **errp) {PCIDevice *pdev = &vdev->pdev;@@ -2199,6 +2220,9 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint8_t pos, Error **errp)vfio_check_af_flr (vdev, pos); ret =pci_add_capability (pdev, cap_id, pos, size, errp);break ; +case PCI_CAP_ID_VNDR : + ret =vfio_add_vendor_specific_cap (vdev, pos, size, errp); +break ;default : ret =pci_add_capability (pdev, cap_id, pos, size, errp);break ;
Cédric Le Goater
AppliedQuoted 49 lines · Alex Williamson
Quoted 46 lines · On Wed, 27 Mar 2024 16:11:37 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 43 lines · Alex Williamson
Quoted 35 lines · On Fri, 22 Mar 2024 12:12:10 +0530 Vinayak Kale <[email protected]> wrote:
In case of migration, during restore operation, qemu checks config space of the pci device with the config space in the migration stream captured during save operation. In case of config space data mismatch, restore operation is failed.
config space check is done in function
get_pci_config_device(). By default VSC (vendor-specific-capability) in config space is checked.Due to qemu's config space check for VSC, live migration is broken across NVIDIA vGPU devices in situation where source and destination host driver is different. In this situation, Vendor Specific Information in VSC varies on the destination to ensure vGPU feature capabilities exposed to the guest driver are compatible with destination host.
If a vfio-pci device is migration capable and vfio-pci vendor driver is OK with volatile Vendor Specific Info in VSC then qemu should exempt config space check for Vendor Specific Info. It is vendor driver's responsibility to ensure that VSC is consistent across migration. Here consistency could mean that VSC format should be same on source and destination, however actual Vendor Specific Info may not be byte-to-byte identical.
This patch skips the check for Vendor Specific Information in VSC for VFIO-PCI device by clearing pdev->cmask[] offsets. Config space check is still enforced for 3 byte VSC header. If cmask[] is not set for an offset, then qemu skips config space check for that offset.
Signed-off-by Vinayak Kale <[email protected]>---
Version Historyv2->v3:
- Config space check skipped only for Vendor Specific Info in VSC, check is still enforced for 3 byte VSC header.
- Updated commit description with live migration failure scenario.v1->v2:
- Limited scope of change to vfio-pci devices instead of all pci devices.Quoted 2 lines
hw/vfio/pci.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)Acked-by Alex Williamson <[email protected]>
Quoted 7 lines
Quoted 5 lines
A very reasonable way to do it.
Reviewed-by Michael S. Tsirkin <[email protected]>Merge through the VFIO tree I presume?
Yep, Cédric said he´d grab it for 9.1. Thanks,
Applied to vfio-next.
Thanks,
C.
Hello Vinayak,
Quoted 49 lines · Cédric Le Goater
Quoted 48 lines · Alex Williamson
Quoted 45 lines · On Wed, 27 Mar 2024 16:11:37 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 42 lines · Alex Williamson
Quoted 35 lines · On Fri, 22 Mar 2024 12:12:10 +0530 Vinayak Kale <[email protected]> wrote:
In case of migration, during restore operation, qemu checks config space of the pci device with the config space in the migration stream captured during save operation. In case of config space data mismatch, restore operation is failed.
config space check is done in function
get_pci_config_device(). By default VSC (vendor-specific-capability) in config space is checked.Due to qemu's config space check for VSC, live migration is broken across NVIDIA vGPU devices in situation where source and destination host driver is different. In this situation, Vendor Specific Information in VSC varies on the destination to ensure vGPU feature capabilities exposed to the guest driver are compatible with destination host.
If a vfio-pci device is migration capable and vfio-pci vendor driver is OK with volatile Vendor Specific Info in VSC then qemu should exempt config space check for Vendor Specific Info. It is vendor driver's responsibility to ensure that VSC is consistent across migration. Here consistency could mean that VSC format should be same on source and destination, however actual Vendor Specific Info may not be byte-to-byte identical.
This patch skips the check for Vendor Specific Information in VSC for VFIO-PCI device by clearing pdev->cmask[] offsets. Config space check is still enforced for 3 byte VSC header. If cmask[] is not set for an offset, then qemu skips config space check for that offset.
Signed-off-by Vinayak Kale <[email protected]>---
Version Historyv2->v3:
- Config space check skipped only for Vendor Specific Info in VSC, check is still enforced for 3 byte VSC header.
- Updated commit description with live migration failure scenario.v1->v2:
- Limited scope of change to vfio-pci devices instead of all pci devices.Quoted 2 lines
hw/vfio/pci.c| 24 ++++++++++++++++++++++++1 file changed, 24 insertions(+)Acked-by Alex Williamson <[email protected]>
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
A very reasonable way to do it.
Reviewed-by Michael S. Tsirkin <[email protected]>Merge through the VFIO tree I presume?
Yep, Cédric said he´d grab it for 9.1. Thanks,
Could you please resend an update of this change adding a machine compatibility property for migration ?
Thanks,
C.
Quoted 55 lines · Cédric Le Goater
Hello Vinayak,
Quoted 51 lines · Cédric Le Goater
Quoted 50 lines · Alex Williamson
Quoted 47 lines · On Wed, 27 Mar 2024 16:11:37 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 37 lines · Alex Williamson
Quoted 35 lines · On Fri, 22 Mar 2024 12:12:10 +0530 Vinayak Kale <[email protected]> wrote:
In case of migration, during restore operation, qemu checks config space of the pci device with the config space in the migration stream captured during save operation. In case of config space data mismatch, restore operation is failed.
config space check is done in function
get_pci_config_device(). By default VSC (vendor-specific-capability) in config space is checked.Due to qemu's config space check for VSC, live migration is broken across NVIDIA vGPU devices in situation where source and destination host driver is different. In this situation, Vendor Specific Information in VSC varies on the destination to ensure vGPU feature capabilities exposed to the guest driver are compatible with destination host.
If a vfio-pci device is migration capable and vfio-pci vendor driver is OK with volatile Vendor Specific Info in VSC then qemu should exempt config space check for Vendor Specific Info. It is vendor driver's responsibility to ensure that VSC is consistent across migration. Here consistency could mean that VSC format should be same on source and destination, however actual Vendor Specific Info may not be byte-to-byte identical.
This patch skips the check for Vendor Specific Information in VSC for VFIO-PCI device by clearing pdev->cmask[] offsets. Config space check is still enforced for 3 byte VSC header. If cmask[] is not set for an offset, then qemu skips config space check for that offset.
Signed-off-by Vinayak Kale <[email protected]>---
Version Historyv2->v3:
- Config space check skipped only for Vendor Specific Info in VSC, check is still enforced for 3 byte VSC header.
- Updated commit description with live migration failure scenario.v1->v2:
- Limited scope of change to vfio-pci devices instead of all pci devices.Quoted 5 lines
Quoted 2 lines
hw/vfio/pci.c| 24 ++++++++++++++++++++++++1 file changed, 24 insertions(+)Acked-by Alex Williamson <[email protected]>A very reasonable way to do it.
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Reviewed-by Michael S. Tsirkin <[email protected]>Merge through the VFIO tree I presume?
Yep, Cédric said he´d grab it for 9.1. Thanks,
Could you please resend an update of this change adding a machine compatibility property for migration ?
Sure, I'll address this in V4.