Skip to contentSkip to conversation

Introduce vendor ops in vfio-pci

RFC PATCH In review Yan Zhao 7 participants 42 messages May 18, 2020 – Jun 22, 2020

Subsystems (4)
lore
Share & tools
lore

Yan Zhao

author
#0 lore raw

v4 cover Introduce vendor ops in vfio-pci

When using vfio-pci to pass through devices, though it's desired to use its default implementations in most of time, it is also sometimes necessary to call vendors specific operations.
For example, in order to do device live migration, the way of dirty pages detection and device state save-restore may be varied from device to device.
Vendors may want to add a vendor device region or may want to intercept writes to a BAR region.
So, in this series, we introduce a way to allow vendors to provide vendor specific ops for VFIO devices and meanwhile export several vfio-pci interfaces as default implementations to simplify code of vendor driver and avoid duplication.

Vendor driver registration/unregistration goes like this:
(1) macros are provided to let vendor drivers register/unregister vfio_pci_vendor_driver_ops to vfio_pci in their module_init() and module_exit().
vfio_pci_vendor_driver_ops contains callbacks probe() and remove() and a pointer to vfio_device_ops.

(2) vendor drivers define their module aliases as "vfio-pci:$vendor_id-$device_id".
E.g. A vendor module for VF devices of Intel(R) Ethernet Controller XL710 family can define its module alias as MODULE_ALIAS("vfio-pci:8086-154c").

(3) when module vfio_pci is bound to a device, it would call modprobe in user space for modules of alias "vfio-pci:$vendor_id-$device_id", which would trigger unloaded vendor drivers to register their vfio_pci_vendor_driver_ops to vfio_pci.
Then it searches registered ops list and calls probe() to test whether this vendor driver supports this physical device.
A success probe() would make bind vfio device to vendor provided vfio_device_ops, which would call exported default implementations in vfio_pci_ops if necessary.

                                        _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

__________ (un)register vendor ops | ___________ ___________ |

|          |<----------------------------|    VF    |   |           |   
| vfio-pci |                           | |  vendor  |   | PF driver |  |
|__________|---------------------------->|  driver  |   |___________|   
     |           probe/remove()        |  -----------          |       |
     |                                                         |         
     |                                 |_ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _|
    \|/                                                       \|/
-----------                                              ------------
|    VF   |                                              |    PF    |
-----------                                              ------------

a typical usage in SRIOV

Ref counts:
(1) vendor drivers must be a module and compiled to depend on module vfio_pci.
(2) In vfio_pci, a successful register would add refs of itself, and a successful unregister would derefs of itself.
(3) In vfio_pci, a successful probe() of a vendor driver would add ref of the vendor module. It derefs of the vendor module after calling remove().
(4) macro provided to make sure vendor module always unregister itself in its module_exit

Those are to prevent below conditions:
a. vfio_pci is unloaded after a successful register from vendor driver.
Though vfio_pci would later call modprobe to ask the vendor module to register again, it cannot help if vendor driver remain as loaded across unloading-loading of vfio_pci.
b. vendor driver unregisters itself after successfully probed by vfio_pci. c. circular dependency between vfio_pci and the vendor driver.
if vfio_pci adds refs to both vfio_pci and vendor driver on a successful register and if vendor driver only do the unregistration in its module_exit, then it would have no chance to do the unregistration.

Patch Overview
patches 1-2 provide register/unregister interfaces for vendor drivers
Patch 3 exports several members in vdev, including vendor_data, and
exports functions in vfio_pci_ops to allow them accessible from vendor drivers.
patches 4-5 export some more vdev members to vendor driver to simplify
their implementations.
patch 6 is from Tina Zhang to define vendor specific Irq type
capability.
patch 7 introduces a new vendor defined irq type
VFIO_IRQ_TYPE_REMAP_BAR_REGION.
patches 8-10
use VF live migration driver for Intel's 710 SRIOV devices as an example of how to implement this vendor ops interface.
patch 8 first let the vendor ops pass through VFs.
patch 9 implements a migration region based on migration protocol
defined in [1][2].
(Some dirty page tracking functions are intentionally commented out and would send out later in future.)
patch 10 serves as an example of how to define vendor specific irq
type. This irq will trigger qemu to dynamic map BAR regions in order to implement software based dirty page track.

Changelog:
RFC v3- RFC v4:
- use exported function to make vendor driver access internal fields of vdev rather than make struct vfio_pci_device public. (Alex)
- add a new interface vfio_pci_get_barmap() to call vfio_pci_setup_barma() and let vfio_pci_setup_barmap() still able to return detailed errno. (Alex)
- removed sample code to pass through igd devices. instead, use the first patch (patch 8/10) of i40e vf migration as an mere pass-through example.
- rebased code to 5.7 and VFIO migration kernel patches v17 and qemu patches v16.
- added a demo of vendor defined irq type.

RFC v2- RFC v3:
- embedding struct vfio_pci_device into struct vfio_pci_device_private. (Alex)

RFC v1- RFC v2:
- renamed mediate ops to vendor ops
- use of request_module and module alias to manage vendor driver load (Alex)
- changed from vfio_pci_ops calling vendor ops
to vendor ops calling default vfio_pci_ops (Alex)
- dropped patches for dynamic traps of BARs. will submit them later.

Links:
[1] VFIO migration kernel v17:

    https://patchwork.kernel.org/cover/11466129/
[2] VFIO migration qemu v16:
    https://patchwork.kernel.org/cover/11456557/

Previous versions:
RFC v3: https://lkml.org/lkml/2020/2/11/142

RFC v2: https://lkml.org/lkml/2020/1/30/956

RFC v1:
kernel part: https://www.spinics.net/lists/kernel/msg3337337.html.
qemu part: https://www.spinics.net/lists/kernel/msg3337337.html.

Tina Zhang (1):
vfio: Define device specific irq type capability

Yan Zhao (9):
vfio/pci: register/unregister vfio_pci_vendor_driver_ops vfio/pci: macros to generate module_init and module_exit for vendor
modules
vfio/pci: export vendor_data, irq_type, num_regions, pdev and
functions in vfio_pci_ops
vfio/pci: let vfio_pci know number of vendor regions and vendor irqs vfio/pci: export vfio_pci_get_barmap
vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION i40e/vf_migration: VF live migration - pass-through VF first i40e/vf_migration: register a migration vendor region i40e/vf_migration: vendor defined irq_type to support dynamic bar map

 drivers/net/ethernet/intel/Kconfig            |  10 +
 drivers/net/ethernet/intel/i40e/Makefile      |   2 +
 .../ethernet/intel/i40e/i40e_vf_migration.c   | 904 ++++++++++++++++++
 .../ethernet/intel/i40e/i40e_vf_migration.h   | 119 +++
 drivers/vfio/pci/vfio_pci.c                   | 181 +++-
 drivers/vfio/pci/vfio_pci_private.h           |   9 +
 drivers/vfio/pci/vfio_pci_rdwr.c              |  10 +
 include/linux/vfio.h                          |  58 ++
 include/uapi/linux/vfio.h                     |  30 +-
 9 files changed, 1311 insertions(+), 12 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.c
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.h
Signature · 1 line
2.17.1
author
#1 lore raw

RFC PATCH v4 1/10 vfio/pci: register/unregister vfio_pci_vendor_driver_ops

vfio_pci_vendor_driver_ops includes two parts:
(1) .probe() and .remove() interface to be called by vfio_pci_probe() and vfio_pci_remove().
(2) pointer to struct vfio_device_ops. It will be registered as ops of vfio device if .probe() succeeds.

---
 drivers/vfio/pci/vfio_pci.c         | 102 +++++++++++++++++++++++++++-
 drivers/vfio/pci/vfio_pci_private.h |   7 ++
 include/linux/vfio.h                |   9 +++
 3 files changed, 117 insertions(+), 1 deletion(-)
drivers/vfio/pci/vfio_pci.c +101 -1
index 6c6b37b5c04e..43d10d34cbc2 100644
@@ -68,6 +68,11 @@ static inline bool vfio_vga_disabled(void)
 #endif
 }
 
+static struct vfio_pci {
+	struct  mutex		vendor_drivers_lock;
+	struct  list_head	vendor_drivers_list;
+} vfio_pci;
+
 /*
  * Our VGA arbiter participation is limited since we don't know anything
  * about the device itself.  However, if the device is the only VGA device
@@ -1570,6 +1575,35 @@ static int vfio_pci_bus_notifier(struct notifier_block *nb,
 	return 0;
 }
 
+static int probe_vendor_drivers(struct vfio_pci_device *vdev)
+{
+	struct vfio_pci_vendor_driver *driver;
+	int ret = -ENODEV;
+
+	request_module("vfio-pci:%x-%x", vdev->pdev->vendor,
+					 vdev->pdev->device);
+
+	mutex_lock(&vfio_pci.vendor_drivers_lock);
+	list_for_each_entry(driver, &vfio_pci.vendor_drivers_list, next) {
+		void *data;
+
+		if (!try_module_get(driver->ops->owner))
+			continue;
+
+		data = driver->ops->probe(vdev->pdev);
+		if (IS_ERR(data)) {
+			module_put(driver->ops->owner);
+			continue;
+		}
+		vdev->vendor_driver = driver;
+		vdev->vendor_data = data;
+		ret = 0;
+		break;
+	}
+	mutex_unlock(&vfio_pci.vendor_drivers_lock);
+	return ret;
+}
+
 static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
 	struct vfio_pci_device *vdev;
@@ -1609,7 +1643,12 @@ static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	mutex_init(&vdev->ioeventfds_lock);
 	INIT_LIST_HEAD(&vdev->ioeventfds_list);
 
-	ret = vfio_add_group_dev(&pdev->dev, &vfio_pci_ops, vdev);
+	if (probe_vendor_drivers(vdev))
+		ret = vfio_add_group_dev(&pdev->dev, &vfio_pci_ops, vdev);
+	else
+		ret = vfio_add_group_dev(&pdev->dev,
+					 vdev->vendor_driver->ops->device_ops,
+					 vdev);
 	if (ret)
 		goto out_free;
 
@@ -1698,6 +1737,11 @@ static void vfio_pci_remove(struct pci_dev *pdev)
 	if (!disable_idle_d3)
 		vfio_pci_set_power_state(vdev, PCI_D0);
 
+	if (vdev->vendor_driver) {
+		vdev->vendor_driver->ops->remove(vdev->vendor_data);
+		module_put(vdev->vendor_driver->ops->owner);
+	}
+
 	kfree(vdev->pm_save);
 	kfree(vdev);
 
@@ -2035,6 +2079,8 @@ static int __init vfio_pci_init(void)
 
 	vfio_pci_fill_ids();
 
+	mutex_init(&vfio_pci.vendor_drivers_lock);
+	INIT_LIST_HEAD(&vfio_pci.vendor_drivers_list);
 	return 0;
 
 out_driver:
@@ -2042,6 +2088,60 @@ static int __init vfio_pci_init(void)
 	return ret;
 }
 
+int __vfio_pci_register_vendor_driver(struct vfio_pci_vendor_driver_ops *ops)
+{
+	struct vfio_pci_vendor_driver *driver, *tmp;
+
+	if (!ops || !ops->device_ops)
+		return -EINVAL;
+
+	driver = kzalloc(sizeof(*driver), GFP_KERNEL);
+	if (!driver)
+		return -ENOMEM;
+
+	driver->ops = ops;
+
+	mutex_lock(&vfio_pci.vendor_drivers_lock);
+
+	/* Check for duplicates */
+	list_for_each_entry(tmp, &vfio_pci.vendor_drivers_list, next) {
+		if (tmp->ops->device_ops == ops->device_ops) {
+			mutex_unlock(&vfio_pci.vendor_drivers_lock);
+			kfree(driver);
+			return -EINVAL;
+		}
+	}
+
+	list_add(&driver->next, &vfio_pci.vendor_drivers_list);
+
+	mutex_unlock(&vfio_pci.vendor_drivers_lock);
+
+	if (!try_module_get(THIS_MODULE))
+		return -ENODEV;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(__vfio_pci_register_vendor_driver);
+
+void vfio_pci_unregister_vendor_driver(struct vfio_device_ops *device_ops)
+{
+	struct vfio_pci_vendor_driver *driver, *tmp;
+
+	mutex_lock(&vfio_pci.vendor_drivers_lock);
+	list_for_each_entry_safe(driver, tmp,
+				 &vfio_pci.vendor_drivers_list, next) {
+		if (driver->ops->device_ops == device_ops) {
+			list_del(&driver->next);
+			mutex_unlock(&vfio_pci.vendor_drivers_lock);
+			kfree(driver);
+			module_put(THIS_MODULE);
+			return;
+		}
+	}
+	mutex_unlock(&vfio_pci.vendor_drivers_lock);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_unregister_vendor_driver);
+
 module_init(vfio_pci_init);
 module_exit(vfio_pci_cleanup);
 
drivers/vfio/pci/vfio_pci_private.h +7 -0
index 36ec69081ecd..7758a20546fa 100644
@@ -92,6 +92,11 @@ struct vfio_pci_vf_token {
 	int			users;
 };
 
+struct vfio_pci_vendor_driver {
+	const struct vfio_pci_vendor_driver_ops *ops;
+	struct list_head			next;
+};
+
 struct vfio_pci_device {
 	struct pci_dev		*pdev;
 	void __iomem		*barmap[PCI_STD_NUM_BARS];
@@ -132,6 +137,8 @@ struct vfio_pci_device {
 	struct list_head	ioeventfds_list;
 	struct vfio_pci_vf_token	*vf_token;
 	struct notifier_block	nb;
+	void			*vendor_data;
+	struct vfio_pci_vendor_driver	*vendor_driver;
 };
 
 #define is_intx(vdev) (vdev->irq_type == VFIO_PCI_INTX_IRQ_INDEX)
include/linux/vfio.h +9 -0
index 38d3c6a8dc7e..3e53deb012b6 100644
@@ -214,4 +214,13 @@ extern int vfio_virqfd_enable(void *opaque,
 			      void *data, struct virqfd **pvirqfd, int fd);
 extern void vfio_virqfd_disable(struct virqfd **pvirqfd);
 
+struct vfio_pci_vendor_driver_ops {
+	char			*name;
+	struct module		*owner;
+	void			*(*probe)(struct pci_dev *pdev);
+	void			(*remove)(void *vendor_data);
+	struct vfio_device_ops *device_ops;
+};
+int __vfio_pci_register_vendor_driver(struct vfio_pci_vendor_driver_ops *ops);
+void vfio_pci_unregister_vendor_driver(struct vfio_device_ops *device_ops);
 #endif /* VFIO_H */
Signature · 1 line
2.17.1
author
#2 lore raw

RFC PATCH v4 2/10 vfio/pci: macros to generate module_init and module_exit for vendor modules

In reply to Yan Zhao

vendor modules call macro module_vfio_pci_register_vendor_handler to generate module_init and module_exit.
It is necessary to ensure that vendor modules always call vfio_pci_register_vendor_driver() on driver loading and vfio_pci_unregister_vendor_driver on driver unloading,
because
(1) at compiling time, there's only a dependency of vendor modules on vfio_pci.
(2) at runtime,
- vendor modules add refs of vfio_pci on a successful calling of vfio_pci_register_vendor_driver() and deref of vfio_pci on a successful calling of vfio_pci_unregister_vendor_driver().
- vfio_pci only adds refs of vendor module on a successful probe of vendor driver.
vfio_pci derefs vendor module when unbinding from a device.

So, after vfio_pci is unbound from a device, the vendor module to that device is free to get unloaded. However, if that vendor module does not call vfio_pci_unregister_vendor_driver() in its module_exit, vfio_pci may hold a stale pointer to vendor module.

---
 include/linux/vfio.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
include/linux/vfio.h +27 -0
index 3e53deb012b6..f3746608c2d9 100644
@@ -223,4 +223,31 @@ struct vfio_pci_vendor_driver_ops {
 };
 int __vfio_pci_register_vendor_driver(struct vfio_pci_vendor_driver_ops *ops);
 void vfio_pci_unregister_vendor_driver(struct vfio_device_ops *device_ops);
+
+#define vfio_pci_register_vendor_driver(__name, __probe, __remove,	\
+					__device_ops)			\
+static struct vfio_pci_vendor_driver_ops  __ops ## _node = {		\
+	.owner		= THIS_MODULE,					\
+	.name		= __name,					\
+	.probe		= __probe,					\
+	.remove		= __remove,					\
+	.device_ops	= __device_ops,					\
+};									\
+__vfio_pci_register_vendor_driver(&__ops ## _node)
+
+#define module_vfio_pci_register_vendor_handler(name, probe, remove,	\
+						device_ops)		\
+static int __init device_ops ## _module_init(void)			\
+{									\
+	vfio_pci_register_vendor_driver(name, probe, remove,		\
+					device_ops);			\
+	return 0;							\
+};									\
+static void __exit device_ops ## _module_exit(void)			\
+{									\
+	vfio_pci_unregister_vendor_driver(device_ops);			\
+};									\
+module_init(device_ops ## _module_init);				\
+module_exit(device_ops ## _module_exit)
+
 #endif /* VFIO_H */
Signature · 1 line
2.17.1
#3 lore raw

Re: [RFC PATCH v4 02/10] vfio/pci: macros to generate module_init and module_exit for vendor modules

Quoted 44 lines · On Sun, 17 May 2020 22:45:10 -0400 Yan Zhao <[email protected]> wrote:

vendor modules call macro module_vfio_pci_register_vendor_handler to generate module_init and module_exit.
It is necessary to ensure that vendor modules always call vfio_pci_register_vendor_driver() on driver loading and vfio_pci_unregister_vendor_driver on driver unloading,
because
(1) at compiling time, there's only a dependency of vendor modules on vfio_pci.
(2) at runtime,
- vendor modules add refs of vfio_pci on a successful calling of vfio_pci_register_vendor_driver() and deref of vfio_pci on a successful calling of vfio_pci_unregister_vendor_driver().
- vfio_pci only adds refs of vendor module on a successful probe of vendor driver.
vfio_pci derefs vendor module when unbinding from a device.

So, after vfio_pci is unbound from a device, the vendor module to that device is free to get unloaded. However, if that vendor module does not call vfio_pci_unregister_vendor_driver() in its module_exit, vfio_pci may hold a stale pointer to vendor module.

---
 include/linux/vfio.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
include/linux/vfio.h +8 -0
index 3e53deb012b6..f3746608c2d9 100644
@@ -223,4 +223,31 @@ struct vfio_pci_vendor_driver_ops {
 };
 int __vfio_pci_register_vendor_driver(struct vfio_pci_vendor_driver_ops *ops);
 void vfio_pci_unregister_vendor_driver(struct vfio_device_ops *device_ops);
+
+#define vfio_pci_register_vendor_driver(__name, __probe, __remove,	\
+					__device_ops)			\
+static struct vfio_pci_vendor_driver_ops  __ops ## _node = {		\
+	.owner		= THIS_MODULE,					\
+	.name		= __name,					\
+	.probe		= __probe,					\
+	.remove		= __remove,					\

include/linux/vfio.h struct vfio_pci_vendor_driver_ops { · line 243

Quoted 10 lines
include/linux/vfio.h +10 -0
+	.device_ops	= __device_ops,					\
+};									\
+__vfio_pci_register_vendor_driver(&__ops ## _node)
+
+#define module_vfio_pci_register_vendor_handler(name, probe, remove,	\
+						device_ops)		\
+static int __init device_ops ## _module_init(void)			\
+{									\
+	vfio_pci_register_vendor_driver(name, probe, remove,		\
+					device_ops);			\

What if this function fails (e.g. with -ENOMEM)?

Quoted 10 lines
include/linux/vfio.h +9 -0
+	return 0;							\
+};									\
+static void __exit device_ops ## _module_exit(void)			\
+{									\
+	vfio_pci_unregister_vendor_driver(device_ops);			\
+};									\
+module_init(device_ops ## _module_init);				\
+module_exit(device_ops ## _module_exit)
+
 #endif /* VFIO_H */
author
#4 lore raw

Re: [RFC PATCH v4 02/10] vfio/pci: macros to generate module_init and module_exit for vendor modules

Quoted 51 lines · Cornelia Huck
Quoted 48 lines · On Sun, 17 May 2020 22:45:10 -0400 Yan Zhao <[email protected]> wrote:

vendor modules call macro module_vfio_pci_register_vendor_handler to generate module_init and module_exit.
It is necessary to ensure that vendor modules always call vfio_pci_register_vendor_driver() on driver loading and vfio_pci_unregister_vendor_driver on driver unloading,
because
(1) at compiling time, there's only a dependency of vendor modules on vfio_pci.
(2) at runtime,
- vendor modules add refs of vfio_pci on a successful calling of vfio_pci_register_vendor_driver() and deref of vfio_pci on a successful calling of vfio_pci_unregister_vendor_driver().
- vfio_pci only adds refs of vendor module on a successful probe of vendor driver.
vfio_pci derefs vendor module when unbinding from a device.

So, after vfio_pci is unbound from a device, the vendor module to that device is free to get unloaded. However, if that vendor module does not call vfio_pci_unregister_vendor_driver() in its module_exit, vfio_pci may hold a stale pointer to vendor module.

---
 include/linux/vfio.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
include/linux/vfio.h +12 -0
index 3e53deb012b6..f3746608c2d9 100644
@@ -223,4 +223,31 @@ struct vfio_pci_vendor_driver_ops {
 };
 int __vfio_pci_register_vendor_driver(struct vfio_pci_vendor_driver_ops *ops);
 void vfio_pci_unregister_vendor_driver(struct vfio_device_ops *device_ops);
+
+#define vfio_pci_register_vendor_driver(__name, __probe, __remove,	\
+					__device_ops)			\
+static struct vfio_pci_vendor_driver_ops  __ops ## _node = {		\
+	.owner		= THIS_MODULE,					\
+	.name		= __name,					\
+	.probe		= __probe,					\
+	.remove		= __remove,					\
+	.device_ops	= __device_ops,					\
+};									\
+__vfio_pci_register_vendor_driver(&__ops ## _node)
+
Quoted 9 lines

include/linux/vfio.h struct vfio_pci_vendor_driver_ops { · line 243

Quoted 6 lines
include/linux/vfio.h +6 -0
+#define module_vfio_pci_register_vendor_handler(name, probe, remove,	\
+						device_ops)		\
+static int __init device_ops ## _module_init(void)			\
+{									\
+	vfio_pci_register_vendor_driver(name, probe, remove,		\
+					device_ops);			\

What if this function fails (e.g. with -ENOMEM)?

right. I need to return error in that case.

Thanks for pointing it out!

Yan

Quoted 11 lines
Quoted 10 lines
include/linux/vfio.h +9 -0
+	return 0;							\
+};									\
+static void __exit device_ops ## _module_exit(void)			\
+{									\
+	vfio_pci_unregister_vendor_driver(device_ops);			\
+};									\
+module_init(device_ops ## _module_init);				\
+module_exit(device_ops ## _module_exit)
+
 #endif /* VFIO_H */
author
#5 lore raw

RFC PATCH v4 3/10 vfio/pci: export vendor_data, irq_type, num_regions, pdev and functions in vfio_pci_ops

In reply to Yan Zhao

export functions vfio_pci_vendor_data(), vfio_pci_irq_type(), vfio_pci_num_regions(), vfio_pci_pdev(), and functions in vfio_pci_ops, so they are able to be called from outside modules and make them a kind of inherited by vfio_device_ops provided by vendor modules

---
 drivers/vfio/pci/vfio_pci.c | 56 +++++++++++++++++++++++++++++++------
 include/linux/vfio.h        | 18 ++++++++++++
 2 files changed, 66 insertions(+), 8 deletions(-)
drivers/vfio/pci/vfio_pci.c +48 -8
index 43d10d34cbc2..290b7ab55ecf 100644
@@ -73,6 +73,38 @@ static struct vfio_pci {
 	struct  list_head	vendor_drivers_list;
 } vfio_pci;
 
+struct pci_dev *vfio_pci_pdev(void *device_data)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	return vdev->pdev;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_pdev);
+
+int vfio_pci_num_regions(void *device_data)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	return vdev->num_regions;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_num_regions);
+
+int vfio_pci_irq_type(void *device_data)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	return vdev->irq_type;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_irq_type);
+
+void *vfio_pci_vendor_data(void *device_data)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	return vdev->vendor_data;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_vendor_data);
+
 /*
  * Our VGA arbiter participation is limited since we don't know anything
  * about the device itself.  However, if the device is the only VGA device
@@ -514,7 +546,7 @@ static void vfio_pci_vf_token_user_add(struct vfio_pci_device *vdev, int val)
 	vfio_device_put(pf_dev);
 }
 
-static void vfio_pci_release(void *device_data)
+void vfio_pci_release(void *device_data)
 {
 	struct vfio_pci_device *vdev = device_data;
 
@@ -530,8 +562,9 @@ static void vfio_pci_release(void *device_data)
 
 	module_put(THIS_MODULE);
 }
+EXPORT_SYMBOL_GPL(vfio_pci_release);
 
-static int vfio_pci_open(void *device_data)
+int vfio_pci_open(void *device_data)
 {
 	struct vfio_pci_device *vdev = device_data;
 	int ret = 0;
@@ -556,6 +589,7 @@ static int vfio_pci_open(void *device_data)
 		module_put(THIS_MODULE);
 	return ret;
 }
+EXPORT_SYMBOL_GPL(vfio_pci_open);
 
 static int vfio_pci_get_irq_count(struct vfio_pci_device *vdev, int irq_type)
 {
@@ -741,7 +775,7 @@ int vfio_pci_register_dev_region(struct vfio_pci_device *vdev,
 	return 0;
 }
 
-static long vfio_pci_ioctl(void *device_data,
+long vfio_pci_ioctl(void *device_data,
 			   unsigned int cmd, unsigned long arg)
 {
 	struct vfio_pci_device *vdev = device_data;
@@ -1253,6 +1287,7 @@ static long vfio_pci_ioctl(void *device_data,
 
 	return -ENOTTY;
 }
+EXPORT_SYMBOL_GPL(vfio_pci_ioctl);
 
 static ssize_t vfio_pci_rw(void *device_data, char __user *buf,
 			   size_t count, loff_t *ppos, bool iswrite)
@@ -1286,7 +1321,7 @@ static ssize_t vfio_pci_rw(void *device_data, char __user *buf,
 	return -EINVAL;
 }
 
-static ssize_t vfio_pci_read(void *device_data, char __user *buf,
+ssize_t vfio_pci_read(void *device_data, char __user *buf,
 			     size_t count, loff_t *ppos)
 {
 	if (!count)
@@ -1294,8 +1329,9 @@ static ssize_t vfio_pci_read(void *device_data, char __user *buf,
 
 	return vfio_pci_rw(device_data, buf, count, ppos, false);
 }
+EXPORT_SYMBOL_GPL(vfio_pci_read);
 
-static ssize_t vfio_pci_write(void *device_data, const char __user *buf,
+ssize_t vfio_pci_write(void *device_data, const char __user *buf,
 			      size_t count, loff_t *ppos)
 {
 	if (!count)
@@ -1303,8 +1339,9 @@ static ssize_t vfio_pci_write(void *device_data, const char __user *buf,
 
 	return vfio_pci_rw(device_data, (char __user *)buf, count, ppos, true);
 }
+EXPORT_SYMBOL_GPL(vfio_pci_write);
 
-static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma)
+int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma)
 {
 	struct vfio_pci_device *vdev = device_data;
 	struct pci_dev *pdev = vdev->pdev;
@@ -1365,8 +1402,9 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma)
 	return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
 			       req_len, vma->vm_page_prot);
 }
+EXPORT_SYMBOL_GPL(vfio_pci_mmap);
 
-static void vfio_pci_request(void *device_data, unsigned int count)
+void vfio_pci_request(void *device_data, unsigned int count)
 {
 	struct vfio_pci_device *vdev = device_data;
 	struct pci_dev *pdev = vdev->pdev;
@@ -1386,6 +1424,7 @@ static void vfio_pci_request(void *device_data, unsigned int count)
 
 	mutex_unlock(&vdev->igate);
 }
+EXPORT_SYMBOL_GPL(vfio_pci_request);
 
 static int vfio_pci_validate_vf_token(struct vfio_pci_device *vdev,
 				      bool vf_token, uuid_t *uuid)
@@ -1482,7 +1521,7 @@ static int vfio_pci_validate_vf_token(struct vfio_pci_device *vdev,
 
 #define VF_TOKEN_ARG "vf_token="
 
-static int vfio_pci_match(void *device_data, char *buf)
+int vfio_pci_match(void *device_data, char *buf)
 {
 	struct vfio_pci_device *vdev = device_data;
 	bool vf_token = false;
@@ -1530,6 +1569,7 @@ static int vfio_pci_match(void *device_data, char *buf)
 
 	return 1; /* Match */
 }
+EXPORT_SYMBOL_GPL(vfio_pci_match);
 
 static const struct vfio_device_ops vfio_pci_ops = {
 	.name		= "vfio-pci",
include/linux/vfio.h +18 -0
index f3746608c2d9..6ededceb1964 100644
@@ -214,6 +214,24 @@ extern int vfio_virqfd_enable(void *opaque,
 			      void *data, struct virqfd **pvirqfd, int fd);
 extern void vfio_virqfd_disable(struct virqfd **pvirqfd);
 
+extern int vfio_pci_irq_type(void *device_data);
+extern int vfio_pci_num_regions(void *device_data);
+extern struct pci_dev *vfio_pci_pdev(void *device_data);
+
+extern long vfio_pci_ioctl(void *device_data,
+			   unsigned int cmd, unsigned long arg);
+extern ssize_t vfio_pci_read(void *device_data, char __user *buf,
+			     size_t count, loff_t *ppos);
+extern ssize_t vfio_pci_write(void *device_data, const char __user *buf,
+			      size_t count, loff_t *ppos);
+extern int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma);
+extern void vfio_pci_request(void *device_data, unsigned int count);
+extern int vfio_pci_open(void *device_data);
+extern void vfio_pci_release(void *device_data);
+extern int vfio_pci_match(void *device_data, char *buf);
+
+extern void *vfio_pci_vendor_data(void *device_data);
+
 struct vfio_pci_vendor_driver_ops {
 	char			*name;
 	struct module		*owner;
Signature · 1 line
2.17.1
author
#6 lore raw

RFC PATCH v4 4/10 vfio/pci: let vfio_pci know number of vendor regions and vendor irqs

In reply to Yan Zhao

This allows a simpler VFIO_DEVICE_GET_INFO ioctl in vendor driver

---
 drivers/vfio/pci/vfio_pci.c         | 23 +++++++++++++++++++++--
 drivers/vfio/pci/vfio_pci_private.h |  2 ++
 include/linux/vfio.h                |  3 +++
 3 files changed, 26 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci.c +21 -2
index 290b7ab55ecf..30137c1c5308 100644
@@ -105,6 +105,24 @@ void *vfio_pci_vendor_data(void *device_data)
 }
 EXPORT_SYMBOL_GPL(vfio_pci_vendor_data);
 
+int vfio_pci_set_vendor_regions(void *device_data, int num_vendor_regions)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	vdev->num_vendor_regions = num_vendor_regions;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_set_vendor_regions);
+
+
+int vfio_pci_set_vendor_irqs(void *device_data, int num_vendor_irqs)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	vdev->num_vendor_irqs = num_vendor_irqs;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_set_vendor_irqs);
 /*
  * Our VGA arbiter participation is limited since we don't know anything
  * about the device itself.  However, if the device is the only VGA device
@@ -797,8 +815,9 @@ long vfio_pci_ioctl(void *device_data,
 		if (vdev->reset_works)
 			info.flags |= VFIO_DEVICE_FLAGS_RESET;
 
-		info.num_regions = VFIO_PCI_NUM_REGIONS + vdev->num_regions;
-		info.num_irqs = VFIO_PCI_NUM_IRQS;
+		info.num_regions = VFIO_PCI_NUM_REGIONS + vdev->num_regions +
+						vdev->num_vendor_regions;
+		info.num_irqs = VFIO_PCI_NUM_IRQS + vdev->num_vendor_irqs;
 
 		return copy_to_user((void __user *)arg, &info, minsz) ?
 			-EFAULT : 0;
drivers/vfio/pci/vfio_pci_private.h +2 -0
index 7758a20546fa..c6cfc4605987 100644
@@ -110,6 +110,8 @@ struct vfio_pci_device {
 	int			num_ctx;
 	int			irq_type;
 	int			num_regions;
+	int			num_vendor_regions;
+	int			num_vendor_irqs;
 	struct vfio_pci_region	*region;
 	u8			msi_qmax;
 	u8			msix_bar;
include/linux/vfio.h +3 -0
index 6ededceb1964..6310c53f9d36 100644
@@ -231,6 +231,9 @@ extern void vfio_pci_release(void *device_data);
 extern int vfio_pci_match(void *device_data, char *buf);
 
 extern void *vfio_pci_vendor_data(void *device_data);
+extern int vfio_pci_set_vendor_regions(void *device_data,
+				       int num_vendor_regions);
+extern int vfio_pci_set_vendor_irqs(void *device_data, int num_vendor_irqs);
 
 struct vfio_pci_vendor_driver_ops {
 	char			*name;
Signature · 1 line
2.17.1
#7 lore raw

Re: [RFC PATCH v4 04/10] vfio/pci: let vfio_pci know number of vendor regions and vendor irqs

Quoted 15 lines · On Sun, 17 May 2020 22:49:44 -0400 Yan Zhao <[email protected]> wrote:

This allows a simpler VFIO_DEVICE_GET_INFO ioctl in vendor driver

---
 drivers/vfio/pci/vfio_pci.c         | 23 +++++++++++++++++++++--
 drivers/vfio/pci/vfio_pci_private.h |  2 ++
 include/linux/vfio.h                |  3 +++
 3 files changed, 26 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci.c +0 -0
index 290b7ab55ecf..30137c1c5308 100644

drivers/vfio/pci/vfio_pci.c void *vfio_pci_vendor_data(void *device_data) · line 112

Quoted 8 lines
drivers/vfio/pci/vfio_pci.c +5 -0
@@ -105,6 +105,24 @@ void *vfio_pci_vendor_data(void *device_data)
 }
 EXPORT_SYMBOL_GPL(vfio_pci_vendor_data);
 
+int vfio_pci_set_vendor_regions(void *device_data, int num_vendor_regions)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	vdev->num_vendor_regions = num_vendor_regions;

Do we need any kind of sanity check here, in case this is called with a bogus value?

drivers/vfio/pci/vfio_pci.c void *vfio_pci_vendor_data(void *device_data) · line 122

Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +10 -0
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_set_vendor_regions);
+
+
+int vfio_pci_set_vendor_irqs(void *device_data, int num_vendor_irqs)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	vdev->num_vendor_irqs = num_vendor_irqs;

Here as well.

Quoted 6 lines
drivers/vfio/pci/vfio_pci.c +3 -0
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_set_vendor_irqs);
 /*
  * Our VGA arbiter participation is limited since we don't know anything
  * about the device itself.  However, if the device is the only VGA device

(...)

author
#8 lore raw

Re: [RFC PATCH v4 04/10] vfio/pci: let vfio_pci know number of vendor regions and vendor irqs

Quoted 21 lines · Cornelia Huck
Quoted 18 lines · On Sun, 17 May 2020 22:49:44 -0400 Yan Zhao <[email protected]> wrote:

This allows a simpler VFIO_DEVICE_GET_INFO ioctl in vendor driver

---
 drivers/vfio/pci/vfio_pci.c         | 23 +++++++++++++++++++++--
 drivers/vfio/pci/vfio_pci_private.h |  2 ++
 include/linux/vfio.h                |  3 +++
 3 files changed, 26 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci.c +0 -0
index 290b7ab55ecf..30137c1c5308 100644
@@ -105,6 +105,24 @@ void *vfio_pci_vendor_data(void *device_data)
 }
 EXPORT_SYMBOL_GPL(vfio_pci_vendor_data);
 
Quoted 9 lines

drivers/vfio/pci/vfio_pci.c void *vfio_pci_vendor_data(void *device_data) · line 112

Quoted 5 lines
drivers/vfio/pci/vfio_pci.c +5 -0
+int vfio_pci_set_vendor_regions(void *device_data, int num_vendor_regions)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	vdev->num_vendor_regions = num_vendor_regions;

Do we need any kind of sanity check here, in case this is called with a bogus value?

you are right. it at least needs to be >=0.
maybe type of "unsigned int" is more appropriate for num_vendor_regions. we don't need to check its max value as QEMU would check it.

Quoted 4 lines
Quoted 4 lines
drivers/vfio/pci/vfio_pci.c +4 -0
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_set_vendor_regions);
+
Quoted 8 lines

drivers/vfio/pci/vfio_pci.c void *vfio_pci_vendor_data(void *device_data) · line 122

Quoted 6 lines
drivers/vfio/pci/vfio_pci.c +6 -0
+
+int vfio_pci_set_vendor_irqs(void *device_data, int num_vendor_irqs)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	vdev->num_vendor_irqs = num_vendor_irqs;

Here as well.

yes. will change the type to "unsigned int".
Thank you for kindly reviewing:)

Yan

Quoted 10 lines
Quoted 7 lines
drivers/vfio/pci/vfio_pci.c +3 -0
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_set_vendor_irqs);
 /*
  * Our VGA arbiter participation is limited since we don't know anything
  * about the device itself.  However, if the device is the only VGA device

(...)

#9 lore raw

Re: [RFC PATCH v4 04/10] vfio/pci: let vfio_pci know number of vendor regions and vendor irqs

Quoted 31 lines · Yan Zhao
Quoted 21 lines · Cornelia Huck
Quoted 18 lines · On Sun, 17 May 2020 22:49:44 -0400 Yan Zhao <[email protected]> wrote:

This allows a simpler VFIO_DEVICE_GET_INFO ioctl in vendor driver

---
 drivers/vfio/pci/vfio_pci.c         | 23 +++++++++++++++++++++--
 drivers/vfio/pci/vfio_pci_private.h |  2 ++
 include/linux/vfio.h                |  3 +++
 3 files changed, 26 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci.c +0 -0
index 290b7ab55ecf..30137c1c5308 100644
@@ -105,6 +105,24 @@ void *vfio_pci_vendor_data(void *device_data)
 }
 EXPORT_SYMBOL_GPL(vfio_pci_vendor_data);
 
Quoted 9 lines

drivers/vfio/pci/vfio_pci.c void *vfio_pci_vendor_data(void *device_data) · line 112

Quoted 5 lines
drivers/vfio/pci/vfio_pci.c +5 -0
+int vfio_pci_set_vendor_regions(void *device_data, int num_vendor_regions)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	vdev->num_vendor_regions = num_vendor_regions;

Do we need any kind of sanity check here, in case this is called with a bogus value?

Quoted 3 lines

you are right. it at least needs to be >=0.
maybe type of "unsigned int" is more appropriate for num_vendor_regions. we don't need to check its max value as QEMU would check it.

That seems like a bad precedent - the caller may not be QEMU.

dme.

Signature · 1 line
I'm not the reason you're looking for redemption.
author
#10 lore raw

Re: [RFC PATCH v4 04/10] vfio/pci: let vfio_pci know number of vendor regions and vendor irqs

Quoted 33 lines · David Edmondson
Quoted 31 lines · Yan Zhao
Quoted 30 lines · Cornelia Huck
Quoted 15 lines · On Sun, 17 May 2020 22:49:44 -0400 Yan Zhao <[email protected]> wrote:

This allows a simpler VFIO_DEVICE_GET_INFO ioctl in vendor driver

---
 drivers/vfio/pci/vfio_pci.c         | 23 +++++++++++++++++++++--
 drivers/vfio/pci/vfio_pci_private.h |  2 ++
 include/linux/vfio.h                |  3 +++
 3 files changed, 26 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci.c +0 -0
index 290b7ab55ecf..30137c1c5308 100644

drivers/vfio/pci/vfio_pci.c void *vfio_pci_vendor_data(void *device_data) · line 112

Quoted 8 lines
drivers/vfio/pci/vfio_pci.c +5 -0
@@ -105,6 +105,24 @@ void *vfio_pci_vendor_data(void *device_data)
 }
 EXPORT_SYMBOL_GPL(vfio_pci_vendor_data);
 
+int vfio_pci_set_vendor_regions(void *device_data, int num_vendor_regions)
+{
+	struct vfio_pci_device *vdev = device_data;
+
+	vdev->num_vendor_regions = num_vendor_regions;

Do we need any kind of sanity check here, in case this is called with a bogus value?

Quoted 6 lines
Quoted 3 lines

you are right. it at least needs to be >=0.
maybe type of "unsigned int" is more appropriate for num_vendor_regions. we don't need to check its max value as QEMU would check it.

That seems like a bad precedent - the caller may not be QEMU.

but the caller has to query that through vfio_pci_ioctl() and at there info.num_regions = VFIO_PCI_NUM_REGIONS + vdev->num_regions + vdev->num_vendor_regions;

info.num_regions is of type __u32.

Thanks
Yan

author
#11 lore raw

RFC PATCH v4 5/10 vfio/pci: export vfio_pci_get_barmap

In reply to Yan Zhao

This allows vendor driver to read/write to bars directly which is useful in security checking condition.

---
 drivers/vfio/pci/vfio_pci_rdwr.c | 10 ++++++++++
 include/linux/vfio.h             |  1 +
 2 files changed, 11 insertions(+)
drivers/vfio/pci/vfio_pci_rdwr.c +10 -0
index a87992892a9f..e4085311ab28 100644
@@ -153,6 +153,16 @@ static int vfio_pci_setup_barmap(struct vfio_pci_device *vdev, int bar)
 	return 0;
 }
 
+void __iomem *vfio_pci_get_barmap(void *device_data, int bar)
+{
+	int ret;
+	struct vfio_pci_device *vdev = device_data;
+
+	ret = vfio_pci_setup_barmap(vdev, bar);
+	return ret ? ERR_PTR(ret) : vdev->barmap[bar];
+}
+EXPORT_SYMBOL_GPL(vfio_pci_get_barmap);
+
 ssize_t vfio_pci_bar_rw(struct vfio_pci_device *vdev, char __user *buf,
 			size_t count, loff_t *ppos, bool iswrite)
 {
include/linux/vfio.h +1 -0
index 6310c53f9d36..0c786fec4602 100644
@@ -234,6 +234,7 @@ extern void *vfio_pci_vendor_data(void *device_data);
 extern int vfio_pci_set_vendor_regions(void *device_data,
 				       int num_vendor_regions);
 extern int vfio_pci_set_vendor_irqs(void *device_data, int num_vendor_irqs);
+extern void __iomem *vfio_pci_get_barmap(void *device_data, int bar);
 
 struct vfio_pci_vendor_driver_ops {
 	char			*name;
Signature · 1 line
2.17.1
#12 lore raw

Re: [RFC PATCH v4 05/10] vfio/pci: export vfio_pci_get_barmap

[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Yan,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on vfio/next]
[also build test WARNING on jkirsher-next-queue/dev-queue linus/master v5.7-rc6 next-20200515] [cannot apply to linux/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Yan-Zhao/Introduce-vendor-ops-in-vfio-pci/20200518-110542
base:   https://github.com/awilliam/linux-vfio.git next
config: x86_64-allyesconfig (attached as .config)
reproduce:
        # apt-get install sparse
        # sparse version: v0.6.1-193-gb8fad4bc-dirty
        # save the attached .config to linux build tree
        make C=1 ARCH=x86_64 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__'

If you fix the issue, kindly add following tag as appropriate

sparse warnings: (new ones prefixed by >>)

>> drivers/vfio/pci/vfio_pci_rdwr.c:162:20: sparse: sparse: incompatible types in conditional expression (different address spaces):
>> drivers/vfio/pci/vfio_pci_rdwr.c:162:20: sparse:    void *
>> drivers/vfio/pci/vfio_pci_rdwr.c:162:20: sparse:    void [noderef] <asn:2> *

vim +162 drivers/vfio/pci/vfio_pci_rdwr.c

155
156 void __iomem *vfio_pci_get_barmap(void *device_data, int bar)
157 {
158 int ret;
159 struct vfio_pci_device *vdev = device_data;
160
161 ret = vfio_pci_setup_barmap(vdev, bar);

 > 162		return ret ? ERR_PTR(ret) : vdev->barmap[bar];

163 }
164 EXPORT_SYMBOL_GPL(vfio_pci_get_barmap);
165

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

author
#13 lore raw

RFC PATCH v4 6/10 vfio: Define device specific irq type capability

In reply to Yan Zhao

From: Tina Zhang <[email protected]>

Cap the number of irqs with fixed indexes and use capability chains to chain device specific irqs.

v2:
- Irq capability index starts from 1.

---
 include/uapi/linux/vfio.h | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
include/uapi/linux/vfio.h +18 -1
index 0fe7c9a6f211..2d0d85c7c4d4 100644
@@ -683,11 +683,27 @@ struct vfio_irq_info {
 #define VFIO_IRQ_INFO_MASKABLE		(1 << 1)
 #define VFIO_IRQ_INFO_AUTOMASKED	(1 << 2)
 #define VFIO_IRQ_INFO_NORESIZE		(1 << 3)
+#define VFIO_IRQ_INFO_FLAG_CAPS		(1 << 4) /* Info supports caps */
 	__u32	index;		/* IRQ index */
 	__u32	count;		/* Number of IRQs within this index */
+	__u32	cap_offset;	/* Offset within info struct of first cap */
 };
 #define VFIO_DEVICE_GET_IRQ_INFO	_IO(VFIO_TYPE, VFIO_BASE + 9)
 
+/*
+ * The irq type capability allows irqs unique to a specific device or
+ * class of devices to be exposed.
+ *
+ * The structures below define version 1 of this capability.
+ */
+#define VFIO_IRQ_INFO_CAP_TYPE      1
+
+struct vfio_irq_info_cap_type {
+	struct vfio_info_cap_header header;
+	__u32 type;     /* global per bus driver */
+	__u32 subtype;  /* type specific */
+};
+
 /**
  * VFIO_DEVICE_SET_IRQS - _IOW(VFIO_TYPE, VFIO_BASE + 10, struct vfio_irq_set)
  *
@@ -789,7 +805,8 @@ enum {
 	VFIO_PCI_MSIX_IRQ_INDEX,
 	VFIO_PCI_ERR_IRQ_INDEX,
 	VFIO_PCI_REQ_IRQ_INDEX,
-	VFIO_PCI_NUM_IRQS
+	VFIO_PCI_NUM_IRQS = 5	/* Fixed user ABI, IRQ indexes >=5 use   */
+				/* device specific cap to define content */
 };
 
 /*
Signature · 1 line
2.17.1
author
#14 lore raw

RFC PATCH v4 7/10 vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

In reply to Yan Zhao

This is a virtual irq type.
vendor driver triggers this irq when it wants to notify userspace to remap PCI BARs.

1. vendor driver triggers this irq and packs the target bar number in the ctx count. i.e. "1 << bar_number".
if a bit is set, the corresponding bar is to be remapped.

2. userspace requery the specified PCI BAR from kernel and if flags of the bar regions are changed, it removes the old subregions and attaches subregions according to the new flags.

3. userspace notifies back to kernel by writing one to the eventfd of this irq.

Please check the corresponding qemu implementation from the reply of this patch, and a sample usage in vendor driver in patch [10/10].

---
 include/uapi/linux/vfio.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
include/uapi/linux/vfio.h +11 -0
index 2d0d85c7c4d4..55895f75d720 100644
@@ -704,6 +704,17 @@ struct vfio_irq_info_cap_type {
 	__u32 subtype;  /* type specific */
 };
 
+/* Bar Region Query IRQ TYPE */
+#define VFIO_IRQ_TYPE_REMAP_BAR_REGION			(1)
+
+/* sub-types for VFIO_IRQ_TYPE_REMAP_BAR_REGION */
+/*
+ * This irq notifies userspace to re-query BAR region and remaps the
+ * subregions.
+ */
+#define VFIO_IRQ_SUBTYPE_REMAP_BAR_REGION	(0)
+
+
 /**
  * VFIO_DEVICE_SET_IRQS - _IOW(VFIO_TYPE, VFIO_BASE + 10, struct vfio_irq_set)
  *
Signature · 1 line
2.17.1
author
#15 lore raw

RFC PATCH v4 1/1 hw/vfio/pci: remap bar region irq

Added an irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION to dynamically query and remap BAR regions.

QEMU decodes the index of the BARs by reading cnt of eventfd. If bit n is set, the corresponding BAR will be requeried and its subregions will be remapped according to the its new flags.

rely on [1] "vfio: Add a funtion to return a specific irq capabilities"
[1] https://www.mail-archive.com/[email protected]/msg621645.html

---
 hw/vfio/common.c              | 50 ++++++++++++++++++++++++
 hw/vfio/pci.c                 | 90 +++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.h                 |  2 +
 include/hw/vfio/vfio-common.h |  2 +
 linux-headers/linux/vfio.h    | 11 ++++++
 5 files changed, 155 insertions(+)
hw/vfio/common.c +50 -0
index a041c3b..cf24293 100644
@@ -1284,6 +1284,56 @@ void vfio_region_unmap(VFIORegion *region)
     }
 }
 
+/*
+ * re-query a region's flags,
+ * and update its mmap'd subregions.
+ * It does not support change a region's size.
+ */
+void vfio_region_reset_mmap(VFIODevice *vbasedev, VFIORegion *region, int index)
+{
+    struct vfio_region_info *new;
+
+    if (!region->mem) {
+        return;
+    }
+
+    if (vfio_get_region_info(vbasedev, index, &new)) {
+        goto out;
+    }
+
+    if (region->size != new->size) {
+        error_report("vfio: resetting of region size is not supported");
+        goto out;
+    }
+
+    if (region->flags == new->flags) {
+        goto out;
+    }
+
+    /* ummap old mmap'd subregions, if any */
+    vfio_region_unmap(region);
+    region->nr_mmaps = 0;
+    g_free(region->mmaps);
+    region->mmaps = NULL;
+
+    /* setup new mmap'd subregions*/
+    region->flags = new->flags;
+    if (vbasedev->no_mmap ||
+            !(region->flags & VFIO_REGION_INFO_FLAG_MMAP)) {
+        goto out;
+    }
+
+    if (vfio_setup_region_sparse_mmaps(region, new)) {
+        region->nr_mmaps = 1;
+        region->mmaps = g_new0(VFIOMmap, region->nr_mmaps);
+        region->mmaps[0].offset = 0;
+        region->mmaps[0].size = region->size;
+    }
+    vfio_region_mmap(region);
+out:
+    g_free(new);
+}
+
 void vfio_region_exit(VFIORegion *region)
 {
     int i;
hw/vfio/pci.c +90 -0

108 lines not on this page · raw message

hw/vfio/pci.h +2 -0
index b148c93..5a1e564 100644
@@ -134,6 +134,7 @@ typedef struct VFIOPCIDevice {
     PCIHostDeviceAddress host;
     EventNotifier err_notifier;
     EventNotifier req_notifier;
+    EventNotifier remap_bar_notifier;
     int (*resetfn)(struct VFIOPCIDevice *);
     uint32_t vendor_id;
     uint32_t device_id;
@@ -157,6 +158,7 @@ typedef struct VFIOPCIDevice {
     uint8_t nv_gpudirect_clique;
     bool pci_aer;
     bool req_enabled;
+    bool remap_bar_enabled;
     bool has_flr;
     bool has_pm_reset;
     bool rom_read_failed;
include/hw/vfio/vfio-common.h +2 -0
index a6283b7..1c16790 100644
@@ -188,6 +188,8 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
 int vfio_region_mmap(VFIORegion *region);
 void vfio_region_mmaps_set_enabled(VFIORegion *region, bool enabled);
 void vfio_region_unmap(VFIORegion *region);
+void vfio_region_reset_mmap(VFIODevice *vbasedev,
+                            VFIORegion *region, int index);
 void vfio_region_exit(VFIORegion *region);
 void vfio_region_finalize(VFIORegion *region);
 void vfio_reset_handler(void *opaque);
linux-headers/linux/vfio.h +11 -0
index 2598a84..2344ca6 100644
@@ -703,6 +703,17 @@ struct vfio_irq_info_cap_type {
 	__u32 subtype;  /* type specific */
 };
 
+/* Bar Region Query IRQ TYPE */
+#define VFIO_IRQ_TYPE_REMAP_BAR_REGION                 (1)
+
+/* sub-types for VFIO_IRQ_TYPE_REMAP_BAR_REGION */
+/*
+ * This irq notifies userspace to re-query BAR region and remaps the
+ * subregions.
+ */
+#define VFIO_IRQ_SUBTYPE_REMAP_BAR_REGION      (0)
+
+
 /**
  * VFIO_DEVICE_SET_IRQS - _IOW(VFIO_TYPE, VFIO_BASE + 10, struct vfio_irq_set)
  *
Signature · 1 line
2.7.4
#16 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

In reply to Yan Zhao
Quoted 31 lines · On Sun, 17 May 2020 22:52:45 -0400 Yan Zhao <[email protected]> wrote:

This is a virtual irq type.
vendor driver triggers this irq when it wants to notify userspace to remap PCI BARs.

1. vendor driver triggers this irq and packs the target bar number in the ctx count. i.e. "1 << bar_number".
if a bit is set, the corresponding bar is to be remapped.

2. userspace requery the specified PCI BAR from kernel and if flags of the bar regions are changed, it removes the old subregions and attaches subregions according to the new flags.

3. userspace notifies back to kernel by writing one to the eventfd of this irq.

Please check the corresponding qemu implementation from the reply of this patch, and a sample usage in vendor driver in patch [10/10].

---
 include/uapi/linux/vfio.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
include/uapi/linux/vfio.h +0 -0
index 2d0d85c7c4d4..55895f75d720 100644
@@ -704,6 +704,17 @@ struct vfio_irq_info_cap_type {
 	__u32 subtype;  /* type specific */
 };

include/uapi/linux/vfio.h struct vfio_irq_info_cap_type { · line 715

Quoted 10 lines
include/uapi/linux/vfio.h +9 -0
 
+/* Bar Region Query IRQ TYPE */
+#define VFIO_IRQ_TYPE_REMAP_BAR_REGION			(1)
+
+/* sub-types for VFIO_IRQ_TYPE_REMAP_BAR_REGION */
+/*
+ * This irq notifies userspace to re-query BAR region and remaps the
+ * subregions.
+ */
+#define VFIO_IRQ_SUBTYPE_REMAP_BAR_REGION	(0)

Hi Yan,

How do we do this in a way that's backwards compatible? Or maybe, how do we perform a handshake between the vendor driver and userspace to indicate this support? Would the vendor driver refuse to change device_state in the migration region if the user has not enabled this IRQ?

Everything you've described in the commit log needs to be in this header, we can't have the usage protocol buried in a commit log. It also seems like this is unnecessarily PCI specific. Can't the count bitmap simply indicate the region index to re-evaluate? Maybe you were worried about running out of bits in the ctx count? An IRQ per region could resolve that, but maybe we could also just add another IRQ for the next bitmap of regions. I assume that the bitmap can indicate multiple regions to re-evaluate, but that should be documented.

Also, what sort of service requirements does this imply? Would the vendor driver send this IRQ when the user tries to set the device_state to _SAVING and therefore we'd require the user to accept, implement the mapping change, and acknowledge the IRQ all while waiting for the write to device_state to return? That implies quite a lot of asynchronous support in the userspace driver. Thanks,

Alex

Quoted 5 lines
include/uapi/linux/vfio.h +2 -0
+
+
 /**
  * VFIO_DEVICE_SET_IRQS - _IOW(VFIO_TYPE, VFIO_BASE + 10, struct vfio_irq_set)
  *
author
#17 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 47 lines · Alex Williamson
Quoted 31 lines · On Sun, 17 May 2020 22:52:45 -0400 Yan Zhao <[email protected]> wrote:

This is a virtual irq type.
vendor driver triggers this irq when it wants to notify userspace to remap PCI BARs.

1. vendor driver triggers this irq and packs the target bar number in the ctx count. i.e. "1 << bar_number".
if a bit is set, the corresponding bar is to be remapped.

2. userspace requery the specified PCI BAR from kernel and if flags of the bar regions are changed, it removes the old subregions and attaches subregions according to the new flags.

3. userspace notifies back to kernel by writing one to the eventfd of this irq.

Please check the corresponding qemu implementation from the reply of this patch, and a sample usage in vendor driver in patch [10/10].

---
 include/uapi/linux/vfio.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
include/uapi/linux/vfio.h +0 -0
index 2d0d85c7c4d4..55895f75d720 100644
@@ -704,6 +704,17 @@ struct vfio_irq_info_cap_type {
 	__u32 subtype;  /* type specific */
 };

include/uapi/linux/vfio.h struct vfio_irq_info_cap_type { · line 715

Quoted 10 lines
include/uapi/linux/vfio.h +9 -0
 
+/* Bar Region Query IRQ TYPE */
+#define VFIO_IRQ_TYPE_REMAP_BAR_REGION			(1)
+
+/* sub-types for VFIO_IRQ_TYPE_REMAP_BAR_REGION */
+/*
+ * This irq notifies userspace to re-query BAR region and remaps the
+ * subregions.
+ */
+#define VFIO_IRQ_SUBTYPE_REMAP_BAR_REGION	(0)

Hi Yan,

Quoted 3 lines

How do we do this in a way that's backwards compatible? Or maybe, how do we perform a handshake between the vendor driver and userspace to indicate this support?

hi Alex
thank you for your thoughtful review!

do you think below sequence can provide enough backwards compatibility?

- on vendor driver opening, it registers an irq of type VFIO_IRQ_TYPE_REMAP_BAR_REGION, and reports to driver vfio-pci there's 1 vendor irq.

- after userspace detects the irq of type VFIO_IRQ_TYPE_REMAP_BAR_REGION it enables it by signaling ACTION_TRIGGER.

- on receiving this ACTION_TRIGGER, vendor driver will try to setup a virqfd to monitor file write to the fd of this irq, enable this irq and return its enabling status to userspace.

Quoted 3 lines

Would the vendor driver refuse to change
device_state in the migration region if the user has not enabled this IRQ?

yes, vendor driver can refuse to change device_state if the irq VFIO_IRQ_TYPE_REMAP_BAR_REGION is not enabled.
in my sample i40e_vf driver (patch 10/10), it implemented this logic like below:

i40e_vf_set_device_state

    |-> case VFIO_DEVICE_STATE_SAVING | VFIO_DEVICE_STATE_RUNNING:
    |          ret = i40e_vf_prepare_dirty_track(i40e_vf_dev);
                              |->ret = i40e_vf_remap_bars(i40e_vf_dev, true);
			                     |->if (!i40e_vf_dev->remap_irq_ctx.init)
                                                    return -ENODEV;

(i40e_vf_dev->remap_irq_ctx.init is set in below path) i40e_vf_ioctl(cmd==VFIO_DEVICE_SET_IRQS)

    |->i40e_vf_set_irq_remap_bars
       |->i40e_vf_enable_remap_bars_irq
           |-> vf_dev->remap_irq_ctx.init = true;
Quoted 3 lines

Everything you've described in the commit log needs to be in this header, we can't have the usage protocol buried in a commit log. It

got it! I'll move all descriptions in commit logs to this header so that readers can understand the whole picture here.

Quoted 2 lines

also seems like this is unnecessarily PCI specific. Can't the count bitmap simply indicate the region index to re-evaluate? Maybe you were

yes, it is possible. but what prevented me from doing it is that it's not easy to write an irq handler in qemu to remap other regions dynamically.

for BAR regions, there're 3 layers as below.
1. bar->mr -->bottom layer
2. bar->region.mem --> slow path
3. bar->region->mmaps[i].mem --> fast path
so, bar remap irq handler can simply re-revaluate the region and remove/re-generate the layer 3 (fast path) without losing track of any guest accesses to the bar regions.

actually so far, the bar remap irq handler in qemu only supports remap mmap'd subregions (layout of mmap'd subregions are re-queried) and not supports updating the whole bar region size.
(do you think updating bar region size is a must?)

however, there are no such fast path and slow path in other regions, so remap handlers for them are region specific.

Quoted 1 line

worried about running out of bits in the ctx count? An IRQ per region

yes. that's also possible :)
but current ctx count is 64bit, so it can support regions of index up to 63. if we don't need to remap dev regions, seems it's enough?

Quoted 3 lines

could resolve that, but maybe we could also just add another IRQ for the next bitmap of regions. I assume that the bitmap can indicate multiple regions to re-evaluate, but that should be documented.

hmm. would you mind elaborating more about it?

Quoted 7 lines

Also, what sort of service requirements does this imply? Would the vendor driver send this IRQ when the user tries to set the device_state to _SAVING and therefore we'd require the user to accept, implement the mapping change, and acknowledge the IRQ all while waiting for the write to device_state to return? That implies quite a lot of asynchronous support in the userspace driver. Thanks,

yes.
(1) when user sets device_state to _SAVING, the vendor driver notifies this IRQ, waits until user IRQ ack is received.
(2) in IRQ handler, user decodes and sends IRQ ack to vendor driver.

if a wait is required in (1) returns, it demands the qemu_mutex_iothread is not locked in migration thread when device_state is set in (1), as before entering (2), acquiring of this mutex is required.

Currently, this lock is not hold in vfio_migration_set_state() at save_setup stage but is hold in stop and copy stage. so we wait in kernel in save_setup stage and not wait in stop stage.
it can be fixed by calling qemu_mutex_unlock_iothread() on entering vfio_migration_set_state() and qemu_mutex_lock_iothread() on leaving vfio_migration_set_state() in qemu.

do you think it's acceptable?

Thanks
Yan

Quoted 8 lines
Quoted 7 lines
include/uapi/linux/vfio.h +2 -0
+
+
 /**
  * VFIO_DEVICE_SET_IRQS - _IOW(VFIO_TYPE, VFIO_BASE + 10, struct vfio_irq_set)
  *
#18 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 60 lines · On Mon, 1 Jun 2020 02:57:26 -0400 Yan Zhao <[email protected]> wrote:
Quoted 47 lines · Alex Williamson
Quoted 31 lines · On Sun, 17 May 2020 22:52:45 -0400 Yan Zhao <[email protected]> wrote:

This is a virtual irq type.
vendor driver triggers this irq when it wants to notify userspace to remap PCI BARs.

1. vendor driver triggers this irq and packs the target bar number in the ctx count. i.e. "1 << bar_number".
if a bit is set, the corresponding bar is to be remapped.

2. userspace requery the specified PCI BAR from kernel and if flags of the bar regions are changed, it removes the old subregions and attaches subregions according to the new flags.

3. userspace notifies back to kernel by writing one to the eventfd of this irq.

Please check the corresponding qemu implementation from the reply of this patch, and a sample usage in vendor driver in patch [10/10].

---
 include/uapi/linux/vfio.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
include/uapi/linux/vfio.h +0 -0
index 2d0d85c7c4d4..55895f75d720 100644
@@ -704,6 +704,17 @@ struct vfio_irq_info_cap_type {
 	__u32 subtype;  /* type specific */
 };

include/uapi/linux/vfio.h struct vfio_irq_info_cap_type { · line 715

Quoted 10 lines
include/uapi/linux/vfio.h +9 -0
 
+/* Bar Region Query IRQ TYPE */
+#define VFIO_IRQ_TYPE_REMAP_BAR_REGION			(1)
+
+/* sub-types for VFIO_IRQ_TYPE_REMAP_BAR_REGION */
+/*
+ * This irq notifies userspace to re-query BAR region and remaps the
+ * subregions.
+ */
+#define VFIO_IRQ_SUBTYPE_REMAP_BAR_REGION	(0)  

Hi Yan,

Quoted 3 lines

How do we do this in a way that's backwards compatible? Or maybe, how do we perform a handshake between the vendor driver and userspace to indicate this support?

hi Alex
thank you for your thoughtful review!

do you think below sequence can provide enough backwards compatibility?

- on vendor driver opening, it registers an irq of type VFIO_IRQ_TYPE_REMAP_BAR_REGION, and reports to driver vfio-pci there's 1 vendor irq.

Quoted 6 lines

- after userspace detects the irq of type VFIO_IRQ_TYPE_REMAP_BAR_REGION it enables it by signaling ACTION_TRIGGER.

- on receiving this ACTION_TRIGGER, vendor driver will try to setup a virqfd to monitor file write to the fd of this irq, enable this irq and return its enabling status to userspace.

I'm not sure I follow here, what's the purpose of the irqfd? When and what does the user signal by writing to the irqfd? Is this an ACK mechanism? Is this a different fd from the signaling eventfd?

Quoted 16 lines
Quoted 3 lines

Would the vendor driver refuse to change
device_state in the migration region if the user has not enabled this IRQ?

yes, vendor driver can refuse to change device_state if the irq VFIO_IRQ_TYPE_REMAP_BAR_REGION is not enabled.
in my sample i40e_vf driver (patch 10/10), it implemented this logic like below:

i40e_vf_set_device_state

    |-> case VFIO_DEVICE_STATE_SAVING | VFIO_DEVICE_STATE_RUNNING:
    |          ret = i40e_vf_prepare_dirty_track(i40e_vf_dev);
                              |->ret = i40e_vf_remap_bars(i40e_vf_dev, true);
			                     |->if (!i40e_vf_dev->remap_irq_ctx.init)
                                                    return -ENODEV;
Quoted 5 lines

(i40e_vf_dev->remap_irq_ctx.init is set in below path) i40e_vf_ioctl(cmd==VFIO_DEVICE_SET_IRQS)

    |->i40e_vf_set_irq_remap_bars
       |->i40e_vf_enable_remap_bars_irq
           |-> vf_dev->remap_irq_ctx.init = true;

This should be a documented aspect of the uapi, not left to vendor discretion to implement.

Quoted 19 lines
Quoted 3 lines

Everything you've described in the commit log needs to be in this header, we can't have the usage protocol buried in a commit log. It

got it! I'll move all descriptions in commit logs to this header so that readers can understand the whole picture here.

Quoted 2 lines

also seems like this is unnecessarily PCI specific. Can't the count bitmap simply indicate the region index to re-evaluate? Maybe you were

yes, it is possible. but what prevented me from doing it is that it's not easy to write an irq handler in qemu to remap other regions dynamically.

for BAR regions, there're 3 layers as below.
1. bar->mr -->bottom layer
2. bar->region.mem --> slow path
3. bar->region->mmaps[i].mem --> fast path
so, bar remap irq handler can simply re-revaluate the region and remove/re-generate the layer 3 (fast path) without losing track of any guest accesses to the bar regions.

Quoted 4 lines

actually so far, the bar remap irq handler in qemu only supports remap mmap'd subregions (layout of mmap'd subregions are re-queried) and not supports updating the whole bar region size.
(do you think updating bar region size is a must?)

It depends on whether our interrupt is defined that the user should re-evaluate the entire region_info or just the spare mmap capability. A device spontaneously changing region size seems like a much more abstract problem. We do need to figure out how to support resizeable BARs, but it seems that would be at the direction of the user, for example emulating the resizeable BAR capability and requiring userspace to re-evaluate the region_info after interacting with that emulation. So long as we specify that this IRQ is limited to re-evaluating the sparse mmap capability for the indicated regions, I don't think we need to handle the remainder of region_info spontaneously changing.

Quoted 2 lines

however, there are no such fast path and slow path in other regions, so remap handlers for them are region specific.

QEMU support for re-evaluating arbitrary regions for sparse mmap changes should not limit our kernel implementation. Maybe it does suggest though that userspace should be informed of the region indexes subject to re-evaluation such that it can choose to ignore this interrupt (and lose the features enabled by the IRQ), if it doesn't support re-evaluating all of the indicated regions. For example the capability could include a bitmap indicating regions that might be signaled and the QEMU driver might skip registering an eventfd via SET_IRQS if support for non-BAR region indexes is indicated as a requirement. I'd really prefer if we can design this to not be limited to PCI BARs.

Quoted 4 lines
Quoted 1 line

worried about running out of bits in the ctx count? An IRQ per region

yes. that's also possible :)
but current ctx count is 64bit, so it can support regions of index up to 63. if we don't need to remap dev regions, seems it's enough?

This is the kind of decision we might look back on 10yrs later and wonder how we were so short sighted, but yes it does seem like enough and we can define additional IRQs for each of the next 64 region indexes if we need too.

Quoted 4 lines
Quoted 3 lines

could resolve that, but maybe we could also just add another IRQ for the next bitmap of regions. I assume that the bitmap can indicate multiple regions to re-evaluate, but that should be documented.

hmm. would you mind elaborating more about it?

I'm just confirming that the usage expectation would allow the user to be signaled with multiple bits in the bitmap set and the user is expected to re-evaluate each region index sparse bitmap.

Quoted 15 lines
Quoted 6 lines

Also, what sort of service requirements does this imply? Would the vendor driver send this IRQ when the user tries to set the device_state to _SAVING and therefore we'd require the user to accept, implement the mapping change, and acknowledge the IRQ all while waiting for the write to device_state to return? That implies quite a lot of asynchronous support in the userspace driver. Thanks,

yes.
(1) when user sets device_state to _SAVING, the vendor driver notifies this IRQ, waits until user IRQ ack is received.
(2) in IRQ handler, user decodes and sends IRQ ack to vendor driver.

if a wait is required in (1) returns, it demands the qemu_mutex_iothread is not locked in migration thread when device_state is set in (1), as before entering (2), acquiring of this mutex is required.

Quoted 8 lines

Currently, this lock is not hold in vfio_migration_set_state() at save_setup stage but is hold in stop and copy stage. so we wait in kernel in save_setup stage and not wait in stop stage.
it can be fixed by calling qemu_mutex_unlock_iothread() on entering vfio_migration_set_state() and qemu_mutex_lock_iothread() on leaving vfio_migration_set_state() in qemu.

do you think it's acceptable?

I'm not thrilled by it, it seems a bit tricky for both userspace and the vendor driver to get right. Userspace needs to handle this eventfd while blocked on write(2) into a region, which for QEMU means additional ioctls to retrieve new REGION_INFO, closing some mmaps, maybe opening other mmaps, which implies new layering of MemoryRegion sub-regions and all of the calls through KVM to implement those address space changes. The vendor driver must also be able to support concurrency of handling the REGION_INFO ioctl, new calls to mmap regions, and maybe vm_ops.close and vm_ops.fault. These regions might also be IOMMU mapped, so re-evaluating the sparse mmap could result in DMA maps and unmaps, which the vendor driver might see via the notify forcing it to unpin pages. How would the vendor driver know when to unblock the write to device_state, would it look for vm_ops.close on infringing vmas or are you thinking of an ACK via irqfd? I wouldn't want to debug lockups as a result of this design :-\

What happens if the mmap re-evaluation occurs asynchronous to the device_state write? The vendor driver can track outstanding mmap vmas to areas it's trying to revoke, so the vendor driver can know when userspace has reached an acceptable state (assuming we require userspace to munmap areas that are no longer valid). We should also consider what we can accomplish by invalidating user mmaps, ex. can we fault them back in on a per-page basis and continue to mark them dirty in the migration state, re-invalidating on each iteration until they've
finally been closed. It seems the vendor driver needs to handle
incrementally closing each mmap anyway, there's no requirement to the user to stop the device (ie. block all access), make these changes, then restart the device. So perhaps the vendor driver can "limp" along until userspace completes the changes. I think we can assume we are in a cooperative environment here, userspace wants to perform a migration, disabling direct access to some regions is for mediating those accesses during migration, not for preventing the user from accessing something they shouldn't have access to, userspace is only delaying the migration or affecting the state of their device by not promptly participating in the protocol.

Another problem I see though is what about p2p DMA? If the vendor driver invalidates an mmap we're removing it from both direct CPU as well as DMA access via the IOMMU. We can't signal to the guest OS that a DMA channel they've been using is suddenly no longer valid. Is QEMU going to need to avoid ever IOMMU mapping device_ram for regions subject to mmap invalidation? That would introduce an undesirable need to choose whether we want to support p2p or migration unless we had an IOMMU that could provide dirty tracking via p2p, right? Thanks,

Alex

author
#19 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 66 lines · Alex Williamson
Quoted 63 lines · On Mon, 1 Jun 2020 02:57:26 -0400 Yan Zhao <[email protected]> wrote:
Quoted 47 lines · Alex Williamson
Quoted 31 lines · On Sun, 17 May 2020 22:52:45 -0400 Yan Zhao <[email protected]> wrote:

This is a virtual irq type.
vendor driver triggers this irq when it wants to notify userspace to remap PCI BARs.

1. vendor driver triggers this irq and packs the target bar number in the ctx count. i.e. "1 << bar_number".
if a bit is set, the corresponding bar is to be remapped.

2. userspace requery the specified PCI BAR from kernel and if flags of the bar regions are changed, it removes the old subregions and attaches subregions according to the new flags.

3. userspace notifies back to kernel by writing one to the eventfd of this irq.

Please check the corresponding qemu implementation from the reply of this patch, and a sample usage in vendor driver in patch [10/10].

---
 include/uapi/linux/vfio.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
include/uapi/linux/vfio.h +0 -0
index 2d0d85c7c4d4..55895f75d720 100644
@@ -704,6 +704,17 @@ struct vfio_irq_info_cap_type {
 	__u32 subtype;  /* type specific */
 };

include/uapi/linux/vfio.h struct vfio_irq_info_cap_type { · line 715

Quoted 10 lines
include/uapi/linux/vfio.h +9 -0
 
+/* Bar Region Query IRQ TYPE */
+#define VFIO_IRQ_TYPE_REMAP_BAR_REGION			(1)
+
+/* sub-types for VFIO_IRQ_TYPE_REMAP_BAR_REGION */
+/*
+ * This irq notifies userspace to re-query BAR region and remaps the
+ * subregions.
+ */
+#define VFIO_IRQ_SUBTYPE_REMAP_BAR_REGION	(0)  

Hi Yan,

Quoted 3 lines

How do we do this in a way that's backwards compatible? Or maybe, how do we perform a handshake between the vendor driver and userspace to indicate this support?

hi Alex
thank you for your thoughtful review!

do you think below sequence can provide enough backwards compatibility?

- on vendor driver opening, it registers an irq of type VFIO_IRQ_TYPE_REMAP_BAR_REGION, and reports to driver vfio-pci there's 1 vendor irq.

- after userspace detects the irq of type VFIO_IRQ_TYPE_REMAP_BAR_REGION it enables it by signaling ACTION_TRIGGER.

Quoted 7 lines
Quoted 3 lines

- on receiving this ACTION_TRIGGER, vendor driver will try to setup a virqfd to monitor file write to the fd of this irq, enable this irq and return its enabling status to userspace.

I'm not sure I follow here, what's the purpose of the irqfd? When and what does the user signal by writing to the irqfd? Is this an ACK mechanism? Is this a different fd from the signaling eventfd?

it's not the kvm irqfd.
in the vendor driver side, once ACTION_TRIGGER is received for the remap irq, interface vfio_virqfd_enable() is called to monitor writes to the eventfd of this irq.

when vendor driver signals the eventfd, remap handler in QEMU is called and it writes to the eventfd after remapping is done. Then the virqfd->handler registered in vendor driver is called to receive the QEMU ack.

Quoted 17 lines
Quoted 17 lines
Quoted 4 lines

Would the vendor driver refuse to change
device_state in the migration region if the user has not enabled this IRQ?

yes, vendor driver can refuse to change device_state if the irq VFIO_IRQ_TYPE_REMAP_BAR_REGION is not enabled.
in my sample i40e_vf driver (patch 10/10), it implemented this logic like below:

i40e_vf_set_device_state

    |-> case VFIO_DEVICE_STATE_SAVING | VFIO_DEVICE_STATE_RUNNING:
    |          ret = i40e_vf_prepare_dirty_track(i40e_vf_dev);
                              |->ret = i40e_vf_remap_bars(i40e_vf_dev, true);
			                     |->if (!i40e_vf_dev->remap_irq_ctx.init)
                                                    return -ENODEV;
Quoted 9 lines
Quoted 5 lines

(i40e_vf_dev->remap_irq_ctx.init is set in below path) i40e_vf_ioctl(cmd==VFIO_DEVICE_SET_IRQS)

    |->i40e_vf_set_irq_remap_bars
       |->i40e_vf_enable_remap_bars_irq
           |-> vf_dev->remap_irq_ctx.init = true;

This should be a documented aspect of the uapi, not left to vendor discretion to implement.

ok. got it.

Quoted 24 lines
Quoted 19 lines
Quoted 3 lines

Everything you've described in the commit log needs to be in this header, we can't have the usage protocol buried in a commit log. It

got it! I'll move all descriptions in commit logs to this header so that readers can understand the whole picture here.

Quoted 2 lines

also seems like this is unnecessarily PCI specific. Can't the count bitmap simply indicate the region index to re-evaluate? Maybe you were

yes, it is possible. but what prevented me from doing it is that it's not easy to write an irq handler in qemu to remap other regions dynamically.

for BAR regions, there're 3 layers as below.
1. bar->mr -->bottom layer
2. bar->region.mem --> slow path
3. bar->region->mmaps[i].mem --> fast path
so, bar remap irq handler can simply re-revaluate the region and remove/re-generate the layer 3 (fast path) without losing track of any guest accesses to the bar regions.

Quoted 4 lines

actually so far, the bar remap irq handler in qemu only supports remap mmap'd subregions (layout of mmap'd subregions are re-queried) and not supports updating the whole bar region size.
(do you think updating bar region size is a must?)

Quoted 10 lines

It depends on whether our interrupt is defined that the user should re-evaluate the entire region_info or just the spare mmap capability. A device spontaneously changing region size seems like a much more abstract problem. We do need to figure out how to support resizeable BARs, but it seems that would be at the direction of the user, for example emulating the resizeable BAR capability and requiring userspace to re-evaluate the region_info after interacting with that emulation. So long as we specify that this IRQ is limited to re-evaluating the sparse mmap capability for the indicated regions, I don't think we need to handle the remainder of region_info spontaneously changing.

got it.

Quoted 15 lines
Quoted 2 lines

however, there are no such fast path and slow path in other regions, so remap handlers for them are region specific.

QEMU support for re-evaluating arbitrary regions for sparse mmap changes should not limit our kernel implementation. Maybe it does suggest though that userspace should be informed of the region indexes subject to re-evaluation such that it can choose to ignore this interrupt (and lose the features enabled by the IRQ), if it doesn't support re-evaluating all of the indicated regions. For example the capability could include a bitmap indicating regions that might be signaled and the QEMU driver might skip registering an eventfd via SET_IRQS if support for non-BAR region indexes is indicated as a requirement. I'd really prefer if we can design this to not be limited to PCI BARs.

what about use the irq_set->start and irq_set->count in ioctl SET_IRQS to notify vendor driver of the supported invalidation range of region indexes? e.g. currently it's irq_set->start = 0, and irq_set->count=6. this SET_IRQS ioctl can be called multiple of times to notify vendor driver all supported ranges.
if vendor driver signals indexes outside of this range, QEMU just ignores the request.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

worried about running out of bits in the ctx count? An IRQ per region

yes. that's also possible :)
but current ctx count is 64bit, so it can support regions of index up to 63. if we don't need to remap dev regions, seems it's enough?

This is the kind of decision we might look back on 10yrs later and wonder how we were so short sighted, but yes it does seem like enough and we can define additional IRQs for each of the next 64 region indexes if we need too.

ok.

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

could resolve that, but maybe we could also just add another IRQ for the next bitmap of regions. I assume that the bitmap can indicate multiple regions to re-evaluate, but that should be documented.

hmm. would you mind elaborating more about it?

I'm just confirming that the usage expectation would allow the user to be signaled with multiple bits in the bitmap set and the user is expected to re-evaluate each region index sparse bitmap.

yes, currently, vendor driver is able to specify multiple bits in the bitmap set.

Quoted 25 lines
Quoted 16 lines
Quoted 7 lines

Also, what sort of service requirements does this imply? Would the vendor driver send this IRQ when the user tries to set the device_state to _SAVING and therefore we'd require the user to accept, implement the mapping change, and acknowledge the IRQ all while waiting for the write to device_state to return? That implies quite a lot of asynchronous support in the userspace driver. Thanks,

yes.
(1) when user sets device_state to _SAVING, the vendor driver notifies this IRQ, waits until user IRQ ack is received.
(2) in IRQ handler, user decodes and sends IRQ ack to vendor driver.

if a wait is required in (1) returns, it demands the qemu_mutex_iothread is not locked in migration thread when device_state is set in (1), as before entering (2), acquiring of this mutex is required.

Quoted 8 lines

Currently, this lock is not hold in vfio_migration_set_state() at save_setup stage but is hold in stop and copy stage. so we wait in kernel in save_setup stage and not wait in stop stage.
it can be fixed by calling qemu_mutex_unlock_iothread() on entering vfio_migration_set_state() and qemu_mutex_lock_iothread() on leaving vfio_migration_set_state() in qemu.

do you think it's acceptable?

Quoted 15 lines

I'm not thrilled by it, it seems a bit tricky for both userspace and the vendor driver to get right. Userspace needs to handle this eventfd while blocked on write(2) into a region, which for QEMU means additional ioctls to retrieve new REGION_INFO, closing some mmaps, maybe opening other mmaps, which implies new layering of MemoryRegion sub-regions and all of the calls through KVM to implement those address space changes. The vendor driver must also be able to support concurrency of handling the REGION_INFO ioctl, new calls to mmap regions, and maybe vm_ops.close and vm_ops.fault. These regions might also be IOMMU mapped, so re-evaluating the sparse mmap could result in DMA maps and unmaps, which the vendor driver might see via the notify forcing it to unpin pages. How would the vendor driver know when to unblock the write to device_state, would it look for vm_ops.close on infringing vmas or are you thinking of an ACK via irqfd? I wouldn't want to debug lockups as a result of this design :-\

hmm, do you think below sequence is acceptable?
1. QEMU sets device_state to PRE_SAVING.
vendor driver signals the remap irq and returns the device_state write.

2. QEMU remap irq handler is invoked to do the region remapping. after that user ack is sent to vendor driver by the handler writing to eventfd.

3. QEMU sets device state to SAVING.
vendor driver returns success if user ack is received or failure after a timeout wait.

Quoted 12 lines

What happens if the mmap re-evaluation occurs asynchronous to the device_state write? The vendor driver can track outstanding mmap vmas to areas it's trying to revoke, so the vendor driver can know when userspace has reached an acceptable state (assuming we require userspace to munmap areas that are no longer valid). We should also consider what we can accomplish by invalidating user mmaps, ex. can we fault them back in on a per-page basis and continue to mark them dirty in the migration state, re-invalidating on each iteration until they've
finally been closed. It seems the vendor driver needs to handle
incrementally closing each mmap anyway, there's no requirement to the user to stop the device (ie. block all access), make these changes,

Quoted 9 lines

then restart the device. So perhaps the vendor driver can "limp" along until userspace completes the changes. I think we can assume we are in a cooperative environment here, userspace wants to perform a migration, disabling direct access to some regions is for mediating those accesses during migration, not for preventing the user from accessing something they shouldn't have access to, userspace is only delaying the migration or affecting the state of their device by not promptly participating in the protocol.

the problem is that the mmap re-evaluation has to be done before device_state is successfully set to SAVING. otherwise, the QEMU may have left save_setup stage and it's too late to start dirty tracking. And the reason for us to trap the BAR regions is not because there're dirty data in this region, it is because we want to know when the device registers mapped in the BARs are written, so we can do dirty page track of system memory in software way.

Quoted 8 lines

Another problem I see though is what about p2p DMA? If the vendor driver invalidates an mmap we're removing it from both direct CPU as well as DMA access via the IOMMU. We can't signal to the guest OS that a DMA channel they've been using is suddenly no longer valid. Is QEMU going to need to avoid ever IOMMU mapping device_ram for regions subject to mmap invalidation? That would introduce an undesirable need to choose whether we want to support p2p or migration unless we had an IOMMU that could provide dirty tracking via p2p, right? Thanks,

yes, if there are device memory mapped in the BARs to be remapped, p2p DMA would be affected. Perhaps it is what vendor driver should be aware of and know what it is doing before sending out the remap irq ? in i40e vf's case, the BAR 0 to be remapped is only for device registers, so is it still good?

Thanks
Yan

#20 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 79 lines · On Tue, 2 Jun 2020 04:28:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 66 lines · Alex Williamson
Quoted 63 lines · On Mon, 1 Jun 2020 02:57:26 -0400 Yan Zhao <[email protected]> wrote:
Quoted 47 lines · Alex Williamson
Quoted 31 lines · On Sun, 17 May 2020 22:52:45 -0400 Yan Zhao <[email protected]> wrote:

This is a virtual irq type.
vendor driver triggers this irq when it wants to notify userspace to remap PCI BARs.

1. vendor driver triggers this irq and packs the target bar number in the ctx count. i.e. "1 << bar_number".
if a bit is set, the corresponding bar is to be remapped.

2. userspace requery the specified PCI BAR from kernel and if flags of the bar regions are changed, it removes the old subregions and attaches subregions according to the new flags.

3. userspace notifies back to kernel by writing one to the eventfd of this irq.

Please check the corresponding qemu implementation from the reply of this patch, and a sample usage in vendor driver in patch [10/10].

---
 include/uapi/linux/vfio.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
include/uapi/linux/vfio.h +0 -0
index 2d0d85c7c4d4..55895f75d720 100644
@@ -704,6 +704,17 @@ struct vfio_irq_info_cap_type {
 	__u32 subtype;  /* type specific */
 };

include/uapi/linux/vfio.h struct vfio_irq_info_cap_type { · line 715

Quoted 10 lines
include/uapi/linux/vfio.h +9 -0
 
+/* Bar Region Query IRQ TYPE */
+#define VFIO_IRQ_TYPE_REMAP_BAR_REGION			(1)
+
+/* sub-types for VFIO_IRQ_TYPE_REMAP_BAR_REGION */
+/*
+ * This irq notifies userspace to re-query BAR region and remaps the
+ * subregions.
+ */
+#define VFIO_IRQ_SUBTYPE_REMAP_BAR_REGION	(0)    

Hi Yan,

Quoted 3 lines

How do we do this in a way that's backwards compatible? Or maybe, how do we perform a handshake between the vendor driver and userspace to indicate this support?

hi Alex
thank you for your thoughtful review!

do you think below sequence can provide enough backwards compatibility?

- on vendor driver opening, it registers an irq of type VFIO_IRQ_TYPE_REMAP_BAR_REGION, and reports to driver vfio-pci there's 1 vendor irq.

- after userspace detects the irq of type VFIO_IRQ_TYPE_REMAP_BAR_REGION it enables it by signaling ACTION_TRIGGER.

Quoted 7 lines
Quoted 3 lines

- on receiving this ACTION_TRIGGER, vendor driver will try to setup a virqfd to monitor file write to the fd of this irq, enable this irq and return its enabling status to userspace.

I'm not sure I follow here, what's the purpose of the irqfd? When and what does the user signal by writing to the irqfd? Is this an ACK mechanism? Is this a different fd from the signaling eventfd?

it's not the kvm irqfd.
in the vendor driver side, once ACTION_TRIGGER is received for the remap irq, interface vfio_virqfd_enable() is called to monitor writes to the eventfd of this irq.

Quoted 4 lines

when vendor driver signals the eventfd, remap handler in QEMU is called and it writes to the eventfd after remapping is done. Then the virqfd->handler registered in vendor driver is called to receive the QEMU ack.

This seems racy to use the same fd as both an eventfd and irqfd, does the host need to wait for the user to service the previous IRQ before sending a new one? Don't we have gaps where the user is either reading or writing where we can lose an interrupt? Does the user also write a bitmap? How do we avoid getting out of sync? Why do we even need this, can't the vendor driver look for vm_ops.close callbacks for the offending vma mmaps?

Quoted 78 lines
Quoted 16 lines
Quoted 16 lines
Quoted 3 lines

Would the vendor driver refuse to change
device_state in the migration region if the user has not enabled this IRQ?

yes, vendor driver can refuse to change device_state if the irq VFIO_IRQ_TYPE_REMAP_BAR_REGION is not enabled.
in my sample i40e_vf driver (patch 10/10), it implemented this logic like below:

i40e_vf_set_device_state

    |-> case VFIO_DEVICE_STATE_SAVING | VFIO_DEVICE_STATE_RUNNING:
    |          ret = i40e_vf_prepare_dirty_track(i40e_vf_dev);
                              |->ret = i40e_vf_remap_bars(i40e_vf_dev, true);
			                     |->if (!i40e_vf_dev->remap_irq_ctx.init)
                                                    return -ENODEV;
Quoted 9 lines
Quoted 5 lines

(i40e_vf_dev->remap_irq_ctx.init is set in below path) i40e_vf_ioctl(cmd==VFIO_DEVICE_SET_IRQS)

    |->i40e_vf_set_irq_remap_bars
       |->i40e_vf_enable_remap_bars_irq
           |-> vf_dev->remap_irq_ctx.init = true;  

This should be a documented aspect of the uapi, not left to vendor discretion to implement.

ok. got it.

Quoted 24 lines
Quoted 19 lines
Quoted 3 lines

Everything you've described in the commit log needs to be in this header, we can't have the usage protocol buried in a commit log. It

got it! I'll move all descriptions in commit logs to this header so that readers can understand the whole picture here.

Quoted 2 lines

also seems like this is unnecessarily PCI specific. Can't the count bitmap simply indicate the region index to re-evaluate? Maybe you were

yes, it is possible. but what prevented me from doing it is that it's not easy to write an irq handler in qemu to remap other regions dynamically.

for BAR regions, there're 3 layers as below.
1. bar->mr -->bottom layer
2. bar->region.mem --> slow path
3. bar->region->mmaps[i].mem --> fast path
so, bar remap irq handler can simply re-revaluate the region and remove/re-generate the layer 3 (fast path) without losing track of any guest accesses to the bar regions.

Quoted 4 lines

actually so far, the bar remap irq handler in qemu only supports remap mmap'd subregions (layout of mmap'd subregions are re-queried) and not supports updating the whole bar region size.
(do you think updating bar region size is a must?)

Quoted 10 lines

It depends on whether our interrupt is defined that the user should re-evaluate the entire region_info or just the spare mmap capability. A device spontaneously changing region size seems like a much more abstract problem. We do need to figure out how to support resizeable BARs, but it seems that would be at the direction of the user, for example emulating the resizeable BAR capability and requiring userspace to re-evaluate the region_info after interacting with that emulation. So long as we specify that this IRQ is limited to re-evaluating the sparse mmap capability for the indicated regions, I don't think we need to handle the remainder of region_info spontaneously changing.

got it.

Quoted 15 lines
Quoted 2 lines

however, there are no such fast path and slow path in other regions, so remap handlers for them are region specific.

QEMU support for re-evaluating arbitrary regions for sparse mmap changes should not limit our kernel implementation. Maybe it does suggest though that userspace should be informed of the region indexes subject to re-evaluation such that it can choose to ignore this interrupt (and lose the features enabled by the IRQ), if it doesn't support re-evaluating all of the indicated regions. For example the capability could include a bitmap indicating regions that might be signaled and the QEMU driver might skip registering an eventfd via SET_IRQS if support for non-BAR region indexes is indicated as a requirement. I'd really prefer if we can design this to not be limited to PCI BARs.

Quoted 7 lines

what about use the irq_set->start and irq_set->count in ioctl SET_IRQS to notify vendor driver of the supported invalidation range of region indexes? e.g. currently it's irq_set->start = 0, and irq_set->count=6. this SET_IRQS ioctl can be called multiple of times to notify vendor driver all supported ranges.
if vendor driver signals indexes outside of this range, QEMU just ignores the request.

You want an IRQ per region? I don't think I understand this proposal. Overloading sub-indexes within a SET_IRQS index is not acceptable. Also, "if vendor driver signals indexes outside of this range, QEMU just ignores the request", if that means that QEMU doesn't handle a request to re-evaluate the sparse mmap capability I think that nullifies the entire proposal.

Quoted 63 lines
Quoted 9 lines
Quoted 4 lines
Quoted 1 line

worried about running out of bits in the ctx count? An IRQ per region

yes. that's also possible :)
but current ctx count is 64bit, so it can support regions of index up to 63. if we don't need to remap dev regions, seems it's enough?

This is the kind of decision we might look back on 10yrs later and wonder how we were so short sighted, but yes it does seem like enough and we can define additional IRQs for each of the next 64 region indexes if we need too.

ok.

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

could resolve that, but maybe we could also just add another IRQ for the next bitmap of regions. I assume that the bitmap can indicate multiple regions to re-evaluate, but that should be documented.

hmm. would you mind elaborating more about it?

I'm just confirming that the usage expectation would allow the user to be signaled with multiple bits in the bitmap set and the user is expected to re-evaluate each region index sparse bitmap.

yes, currently, vendor driver is able to specify multiple bits in the bitmap set.

Quoted 25 lines
Quoted 16 lines
Quoted 7 lines

Also, what sort of service requirements does this imply? Would the vendor driver send this IRQ when the user tries to set the device_state to _SAVING and therefore we'd require the user to accept, implement the mapping change, and acknowledge the IRQ all while waiting for the write to device_state to return? That implies quite a lot of asynchronous support in the userspace driver. Thanks,

yes.
(1) when user sets device_state to _SAVING, the vendor driver notifies this IRQ, waits until user IRQ ack is received.
(2) in IRQ handler, user decodes and sends IRQ ack to vendor driver.

if a wait is required in (1) returns, it demands the qemu_mutex_iothread is not locked in migration thread when device_state is set in (1), as before entering (2), acquiring of this mutex is required.

Quoted 8 lines

Currently, this lock is not hold in vfio_migration_set_state() at save_setup stage but is hold in stop and copy stage. so we wait in kernel in save_setup stage and not wait in stop stage.
it can be fixed by calling qemu_mutex_unlock_iothread() on entering vfio_migration_set_state() and qemu_mutex_lock_iothread() on leaving vfio_migration_set_state() in qemu.

do you think it's acceptable?

Quoted 15 lines

I'm not thrilled by it, it seems a bit tricky for both userspace and the vendor driver to get right. Userspace needs to handle this eventfd while blocked on write(2) into a region, which for QEMU means additional ioctls to retrieve new REGION_INFO, closing some mmaps, maybe opening other mmaps, which implies new layering of MemoryRegion sub-regions and all of the calls through KVM to implement those address space changes. The vendor driver must also be able to support concurrency of handling the REGION_INFO ioctl, new calls to mmap regions, and maybe vm_ops.close and vm_ops.fault. These regions might also be IOMMU mapped, so re-evaluating the sparse mmap could result in DMA maps and unmaps, which the vendor driver might see via the notify forcing it to unpin pages. How would the vendor driver know when to unblock the write to device_state, would it look for vm_ops.close on infringing vmas or are you thinking of an ACK via irqfd? I wouldn't want to debug lockups as a result of this design :-\

Quoted 4 lines

hmm, do you think below sequence is acceptable?
1. QEMU sets device_state to PRE_SAVING.
vendor driver signals the remap irq and returns the device_state write.

We have no PRE_SAVING device state. We haven't even gotten an implementation of the agreed migration protocol into mainline and we're already abandoning it?

Quoted 2 lines

2. QEMU remap irq handler is invoked to do the region remapping. after that user ack is sent to vendor driver by the handler writing to eventfd.

Why would we even need an remap IRQ, if we're going to throw away the migration protocol we could just define that the user should re-evaluate the sparse mmaps when switching to _SAVING.

Quoted 3 lines

3. QEMU sets device state to SAVING.
vendor driver returns success if user ack is received or failure after a timeout wait.

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where the existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the extent that a vendor driver requiring this will automatically fail migration.

Quoted 20 lines
Quoted 11 lines

What happens if the mmap re-evaluation occurs asynchronous to the device_state write? The vendor driver can track outstanding mmap vmas to areas it's trying to revoke, so the vendor driver can know when userspace has reached an acceptable state (assuming we require userspace to munmap areas that are no longer valid). We should also consider what we can accomplish by invalidating user mmaps, ex. can we fault them back in on a per-page basis and continue to mark them dirty in the migration state, re-invalidating on each iteration until they've
finally been closed. It seems the vendor driver needs to handle
incrementally closing each mmap anyway, there's no requirement to the user to stop the device (ie. block all access), make these changes,

Quoted 9 lines

then restart the device. So perhaps the vendor driver can "limp" along until userspace completes the changes. I think we can assume we are in a cooperative environment here, userspace wants to perform a migration, disabling direct access to some regions is for mediating those accesses during migration, not for preventing the user from accessing something they shouldn't have access to, userspace is only delaying the migration or affecting the state of their device by not promptly participating in the protocol.

Quoted 7 lines

the problem is that the mmap re-evaluation has to be done before device_state is successfully set to SAVING. otherwise, the QEMU may have left save_setup stage and it's too late to start dirty tracking. And the reason for us to trap the BAR regions is not because there're dirty data in this region, it is because we want to know when the device registers mapped in the BARs are written, so we can do dirty page track of system memory in software way.

I think my proposal above resolves this.

Quoted 9 lines
Quoted 8 lines

Another problem I see though is what about p2p DMA? If the vendor driver invalidates an mmap we're removing it from both direct CPU as well as DMA access via the IOMMU. We can't signal to the guest OS that a DMA channel they've been using is suddenly no longer valid. Is QEMU going to need to avoid ever IOMMU mapping device_ram for regions subject to mmap invalidation? That would introduce an undesirable need to choose whether we want to support p2p or migration unless we had an IOMMU that could provide dirty tracking via p2p, right? Thanks,

Quoted 5 lines

yes, if there are device memory mapped in the BARs to be remapped, p2p DMA would be affected. Perhaps it is what vendor driver should be aware of and know what it is doing before sending out the remap irq ? in i40e vf's case, the BAR 0 to be remapped is only for device registers, so is it still good?

No, we can't design the interface based on one vendor driver's implementation of the interface or the requirements of a single device. If we took the approach above where the user is provided both the normal sparse mmap and the _SAVING sparse mmap, perhaps QEMU could avoid DMA mapping portions that don't exist in the _SAVING version, at least then the p2p DMA mappings would be consistent across the transition. QEMU might be able to combine the sparse mmap maps such that it can easily drop ranges not present during _SAVING. QEMU would need to munmap() the dropped ranges rather than simply mark the MemoryRegion disabled though for the vendor driver to have visibility of the vm_ops.close callback. Thanks,

Alex

author
#21 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 12 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where the existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the extent that a vendor driver requiring this will automatically fail migration.

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

Quoted 28 lines
Quoted 20 lines
Quoted 11 lines

What happens if the mmap re-evaluation occurs asynchronous to the device_state write? The vendor driver can track outstanding mmap vmas to areas it's trying to revoke, so the vendor driver can know when userspace has reached an acceptable state (assuming we require userspace to munmap areas that are no longer valid). We should also consider what we can accomplish by invalidating user mmaps, ex. can we fault them back in on a per-page basis and continue to mark them dirty in the migration state, re-invalidating on each iteration until they've
finally been closed. It seems the vendor driver needs to handle
incrementally closing each mmap anyway, there's no requirement to the user to stop the device (ie. block all access), make these changes,

Quoted 9 lines

then restart the device. So perhaps the vendor driver can "limp" along until userspace completes the changes. I think we can assume we are in a cooperative environment here, userspace wants to perform a migration, disabling direct access to some regions is for mediating those accesses during migration, not for preventing the user from accessing something they shouldn't have access to, userspace is only delaying the migration or affecting the state of their device by not promptly participating in the protocol.

Quoted 7 lines

the problem is that the mmap re-evaluation has to be done before device_state is successfully set to SAVING. otherwise, the QEMU may have left save_setup stage and it's too late to start dirty tracking. And the reason for us to trap the BAR regions is not because there're dirty data in this region, it is because we want to know when the device registers mapped in the BARs are written, so we can do dirty page track of system memory in software way.

Quoted 2 lines

I think my proposal above resolves this.

yes.

Quoted 15 lines
Quoted 9 lines
Quoted 8 lines

Another problem I see though is what about p2p DMA? If the vendor driver invalidates an mmap we're removing it from both direct CPU as well as DMA access via the IOMMU. We can't signal to the guest OS that a DMA channel they've been using is suddenly no longer valid. Is QEMU going to need to avoid ever IOMMU mapping device_ram for regions subject to mmap invalidation? That would introduce an undesirable need to choose whether we want to support p2p or migration unless we had an IOMMU that could provide dirty tracking via p2p, right? Thanks,

Quoted 5 lines

yes, if there are device memory mapped in the BARs to be remapped, p2p DMA would be affected. Perhaps it is what vendor driver should be aware of and know what it is doing before sending out the remap irq ? in i40e vf's case, the BAR 0 to be remapped is only for device registers, so is it still good?

Quoted 12 lines

No, we can't design the interface based on one vendor driver's implementation of the interface or the requirements of a single device. If we took the approach above where the user is provided both the normal sparse mmap and the _SAVING sparse mmap, perhaps QEMU could avoid DMA mapping portions that don't exist in the _SAVING version, at least then the p2p DMA mappings would be consistent across the transition. QEMU might be able to combine the sparse mmap maps such that it can easily drop ranges not present during _SAVING. QEMU would need to munmap() the dropped ranges rather than simply mark the MemoryRegion disabled though for the vendor driver to have visibility of the vm_ops.close callback. Thanks,

ok. got it! thanks you!

Yan

#22 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 13 lines · On Tue, 2 Jun 2020 21:40:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 12 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where the existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the extent that a vendor driver requiring this will automatically fail migration.

Quoted 6 lines

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

In broad strokes, I don't think we've found the right solution yet. I really question whether it's supportable to parcel out vfio-pci like this and I don't know how I'd support unraveling whether we have a bug in vfio-pci, the vendor driver, or how the vendor driver is making use of vfio-pci.

Let me also ask, why does any of this need to be in the kernel? We spend 5 patches slicing up vfio-pci so that we can register a vendor driver and have that vendor driver call into vfio-pci as it sees fit. We have two patches creating device specific interrupts and a BAR remapping scheme that we've decided we don't need. That brings us to the actual i40e vendor driver, where the first patch is simply making the vendor driver work like vfio-pci already does, the second patch is handling the migration region, and the third patch is implementing the BAR remapping IRQ that we decided we don't need. It's difficult to actually find the small bit of code that's required to support migration outside of just dealing with the protocol we've defined to expose this from the kernel. So why are we trying to do this in the kernel? We have quirk support in QEMU, we can easily flip MemoryRegions on and off, etc. What access to the device outside of what vfio-pci provides to the user, and therefore QEMU, is necessary to implement this migration support for i40e VFs? Is this just an exercise in making use of the migration interface? Thanks,

Alex

author
#23 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 38 lines · Alex Williamson
Quoted 13 lines · On Tue, 2 Jun 2020 21:40:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 12 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where the existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the extent that a vendor driver requiring this will automatically fail migration.

Quoted 6 lines

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

In broad strokes, I don't think we've found the right solution yet. I really question whether it's supportable to parcel out vfio-pci like this and I don't know how I'd support unraveling whether we have a bug in vfio-pci, the vendor driver, or how the vendor driver is making use of vfio-pci.

Let me also ask, why does any of this need to be in the kernel? We spend 5 patches slicing up vfio-pci so that we can register a vendor driver and have that vendor driver call into vfio-pci as it sees fit. We have two patches creating device specific interrupts and a BAR remapping scheme that we've decided we don't need. That brings us to the actual i40e vendor driver, where the first patch is simply making the vendor driver work like vfio-pci already does, the second patch is handling the migration region, and the third patch is implementing the BAR remapping IRQ that we decided we don't need. It's difficult to

Quoted 9 lines

actually find the small bit of code that's required to support migration outside of just dealing with the protocol we've defined to expose this from the kernel. So why are we trying to do this in the kernel? We have quirk support in QEMU, we can easily flip MemoryRegions on and off, etc. What access to the device outside of what vfio-pci provides to the user, and therefore QEMU, is necessary to implement this migration support for i40e VFs? Is this just an exercise in making use of the migration interface? Thanks,

hi Alex

There was a description of intention of this series in RFC v1
(https://www.spinics.net/lists/kernel/msg3337337.html).
sorry, I didn't include it in starting from RFC v2.

"
The reason why we don't choose the way of writing mdev parent driver is that
(1) VFs are almost all the time directly passthroughed. Directly binding to vfio-pci can make most of the code shared/reused. If we write a vendor specific mdev parent driver, most of the code (like passthrough style of rw/mmap) still needs to be copied from vfio-pci driver, which is actually a duplicated and tedious work.
(2) For features like dynamically trap/untrap pci bars, if they are in vfio-pci, they can be available to most people without repeated code copying and re-testing.
(3) with a 1:1 mdev driver which passes through VFs most of the time, people have to decide whether to bind VFs to vfio-pci or mdev parent driver before it runs into a real migration need. However, if vfio-pci is bound initially, they have no chance to do live migration when there's a need later.
"
particularly, there're some devices (like NVMe) they purely reply on vfio-pci to do device pass-through and they have no standalone parent driver to do mdev way.

I think live migration is a general requirement for most devices and to interact with the migration interface requires vendor drivers to do device specific tasks like geting/seting device state, starting/stopping devices, tracking dirty data, report migration capabilities... all those works need be in kernel.
do you think it's better to create numerous vendor quirks in vfio-pci?

as to this series, though patch 9/10 currently only demos reporting a migration region, it actually shows the capability iof vendor driver to customize device regions. e.g. in patch 10/10, it customizes the BAR0 to be read/write. and though we abandoned the REMAP BAR irq_type in patch 10/10 for migration purpose, I have to say this irq_type has its usage in other use cases, where synchronization is not a hard requirement and all it needs is a notification channel from kernel to use. this series just provides a possibility for vendors to customize device regions and irqs.

for interfaces exported in patch 3/10-5/10, they anyway need to be exported for writing mdev parent drivers that pass through devices at normal time to avoid duplication. and yes, your worry about identification of bug sources is reasonable. but if a device is binding to vfio-pci with a vendor module loaded, and there's a bug, they can do at least two ways to identify if it's a bug in vfio-pci itself.
(1) prevent vendor modules from loading and see if the problem exists with pure vfio-pci.
(2) do what's demoed in patch 8/10, i.e. do nothing but simply pass all operations to vfio-pci.

so, do you think this series has its merit and we can continue improving it?

Thanks
Yan

#24 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 50 lines · On Wed, 3 Jun 2020 22:42:28 -0400 Yan Zhao <[email protected]> wrote:
Quoted 38 lines · Alex Williamson
Quoted 13 lines · On Tue, 2 Jun 2020 21:40:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 12 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where the existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the extent that a vendor driver requiring this will automatically fail migration.

Quoted 6 lines

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

In broad strokes, I don't think we've found the right solution yet. I really question whether it's supportable to parcel out vfio-pci like this and I don't know how I'd support unraveling whether we have a bug in vfio-pci, the vendor driver, or how the vendor driver is making use of vfio-pci.

Let me also ask, why does any of this need to be in the kernel? We spend 5 patches slicing up vfio-pci so that we can register a vendor driver and have that vendor driver call into vfio-pci as it sees fit. We have two patches creating device specific interrupts and a BAR remapping scheme that we've decided we don't need. That brings us to the actual i40e vendor driver, where the first patch is simply making the vendor driver work like vfio-pci already does, the second patch is handling the migration region, and the third patch is implementing the BAR remapping IRQ that we decided we don't need. It's difficult to

Quoted 9 lines

actually find the small bit of code that's required to support migration outside of just dealing with the protocol we've defined to expose this from the kernel. So why are we trying to do this in the kernel? We have quirk support in QEMU, we can easily flip MemoryRegions on and off, etc. What access to the device outside of what vfio-pci provides to the user, and therefore QEMU, is necessary to implement this migration support for i40e VFs? Is this just an exercise in making use of the migration interface? Thanks,

hi Alex

Quoted 7 lines

There was a description of intention of this series in RFC v1
(https://www.spinics.net/lists/kernel/msg3337337.html).
sorry, I didn't include it in starting from RFC v2.

"
The reason why we don't choose the way of writing mdev parent driver is that

I didn't mention an mdev approach, I'm asking what are we accomplishing by doing this in the kernel at all versus exposing the device as normal through vfio-pci and providing the migration support in QEMU. Are you actually leveraging having some sort of access to the PF in supporting migration of the VF? Is vfio-pci masking the device in a way that prevents migrating the state from QEMU?

Quoted 18 lines

(1) VFs are almost all the time directly passthroughed. Directly binding to vfio-pci can make most of the code shared/reused. If we write a vendor specific mdev parent driver, most of the code (like passthrough style of rw/mmap) still needs to be copied from vfio-pci driver, which is actually a duplicated and tedious work.
(2) For features like dynamically trap/untrap pci bars, if they are in vfio-pci, they can be available to most people without repeated code copying and re-testing.
(3) with a 1:1 mdev driver which passes through VFs most of the time, people have to decide whether to bind VFs to vfio-pci or mdev parent driver before it runs into a real migration need. However, if vfio-pci is bound initially, they have no chance to do live migration when there's a need later.
"
particularly, there're some devices (like NVMe) they purely reply on vfio-pci to do device pass-through and they have no standalone parent driver to do mdev way.

Quoted 5 lines

I think live migration is a general requirement for most devices and to interact with the migration interface requires vendor drivers to do device specific tasks like geting/seting device state, starting/stopping devices, tracking dirty data, report migration capabilities... all those works need be in kernel.

I think Alex Graf proved they don't necessarily need to be done in kernel back in 2015: https://www.youtube.com/watch?v=4RFsSgzuFso
He was able to achieve i40e VF live migration by only hacking QEMU. In this series you're allowing a vendor driver to interpose itself between the user (QEMU) and vfio-pci such that we switch to the vendor code during migration. Why can't that interpose layer be in QEMU rather than the kernel? It seems that it only must be in the kernel if we need to provide migration state via backdoor, perhaps like going through the PF. So what access to the i40e VF device is not provided to the user through vfio-pci that is necessary to implement migration of this device? The tasks listed above are mostly standard device driver activities and clearly vfio-pci allows userspace device drivers.

Quoted 1 line

do you think it's better to create numerous vendor quirks in vfio-pci?

In QEMU, perhaps. Alternatively, let's look at exactly what access is not provided through vfio-pci that's necessary for this and decide if we want to enable that access or if cracking vfio-pci wide open for vendor drivers to pick and choose when and how to use it is really the right answer.

Quoted 9 lines

as to this series, though patch 9/10 currently only demos reporting a migration region, it actually shows the capability iof vendor driver to customize device regions. e.g. in patch 10/10, it customizes the BAR0 to be read/write. and though we abandoned the REMAP BAR irq_type in patch 10/10 for migration purpose, I have to say this irq_type has its usage in other use cases, where synchronization is not a hard requirement and all it needs is a notification channel from kernel to use. this series just provides a possibility for vendors to customize device regions and irqs.

I don't disagree that a device specific interrupt might be useful, but I would object to implementing this one only as an artificial use case. We can wait for a legitimate use case to implement that.

Quoted 3 lines

for interfaces exported in patch 3/10-5/10, they anyway need to be exported for writing mdev parent drivers that pass through devices at normal time to avoid duplication. and yes, your worry about

Where are those parent drivers? What are their actual requirements?

Quoted 7 lines

identification of bug sources is reasonable. but if a device is binding to vfio-pci with a vendor module loaded, and there's a bug, they can do at least two ways to identify if it's a bug in vfio-pci itself.
(1) prevent vendor modules from loading and see if the problem exists with pure vfio-pci.
(2) do what's demoed in patch 8/10, i.e. do nothing but simply pass all operations to vfio-pci.

The code split is still extremely ad-hoc, there's no API. An mdev driver isn't even a sub-driver of vfio-pci like you're trying to accomplish here, there would need to be a much more defined API when the base device isn't even a vfio_pci_device. I don't see how this series would directly enable an mdev use case.

Quoted 2 lines

so, do you think this series has its merit and we can continue improving it?

I think this series is trying to push an artificial use case that is perhaps better done in userspace. What is the actual interaction with the VF device that can only be done in the host kernel for this example? Thanks,

Alex

#25 lore raw

RE: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 153 lines

From: Alex Williamson <[email protected]>
Sent: Thursday, June 4, 2020 12:11 PM

Quoted 9 lines · On Wed, 3 Jun 2020 22:42:28 -0400 Yan Zhao <[email protected]> wrote:
Quoted 8 lines · Alex Williamson
Quoted 5 lines · On Tue, 2 Jun 2020 21:40:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 4 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where

the

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the

extent that a vendor driver requiring this will automatically fail migration.

Quoted 35 lines
Quoted 24 lines
Quoted 7 lines
Quoted 1 line

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

In broad strokes, I don't think we've found the right solution yet. I really question whether it's supportable to parcel out vfio-pci like this and I don't know how I'd support unraveling whether we have a bug in vfio-pci, the vendor driver, or how the vendor driver is making use of vfio-pci.

Let me also ask, why does any of this need to be in the kernel? We spend 5 patches slicing up vfio-pci so that we can register a vendor driver and have that vendor driver call into vfio-pci as it sees fit. We have two patches creating device specific interrupts and a BAR remapping scheme that we've decided we don't need. That brings us to the actual i40e vendor driver, where the first patch is simply making the vendor driver work like vfio-pci already does, the second patch is handling the migration region, and the third patch is implementing the BAR remapping IRQ that we decided we don't need. It's difficult to actually find the small bit of code that's

Quoted 9 lines

required to support migration outside of just dealing with the protocol we've defined to expose this from the kernel. So why are we trying to do this in the kernel? We have quirk support in QEMU, we can easily flip MemoryRegions on and off, etc. What access to the device outside of what vfio-pci provides to the user, and therefore QEMU, is necessary to implement this migration support for i40e VFs? Is this just an exercise in making use of the migration interface? Thanks,

hi Alex

Quoted 7 lines

There was a description of intention of this series in RFC v1
(https://www.spinics.net/lists/kernel/msg3337337.html).
sorry, I didn't include it in starting from RFC v2.

"
The reason why we don't choose the way of writing mdev parent driver is that

I didn't mention an mdev approach, I'm asking what are we accomplishing by doing this in the kernel at all versus exposing the device as normal through vfio-pci and providing the migration support in QEMU. Are you actually leveraging having some sort of access to the PF in supporting migration of the VF? Is vfio-pci masking the device in a way that prevents migrating the state from QEMU?

Quoted 18 lines

(1) VFs are almost all the time directly passthroughed. Directly binding to vfio-pci can make most of the code shared/reused. If we write a vendor specific mdev parent driver, most of the code (like passthrough style of rw/mmap) still needs to be copied from vfio-pci driver, which is actually a duplicated and tedious work.
(2) For features like dynamically trap/untrap pci bars, if they are in vfio-pci, they can be available to most people without repeated code copying and re-testing.
(3) with a 1:1 mdev driver which passes through VFs most of the time, people have to decide whether to bind VFs to vfio-pci or mdev parent driver before it runs into a real migration need. However, if vfio-pci is bound initially, they have no chance to do live migration when there's a need later.
"
particularly, there're some devices (like NVMe) they purely reply on vfio-pci to do device pass-through and they have no standalone parent driver to do mdev way.

Quoted 5 lines

I think live migration is a general requirement for most devices and to interact with the migration interface requires vendor drivers to do device specific tasks like geting/seting device state, starting/stopping devices, tracking dirty data, report migration capabilities... all those works need be in kernel.

I think Alex Graf proved they don't necessarily need to be done in kernel back in 2015: https://www.youtube.com/watch?v=4RFsSgzuFso
He was able to achieve i40e VF live migration by only hacking QEMU. In this series you're allowing a vendor driver to interpose itself between the user (QEMU) and vfio-pci such that we switch to the vendor code during migration. Why can't that interpose layer be in QEMU rather than the kernel? It seems that it only must be in the kernel if we need to provide migration state via backdoor, perhaps like going through the PF. So what access to the i40e VF device is not provided to the user through vfio-pci that is necessary to implement migration of this device? The tasks listed above are mostly standard device driver activities and clearly vfio-pci allows userspace device drivers.

Quoted 1 line

do you think it's better to create numerous vendor quirks in vfio-pci?

In QEMU, perhaps. Alternatively, let's look at exactly what access is not provided through vfio-pci that's necessary for this and decide if we want to enable that access or if cracking vfio-pci wide open for vendor drivers to pick and choose when and how to use it is really the right answer.

Quoted 10 lines

as to this series, though patch 9/10 currently only demos reporting a migration region, it actually shows the capability iof vendor driver to customize device regions. e.g. in patch 10/10, it customizes the BAR0 to be read/write. and though we abandoned the REMAP BAR irq_type in patch
10/10 for migration purpose, I have to say this irq_type has its usage in other use cases, where synchronization is not a hard requirement and all it needs is a notification channel from kernel to use. this series just provides a possibility for vendors to customize device regions and irqs.

I don't disagree that a device specific interrupt might be useful, but I would object to implementing this one only as an artificial use case. We can wait for a legitimate use case to implement that.

Quoted 3 lines

for interfaces exported in patch 3/10-5/10, they anyway need to be exported for writing mdev parent drivers that pass through devices at normal time to avoid duplication. and yes, your worry about

Where are those parent drivers? What are their actual requirements?

Quoted 7 lines

identification of bug sources is reasonable. but if a device is binding to vfio-pci with a vendor module loaded, and there's a bug, they can do at least two ways to identify if it's a bug in vfio-pci itself.
(1) prevent vendor modules from loading and see if the problem exists with pure vfio-pci.
(2) do what's demoed in patch 8/10, i.e. do nothing but simply pass all operations to vfio-pci.

The code split is still extremely ad-hoc, there's no API. An mdev driver isn't even a sub-driver of vfio-pci like you're trying to accomplish here, there would need to be a much more defined API when the base device isn't even a vfio_pci_device. I don't see how this series would directly enable an mdev use case.

Quoted 6 lines
Quoted 2 lines

so, do you think this series has its merit and we can continue improving it?

I think this series is trying to push an artificial use case that is perhaps better done in userspace. What is the actual interaction with the VF device that can only be done in the host kernel for this example? Thanks,

Hi Alex,

As shared in KVM Forum last November(https://www.youtube.com/watch?v=aiCCUFXxVEA), we already have one PoC working internally. This series is part of that, if going well, we plan to support it in our future network, storage, security etc. device drivers.

This series has two enhancements to support passthrough device live migration:
general support for SR-IOV live migration and Software assisted dirty page tracking. We tried PoC for other solutions too, but this series seems to work the best balancing on feasibility, code duplication, performance etc.

We are more focusing on enabling our latest E810 NIC product now, but we will check again how we could make it public earlier, as low quality i40e PoC or formal E810 driver, so you may see "the actual interaction" more clearly.

Thanks,
--Shaopeng

Quoted 2 lines

Alex

#26 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 166 lines · Shaopeng
Quoted 153 lines

From: Alex Williamson <[email protected]>
Sent: Thursday, June 4, 2020 12:11 PM

Quoted 9 lines · On Wed, 3 Jun 2020 22:42:28 -0400 Yan Zhao <[email protected]> wrote:
Quoted 8 lines · Alex Williamson
Quoted 5 lines · On Tue, 2 Jun 2020 21:40:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 4 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where

the

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the

extent that a vendor driver requiring this will automatically fail migration.

Quoted 35 lines
Quoted 24 lines
Quoted 7 lines
Quoted 1 line

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

In broad strokes, I don't think we've found the right solution yet. I really question whether it's supportable to parcel out vfio-pci like this and I don't know how I'd support unraveling whether we have a bug in vfio-pci, the vendor driver, or how the vendor driver is making use of vfio-pci.

Let me also ask, why does any of this need to be in the kernel? We spend 5 patches slicing up vfio-pci so that we can register a vendor driver and have that vendor driver call into vfio-pci as it sees fit. We have two patches creating device specific interrupts and a BAR remapping scheme that we've decided we don't need. That brings us to the actual i40e vendor driver, where the first patch is simply making the vendor driver work like vfio-pci already does, the second patch is handling the migration region, and the third patch is implementing the BAR remapping IRQ that we decided we don't need. It's difficult to actually find the small bit of code that's

Quoted 9 lines

required to support migration outside of just dealing with the protocol we've defined to expose this from the kernel. So why are we trying to do this in the kernel? We have quirk support in QEMU, we can easily flip MemoryRegions on and off, etc. What access to the device outside of what vfio-pci provides to the user, and therefore QEMU, is necessary to implement this migration support for i40e VFs? Is this just an exercise in making use of the migration interface? Thanks,

hi Alex

Quoted 7 lines

There was a description of intention of this series in RFC v1
(https://www.spinics.net/lists/kernel/msg3337337.html).
sorry, I didn't include it in starting from RFC v2.

"
The reason why we don't choose the way of writing mdev parent driver is that

I didn't mention an mdev approach, I'm asking what are we accomplishing by doing this in the kernel at all versus exposing the device as normal through vfio-pci and providing the migration support in QEMU. Are you actually leveraging having some sort of access to the PF in supporting migration of the VF? Is vfio-pci masking the device in a way that prevents migrating the state from QEMU?

Quoted 18 lines

(1) VFs are almost all the time directly passthroughed. Directly binding to vfio-pci can make most of the code shared/reused. If we write a vendor specific mdev parent driver, most of the code (like passthrough style of rw/mmap) still needs to be copied from vfio-pci driver, which is actually a duplicated and tedious work.
(2) For features like dynamically trap/untrap pci bars, if they are in vfio-pci, they can be available to most people without repeated code copying and re-testing.
(3) with a 1:1 mdev driver which passes through VFs most of the time, people have to decide whether to bind VFs to vfio-pci or mdev parent driver before it runs into a real migration need. However, if vfio-pci is bound initially, they have no chance to do live migration when there's a need later.
"
particularly, there're some devices (like NVMe) they purely reply on vfio-pci to do device pass-through and they have no standalone parent driver to do mdev way.

Quoted 5 lines

I think live migration is a general requirement for most devices and to interact with the migration interface requires vendor drivers to do device specific tasks like geting/seting device state, starting/stopping devices, tracking dirty data, report migration capabilities... all those works need be in kernel.

I think Alex Graf proved they don't necessarily need to be done in kernel back in 2015: https://www.youtube.com/watch?v=4RFsSgzuFso
He was able to achieve i40e VF live migration by only hacking QEMU. In this series you're allowing a vendor driver to interpose itself between the user (QEMU) and vfio-pci such that we switch to the vendor code during migration. Why can't that interpose layer be in QEMU rather than the kernel? It seems that it only must be in the kernel if we need to provide migration state via backdoor, perhaps like going through the PF. So what access to the i40e VF device is not provided to the user through vfio-pci that is necessary to implement migration of this device? The tasks listed above are mostly standard device driver activities and clearly vfio-pci allows userspace device drivers.

Quoted 1 line

do you think it's better to create numerous vendor quirks in vfio-pci?

In QEMU, perhaps. Alternatively, let's look at exactly what access is not provided through vfio-pci that's necessary for this and decide if we want to enable that access or if cracking vfio-pci wide open for vendor drivers to pick and choose when and how to use it is really the right answer.

Quoted 10 lines

as to this series, though patch 9/10 currently only demos reporting a migration region, it actually shows the capability iof vendor driver to customize device regions. e.g. in patch 10/10, it customizes the BAR0 to be read/write. and though we abandoned the REMAP BAR irq_type in patch
10/10 for migration purpose, I have to say this irq_type has its usage in other use cases, where synchronization is not a hard requirement and all it needs is a notification channel from kernel to use. this series just provides a possibility for vendors to customize device regions and irqs.

I don't disagree that a device specific interrupt might be useful, but I would object to implementing this one only as an artificial use case. We can wait for a legitimate use case to implement that.

Quoted 3 lines

for interfaces exported in patch 3/10-5/10, they anyway need to be exported for writing mdev parent drivers that pass through devices at normal time to avoid duplication. and yes, your worry about

Where are those parent drivers? What are their actual requirements?

Quoted 7 lines

identification of bug sources is reasonable. but if a device is binding to vfio-pci with a vendor module loaded, and there's a bug, they can do at least two ways to identify if it's a bug in vfio-pci itself.
(1) prevent vendor modules from loading and see if the problem exists with pure vfio-pci.
(2) do what's demoed in patch 8/10, i.e. do nothing but simply pass all operations to vfio-pci.

The code split is still extremely ad-hoc, there's no API. An mdev driver isn't even a sub-driver of vfio-pci like you're trying to accomplish here, there would need to be a much more defined API when the base device isn't even a vfio_pci_device. I don't see how this series would directly enable an mdev use case.

Quoted 6 lines
Quoted 2 lines

so, do you think this series has its merit and we can continue improving it?

I think this series is trying to push an artificial use case that is perhaps better done in userspace. What is the actual interaction with the VF device that can only be done in the host kernel for this example? Thanks,

Hi Alex,

As shared in KVM Forum last November(https://www.youtube.com/watch?v=aiCCUFXxVEA), we already have one PoC working internally. This series is part of that, if going well, we plan to support it in our future network, storage, security etc. device drivers.

Quoted 8 lines

This series has two enhancements to support passthrough device live migration:
general support for SR-IOV live migration and Software assisted dirty page tracking. We tried PoC for other solutions too, but this series seems to work the best balancing on feasibility, code duplication, performance etc.

We are more focusing on enabling our latest E810 NIC product now, but we will check again how we could make it public earlier, as low quality i40e PoC or formal E810 driver, so you may see "the actual interaction" more clearly.

"General support for SR-IOV live migration" is not a thing, there's always device specific code. "Software assisted dirty page tracking" implies to me trapping and emulating device accesses in order to learn about DMA targets, which is something that we can also do in QEMU.

In your list of "balancing on feasibility, code duplication, performance, etc", an explicit mention of security is strikingly lacking. The first two are rather obvious, it's more feasible to implement features like migration for any device once the code necessary for such a feature is buried in a vendor driver surrounded by a small development community. The actual protocol of the device state is hidden behind an interface that's opaque to userspace and requires trust that the vendor has implemented a robust scheme and ultimately relying on the vendor for ongoing support. Reducing code duplication by exporting the guts of vfio-pci makes that even easier. Vendors drivers get to ad-hoc pick and choose how and when they interact with vfio-pci, leaving vfio-pci with objects and device state it can't really trust. The performance claim is harder to justify as the path to trapping a region in the kernel vendor driver necessarily passes through trapping that same region in QEMU. Maintaining a dirty bitmap in the vfio IOMMU backend and later retrieving it via the dirty page tracking actually sounds like more overhead than setting dirty bits within QEMU.

But then there's the entire security aspect where the same thing that I think makes this more feasible for the vendor driver opens the door for a much larger attack surface from the user (ie. we're intentionally choosing to open the door to vendor drivers running privileged inside the host kernel, reviewed and supported by smaller communities). Additionally, by masquerading these vendor drivers behind vfio-pci, we simplify usage for the user, but also prevent them from making an informed decision of the security risk of binding a device to "vfio-pci". Is it really vfio-pci, or is it i40e_vf_migration, which may or may not share common code with vfio-pci? VFIO already supports a plugin bus driver architecture, vendors can already supply their own, but users/admin must choose to use it. I originally thought that's something we should abstract for the user, but I'm beginning to see that might actually be the one remaining leverage point we have to architect secure interfaces rather than turning vfio into a playground of insecure and abandoned vendor drivers.

VFIO migration support is trying to provide a solution for device specific state and dirty page tracking, but I don't think the desire to make use of this migration interface in and of itself as justification for an in-kernel vendor driver. We assume with mdevs that we're exposing a portion of a device to the user and the mediation of that portion of the device contains state and awareness of page dirtying that we can extract for migration. Implementing that state awareness and dirty page tracking in the host kernel for the sole purpose of being able to extract it via the migration interface seems like a non-goal and unwise security choice. Thanks,

Alex

author
#27 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

In reply to Alex Williamson
Quoted 61 lines · Alex Williamson
Quoted 50 lines · On Wed, 3 Jun 2020 22:42:28 -0400 Yan Zhao <[email protected]> wrote:
Quoted 38 lines · Alex Williamson
Quoted 13 lines · On Tue, 2 Jun 2020 21:40:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 12 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where the existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the extent that a vendor driver requiring this will automatically fail migration.

Quoted 6 lines

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

In broad strokes, I don't think we've found the right solution yet. I really question whether it's supportable to parcel out vfio-pci like this and I don't know how I'd support unraveling whether we have a bug in vfio-pci, the vendor driver, or how the vendor driver is making use of vfio-pci.

Let me also ask, why does any of this need to be in the kernel? We spend 5 patches slicing up vfio-pci so that we can register a vendor driver and have that vendor driver call into vfio-pci as it sees fit. We have two patches creating device specific interrupts and a BAR remapping scheme that we've decided we don't need. That brings us to the actual i40e vendor driver, where the first patch is simply making the vendor driver work like vfio-pci already does, the second patch is handling the migration region, and the third patch is implementing the BAR remapping IRQ that we decided we don't need. It's difficult to

Quoted 9 lines

actually find the small bit of code that's required to support migration outside of just dealing with the protocol we've defined to expose this from the kernel. So why are we trying to do this in the kernel? We have quirk support in QEMU, we can easily flip MemoryRegions on and off, etc. What access to the device outside of what vfio-pci provides to the user, and therefore QEMU, is necessary to implement this migration support for i40e VFs? Is this just an exercise in making use of the migration interface? Thanks,

hi Alex

Quoted 7 lines

There was a description of intention of this series in RFC v1
(https://www.spinics.net/lists/kernel/msg3337337.html).
sorry, I didn't include it in starting from RFC v2.

"
The reason why we don't choose the way of writing mdev parent driver is that

Quoted 7 lines

I didn't mention an mdev approach, I'm asking what are we accomplishing by doing this in the kernel at all versus exposing the device as normal through vfio-pci and providing the migration support in QEMU. Are you actually leveraging having some sort of access to the PF in supporting migration of the VF? Is vfio-pci masking the device in a way that prevents migrating the state from QEMU?

yes, communication to PF is required. VF state is managed by PF and is queried from PF when VF is stopped.

migration support in QEMU seems only suitable to devices with dirty pages and device state available by reading/writing device MMIOs, which is not the case for most devices.

Quoted 24 lines
Quoted 18 lines

(1) VFs are almost all the time directly passthroughed. Directly binding to vfio-pci can make most of the code shared/reused. If we write a vendor specific mdev parent driver, most of the code (like passthrough style of rw/mmap) still needs to be copied from vfio-pci driver, which is actually a duplicated and tedious work.
(2) For features like dynamically trap/untrap pci bars, if they are in vfio-pci, they can be available to most people without repeated code copying and re-testing.
(3) with a 1:1 mdev driver which passes through VFs most of the time, people have to decide whether to bind VFs to vfio-pci or mdev parent driver before it runs into a real migration need. However, if vfio-pci is bound initially, they have no chance to do live migration when there's a need later.
"
particularly, there're some devices (like NVMe) they purely reply on vfio-pci to do device pass-through and they have no standalone parent driver to do mdev way.

Quoted 5 lines

I think live migration is a general requirement for most devices and to interact with the migration interface requires vendor drivers to do device specific tasks like geting/seting device state, starting/stopping devices, tracking dirty data, report migration capabilities... all those works need be in kernel.

Quoted 3 lines

I think Alex Graf proved they don't necessarily need to be done in kernel back in 2015: https://www.youtube.com/watch?v=4RFsSgzuFso
He was able to achieve i40e VF live migration by only hacking QEMU. In

I checked the qemu code. https://github.com/agraf/qemu/tree/vfio-i40vf.
a new vfio-i40e device type is registered as a child type of vfio-pci, as well as its exclusive savevm handlers, which are not compatible to Kirti's general VFIO live migration framework.

Quoted 10 lines

this series you're allowing a vendor driver to interpose itself between the user (QEMU) and vfio-pci such that we switch to the vendor code during migration. Why can't that interpose layer be in QEMU rather than the kernel? It seems that it only must be in the kernel if we need to provide migration state via backdoor, perhaps like going through the PF. So what access to the i40e VF device is not provided to the user through vfio-pci that is necessary to implement migration of this device? The tasks listed above are mostly standard device driver activities and clearly vfio-pci allows userspace device drivers.

tasks like interacting with PF driver, preparing resources and tracking dirty pages in device internal memory, detecting of whether dirty page is able to be tracked by hardware and reporting migration capabilities, exposing hardware dirty bitmap buffer... all those are hard to be done in QEMU.

maintaining migration code in kernel can also allow vendors to re-use common code for devices across generations. e.g. for i40e, in some generations, software dirty page track is used, some generations hardware dirty track is enabled and some other generations leveraging IOMMU A/D bit is feasible. is QEMU quirks allowing such flexibility as in kernel?

besides, migration version string as a sysfs attribute requires a vendor driver to generate.

Quoted 8 lines
Quoted 1 line

do you think it's better to create numerous vendor quirks in vfio-pci?

In QEMU, perhaps. Alternatively, let's look at exactly what access is not provided through vfio-pci that's necessary for this and decide if we want to enable that access or if cracking vfio-pci wide open for vendor drivers to pick and choose when and how to use it is really the right answer.

I think the position of vendor modules is just like vfio_pci_igd.c under vfio-pci. the difference is that the vendor modules are able to be dynamically loaded outside of vfio-pci.

Quoted 10 lines
Quoted 9 lines

as to this series, though patch 9/10 currently only demos reporting a migration region, it actually shows the capability iof vendor driver to customize device regions. e.g. in patch 10/10, it customizes the BAR0 to be read/write. and though we abandoned the REMAP BAR irq_type in patch 10/10 for migration purpose, I have to say this irq_type has its usage in other use cases, where synchronization is not a hard requirement and all it needs is a notification channel from kernel to use. this series just provides a possibility for vendors to customize device regions and irqs.

Quoted 4 lines

I don't disagree that a device specific interrupt might be useful, but I would object to implementing this one only as an artificial use case. We can wait for a legitimate use case to implement that.

ok. sure.

Quoted 6 lines
Quoted 3 lines

for interfaces exported in patch 3/10-5/10, they anyway need to be exported for writing mdev parent drivers that pass through devices at normal time to avoid duplication. and yes, your worry about

Where are those parent drivers? What are their actual requirements?

if this way of registering vendor ops to vfio-pci is not permitted, vendors have to resort to writing its mdev parent drivers for VFs. Those parent drivers need to pass through VFs at normal time, doing exactly what vfio-pci does and only doing what vendor ops does during migration.

if vfio-pci could export common code to those parent drivers, lots of duplicated code can be avoided.

Quoted 8 lines
Quoted 7 lines

identification of bug sources is reasonable. but if a device is binding to vfio-pci with a vendor module loaded, and there's a bug, they can do at least two ways to identify if it's a bug in vfio-pci itself.
(1) prevent vendor modules from loading and see if the problem exists with pure vfio-pci.
(2) do what's demoed in patch 8/10, i.e. do nothing but simply pass all operations to vfio-pci.

Quoted 6 lines

The code split is still extremely ad-hoc, there's no API. An mdev driver isn't even a sub-driver of vfio-pci like you're trying to accomplish here, there would need to be a much more defined API when the base device isn't even a vfio_pci_device. I don't see how this series would directly enable an mdev use case.

similar to Yi's series https://patchwork.kernel.org/patch/11320841/.
we can parcel the vdev creation code in vfio_pci_probe() to allow calling from mdev parent probe routine. (of course, also need to parcel code to free vdev) e.g.

void *vfio_pci_alloc_vdev(struct pci_dev *pdev, const struct pci_device_id *id)

{
	struct vfio_pci_device *vdev;
        vdev = kzalloc(sizeof(*vdev), GFP_KERNEL);
        if (!vdev) {
                ret = -ENOMEM;
                goto out_group_put;
        }

        vdev->pdev = pdev;
        vdev->irq_type = VFIO_PCI_NUM_IRQS;
        mutex_init(&vdev->igate);
        spin_lock_init(&vdev->irqlock);
        mutex_init(&vdev->ioeventfds_lock);
        INIT_LIST_HEAD(&vdev->ioeventfds_list);
	...
	vfio_pci_probe_power_state(vdev);

        if (!disable_idle_d3) {
                vfio_pci_set_power_state(vdev, PCI_D0);
                vfio_pci_set_power_state(vdev, PCI_D3hot);
        }
	return vdev;
}

static int vfio_mdev_pci_driver_probe(struct pci_dev *pdev, const struct pci_device_id *id)) {

       void *vdev = vfio_pci_alloc_vdev(pdev, id);

       //save the vdev pointer 

}
then all the exported interfaces from this series can also benefit the mdev use case.

Quoted 8 lines
Quoted 2 lines

so, do you think this series has its merit and we can continue improving it?

I think this series is trying to push an artificial use case that is perhaps better done in userspace. What is the actual interaction with the VF device that can only be done in the host kernel for this example? Thanks,

yes, as with shaopeng's reply, looking forward to their early release of code.
Besides, there're other real use cases as well, like NVMe, QAT. we can show the interactions for those devices as well.

Thanks
Yan

#28 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 69 lines · On Thu, 4 Jun 2020 22:02:31 -0400 Yan Zhao <[email protected]> wrote:
Quoted 61 lines · Alex Williamson
Quoted 50 lines · On Wed, 3 Jun 2020 22:42:28 -0400 Yan Zhao <[email protected]> wrote:
Quoted 38 lines · Alex Williamson
Quoted 13 lines · On Tue, 2 Jun 2020 21:40:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 12 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where the existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the extent that a vendor driver requiring this will automatically fail migration.

Quoted 6 lines

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

In broad strokes, I don't think we've found the right solution yet. I really question whether it's supportable to parcel out vfio-pci like this and I don't know how I'd support unraveling whether we have a bug in vfio-pci, the vendor driver, or how the vendor driver is making use of vfio-pci.

Let me also ask, why does any of this need to be in the kernel? We spend 5 patches slicing up vfio-pci so that we can register a vendor driver and have that vendor driver call into vfio-pci as it sees fit. We have two patches creating device specific interrupts and a BAR remapping scheme that we've decided we don't need. That brings us to the actual i40e vendor driver, where the first patch is simply making the vendor driver work like vfio-pci already does, the second patch is handling the migration region, and the third patch is implementing the BAR remapping IRQ that we decided we don't need. It's difficult to

Quoted 9 lines

actually find the small bit of code that's required to support migration outside of just dealing with the protocol we've defined to expose this from the kernel. So why are we trying to do this in the kernel? We have quirk support in QEMU, we can easily flip MemoryRegions on and off, etc. What access to the device outside of what vfio-pci provides to the user, and therefore QEMU, is necessary to implement this migration support for i40e VFs? Is this just an exercise in making use of the migration interface? Thanks,

hi Alex

Quoted 7 lines

There was a description of intention of this series in RFC v1
(https://www.spinics.net/lists/kernel/msg3337337.html).
sorry, I didn't include it in starting from RFC v2.

"
The reason why we don't choose the way of writing mdev parent driver is that

Quoted 7 lines

I didn't mention an mdev approach, I'm asking what are we accomplishing by doing this in the kernel at all versus exposing the device as normal through vfio-pci and providing the migration support in QEMU. Are you actually leveraging having some sort of access to the PF in supporting migration of the VF? Is vfio-pci masking the device in a way that prevents migrating the state from QEMU?

Quoted 6 lines

yes, communication to PF is required. VF state is managed by PF and is queried from PF when VF is stopped.

migration support in QEMU seems only suitable to devices with dirty pages and device state available by reading/writing device MMIOs, which is not the case for most devices.

Post code for such a device.

Quoted 24 lines
Quoted 24 lines
Quoted 23 lines

(1) VFs are almost all the time directly passthroughed. Directly binding to vfio-pci can make most of the code shared/reused. If we write a vendor specific mdev parent driver, most of the code (like passthrough style of rw/mmap) still needs to be copied from vfio-pci driver, which is actually a duplicated and tedious work.
(2) For features like dynamically trap/untrap pci bars, if they are in vfio-pci, they can be available to most people without repeated code copying and re-testing.
(3) with a 1:1 mdev driver which passes through VFs most of the time, people have to decide whether to bind VFs to vfio-pci or mdev parent driver before it runs into a real migration need. However, if vfio-pci is bound initially, they have no chance to do live migration when there's a need later.
"
particularly, there're some devices (like NVMe) they purely reply on vfio-pci to do device pass-through and they have no standalone parent driver to do mdev way.

I think live migration is a general requirement for most devices and to interact with the migration interface requires vendor drivers to do device specific tasks like geting/seting device state, starting/stopping devices, tracking dirty data, report migration capabilities... all those works need be in kernel.

Quoted 7 lines
Quoted 3 lines

I think Alex Graf proved they don't necessarily need to be done in kernel back in 2015: https://www.youtube.com/watch?v=4RFsSgzuFso
He was able to achieve i40e VF live migration by only hacking QEMU. In

I checked the qemu code. https://github.com/agraf/qemu/tree/vfio-i40vf.
a new vfio-i40e device type is registered as a child type of vfio-pci, as well as its exclusive savevm handlers, which are not compatible to Kirti's general VFIO live migration framework.

Obviously, saved state is managed within QEMU. We've already seen pushback to using mdev as a means to implement emulation in the kernel. A vfio migration interface is not an excuse to move everything to in-kernel drivers just to make use of it. IF migration support can be achieved for a device within QEMU, then that's the correct place to put it.

Quoted 10 lines
Quoted 10 lines

this series you're allowing a vendor driver to interpose itself between the user (QEMU) and vfio-pci such that we switch to the vendor code during migration. Why can't that interpose layer be in QEMU rather than the kernel? It seems that it only must be in the kernel if we need to provide migration state via backdoor, perhaps like going through the PF. So what access to the i40e VF device is not provided to the user through vfio-pci that is necessary to implement migration of this device? The tasks listed above are mostly standard device driver activities and clearly vfio-pci allows userspace device drivers.

Quoted 4 lines

tasks like interacting with PF driver, preparing resources and tracking dirty pages in device internal memory, detecting of whether dirty page is able to be tracked by hardware and reporting migration capabilities, exposing hardware dirty bitmap buffer... all those are hard to be done in QEMU.

Something being easier to do in the kernel does not automatically make the kernel the right place to do it. The kernel manages resources, so if access through a PF, where the PF is a shared resources, is necessary then those aspects might justify a kernel interface. We should also consider that the kernel presents a much richer attack vector. QEMU is already confined and a single set of ioctls through vfio-pci is much easier to audit for security than allowing every vendor driver to re-implement their own version. Attempting to re-use vfio-pci code is an effort to contain that risk, but I think it ends up turning into a Frankenstein's monster of intermingled dependencies without a defined API.

Quoted 5 lines

maintaining migration code in kernel can also allow vendors to re-use common code for devices across generations. e.g. for i40e, in some generations, software dirty page track is used, some generations hardware dirty track is enabled and some other generations leveraging IOMMU A/D bit is feasible. is QEMU quirks allowing such flexibility as in kernel?

These arguments all sound like excuses, ie. hiding migration code in the kernel for convenience. Obviously we can re-use code between devices in QEMU. What I think I see happening here is using the vfio migration interface as an excuse to push more code into the kernel, and the vfio-pci vendor extensions are a mechanism to masquerade behind a known driver and avoid defining interfaces for specific features.

Quoted 2 lines

besides, migration version string as a sysfs attribute requires a vendor driver to generate.

We don't have that yet anyway, and it's also a false dependency. The external version string is required _because_ the migration backend is not provided _within_ QEMU. If QEMU manages generating the migration data for a device, we don't need an external version string.

Quoted 11 lines
Quoted 8 lines
Quoted 1 line

do you think it's better to create numerous vendor quirks in vfio-pci?

In QEMU, perhaps. Alternatively, let's look at exactly what access is not provided through vfio-pci that's necessary for this and decide if we want to enable that access or if cracking vfio-pci wide open for vendor drivers to pick and choose when and how to use it is really the right answer.

I think the position of vendor modules is just like vfio_pci_igd.c under vfio-pci. the difference is that the vendor modules are able to be dynamically loaded outside of vfio-pci.

No, this is entirely false. vfio_pci_igd provides two supplemental, read-only regions necessary to satisfy some of the dependencies of the guest driver. It does not attempt to take over the device.

Quoted 22 lines
Quoted 10 lines
Quoted 9 lines

as to this series, though patch 9/10 currently only demos reporting a migration region, it actually shows the capability iof vendor driver to customize device regions. e.g. in patch 10/10, it customizes the BAR0 to be read/write. and though we abandoned the REMAP BAR irq_type in patch 10/10 for migration purpose, I have to say this irq_type has its usage in other use cases, where synchronization is not a hard requirement and all it needs is a notification channel from kernel to use. this series just provides a possibility for vendors to customize device regions and irqs.

Quoted 4 lines

I don't disagree that a device specific interrupt might be useful, but I would object to implementing this one only as an artificial use case. We can wait for a legitimate use case to implement that.

ok. sure.

Quoted 6 lines
Quoted 3 lines

for interfaces exported in patch 3/10-5/10, they anyway need to be exported for writing mdev parent drivers that pass through devices at normal time to avoid duplication. and yes, your worry about

Where are those parent drivers? What are their actual requirements?

Quoted 7 lines

if this way of registering vendor ops to vfio-pci is not permitted, vendors have to resort to writing its mdev parent drivers for VFs. Those parent drivers need to pass through VFs at normal time, doing exactly what vfio-pci does and only doing what vendor ops does during migration.

if vfio-pci could export common code to those parent drivers, lots of duplicated code can be avoided.

There are two sides to this argument though. We could also argue that mdev has already made it too easy to implement device emulation in the kernel, the barrier is that such emulation is more transparent because it does require a fair bit of code duplication from vfio-pci. If we make it easier to simply re-use vfio-pci for much of this, and even take it a step further by allowing vendor drivers to masquerade behind vfio-pci, then we're creating an environment where vendors don't need to work with QEMU to get their device emulation accepted. They can write their own vendor drivers, which are now simplified and sanctioned by exported functions in vfio-pci. They can do this easily and open up massive attack vectors, hiding behind vfio-pci.

I know that I was advocating avoiding user driver confusion, ie. does the user bind a device to vfio-pci, i40e_vf_vfio, etc, but maybe that's the barrier we need such that a user can make an informed decision about what they're actually using. If a vendor then wants to implement a feature in vfio-pci, we'll need to architect an interface for it rather than letting them pick and choose which pieces of vfio-pci to override.

Quoted 47 lines
Quoted 8 lines
Quoted 7 lines

identification of bug sources is reasonable. but if a device is binding to vfio-pci with a vendor module loaded, and there's a bug, they can do at least two ways to identify if it's a bug in vfio-pci itself.
(1) prevent vendor modules from loading and see if the problem exists with pure vfio-pci.
(2) do what's demoed in patch 8/10, i.e. do nothing but simply pass all operations to vfio-pci.

Quoted 6 lines

The code split is still extremely ad-hoc, there's no API. An mdev driver isn't even a sub-driver of vfio-pci like you're trying to accomplish here, there would need to be a much more defined API when the base device isn't even a vfio_pci_device. I don't see how this series would directly enable an mdev use case.

similar to Yi's series https://patchwork.kernel.org/patch/11320841/.
we can parcel the vdev creation code in vfio_pci_probe() to allow calling from mdev parent probe routine. (of course, also need to parcel code to free vdev) e.g.

void *vfio_pci_alloc_vdev(struct pci_dev *pdev, const struct pci_device_id *id)

{
	struct vfio_pci_device *vdev;
        vdev = kzalloc(sizeof(*vdev), GFP_KERNEL);
        if (!vdev) {
                ret = -ENOMEM;
                goto out_group_put;
        }

        vdev->pdev = pdev;
        vdev->irq_type = VFIO_PCI_NUM_IRQS;
        mutex_init(&vdev->igate);
        spin_lock_init(&vdev->irqlock);
        mutex_init(&vdev->ioeventfds_lock);
        INIT_LIST_HEAD(&vdev->ioeventfds_list);
	...
	vfio_pci_probe_power_state(vdev);

        if (!disable_idle_d3) {
                vfio_pci_set_power_state(vdev, PCI_D0);
                vfio_pci_set_power_state(vdev, PCI_D3hot);
        }
	return vdev;
}

static int vfio_mdev_pci_driver_probe(struct pci_dev *pdev, const struct pci_device_id *id)) {

Quoted 7 lines
       void *vdev = vfio_pci_alloc_vdev(pdev, id);

       //save the vdev pointer 

}
then all the exported interfaces from this series can also benefit the mdev use case.

You need to convince me that we're not just doing this for the sake of re-using a migration interface. We do need vendor specific drivers to support migration, but implementing those vendor specific drivers in the kernel just because we have that interface is the wrong answer. If we can implement that device specific migration support in QEMU and limit the attack surface from the hypervisor or guest into the host kernel, that's a better answer. As I've noted above, I'm afraid all of these attempts to parcel out vfio-pci are only going to serve to proliferate vendor modules that have limited community review, expand the attack surface, and potentially harm the vfio ecosystem overall through bad actors and reduced autonomy. Thanks,

Alex

author
#29 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 72 lines · Alex Williamson
Quoted 69 lines · On Thu, 4 Jun 2020 22:02:31 -0400 Yan Zhao <[email protected]> wrote:
Quoted 68 lines · Alex Williamson
Quoted 50 lines · On Wed, 3 Jun 2020 22:42:28 -0400 Yan Zhao <[email protected]> wrote:
Quoted 38 lines · Alex Williamson
Quoted 13 lines · On Tue, 2 Jun 2020 21:40:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 12 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where the existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the extent that a vendor driver requiring this will automatically fail migration.

Quoted 6 lines

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

In broad strokes, I don't think we've found the right solution yet. I really question whether it's supportable to parcel out vfio-pci like this and I don't know how I'd support unraveling whether we have a bug in vfio-pci, the vendor driver, or how the vendor driver is making use of vfio-pci.

Let me also ask, why does any of this need to be in the kernel? We spend 5 patches slicing up vfio-pci so that we can register a vendor driver and have that vendor driver call into vfio-pci as it sees fit. We have two patches creating device specific interrupts and a BAR remapping scheme that we've decided we don't need. That brings us to the actual i40e vendor driver, where the first patch is simply making the vendor driver work like vfio-pci already does, the second patch is handling the migration region, and the third patch is implementing the BAR remapping IRQ that we decided we don't need. It's difficult to

Quoted 9 lines

actually find the small bit of code that's required to support migration outside of just dealing with the protocol we've defined to expose this from the kernel. So why are we trying to do this in the kernel? We have quirk support in QEMU, we can easily flip MemoryRegions on and off, etc. What access to the device outside of what vfio-pci provides to the user, and therefore QEMU, is necessary to implement this migration support for i40e VFs? Is this just an exercise in making use of the migration interface? Thanks,

hi Alex

Quoted 7 lines

There was a description of intention of this series in RFC v1
(https://www.spinics.net/lists/kernel/msg3337337.html).
sorry, I didn't include it in starting from RFC v2.

"
The reason why we don't choose the way of writing mdev parent driver is that

I didn't mention an mdev approach, I'm asking what are we accomplishing by doing this in the kernel at all versus exposing the device as normal through vfio-pci and providing the migration support in QEMU. Are you actually leveraging having some sort of access to the PF in supporting migration of the VF? Is vfio-pci masking the device in a way that prevents migrating the state from QEMU?

Quoted 9 lines
Quoted 6 lines

yes, communication to PF is required. VF state is managed by PF and is queried from PF when VF is stopped.

migration support in QEMU seems only suitable to devices with dirty pages and device state available by reading/writing device MMIOs, which is not the case for most devices.

Post code for such a device.

hi Alex,
There's an example in i40e vf. virtual channel related resources are in guest memory. dirty page tracking requires the info stored in those guest memory.

there're two ways to get the resources addresses:
(1) always trap VF registers related. as in Alex Graf's qemu code.

starting from beginning, it tracks rw of Admin Queue Configuration registers. Then in the write handler vfio_i40evf_aq_mmio_mem_region_write(), guest commands are processed to record the guest dma addresses of the virtual channel related resources.
e.g. vdev->vsi_config is read from the guest dma addr contained in command I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES.

vfio_i40evf_initfn()
{
...
memory_region_init_io(&vdev->aq_mmio_mem, OBJECT(dev),

                          &vfio_i40evf_aq_mmio_mem_region_ops,
                          vdev, "i40evf AQ config",
                          I40E_VFGEN_RSTAT - I40E_VF_ARQBAH1);

...
}

vfio_i40evf_aq_mmio_mem_region_write()
{
...

    switch (addr) {
    case I40E_VF_ARQBAH1:
    case I40E_VF_ARQBAL1:
    case I40E_VF_ARQH1:
    case I40E_VF_ARQLEN1:
    case I40E_VF_ARQT1:
    case I40E_VF_ATQBAH1:
    case I40E_VF_ATQBAL1:
    case I40E_VF_ATQH1:
    case I40E_VF_ATQT1:
    case I40E_VF_ATQLEN1:
        vfio_i40evf_vw32(vdev, addr, data);
        vfio_i40e_aq_update(vdev); ==> update & process atq commands
        break;
    default:
        vfio_i40evf_w32(vdev, addr, data);
        break;
    }
}
vfio_i40e_aq_update(vdev)
	|->vfio_i40e_atq_process_one(vdev, vfio_i40evf_vr32(vdev, I40E_VF_ATQH1)
		|-> hwaddr addr = vfio_i40e_get_atqba(vdev) + (index * sizeof(desc));
		|   pci_dma_read(pdev, addr, &desc, sizeof(desc));//read guest's command
		|   vfio_i40e_record_atq_cmd(vdev, pdev, &desc)
vfio_i40e_record_atq_cmd(...I40eAdminQueueDescriptor *desc) {
	data_addr = desc->params.external.addr_high;
	...

	switch (desc->cookie_high) {
	...
	case I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES:
	pci_dma_read(pdev, data_addr, &vdev->vsi_config,
		         MIN(desc->datalen, sizeof(vdev->vsi_config)));
	...
	}
	...
}

(2) pass through all guest MMIO accesses and only do MMIO trap when migration is about to start.
This is the way we're using in the host vfio-pci vendor driver (or mdev parent driver) of i40e vf device (sorry for no public code available still).

when migration is about to start, it's already too late to get the guest dma address for those virtual channel related resources merely by MMIO trapping, so we have to ask for them from PF.

<...>

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

for interfaces exported in patch 3/10-5/10, they anyway need to be exported for writing mdev parent drivers that pass through devices at normal time to avoid duplication. and yes, your worry about

Where are those parent drivers? What are their actual requirements?

Quoted 7 lines

if this way of registering vendor ops to vfio-pci is not permitted, vendors have to resort to writing its mdev parent drivers for VFs. Those parent drivers need to pass through VFs at normal time, doing exactly what vfio-pci does and only doing what vendor ops does during migration.

if vfio-pci could export common code to those parent drivers, lots of duplicated code can be avoided.

Quoted 12 lines

There are two sides to this argument though. We could also argue that mdev has already made it too easy to implement device emulation in the kernel, the barrier is that such emulation is more transparent because it does require a fair bit of code duplication from vfio-pci. If we make it easier to simply re-use vfio-pci for much of this, and even take it a step further by allowing vendor drivers to masquerade behind vfio-pci, then we're creating an environment where vendors don't need to work with QEMU to get their device emulation accepted. They can write their own vendor drivers, which are now simplified and sanctioned by exported functions in vfio-pci. They can do this easily and open up massive attack vectors, hiding behind vfio-pci.

your concern is reasonable.

Quoted 63 lines

I know that I was advocating avoiding user driver confusion, ie. does the user bind a device to vfio-pci, i40e_vf_vfio, etc, but maybe that's the barrier we need such that a user can make an informed decision about what they're actually using. If a vendor then wants to implement a feature in vfio-pci, we'll need to architect an interface for it rather than letting them pick and choose which pieces of vfio-pci to override.

Quoted 47 lines
Quoted 8 lines
Quoted 7 lines

identification of bug sources is reasonable. but if a device is binding to vfio-pci with a vendor module loaded, and there's a bug, they can do at least two ways to identify if it's a bug in vfio-pci itself.
(1) prevent vendor modules from loading and see if the problem exists with pure vfio-pci.
(2) do what's demoed in patch 8/10, i.e. do nothing but simply pass all operations to vfio-pci.

Quoted 6 lines

The code split is still extremely ad-hoc, there's no API. An mdev driver isn't even a sub-driver of vfio-pci like you're trying to accomplish here, there would need to be a much more defined API when the base device isn't even a vfio_pci_device. I don't see how this series would directly enable an mdev use case.

similar to Yi's series https://patchwork.kernel.org/patch/11320841/.
we can parcel the vdev creation code in vfio_pci_probe() to allow calling from mdev parent probe routine. (of course, also need to parcel code to free vdev) e.g.

void *vfio_pci_alloc_vdev(struct pci_dev *pdev, const struct pci_device_id *id)

{
	struct vfio_pci_device *vdev;
        vdev = kzalloc(sizeof(*vdev), GFP_KERNEL);
        if (!vdev) {
                ret = -ENOMEM;
                goto out_group_put;
        }

        vdev->pdev = pdev;
        vdev->irq_type = VFIO_PCI_NUM_IRQS;
        mutex_init(&vdev->igate);
        spin_lock_init(&vdev->irqlock);
        mutex_init(&vdev->ioeventfds_lock);
        INIT_LIST_HEAD(&vdev->ioeventfds_list);
	...
	vfio_pci_probe_power_state(vdev);

        if (!disable_idle_d3) {
                vfio_pci_set_power_state(vdev, PCI_D0);
                vfio_pci_set_power_state(vdev, PCI_D3hot);
        }
	return vdev;
}

static int vfio_mdev_pci_driver_probe(struct pci_dev *pdev, const struct pci_device_id *id)) {

Quoted 7 lines
       void *vdev = vfio_pci_alloc_vdev(pdev, id);

       //save the vdev pointer 

}
then all the exported interfaces from this series can also benefit the mdev use case.

Quoted 12 lines

You need to convince me that we're not just doing this for the sake of re-using a migration interface. We do need vendor specific drivers to support migration, but implementing those vendor specific drivers in the kernel just because we have that interface is the wrong answer. If we can implement that device specific migration support in QEMU and limit the attack surface from the hypervisor or guest into the host kernel, that's a better answer. As I've noted above, I'm afraid all of these attempts to parcel out vfio-pci are only going to serve to proliferate vendor modules that have limited community review, expand the attack surface, and potentially harm the vfio ecosystem overall through bad actors and reduced autonomy. Thanks,

The requirement to access PF as mentioned above is one of the reason for us to implement the emulation in kernel.
Another reason is that we don't want to duplicate a lot of kernel logic in QEMU as what'd done in Alex Graf's "vfio-i40e". then QEMU has to be updated along kernel driver changing. The effort for maintenance and version matching is a big burden to vendors.
But you are right, there're less review in virtualization side to code under vendor specific directory. That's also the pulse for us to propose common helper APIs for them to call, not only for convenience and duplication-less, but also for code with full review.

would you mind giving us some suggestions for where to go?

Thanks
Yan

#30 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 280 lines · On Wed, 10 Jun 2020 01:23:14 -0400 Yan Zhao <[email protected]> wrote:
Quoted 72 lines · Alex Williamson
Quoted 69 lines · On Thu, 4 Jun 2020 22:02:31 -0400 Yan Zhao <[email protected]> wrote:
Quoted 68 lines · Alex Williamson
Quoted 50 lines · On Wed, 3 Jun 2020 22:42:28 -0400 Yan Zhao <[email protected]> wrote:
Quoted 38 lines · Alex Williamson
Quoted 13 lines · On Tue, 2 Jun 2020 21:40:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 12 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where the existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the extent that a vendor driver requiring this will automatically fail migration.

Quoted 6 lines

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

In broad strokes, I don't think we've found the right solution yet. I really question whether it's supportable to parcel out vfio-pci like this and I don't know how I'd support unraveling whether we have a bug in vfio-pci, the vendor driver, or how the vendor driver is making use of vfio-pci.

Let me also ask, why does any of this need to be in the kernel? We spend 5 patches slicing up vfio-pci so that we can register a vendor driver and have that vendor driver call into vfio-pci as it sees fit. We have two patches creating device specific interrupts and a BAR remapping scheme that we've decided we don't need. That brings us to the actual i40e vendor driver, where the first patch is simply making the vendor driver work like vfio-pci already does, the second patch is handling the migration region, and the third patch is implementing the BAR remapping IRQ that we decided we don't need. It's difficult to

Quoted 9 lines

actually find the small bit of code that's required to support migration outside of just dealing with the protocol we've defined to expose this from the kernel. So why are we trying to do this in the kernel? We have quirk support in QEMU, we can easily flip MemoryRegions on and off, etc. What access to the device outside of what vfio-pci provides to the user, and therefore QEMU, is necessary to implement this migration support for i40e VFs? Is this just an exercise in making use of the migration interface? Thanks,

hi Alex

Quoted 7 lines

There was a description of intention of this series in RFC v1
(https://www.spinics.net/lists/kernel/msg3337337.html).
sorry, I didn't include it in starting from RFC v2.

"
The reason why we don't choose the way of writing mdev parent driver is that

I didn't mention an mdev approach, I'm asking what are we accomplishing by doing this in the kernel at all versus exposing the device as normal through vfio-pci and providing the migration support in QEMU. Are you actually leveraging having some sort of access to the PF in supporting migration of the VF? Is vfio-pci masking the device in a way that prevents migrating the state from QEMU?

Quoted 9 lines
Quoted 6 lines

yes, communication to PF is required. VF state is managed by PF and is queried from PF when VF is stopped.

migration support in QEMU seems only suitable to devices with dirty pages and device state available by reading/writing device MMIOs, which is not the case for most devices.

Post code for such a device.

hi Alex,
There's an example in i40e vf. virtual channel related resources are in guest memory. dirty page tracking requires the info stored in those guest memory.

there're two ways to get the resources addresses:
(1) always trap VF registers related. as in Alex Graf's qemu code.

starting from beginning, it tracks rw of Admin Queue Configuration registers. Then in the write handler vfio_i40evf_aq_mmio_mem_region_write(), guest commands are processed to record the guest dma addresses of the virtual channel related resources.
e.g. vdev->vsi_config is read from the guest dma addr contained in command I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES.

vfio_i40evf_initfn()
{
...
memory_region_init_io(&vdev->aq_mmio_mem, OBJECT(dev),

                          &vfio_i40evf_aq_mmio_mem_region_ops,
                          vdev, "i40evf AQ config",
                          I40E_VFGEN_RSTAT - I40E_VF_ARQBAH1);

...
}

vfio_i40evf_aq_mmio_mem_region_write()

{
   ...
    switch (addr) {
    case I40E_VF_ARQBAH1:
    case I40E_VF_ARQBAL1:
    case I40E_VF_ARQH1:
    case I40E_VF_ARQLEN1:
    case I40E_VF_ARQT1:
    case I40E_VF_ATQBAH1:
    case I40E_VF_ATQBAL1:
    case I40E_VF_ATQH1:
    case I40E_VF_ATQT1:
    case I40E_VF_ATQLEN1:
        vfio_i40evf_vw32(vdev, addr, data);
        vfio_i40e_aq_update(vdev); ==> update & process atq commands
        break;
    default:
        vfio_i40evf_w32(vdev, addr, data);
        break;
    }
}
vfio_i40e_aq_update(vdev)
	|->vfio_i40e_atq_process_one(vdev, vfio_i40evf_vr32(vdev, I40E_VF_ATQH1)
		|-> hwaddr addr = vfio_i40e_get_atqba(vdev) + (index * sizeof(desc));
		|   pci_dma_read(pdev, addr, &desc, sizeof(desc));//read guest's command
		|   vfio_i40e_record_atq_cmd(vdev, pdev, &desc)
vfio_i40e_record_atq_cmd(...I40eAdminQueueDescriptor *desc) {
	data_addr = desc->params.external.addr_high;
	...

	switch (desc->cookie_high) {
	...
	case I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES:
	pci_dma_read(pdev, data_addr, &vdev->vsi_config,
		         MIN(desc->datalen, sizeof(vdev->vsi_config)));
	...
	}
	...
}

(2) pass through all guest MMIO accesses and only do MMIO trap when migration is about to start.
This is the way we're using in the host vfio-pci vendor driver (or mdev parent driver) of i40e vf device (sorry for no public code available still).

when migration is about to start, it's already too late to get the guest dma address for those virtual channel related resources merely by MMIO trapping, so we have to ask for them from PF.

<...>

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

for interfaces exported in patch 3/10-5/10, they anyway need to be exported for writing mdev parent drivers that pass through devices at normal time to avoid duplication. and yes, your worry about

Where are those parent drivers? What are their actual requirements?

Quoted 7 lines

if this way of registering vendor ops to vfio-pci is not permitted, vendors have to resort to writing its mdev parent drivers for VFs. Those parent drivers need to pass through VFs at normal time, doing exactly what vfio-pci does and only doing what vendor ops does during migration.

if vfio-pci could export common code to those parent drivers, lots of duplicated code can be avoided.

Quoted 12 lines

There are two sides to this argument though. We could also argue that mdev has already made it too easy to implement device emulation in the kernel, the barrier is that such emulation is more transparent because it does require a fair bit of code duplication from vfio-pci. If we make it easier to simply re-use vfio-pci for much of this, and even take it a step further by allowing vendor drivers to masquerade behind vfio-pci, then we're creating an environment where vendors don't need to work with QEMU to get their device emulation accepted. They can write their own vendor drivers, which are now simplified and sanctioned by exported functions in vfio-pci. They can do this easily and open up massive attack vectors, hiding behind vfio-pci.

your concern is reasonable.

Quoted 63 lines

I know that I was advocating avoiding user driver confusion, ie. does the user bind a device to vfio-pci, i40e_vf_vfio, etc, but maybe that's the barrier we need such that a user can make an informed decision about what they're actually using. If a vendor then wants to implement a feature in vfio-pci, we'll need to architect an interface for it rather than letting them pick and choose which pieces of vfio-pci to override.

Quoted 47 lines
Quoted 8 lines
Quoted 7 lines

identification of bug sources is reasonable. but if a device is binding to vfio-pci with a vendor module loaded, and there's a bug, they can do at least two ways to identify if it's a bug in vfio-pci itself.
(1) prevent vendor modules from loading and see if the problem exists with pure vfio-pci.
(2) do what's demoed in patch 8/10, i.e. do nothing but simply pass all operations to vfio-pci.

Quoted 6 lines

The code split is still extremely ad-hoc, there's no API. An mdev driver isn't even a sub-driver of vfio-pci like you're trying to accomplish here, there would need to be a much more defined API when the base device isn't even a vfio_pci_device. I don't see how this series would directly enable an mdev use case.

similar to Yi's series https://patchwork.kernel.org/patch/11320841/.
we can parcel the vdev creation code in vfio_pci_probe() to allow calling from mdev parent probe routine. (of course, also need to parcel code to free vdev) e.g.

void *vfio_pci_alloc_vdev(struct pci_dev *pdev, const struct pci_device_id *id)

{
	struct vfio_pci_device *vdev;
        vdev = kzalloc(sizeof(*vdev), GFP_KERNEL);
        if (!vdev) {
                ret = -ENOMEM;
                goto out_group_put;
        }

        vdev->pdev = pdev;
        vdev->irq_type = VFIO_PCI_NUM_IRQS;
        mutex_init(&vdev->igate);
        spin_lock_init(&vdev->irqlock);
        mutex_init(&vdev->ioeventfds_lock);
        INIT_LIST_HEAD(&vdev->ioeventfds_list);
	...
	vfio_pci_probe_power_state(vdev);

        if (!disable_idle_d3) {
                vfio_pci_set_power_state(vdev, PCI_D0);
                vfio_pci_set_power_state(vdev, PCI_D3hot);
        }
	return vdev;
}

static int vfio_mdev_pci_driver_probe(struct pci_dev *pdev, const struct pci_device_id *id)) {

Quoted 7 lines
       void *vdev = vfio_pci_alloc_vdev(pdev, id);

       //save the vdev pointer 

}
then all the exported interfaces from this series can also benefit the mdev use case.

Quoted 12 lines

You need to convince me that we're not just doing this for the sake of re-using a migration interface. We do need vendor specific drivers to support migration, but implementing those vendor specific drivers in the kernel just because we have that interface is the wrong answer. If we can implement that device specific migration support in QEMU and limit the attack surface from the hypervisor or guest into the host kernel, that's a better answer. As I've noted above, I'm afraid all of these attempts to parcel out vfio-pci are only going to serve to proliferate vendor modules that have limited community review, expand the attack surface, and potentially harm the vfio ecosystem overall through bad actors and reduced autonomy. Thanks,

The requirement to access PF as mentioned above is one of the reason for us to implement the emulation in kernel.
Another reason is that we don't want to duplicate a lot of kernel logic in QEMU as what'd done in Alex Graf's "vfio-i40e". then QEMU has to be updated along kernel driver changing. The effort for maintenance and version matching is a big burden to vendors.
But you are right, there're less review in virtualization side to code under vendor specific directory. That's also the pulse for us to propose common helper APIs for them to call, not only for convenience and duplication-less, but also for code with full review.

Quoted 1 line

would you mind giving us some suggestions for where to go?

Not duplicating kernel code into userspace isn't a great excuse. What we need to do to emulate a device is not an exact mapping to what a driver for that device needs to do. If we need to keep the device driver and the emulation in sync then we haven't done a good job with the emulation. What would it look like if we only had an additional device specific region on the vfio device fd we could use to get the descriptor information we need from the PF? This would be more inline with the quirks we provide for IGD assignment. Thanks,

Alex

author
#31 lore raw

Re: [RFC PATCH v4 07/10] vfio/pci: introduce a new irq type VFIO_IRQ_TYPE_REMAP_BAR_REGION

Quoted 285 lines · Alex Williamson
Quoted 280 lines · On Wed, 10 Jun 2020 01:23:14 -0400 Yan Zhao <[email protected]> wrote:
Quoted 72 lines · Alex Williamson
Quoted 69 lines · On Thu, 4 Jun 2020 22:02:31 -0400 Yan Zhao <[email protected]> wrote:
Quoted 68 lines · Alex Williamson
Quoted 50 lines · On Wed, 3 Jun 2020 22:42:28 -0400 Yan Zhao <[email protected]> wrote:
Quoted 38 lines · Alex Williamson
Quoted 13 lines · On Tue, 2 Jun 2020 21:40:58 -0400 Yan Zhao <[email protected]> wrote:
Quoted 12 lines · Alex Williamson

I'm not at all happy with this. Why do we need to hide the migration sparse mmap from the user until migration time? What if instead we introduced a new VFIO_REGION_INFO_CAP_SPARSE_MMAP_SAVING capability where the existing capability is the normal runtime sparse setup and the user is required to use this new one prior to enabled device_state with _SAVING. The vendor driver could then simply track mmap vmas to the region and refuse to change device_state if there are outstanding mmaps conflicting with the _SAVING sparse mmap layout. No new IRQs required, no new irqfds, an incremental change to the protocol, backwards compatible to the extent that a vendor driver requiring this will automatically fail migration.

Quoted 6 lines

right. looks we need to use this approach to solve the problem. thanks for your guide.
so I'll abandon the current remap irq way for dirty tracking during live migration.
but anyway, it demos how to customize irq_types in vendor drivers. then, what do you think about patches 1-5?

In broad strokes, I don't think we've found the right solution yet. I really question whether it's supportable to parcel out vfio-pci like this and I don't know how I'd support unraveling whether we have a bug in vfio-pci, the vendor driver, or how the vendor driver is making use of vfio-pci.

Let me also ask, why does any of this need to be in the kernel? We spend 5 patches slicing up vfio-pci so that we can register a vendor driver and have that vendor driver call into vfio-pci as it sees fit. We have two patches creating device specific interrupts and a BAR remapping scheme that we've decided we don't need. That brings us to the actual i40e vendor driver, where the first patch is simply making the vendor driver work like vfio-pci already does, the second patch is handling the migration region, and the third patch is implementing the BAR remapping IRQ that we decided we don't need. It's difficult to

Quoted 9 lines

actually find the small bit of code that's required to support migration outside of just dealing with the protocol we've defined to expose this from the kernel. So why are we trying to do this in the kernel? We have quirk support in QEMU, we can easily flip MemoryRegions on and off, etc. What access to the device outside of what vfio-pci provides to the user, and therefore QEMU, is necessary to implement this migration support for i40e VFs? Is this just an exercise in making use of the migration interface? Thanks,

hi Alex

Quoted 7 lines

There was a description of intention of this series in RFC v1
(https://www.spinics.net/lists/kernel/msg3337337.html).
sorry, I didn't include it in starting from RFC v2.

"
The reason why we don't choose the way of writing mdev parent driver is that

I didn't mention an mdev approach, I'm asking what are we accomplishing by doing this in the kernel at all versus exposing the device as normal through vfio-pci and providing the migration support in QEMU. Are you actually leveraging having some sort of access to the PF in supporting migration of the VF? Is vfio-pci masking the device in a way that prevents migrating the state from QEMU?

Quoted 9 lines
Quoted 6 lines

yes, communication to PF is required. VF state is managed by PF and is queried from PF when VF is stopped.

migration support in QEMU seems only suitable to devices with dirty pages and device state available by reading/writing device MMIOs, which is not the case for most devices.

Post code for such a device.

hi Alex,
There's an example in i40e vf. virtual channel related resources are in guest memory. dirty page tracking requires the info stored in those guest memory.

there're two ways to get the resources addresses:
(1) always trap VF registers related. as in Alex Graf's qemu code.

starting from beginning, it tracks rw of Admin Queue Configuration registers. Then in the write handler vfio_i40evf_aq_mmio_mem_region_write(), guest commands are processed to record the guest dma addresses of the virtual channel related resources.
e.g. vdev->vsi_config is read from the guest dma addr contained in command I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES.

vfio_i40evf_initfn()
{
...
memory_region_init_io(&vdev->aq_mmio_mem, OBJECT(dev),

                          &vfio_i40evf_aq_mmio_mem_region_ops,
                          vdev, "i40evf AQ config",
                          I40E_VFGEN_RSTAT - I40E_VF_ARQBAH1);

...
}

vfio_i40evf_aq_mmio_mem_region_write()

{
   ...
    switch (addr) {
    case I40E_VF_ARQBAH1:
    case I40E_VF_ARQBAL1:
    case I40E_VF_ARQH1:
    case I40E_VF_ARQLEN1:
    case I40E_VF_ARQT1:
    case I40E_VF_ATQBAH1:
    case I40E_VF_ATQBAL1:
    case I40E_VF_ATQH1:
    case I40E_VF_ATQT1:
    case I40E_VF_ATQLEN1:
        vfio_i40evf_vw32(vdev, addr, data);
        vfio_i40e_aq_update(vdev); ==> update & process atq commands
        break;
    default:
        vfio_i40evf_w32(vdev, addr, data);
        break;
    }
}
vfio_i40e_aq_update(vdev)
	|->vfio_i40e_atq_process_one(vdev, vfio_i40evf_vr32(vdev, I40E_VF_ATQH1)
		|-> hwaddr addr = vfio_i40e_get_atqba(vdev) + (index * sizeof(desc));
		|   pci_dma_read(pdev, addr, &desc, sizeof(desc));//read guest's command
		|   vfio_i40e_record_atq_cmd(vdev, pdev, &desc)
vfio_i40e_record_atq_cmd(...I40eAdminQueueDescriptor *desc) {
	data_addr = desc->params.external.addr_high;
	...

	switch (desc->cookie_high) {
	...
	case I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES:
	pci_dma_read(pdev, data_addr, &vdev->vsi_config,
		         MIN(desc->datalen, sizeof(vdev->vsi_config)));
	...
	}
	...
}

(2) pass through all guest MMIO accesses and only do MMIO trap when migration is about to start.
This is the way we're using in the host vfio-pci vendor driver (or mdev parent driver) of i40e vf device (sorry for no public code available still).

when migration is about to start, it's already too late to get the guest dma address for those virtual channel related resources merely by MMIO trapping, so we have to ask for them from PF.

<...>

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

for interfaces exported in patch 3/10-5/10, they anyway need to be exported for writing mdev parent drivers that pass through devices at normal time to avoid duplication. and yes, your worry about

Where are those parent drivers? What are their actual requirements?

Quoted 7 lines

if this way of registering vendor ops to vfio-pci is not permitted, vendors have to resort to writing its mdev parent drivers for VFs. Those parent drivers need to pass through VFs at normal time, doing exactly what vfio-pci does and only doing what vendor ops does during migration.

if vfio-pci could export common code to those parent drivers, lots of duplicated code can be avoided.

Quoted 12 lines

There are two sides to this argument though. We could also argue that mdev has already made it too easy to implement device emulation in the kernel, the barrier is that such emulation is more transparent because it does require a fair bit of code duplication from vfio-pci. If we make it easier to simply re-use vfio-pci for much of this, and even take it a step further by allowing vendor drivers to masquerade behind vfio-pci, then we're creating an environment where vendors don't need to work with QEMU to get their device emulation accepted. They can write their own vendor drivers, which are now simplified and sanctioned by exported functions in vfio-pci. They can do this easily and open up massive attack vectors, hiding behind vfio-pci.

your concern is reasonable.

Quoted 63 lines

I know that I was advocating avoiding user driver confusion, ie. does the user bind a device to vfio-pci, i40e_vf_vfio, etc, but maybe that's the barrier we need such that a user can make an informed decision about what they're actually using. If a vendor then wants to implement a feature in vfio-pci, we'll need to architect an interface for it rather than letting them pick and choose which pieces of vfio-pci to override.

Quoted 47 lines
Quoted 8 lines
Quoted 7 lines

identification of bug sources is reasonable. but if a device is binding to vfio-pci with a vendor module loaded, and there's a bug, they can do at least two ways to identify if it's a bug in vfio-pci itself.
(1) prevent vendor modules from loading and see if the problem exists with pure vfio-pci.
(2) do what's demoed in patch 8/10, i.e. do nothing but simply pass all operations to vfio-pci.

Quoted 6 lines

The code split is still extremely ad-hoc, there's no API. An mdev driver isn't even a sub-driver of vfio-pci like you're trying to accomplish here, there would need to be a much more defined API when the base device isn't even a vfio_pci_device. I don't see how this series would directly enable an mdev use case.

similar to Yi's series https://patchwork.kernel.org/patch/11320841/.
we can parcel the vdev creation code in vfio_pci_probe() to allow calling from mdev parent probe routine. (of course, also need to parcel code to free vdev) e.g.

void *vfio_pci_alloc_vdev(struct pci_dev *pdev, const struct pci_device_id *id)

{
	struct vfio_pci_device *vdev;
        vdev = kzalloc(sizeof(*vdev), GFP_KERNEL);
        if (!vdev) {
                ret = -ENOMEM;
                goto out_group_put;
        }

        vdev->pdev = pdev;
        vdev->irq_type = VFIO_PCI_NUM_IRQS;
        mutex_init(&vdev->igate);
        spin_lock_init(&vdev->irqlock);
        mutex_init(&vdev->ioeventfds_lock);
        INIT_LIST_HEAD(&vdev->ioeventfds_list);
	...
	vfio_pci_probe_power_state(vdev);

        if (!disable_idle_d3) {
                vfio_pci_set_power_state(vdev, PCI_D0);
                vfio_pci_set_power_state(vdev, PCI_D3hot);
        }
	return vdev;
}

static int vfio_mdev_pci_driver_probe(struct pci_dev *pdev, const struct pci_device_id *id)) {

Quoted 7 lines
       void *vdev = vfio_pci_alloc_vdev(pdev, id);

       //save the vdev pointer 

}
then all the exported interfaces from this series can also benefit the mdev use case.

Quoted 12 lines

You need to convince me that we're not just doing this for the sake of re-using a migration interface. We do need vendor specific drivers to support migration, but implementing those vendor specific drivers in the kernel just because we have that interface is the wrong answer. If we can implement that device specific migration support in QEMU and limit the attack surface from the hypervisor or guest into the host kernel, that's a better answer. As I've noted above, I'm afraid all of these attempts to parcel out vfio-pci are only going to serve to proliferate vendor modules that have limited community review, expand the attack surface, and potentially harm the vfio ecosystem overall through bad actors and reduced autonomy. Thanks,

The requirement to access PF as mentioned above is one of the reason for us to implement the emulation in kernel.
Another reason is that we don't want to duplicate a lot of kernel logic in QEMU as what'd done in Alex Graf's "vfio-i40e". then QEMU has to be updated along kernel driver changing. The effort for maintenance and version matching is a big burden to vendors.
But you are right, there're less review in virtualization side to code under vendor specific directory. That's also the pulse for us to propose common helper APIs for them to call, not only for convenience and duplication-less, but also for code with full review.

Quoted 1 line

would you mind giving us some suggestions for where to go?

Quoted 9 lines

Not duplicating kernel code into userspace isn't a great excuse. What we need to do to emulate a device is not an exact mapping to what a driver for that device needs to do. If we need to keep the device driver and the emulation in sync then we haven't done a good job with the emulation. What would it look like if we only had an additional device specific region on the vfio device fd we could use to get the descriptor information we need from the PF? This would be more inline with the quirks we provide for IGD assignment. Thanks,

hi Alex
Thanks for this suggestion.
As migration region is a generic vendor region, do you think below way to specify device specific region is acceptable?

(1) provide/export an interface to let vendor driver register its device specific region or substitute get_region_info/rw/mmap of existing regions.
(2) export vfio_pci_default_rw(), vfio_pci_default_mmap() to called from both vendor driver handlers and vfio-pci.

Or do you still prefer to adding quirks per device so you can have a better review of all code?

we can add a disable flag to disable regions registered/modified by vendor drivers in bullet (1) for debug purpose.

Thanks
Yan

author
#32 lore raw

RFC PATCH v4 8/10 i40e/vf_migration: VF live migration - pass-through VF first

In reply to Yan Zhao

This driver intercepts all device operations as long as it's probed successfully by vfio-pci driver.

It will process regions and irqs of its interest and then forward operations to default handlers exported from vfio pci if it wishes to.

In this patch, this driver does nothing but pass through VFs to guest by calling to exported handlers from driver vfio-pci.

Cc Shaopeng He <[email protected]>
---
 drivers/net/ethernet/intel/Kconfig            |  10 ++
 drivers/net/ethernet/intel/i40e/Makefile      |   2 +
 .../ethernet/intel/i40e/i40e_vf_migration.c   | 165 ++++++++++++++++++
 .../ethernet/intel/i40e/i40e_vf_migration.h   |  59 +++++++
 4 files changed, 236 insertions(+)
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.c
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.h
drivers/net/ethernet/intel/Kconfig +10 -0
index ad34e4335df2..31780d9a59f1 100644
@@ -264,6 +264,16 @@ config I40E_DCB
 
 	  If unsure, say N.
 
+config I40E_VF_MIGRATION
+	tristate "XL710 Family VF live migration support -- loadable modules only"
+	depends on I40E && VFIO_PCI && m
+	help
+	  Say m if you want to enable live migration of
+	  Virtual Functions of Intel(R) Ethernet Controller XL710
+	  Family of devices. It must be a module.
+	  This module serves as vendor module of module vfio_pci.
+	  VFs bind to module vfio_pci directly.
+
 # this is here to allow seamless migration from I40EVF --> IAVF name
 # so that CONFIG_IAVF symbol will always mirror the state of CONFIG_I40EVF
 config IAVF
drivers/net/ethernet/intel/i40e/Makefile +2 -0
index 2f21b3e89fd0..b80c224c2602 100644
@@ -27,3 +27,5 @@ i40e-objs := i40e_main.o \
 	i40e_xsk.o
 
 i40e-$(CONFIG_I40E_DCB) += i40e_dcb.o i40e_dcb_nl.o
+
+obj-$(CONFIG_I40E_VF_MIGRATION) += i40e_vf_migration.o
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +165 -0

165 lines not on this page · raw message

drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +59 -0

59 lines not on this page · raw message

Signature · 1 line
2.17.1
#33 lore raw

Re: [RFC PATCH v4 08/10] i40e/vf_migration: VF live migration - pass-through VF first

[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Yan,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on vfio/next]
[also build test WARNING on jkirsher-next-queue/dev-queue linus/master v5.7-rc6 next-20200515] [cannot apply to linux/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Yan-Zhao/Introduce-vendor-ops-in-vfio-pci/20200518-110542
base:   https://github.com/awilliam/linux-vfio.git next
config: x86_64-allyesconfig (attached as .config)
reproduce:
        # apt-get install sparse
        # sparse version: v0.6.1-193-gb8fad4bc-dirty
        # save the attached .config to linux build tree
        make C=1 ARCH=x86_64 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__'

If you fix the issue, kindly add following tag as appropriate

sparse warnings: (new ones prefixed by >>)

>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:54:6: sparse: sparse: symbol 'i40e_vf_release' was not declared. Should it be static?
>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:108:6: sparse: sparse: symbol 'i40e_vf_probe' was not declared. Should it be static?

Please review and possibly fold the followup patch.

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#34 lore raw

[RFC PATCH] i40e/vf_migration: i40e_vf_release() can be static

In reply to Yan Zhao
Signed-off-by: kbuild test robot <[email protected]>
---
 i40e_vf_migration.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c +2 -2
index 96026dcf5c9df..d222c7531fa6e 100644
@@ -51,7 +51,7 @@ static int i40e_vf_open(void *device_data)
 	return ret;
 }
 
-void i40e_vf_release(void *device_data)
+static void i40e_vf_release(void *device_data)
 {
 	struct i40e_vf_migration *i40e_vf_dev =
 		vfio_pci_vendor_data(device_data);
@@ -105,7 +105,7 @@ static struct vfio_device_ops i40e_vf_device_ops_node = {
 	.request	= i40e_vf_request,
 };
 
-void *i40e_vf_probe(struct pci_dev *pdev)
+static void *i40e_vf_probe(struct pci_dev *pdev)
 {
 	struct i40e_vf_migration *i40e_vf_dev = NULL;
 	struct pci_dev *pf_dev, *vf_dev;
#35 lore raw

Re: [RFC PATCH v4 08/10] i40e/vf_migration: VF live migration - pass-through VF first

In reply to Yan Zhao

Hi Yan,

few nits below...

Quoted 75 lines · Yan Zhao

This driver intercepts all device operations as long as it's probed successfully by vfio-pci driver.

It will process regions and irqs of its interest and then forward operations to default handlers exported from vfio pci if it wishes to.

In this patch, this driver does nothing but pass through VFs to guest by calling to exported handlers from driver vfio-pci.

Cc Shaopeng He <[email protected]>
---
 drivers/net/ethernet/intel/Kconfig            |  10 ++
 drivers/net/ethernet/intel/i40e/Makefile      |   2 +
 .../ethernet/intel/i40e/i40e_vf_migration.c   | 165 ++++++++++++++++++
 .../ethernet/intel/i40e/i40e_vf_migration.h   |  59 +++++++
 4 files changed, 236 insertions(+)
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.c
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.h
drivers/net/ethernet/intel/Kconfig +10 -0
index ad34e4335df2..31780d9a59f1 100644
@@ -264,6 +264,16 @@ config I40E_DCB
 
 	  If unsure, say N.
 
+config I40E_VF_MIGRATION
+	tristate "XL710 Family VF live migration support -- loadable modules only"
+	depends on I40E && VFIO_PCI && m
+	help
+	  Say m if you want to enable live migration of
+	  Virtual Functions of Intel(R) Ethernet Controller XL710
+	  Family of devices. It must be a module.
+	  This module serves as vendor module of module vfio_pci.
+	  VFs bind to module vfio_pci directly.
+
 # this is here to allow seamless migration from I40EVF --> IAVF name
 # so that CONFIG_IAVF symbol will always mirror the state of CONFIG_I40EVF
 config IAVF
drivers/net/ethernet/intel/i40e/Makefile +2 -0
index 2f21b3e89fd0..b80c224c2602 100644
@@ -27,3 +27,5 @@ i40e-objs := i40e_main.o \
 	i40e_xsk.o
 
 i40e-$(CONFIG_I40E_DCB) += i40e_dcb.o i40e_dcb_nl.o
+
+obj-$(CONFIG_I40E_VF_MIGRATION) += i40e_vf_migration.o
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +17 -0
new file mode 100644
index 000000000000..96026dcf5c9d
@@ -0,0 +1,165 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2013 - 2019 Intel Corporation. */
+
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/vfio.h>
+#include <linux/pci.h>
+#include <linux/eventfd.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/sysfs.h>
+#include <linux/file.h>
+#include <linux/pci.h>
+
+#include "i40e.h"
+#include "i40e_vf_migration.h"
+

drivers/net/ethernet/intel/i40e/i40e_vf_migration.c line 27

Quoted 10 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +10 -0
+#define VERSION_STRING  "0.1"
+#define DRIVER_AUTHOR   "Intel Corporation"
+
+static int i40e_vf_open(void *device_data)
+{
+	struct i40e_vf_migration *i40e_vf_dev =
+		vfio_pci_vendor_data(device_data);
+	int ret;
+	struct vfio_device_migration_info *mig_ctl = NULL;
+

"mig_ctl" is not used in this function. Shouldn't this declaration be put into the next patch?

Quoted 111 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +111 -0
+	if (!try_module_get(THIS_MODULE))
+		return -ENODEV;
+
+	mutex_lock(&i40e_vf_dev->reflock);
+	if (!i40e_vf_dev->refcnt) {
+		vfio_pci_set_vendor_regions(device_data, 0);
+		vfio_pci_set_vendor_irqs(device_data, 0);
+	}
+
+	ret = vfio_pci_open(device_data);
+	if (ret)
+		goto error;
+
+	i40e_vf_dev->refcnt++;
+	mutex_unlock(&i40e_vf_dev->reflock);
+	return 0;
+error:
+	if (!i40e_vf_dev->refcnt) {
+		vfio_pci_set_vendor_regions(device_data, 0);
+		vfio_pci_set_vendor_irqs(device_data, 0);
+	}
+	module_put(THIS_MODULE);
+	mutex_unlock(&i40e_vf_dev->reflock);
+	return ret;
+}
+
+void i40e_vf_release(void *device_data)
+{
+	struct i40e_vf_migration *i40e_vf_dev =
+		vfio_pci_vendor_data(device_data);
+
+	mutex_lock(&i40e_vf_dev->reflock);
+	if (!--i40e_vf_dev->refcnt) {
+		vfio_pci_set_vendor_regions(device_data, 0);
+		vfio_pci_set_vendor_irqs(device_data, 0);
+	}
+	vfio_pci_release(device_data);
+	mutex_unlock(&i40e_vf_dev->reflock);
+	module_put(THIS_MODULE);
+}
+
+static long i40e_vf_ioctl(void *device_data,
+			  unsigned int cmd, unsigned long arg)
+{
+	return vfio_pci_ioctl(device_data, cmd, arg);
+}
+
+static ssize_t i40e_vf_read(void *device_data, char __user *buf,
+			    size_t count, loff_t *ppos)
+{
+	return vfio_pci_read(device_data, buf, count, ppos);
+}
+
+static ssize_t i40e_vf_write(void *device_data, const char __user *buf,
+			     size_t count, loff_t *ppos)
+{
+	return vfio_pci_write(device_data, buf, count, ppos);
+}
+
+static int i40e_vf_mmap(void *device_data, struct vm_area_struct *vma)
+{
+	return vfio_pci_mmap(device_data, vma);
+}
+
+static void i40e_vf_request(void *device_data, unsigned int count)
+{
+	vfio_pci_request(device_data, count);
+}
+
+static struct vfio_device_ops i40e_vf_device_ops_node = {
+	.name		= "i40e_vf",
+	.open		= i40e_vf_open,
+	.release	= i40e_vf_release,
+	.ioctl		= i40e_vf_ioctl,
+	.read		= i40e_vf_read,
+	.write		= i40e_vf_write,
+	.mmap		= i40e_vf_mmap,
+	.request	= i40e_vf_request,
+};
+
+void *i40e_vf_probe(struct pci_dev *pdev)
+{
+	struct i40e_vf_migration *i40e_vf_dev = NULL;
+	struct pci_dev *pf_dev, *vf_dev;
+	struct i40e_pf *pf;
+	struct i40e_vf *vf;
+	unsigned int vf_devfn, devfn;
+	int vf_id = -1;
+	int i;
+
+	pf_dev = pdev->physfn;
+	pf = pci_get_drvdata(pf_dev);
+	vf_dev = pdev;
+	vf_devfn = vf_dev->devfn;
+
+	for (i = 0; i < pci_num_vf(pf_dev); i++) {
+		devfn = (pf_dev->devfn + pf_dev->sriov->offset +
+			 pf_dev->sriov->stride * i) & 0xff;
+		if (devfn == vf_devfn) {
+			vf_id = i;
+			break;
+		}
+	}
+
+	if (vf_id == -1)
+		return ERR_PTR(-EINVAL);
+
+	i40e_vf_dev = kzalloc(sizeof(*i40e_vf_dev), GFP_KERNEL);
+
+	if (!i40e_vf_dev)
+		return ERR_PTR(-ENOMEM);

drivers/net/ethernet/intel/i40e/i40e_vf_migration.c line 148

Quoted 10 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +10 -0
+
+	i40e_vf_dev->vf_id = vf_id;
+	i40e_vf_dev->vf_vendor = pdev->vendor;
+	i40e_vf_dev->vf_device = pdev->device;
+	i40e_vf_dev->pf_dev = pf_dev;
+	i40e_vf_dev->vf_dev = vf_dev;
+	mutex_init(&i40e_vf_dev->reflock);
+
+	vf = &pf->vf[vf_id];
+

"vf" is also not used in this function...

Quoted 59 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +17 -0
+	return i40e_vf_dev;
+}
+
+static void i40e_vf_remove(void *vendor_data)
+{
+	kfree(vendor_data);
+}
+
+#define i40e_vf_device_ops (&i40e_vf_device_ops_node)
+module_vfio_pci_register_vendor_handler("I40E VF", i40e_vf_probe,
+					i40e_vf_remove, i40e_vf_device_ops);
+
+MODULE_ALIAS("vfio-pci:8086-154c");
+MODULE_LICENSE("GPL v2");
+MODULE_INFO(supported, "Vendor driver of vfio pci to support VF live migration");
+MODULE_VERSION(VERSION_STRING);
+MODULE_AUTHOR(DRIVER_AUTHOR);
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +36 -0
new file mode 100644
index 000000000000..696d40601ec3
@@ -0,0 +1,59 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2013 - 2019 Intel Corporation. */
+
+#ifndef I40E_MIG_H
+#define I40E_MIG_H
+
+#include <linux/pci.h>
+#include <linux/vfio.h>
+#include <linux/mdev.h>
+
+#include "i40e.h"
+#include "i40e_txrx.h"
+
+/* helper macros copied from vfio-pci */
+#define VFIO_PCI_OFFSET_SHIFT   40
+#define VFIO_PCI_OFFSET_TO_INDEX(off)   ((off) >> VFIO_PCI_OFFSET_SHIFT)
+#define VFIO_PCI_INDEX_TO_OFFSET(index)	((u64)(index) << VFIO_PCI_OFFSET_SHIFT)
+#define VFIO_PCI_OFFSET_MASK    (((u64)(1) << VFIO_PCI_OFFSET_SHIFT) - 1)
+
+/* Single Root I/O Virtualization */
+struct pci_sriov {
+	int		pos;		/* Capability position */
+	int		nres;		/* Number of resources */
+	u32		cap;		/* SR-IOV Capabilities */
+	u16		ctrl;		/* SR-IOV Control */
+	u16		total_VFs;	/* Total VFs associated with the PF */
+	u16		initial_VFs;	/* Initial VFs associated with the PF */
+	u16		num_VFs;	/* Number of VFs available */
+	u16		offset;		/* First VF Routing ID offset */
+	u16		stride;		/* Following VF stride */
+	u16		vf_device;	/* VF device ID */
+	u32		pgsz;		/* Page size for BAR alignment */
+	u8		link;		/* Function Dependency Link */
+	u8		max_VF_buses;	/* Max buses consumed by VFs */
+	u16		driver_max_VFs;	/* Max num VFs driver supports */
+	struct pci_dev	*dev;		/* Lowest numbered PF */

drivers/net/ethernet/intel/i40e/i40e_vf_migration.h line 46

Quoted 10 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +10 -0
+	struct pci_dev	*self;		/* This PF */
+	u32		cfg_size;	/* VF config space size */
+	u32		class;		/* VF device */
+	u8		hdr_type;	/* VF header type */
+	u16		subsystem_vendor; /* VF subsystem vendor */
+	u16		subsystem_device; /* VF subsystem device */
+	resource_size_t	barsz[PCI_SRIOV_NUM_BARS];	/* VF BAR size */
+	bool		drivers_autoprobe; /* Auto probing of VFs by driver */
+};
+

Can "struct pci_sriov" be extracted for common use? This should not be exclusive for "i40e_vf migration support".

drivers/net/ethernet/intel/i40e/i40e_vf_migration.h line 55

Quoted 9 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +9 -0
+struct i40e_vf_migration {
+	__u32				vf_vendor;
+	__u32				vf_device;
+	__u32				handle;
+	struct pci_dev			*pf_dev;
+	struct pci_dev			*vf_dev;
+	int				vf_id;
+	int				refcnt;
+	struct				mutex reflock; /*mutex protect refcnt */

^ ^

stray ' '

Quoted 5 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +4 -0
+};
+
+#endif /* I40E_MIG_H */
+
Signature · 2 lines
Thanks,
Xiang
author
#36 lore raw

Re: [RFC PATCH v4 08/10] i40e/vf_migration: VF live migration - pass-through VF first

Quoted 85 lines · Xiang Zheng

Hi Yan,

few nits below...

Quoted 80 lines · Yan Zhao

This driver intercepts all device operations as long as it's probed successfully by vfio-pci driver.

It will process regions and irqs of its interest and then forward operations to default handlers exported from vfio pci if it wishes to.

In this patch, this driver does nothing but pass through VFs to guest by calling to exported handlers from driver vfio-pci.

Cc Shaopeng He <[email protected]>
---
 drivers/net/ethernet/intel/Kconfig            |  10 ++
 drivers/net/ethernet/intel/i40e/Makefile      |   2 +
 .../ethernet/intel/i40e/i40e_vf_migration.c   | 165 ++++++++++++++++++
 .../ethernet/intel/i40e/i40e_vf_migration.h   |  59 +++++++
 4 files changed, 236 insertions(+)
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.c
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.h
drivers/net/ethernet/intel/Kconfig +10 -0
index ad34e4335df2..31780d9a59f1 100644
@@ -264,6 +264,16 @@ config I40E_DCB
 
 	  If unsure, say N.
 
+config I40E_VF_MIGRATION
+	tristate "XL710 Family VF live migration support -- loadable modules only"
+	depends on I40E && VFIO_PCI && m
+	help
+	  Say m if you want to enable live migration of
+	  Virtual Functions of Intel(R) Ethernet Controller XL710
+	  Family of devices. It must be a module.
+	  This module serves as vendor module of module vfio_pci.
+	  VFs bind to module vfio_pci directly.
+
 # this is here to allow seamless migration from I40EVF --> IAVF name
 # so that CONFIG_IAVF symbol will always mirror the state of CONFIG_I40EVF
 config IAVF
drivers/net/ethernet/intel/i40e/Makefile +2 -0
index 2f21b3e89fd0..b80c224c2602 100644
@@ -27,3 +27,5 @@ i40e-objs := i40e_main.o \
 	i40e_xsk.o
 
 i40e-$(CONFIG_I40E_DCB) += i40e_dcb.o i40e_dcb_nl.o
+
+obj-$(CONFIG_I40E_VF_MIGRATION) += i40e_vf_migration.o
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +22 -0
new file mode 100644
index 000000000000..96026dcf5c9d
@@ -0,0 +1,165 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2013 - 2019 Intel Corporation. */
+
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/vfio.h>
+#include <linux/pci.h>
+#include <linux/eventfd.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/sysfs.h>
+#include <linux/file.h>
+#include <linux/pci.h>
+
+#include "i40e.h"
+#include "i40e_vf_migration.h"
+
+#define VERSION_STRING  "0.1"
+#define DRIVER_AUTHOR   "Intel Corporation"
+
+static int i40e_vf_open(void *device_data)
+{
Quoted 9 lines

drivers/net/ethernet/intel/i40e/i40e_vf_migration.c line 27

Quoted 5 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +5 -0
+	struct i40e_vf_migration *i40e_vf_dev =
+		vfio_pci_vendor_data(device_data);
+	int ret;
+	struct vfio_device_migration_info *mig_ctl = NULL;
+

"mig_ctl" is not used in this function. Shouldn't this declaration be put into the next patch?

right. thanks!

Quoted 115 lines
Quoted 115 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +115 -0
+	if (!try_module_get(THIS_MODULE))
+		return -ENODEV;
+
+	mutex_lock(&i40e_vf_dev->reflock);
+	if (!i40e_vf_dev->refcnt) {
+		vfio_pci_set_vendor_regions(device_data, 0);
+		vfio_pci_set_vendor_irqs(device_data, 0);
+	}
+
+	ret = vfio_pci_open(device_data);
+	if (ret)
+		goto error;
+
+	i40e_vf_dev->refcnt++;
+	mutex_unlock(&i40e_vf_dev->reflock);
+	return 0;
+error:
+	if (!i40e_vf_dev->refcnt) {
+		vfio_pci_set_vendor_regions(device_data, 0);
+		vfio_pci_set_vendor_irqs(device_data, 0);
+	}
+	module_put(THIS_MODULE);
+	mutex_unlock(&i40e_vf_dev->reflock);
+	return ret;
+}
+
+void i40e_vf_release(void *device_data)
+{
+	struct i40e_vf_migration *i40e_vf_dev =
+		vfio_pci_vendor_data(device_data);
+
+	mutex_lock(&i40e_vf_dev->reflock);
+	if (!--i40e_vf_dev->refcnt) {
+		vfio_pci_set_vendor_regions(device_data, 0);
+		vfio_pci_set_vendor_irqs(device_data, 0);
+	}
+	vfio_pci_release(device_data);
+	mutex_unlock(&i40e_vf_dev->reflock);
+	module_put(THIS_MODULE);
+}
+
+static long i40e_vf_ioctl(void *device_data,
+			  unsigned int cmd, unsigned long arg)
+{
+	return vfio_pci_ioctl(device_data, cmd, arg);
+}
+
+static ssize_t i40e_vf_read(void *device_data, char __user *buf,
+			    size_t count, loff_t *ppos)
+{
+	return vfio_pci_read(device_data, buf, count, ppos);
+}
+
+static ssize_t i40e_vf_write(void *device_data, const char __user *buf,
+			     size_t count, loff_t *ppos)
+{
+	return vfio_pci_write(device_data, buf, count, ppos);
+}
+
+static int i40e_vf_mmap(void *device_data, struct vm_area_struct *vma)
+{
+	return vfio_pci_mmap(device_data, vma);
+}
+
+static void i40e_vf_request(void *device_data, unsigned int count)
+{
+	vfio_pci_request(device_data, count);
+}
+
+static struct vfio_device_ops i40e_vf_device_ops_node = {
+	.name		= "i40e_vf",
+	.open		= i40e_vf_open,
+	.release	= i40e_vf_release,
+	.ioctl		= i40e_vf_ioctl,
+	.read		= i40e_vf_read,
+	.write		= i40e_vf_write,
+	.mmap		= i40e_vf_mmap,
+	.request	= i40e_vf_request,
+};
+
+void *i40e_vf_probe(struct pci_dev *pdev)
+{
+	struct i40e_vf_migration *i40e_vf_dev = NULL;
+	struct pci_dev *pf_dev, *vf_dev;
+	struct i40e_pf *pf;
+	struct i40e_vf *vf;
+	unsigned int vf_devfn, devfn;
+	int vf_id = -1;
+	int i;
+
+	pf_dev = pdev->physfn;
+	pf = pci_get_drvdata(pf_dev);
+	vf_dev = pdev;
+	vf_devfn = vf_dev->devfn;
+
+	for (i = 0; i < pci_num_vf(pf_dev); i++) {
+		devfn = (pf_dev->devfn + pf_dev->sriov->offset +
+			 pf_dev->sriov->stride * i) & 0xff;
+		if (devfn == vf_devfn) {
+			vf_id = i;
+			break;
+		}
+	}
+
+	if (vf_id == -1)
+		return ERR_PTR(-EINVAL);
+
+	i40e_vf_dev = kzalloc(sizeof(*i40e_vf_dev), GFP_KERNEL);
+
+	if (!i40e_vf_dev)
+		return ERR_PTR(-ENOMEM);
+
+	i40e_vf_dev->vf_id = vf_id;
+	i40e_vf_dev->vf_vendor = pdev->vendor;
+	i40e_vf_dev->vf_device = pdev->device;
Quoted 9 lines

drivers/net/ethernet/intel/i40e/i40e_vf_migration.c line 148

Quoted 6 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +6 -0
+	i40e_vf_dev->pf_dev = pf_dev;
+	i40e_vf_dev->vf_dev = vf_dev;
+	mutex_init(&i40e_vf_dev->reflock);
+
+	vf = &pf->vf[vf_id];
+

"vf" is also not used in this function...

yes, thanks.

Quoted 64 lines
Quoted 64 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c new file +17 -0
+	return i40e_vf_dev;
+}
+
+static void i40e_vf_remove(void *vendor_data)
+{
+	kfree(vendor_data);
+}
+
+#define i40e_vf_device_ops (&i40e_vf_device_ops_node)
+module_vfio_pci_register_vendor_handler("I40E VF", i40e_vf_probe,
+					i40e_vf_remove, i40e_vf_device_ops);
+
+MODULE_ALIAS("vfio-pci:8086-154c");
+MODULE_LICENSE("GPL v2");
+MODULE_INFO(supported, "Vendor driver of vfio pci to support VF live migration");
+MODULE_VERSION(VERSION_STRING);
+MODULE_AUTHOR(DRIVER_AUTHOR);
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +41 -0
new file mode 100644
index 000000000000..696d40601ec3
@@ -0,0 +1,59 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2013 - 2019 Intel Corporation. */
+
+#ifndef I40E_MIG_H
+#define I40E_MIG_H
+
+#include <linux/pci.h>
+#include <linux/vfio.h>
+#include <linux/mdev.h>
+
+#include "i40e.h"
+#include "i40e_txrx.h"
+
+/* helper macros copied from vfio-pci */
+#define VFIO_PCI_OFFSET_SHIFT   40
+#define VFIO_PCI_OFFSET_TO_INDEX(off)   ((off) >> VFIO_PCI_OFFSET_SHIFT)
+#define VFIO_PCI_INDEX_TO_OFFSET(index)	((u64)(index) << VFIO_PCI_OFFSET_SHIFT)
+#define VFIO_PCI_OFFSET_MASK    (((u64)(1) << VFIO_PCI_OFFSET_SHIFT) - 1)
+
+/* Single Root I/O Virtualization */
+struct pci_sriov {
+	int		pos;		/* Capability position */
+	int		nres;		/* Number of resources */
+	u32		cap;		/* SR-IOV Capabilities */
+	u16		ctrl;		/* SR-IOV Control */
+	u16		total_VFs;	/* Total VFs associated with the PF */
+	u16		initial_VFs;	/* Initial VFs associated with the PF */
+	u16		num_VFs;	/* Number of VFs available */
+	u16		offset;		/* First VF Routing ID offset */
+	u16		stride;		/* Following VF stride */
+	u16		vf_device;	/* VF device ID */
+	u32		pgsz;		/* Page size for BAR alignment */
+	u8		link;		/* Function Dependency Link */
+	u8		max_VF_buses;	/* Max buses consumed by VFs */
+	u16		driver_max_VFs;	/* Max num VFs driver supports */
+	struct pci_dev	*dev;		/* Lowest numbered PF */
+	struct pci_dev	*self;		/* This PF */
+	u32		cfg_size;	/* VF config space size */
+	u32		class;		/* VF device */
+	u8		hdr_type;	/* VF header type */
+	u16		subsystem_vendor; /* VF subsystem vendor */
Quoted 9 lines

drivers/net/ethernet/intel/i40e/i40e_vf_migration.h line 46

Quoted 5 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +5 -0
+	u16		subsystem_device; /* VF subsystem device */
+	resource_size_t	barsz[PCI_SRIOV_NUM_BARS];	/* VF BAR size */
+	bool		drivers_autoprobe; /* Auto probing of VFs by driver */
+};
+

Can "struct pci_sriov" be extracted for common use? This should not be exclusive for "i40e_vf migration support".

the definition of this structure is actually in driver/pci/pci.h. maybe removing the copy here and use below include is better? #include "../../../../pci/pci.h"

Quoted 11 lines

drivers/net/ethernet/intel/i40e/i40e_vf_migration.h line 55

Quoted 9 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +9 -0
+struct i40e_vf_migration {
+	__u32				vf_vendor;
+	__u32				vf_device;
+	__u32				handle;
+	struct pci_dev			*pf_dev;
+	struct pci_dev			*vf_dev;
+	int				vf_id;
+	int				refcnt;
+	struct				mutex reflock; /*mutex protect refcnt */

^ ^

Quoted 2 lines

stray ' '

got it!

thanks for review.

Yan

Quoted 10 lines
Quoted 5 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +4 -0
+};
+
+#endif /* I40E_MIG_H */
+
Signature · 2 lines
Thanks,
Xiang
#37 lore raw

Re: [RFC PATCH v4 08/10] i40e/vf_migration: VF live migration - pass-through VF first

Quoted 35 lines · Yan Zhao
Quoted 34 lines · Xiang Zheng

Hi Yan,

few nits below...

Quoted 29 lines · Yan Zhao

This driver intercepts all device operations as long as it's probed successfully by vfio-pci driver.

It will process regions and irqs of its interest and then forward operations to default handlers exported from vfio pci if it wishes to.

In this patch, this driver does nothing but pass through VFs to guest by calling to exported handlers from driver vfio-pci.

Cc Shaopeng He <[email protected]>
---
 drivers/net/ethernet/intel/Kconfig            |  10 ++
 drivers/net/ethernet/intel/i40e/Makefile      |   2 +
 .../ethernet/intel/i40e/i40e_vf_migration.c   | 165 ++++++++++++++++++
 .../ethernet/intel/i40e/i40e_vf_migration.h   |  59 +++++++
 4 files changed, 236 insertions(+)
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.c
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.h
drivers/net/ethernet/intel/Kconfig +0 -0
index ad34e4335df2..31780d9a59f1 100644
@@ -264,6 +264,16 @@ config I40E_DCB
 
 	  If unsure, say N.
 

[...]

Quoted 56 lines
Quoted 47 lines
Quoted 47 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +41 -0
new file mode 100644
index 000000000000..696d40601ec3
@@ -0,0 +1,59 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2013 - 2019 Intel Corporation. */
+
+#ifndef I40E_MIG_H
+#define I40E_MIG_H
+
+#include <linux/pci.h>
+#include <linux/vfio.h>
+#include <linux/mdev.h>
+
+#include "i40e.h"
+#include "i40e_txrx.h"
+
+/* helper macros copied from vfio-pci */
+#define VFIO_PCI_OFFSET_SHIFT   40
+#define VFIO_PCI_OFFSET_TO_INDEX(off)   ((off) >> VFIO_PCI_OFFSET_SHIFT)
+#define VFIO_PCI_INDEX_TO_OFFSET(index)	((u64)(index) << VFIO_PCI_OFFSET_SHIFT)
+#define VFIO_PCI_OFFSET_MASK    (((u64)(1) << VFIO_PCI_OFFSET_SHIFT) - 1)
+
+/* Single Root I/O Virtualization */
+struct pci_sriov {
+	int		pos;		/* Capability position */
+	int		nres;		/* Number of resources */
+	u32		cap;		/* SR-IOV Capabilities */
+	u16		ctrl;		/* SR-IOV Control */
+	u16		total_VFs;	/* Total VFs associated with the PF */
+	u16		initial_VFs;	/* Initial VFs associated with the PF */
+	u16		num_VFs;	/* Number of VFs available */
+	u16		offset;		/* First VF Routing ID offset */
+	u16		stride;		/* Following VF stride */
+	u16		vf_device;	/* VF device ID */
+	u32		pgsz;		/* Page size for BAR alignment */
+	u8		link;		/* Function Dependency Link */
+	u8		max_VF_buses;	/* Max buses consumed by VFs */
+	u16		driver_max_VFs;	/* Max num VFs driver supports */
+	struct pci_dev	*dev;		/* Lowest numbered PF */
+	struct pci_dev	*self;		/* This PF */
+	u32		cfg_size;	/* VF config space size */
+	u32		class;		/* VF device */
+	u8		hdr_type;	/* VF header type */
+	u16		subsystem_vendor; /* VF subsystem vendor */
Quoted 9 lines

drivers/net/ethernet/intel/i40e/i40e_vf_migration.h line 46

Quoted 5 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +5 -0
+	u16		subsystem_device; /* VF subsystem device */                                                                                   
+	resource_size_t	barsz[PCI_SRIOV_NUM_BARS];	/* VF BAR size */
+	bool		drivers_autoprobe; /* Auto probing of VFs by driver */
+};
+

Can "struct pci_sriov" be extracted for common use? This should not be exclusive for "i40e_vf migration support".

Quoted 4 lines

the definition of this structure is actually in driver/pci/pci.h. maybe removing the copy here and use below include is better? #include "../../../../pci/pci.h"

How about moving the definition from driver/pci/pci.h into include/linux/pci.h? So we can just include "linux/pci.h" and removing the copy here.

Signature · 2 lines
Thanks,
Xiang
author
#38 lore raw

Re: [RFC PATCH v4 08/10] i40e/vf_migration: VF live migration - pass-through VF first

Quoted 97 lines · Xiang Zheng
Quoted 35 lines · Yan Zhao
Quoted 34 lines · Xiang Zheng

Hi Yan,

few nits below...

Quoted 29 lines · Yan Zhao

This driver intercepts all device operations as long as it's probed successfully by vfio-pci driver.

It will process regions and irqs of its interest and then forward operations to default handlers exported from vfio pci if it wishes to.

In this patch, this driver does nothing but pass through VFs to guest by calling to exported handlers from driver vfio-pci.

Cc Shaopeng He <[email protected]>
---
 drivers/net/ethernet/intel/Kconfig            |  10 ++
 drivers/net/ethernet/intel/i40e/Makefile      |   2 +
 .../ethernet/intel/i40e/i40e_vf_migration.c   | 165 ++++++++++++++++++
 .../ethernet/intel/i40e/i40e_vf_migration.h   |  59 +++++++
 4 files changed, 236 insertions(+)
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.c
 create mode 100644 drivers/net/ethernet/intel/i40e/i40e_vf_migration.h
drivers/net/ethernet/intel/Kconfig +0 -0
index ad34e4335df2..31780d9a59f1 100644
@@ -264,6 +264,16 @@ config I40E_DCB
 
 	  If unsure, say N.
 

[...]

Quoted 56 lines
Quoted 56 lines
Quoted 42 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +36 -0
new file mode 100644
index 000000000000..696d40601ec3
@@ -0,0 +1,59 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2013 - 2019 Intel Corporation. */
+
+#ifndef I40E_MIG_H
+#define I40E_MIG_H
+
+#include <linux/pci.h>
+#include <linux/vfio.h>
+#include <linux/mdev.h>
+
+#include "i40e.h"
+#include "i40e_txrx.h"
+
+/* helper macros copied from vfio-pci */
+#define VFIO_PCI_OFFSET_SHIFT   40
+#define VFIO_PCI_OFFSET_TO_INDEX(off)   ((off) >> VFIO_PCI_OFFSET_SHIFT)
+#define VFIO_PCI_INDEX_TO_OFFSET(index)	((u64)(index) << VFIO_PCI_OFFSET_SHIFT)
+#define VFIO_PCI_OFFSET_MASK    (((u64)(1) << VFIO_PCI_OFFSET_SHIFT) - 1)
+
+/* Single Root I/O Virtualization */
+struct pci_sriov {
+	int		pos;		/* Capability position */
+	int		nres;		/* Number of resources */
+	u32		cap;		/* SR-IOV Capabilities */
+	u16		ctrl;		/* SR-IOV Control */
+	u16		total_VFs;	/* Total VFs associated with the PF */
+	u16		initial_VFs;	/* Initial VFs associated with the PF */
+	u16		num_VFs;	/* Number of VFs available */
+	u16		offset;		/* First VF Routing ID offset */
+	u16		stride;		/* Following VF stride */
+	u16		vf_device;	/* VF device ID */
+	u32		pgsz;		/* Page size for BAR alignment */
+	u8		link;		/* Function Dependency Link */
+	u8		max_VF_buses;	/* Max buses consumed by VFs */
+	u16		driver_max_VFs;	/* Max num VFs driver supports */
+	struct pci_dev	*dev;		/* Lowest numbered PF */

drivers/net/ethernet/intel/i40e/i40e_vf_migration.h line 46

Quoted 10 lines
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h new file +10 -0
+	struct pci_dev	*self;		/* This PF */
+	u32		cfg_size;	/* VF config space size */
+	u32		class;		/* VF device */
+	u8		hdr_type;	/* VF header type */
+	u16		subsystem_vendor; /* VF subsystem vendor */
+	u16		subsystem_device; /* VF subsystem device */                                                                                   
+	resource_size_t	barsz[PCI_SRIOV_NUM_BARS];	/* VF BAR size */
+	bool		drivers_autoprobe; /* Auto probing of VFs by driver */
+};
+

Can "struct pci_sriov" be extracted for common use? This should not be exclusive for "i40e_vf migration support".

Quoted 8 lines
Quoted 4 lines

the definition of this structure is actually in driver/pci/pci.h. maybe removing the copy here and use below include is better? #include "../../../../pci/pci.h"

How about moving the definition from driver/pci/pci.h into include/linux/pci.h? So we can just include "linux/pci.h" and removing the copy here.

I prefer to leaving it in drivers/pci/pci.h for now.

Thanks
Yan

author
#39 lore raw

RFC PATCH v4 9/10 i40e/vf_migration: register a migration vendor region

In reply to Yan Zhao

This patch let the vendor driver register a migration region, so that the migration detection code in userspace will be able to see this region and triggers the migration flow according to VFIO migration protocol.

This migration region works based on VFIO migration series with some minor fixes:
[1] kernel v17: https://patchwork.kernel.org/cover/11466129/
[2] qemu v16: https://patchwork.kernel.org/cover/11456557/

---
 .../ethernet/intel/i40e/i40e_vf_migration.c   | 429 +++++++++++++++++-
 .../ethernet/intel/i40e/i40e_vf_migration.h   |  34 ++
 2 files changed, 460 insertions(+), 3 deletions(-)
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c +426 -3
index 96026dcf5c9d..107a291909b3 100644
@@ -17,6 +17,351 @@
 
 #define VERSION_STRING  "0.1"
 #define DRIVER_AUTHOR   "Intel Corporation"
+#define TEST_DIRTY_IOVA_PFN 0
+
+static int i40e_vf_iommu_notifier(struct notifier_block *nb,
+				  unsigned long action, void *data)
+{
+	if (action == VFIO_IOMMU_NOTIFY_DMA_UNMAP) {
+		struct vfio_iommu_type1_dma_unmap *unmap = data;
+		unsigned long iova_pfn, end_iova_pfn;
+
+		iova_pfn = unmap->iova >> PAGE_SHIFT;
+		end_iova_pfn = iova_pfn + unmap->size / PAGE_SIZE;
+
+		pr_info("DMA UNMAP iova_pfn=%lx, end=%lx\n", iova_pfn,
+			end_iova_pfn);
+	}
+
+	return NOTIFY_OK;
+}
+
+/* transient pinning a page
+ */
+static int i40e_vf_set_page_dirty(struct i40e_vf_migration *i40e_vf_dev,
+				  unsigned long dirty_iova_pfn)
+{
+	unsigned long dirty_pfn, cnt = 1;
+	int ret;
+
+	ret = vfio_group_pin_pages(i40e_vf_dev->vfio_group,
+				   &dirty_iova_pfn, cnt,
+				   IOMMU_READ | IOMMU_WRITE, &dirty_pfn);
+	if (ret != cnt) {
+		pr_err("failed to track dirty of page of iova pfn %lx\n",
+		       dirty_iova_pfn);
+		return ret < 0 ? ret : -EFAULT;
+	}
+
+	vfio_group_unpin_pages(i40e_vf_dev->vfio_group, &dirty_iova_pfn, cnt);
+
+	return 0;
+}
+
+/* alloc dirty page tracking resources and
+ * do the first round dirty page scanning
+ */
+static int i40e_vf_prepare_dirty_track(struct i40e_vf_migration *i40e_vf_dev)
+{
+	struct vfio_group *vfio_group;
+	unsigned long events;
+	int ret;
+	struct device *dev = &i40e_vf_dev->vf_dev->dev;
+
+	if (i40e_vf_dev->in_dirty_track) {
+		pr_warn("%s, previous dirty track resources found\n",
+			__func__);
+		return 0;
+	}
+
+	i40e_vf_dev->iommu_notifier.notifier_call = i40e_vf_iommu_notifier;
+
+	events = VFIO_IOMMU_NOTIFY_DMA_UNMAP;
+	ret = vfio_register_notifier(dev, VFIO_IOMMU_NOTIFY, &events,
+				     &i40e_vf_dev->iommu_notifier);
+	if (ret) {
+		pr_err("failed to register vfio iommu notifier\n");
+		return ret;
+	}
+
+	vfio_group = vfio_group_get_external_user_from_dev(dev);
+	if (IS_ERR_OR_NULL(vfio_group)) {
+		ret = PTR_ERR(vfio_group);
+		pr_err("failed to get vfio group from dev\n");
+		goto out;
+	}
+
+	i40e_vf_dev->vfio_group = vfio_group;
+
+	ret = i40e_vf_set_page_dirty(i40e_vf_dev, TEST_DIRTY_IOVA_PFN);
+
+	if (ret) {
+		pr_err("failed to set dirty for test page\n");
+		goto out_group;
+	}
+
+	i40e_vf_dev->in_dirty_track = true;
+	return 0;
+
+out_group:
+	vfio_unregister_notifier(dev, VFIO_IOMMU_NOTIFY,
+				 &i40e_vf_dev->iommu_notifier);
+out:
+	vfio_group_put_external_user(i40e_vf_dev->vfio_group);
+	return ret;
+}
+
+static void i40e_vf_stop_dirty_track(struct i40e_vf_migration *i40e_vf_dev)
+{
+	if (!i40e_vf_dev->in_dirty_track)
+		return;
+
+	vfio_unregister_notifier(&i40e_vf_dev->vf_dev->dev,
+				 VFIO_IOMMU_NOTIFY,
+				 &i40e_vf_dev->iommu_notifier);
+	vfio_group_put_external_user(i40e_vf_dev->vfio_group);
+	i40e_vf_dev->in_dirty_track = false;
+}
+
+static size_t i40e_vf_set_device_state(struct i40e_vf_migration *i40e_vf_dev,
+				       u32 state)
+{
+	int ret = 0;
+	struct vfio_device_migration_info *mig_ctl = i40e_vf_dev->mig_ctl;
+
+	if (state == mig_ctl->device_state)
+		return 0;
+
+	switch (state) {
+	case VFIO_DEVICE_STATE_RUNNING:
+		break;
+	case VFIO_DEVICE_STATE_SAVING | VFIO_DEVICE_STATE_RUNNING:
+		ret = i40e_vf_prepare_dirty_track(i40e_vf_dev);
+		break;
+	case VFIO_DEVICE_STATE_SAVING:
+		// do the last round of dirty page scanning
+		break;
+	case VFIO_DEVICE_STATE_STOP:
+		// release dirty page tracking resources
+		if (mig_ctl->device_state == VFIO_DEVICE_STATE_SAVING)
+			i40e_vf_stop_dirty_track(i40e_vf_dev);
+		break;
+	case VFIO_DEVICE_STATE_RESUMING:
+		break;
+	default:
+		ret = -EFAULT;
+	}
+
+	if (!ret)
+		mig_ctl->device_state = state;
+
+	return ret;
+}
+
+static
+ssize_t i40e_vf_region_migration_rw(struct i40e_vf_migration *i40e_vf_dev,
+				    char __user *buf, size_t count,
+				    loff_t *ppos, bool iswrite)
+{
+#define VDM_OFFSET(x) offsetof(struct vfio_device_migration_info, x)
+	struct vfio_device_migration_info *mig_ctl = i40e_vf_dev->mig_ctl;
+	u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	int ret = 0;
+
+	switch (pos) {
+	case VDM_OFFSET(device_state):
+		if (count != sizeof(mig_ctl->device_state)) {
+			ret = -EINVAL;
+			break;
+		}
+
+		if (iswrite) {
+			u32 device_state;
+
+			if (copy_from_user(&device_state, buf, count)) {
+				ret = -EFAULT;
+				break;
+			}
+
+			ret = i40e_vf_set_device_state(i40e_vf_dev,
+						       device_state) ?
+						       ret : count;
+		} else {
+			ret = copy_to_user(buf, &mig_ctl->device_state,
+					   count) ? -EFAULT : count;
+		}
+		break;
+
+	case VDM_OFFSET(reserved):
+		ret = -EFAULT;
+		break;
+
+	case VDM_OFFSET(pending_bytes):
+		{
+			if (count != sizeof(mig_ctl->pending_bytes)) {
+				ret = -EINVAL;
+				break;
+			}
+
+			if (iswrite)
+				ret = -EFAULT;
+			else
+				ret = copy_to_user(buf,
+						   &mig_ctl->pending_bytes,
+						   count) ? -EFAULT : count;
+
+			break;
+		}
+
+	case VDM_OFFSET(data_offset):
+		{
+			/* as we don't support device internal dirty data
+			 * and our pending_bytes is always 0,
+			 * return error here.
+			 */
+			ret = -EFAULT;
+			break;
+		}
+	case VDM_OFFSET(data_size):
+		if (count != sizeof(mig_ctl->data_size)) {
+			ret = -EINVAL;
+			break;
+		}
+
+		if (iswrite)
+			ret = copy_from_user(&mig_ctl->data_size, buf, count) ?
+					     -EFAULT : count;
+		else
+			ret = copy_to_user(buf, &mig_ctl->data_size, count) ?
+					   -EFAULT : count;
+		break;
+
+	default:
+		ret = -EFAULT;
+		break;
+	}
+	return ret;
+}
+
+static
+int i40e_vf_region_migration_mmap(struct i40e_vf_migration *i40e_vf_dev,
+				  struct i40e_vf_region *region,
+				  struct vm_area_struct *vma)
+{
+	return -EFAULT;
+}
+
+static
+void i40e_vf_region_migration_release(struct i40e_vf_migration *i40e_vf_dev,
+				      struct i40e_vf_region *region)
+{
+	kfree(i40e_vf_dev->mig_ctl);
+	i40e_vf_dev->mig_ctl = NULL;
+}
+
+static const struct i40e_vf_region_ops i40e_vf_region_ops_migration = {
+	.rw		= i40e_vf_region_migration_rw,
+	.release	= i40e_vf_region_migration_release,
+	.mmap		= i40e_vf_region_migration_mmap,
+};
+
+static int i40e_vf_register_region(struct i40e_vf_migration *i40e_vf_dev,
+				   unsigned int type, unsigned int subtype,
+				   const struct i40e_vf_region_ops *ops,
+				   size_t size, u32 flags, void *data)
+{
+	struct i40e_vf_region *regions;
+
+	regions = krealloc(i40e_vf_dev->regions,
+			   (i40e_vf_dev->num_regions + 1) * sizeof(*regions),
+			   GFP_KERNEL);
+	if (!regions)
+		return -ENOMEM;
+
+	i40e_vf_dev->regions = regions;
+	regions[i40e_vf_dev->num_regions].type = type;
+	regions[i40e_vf_dev->num_regions].subtype = subtype;
+	regions[i40e_vf_dev->num_regions].ops = ops;
+	regions[i40e_vf_dev->num_regions].size = size;
+	regions[i40e_vf_dev->num_regions].flags = flags;
+	regions[i40e_vf_dev->num_regions].data = data;
+	i40e_vf_dev->num_regions++;
+	return 0;
+}
+
+static long i40e_vf_get_region_info(void *device_data,
+				    unsigned int cmd, unsigned long arg)
+{
+	struct vfio_region_info info;
+	struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
+	int index, ret;
+	struct vfio_region_info_cap_type cap_type = {
+		.header.id = VFIO_REGION_INFO_CAP_TYPE,
+		.header.version = 1 };
+	struct i40e_vf_region *regions;
+	int num_vdev_regions = vfio_pci_num_regions(device_data);
+	unsigned long minsz;
+	struct i40e_vf_migration *i40e_vf_dev =
+		vfio_pci_vendor_data(device_data);
+
+	minsz = offsetofend(struct vfio_region_info, offset);
+
+	if (cmd != VFIO_DEVICE_GET_REGION_INFO)
+		return -EINVAL;
+	if (copy_from_user(&info, (void __user *)arg, minsz))
+		return -EFAULT;
+	if (info.argsz < minsz)
+		return -EINVAL;
+	if (info.index < VFIO_PCI_NUM_REGIONS + num_vdev_regions)
+		goto default_handle;
+
+	index = info.index - VFIO_PCI_NUM_REGIONS - num_vdev_regions;
+	if (index > i40e_vf_dev->num_regions)
+		return -EINVAL;
+
+	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
+	regions = i40e_vf_dev->regions;
+	info.size = regions[index].size;
+	info.flags = regions[index].flags;
+	cap_type.type = regions[index].type;
+	cap_type.subtype = regions[index].subtype;
+
+	ret = vfio_info_add_capability(&caps, &cap_type.header,
+				       sizeof(cap_type));
+	if (ret)
+		return ret;
+
+	if (regions[index].ops->add_cap) {
+		ret = regions[index].ops->add_cap(i40e_vf_dev,
+				&regions[index], &caps);
+		if (ret)
+			return ret;
+	}
+
+	if (caps.size) {
+		info.flags |= VFIO_REGION_INFO_FLAG_CAPS;
+		if (info.argsz < sizeof(info) + caps.size) {
+			info.argsz = sizeof(info) + caps.size;
+			info.cap_offset = 0;
+		} else {
+			vfio_info_cap_shift(&caps, sizeof(info));
+			if (copy_to_user((void __user *)arg + sizeof(info),
+					 caps.buf, caps.size)) {
+				kfree(caps.buf);
+				return -EFAULT;
+			}
+			info.cap_offset = sizeof(info);
+		}
+
+		kfree(caps.buf);
+	}
+
+	return copy_to_user((void __user *)arg, &info, minsz) ?
+		-EFAULT : 0;
+
+default_handle:
+	return vfio_pci_ioctl(device_data, cmd, arg);
+}
 
 static int i40e_vf_open(void *device_data)
 {
@@ -30,7 +375,26 @@ static int i40e_vf_open(void *device_data)
 
 	mutex_lock(&i40e_vf_dev->reflock);
 	if (!i40e_vf_dev->refcnt) {
-		vfio_pci_set_vendor_regions(device_data, 0);
+		mig_ctl = kzalloc(sizeof(*mig_ctl), GFP_KERNEL);
+		if (!mig_ctl) {
+			ret = -ENOMEM;
+			goto error;
+		}
+
+		ret = i40e_vf_register_region(i40e_vf_dev,
+					      VFIO_REGION_TYPE_MIGRATION,
+					      VFIO_REGION_SUBTYPE_MIGRATION,
+					      &i40e_vf_region_ops_migration,
+					      MIGRATION_REGION_SZ,
+					      VFIO_REGION_INFO_FLAG_READ |
+					      VFIO_REGION_INFO_FLAG_WRITE,
+					      NULL);
+		if (ret)
+			goto error;
+
+		i40e_vf_dev->mig_ctl = mig_ctl;
+		vfio_pci_set_vendor_regions(device_data,
+					    i40e_vf_dev->num_regions);
 		vfio_pci_set_vendor_irqs(device_data, 0);
 	}
 
@@ -43,6 +407,10 @@ static int i40e_vf_open(void *device_data)
 	return 0;
 error:
 	if (!i40e_vf_dev->refcnt) {
+		kfree(mig_ctl);
+		kfree(i40e_vf_dev->regions);
+		i40e_vf_dev->num_regions = 0;
+		i40e_vf_dev->regions = NULL;
 		vfio_pci_set_vendor_regions(device_data, 0);
 		vfio_pci_set_vendor_irqs(device_data, 0);
 	}
@@ -56,8 +424,17 @@ void i40e_vf_release(void *device_data)
 	struct i40e_vf_migration *i40e_vf_dev =
 		vfio_pci_vendor_data(device_data);
 
+	i40e_vf_stop_dirty_track(i40e_vf_dev);
 	mutex_lock(&i40e_vf_dev->reflock);
 	if (!--i40e_vf_dev->refcnt) {
+		int i;
+
+		for (i = 0; i < i40e_vf_dev->num_regions; i++)
+			i40e_vf_dev->regions[i].ops->release(i40e_vf_dev,
+						&i40e_vf_dev->regions[i]);
+		i40e_vf_dev->num_regions = 0;
+		kfree(i40e_vf_dev->regions);
+		i40e_vf_dev->regions = NULL;
 		vfio_pci_set_vendor_regions(device_data, 0);
 		vfio_pci_set_vendor_irqs(device_data, 0);
 	}
@@ -69,19 +446,65 @@ void i40e_vf_release(void *device_data)
 static long i40e_vf_ioctl(void *device_data,
 			  unsigned int cmd, unsigned long arg)
 {
+	if (cmd == VFIO_DEVICE_GET_REGION_INFO)
+		return i40e_vf_get_region_info(device_data, cmd, arg);
+
 	return vfio_pci_ioctl(device_data, cmd, arg);
 }
 
 static ssize_t i40e_vf_read(void *device_data, char __user *buf,
 			    size_t count, loff_t *ppos)
 {
-	return vfio_pci_read(device_data, buf, count, ppos);
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct i40e_vf_migration *i40e_vf_dev =
+		vfio_pci_vendor_data(device_data);
+	struct i40e_vf_region *region;
+	int num_vdev_regions = vfio_pci_num_regions(device_data);
+	int num_vendor_region = i40e_vf_dev->num_regions;
+
+	if (index < VFIO_PCI_NUM_REGIONS + num_vdev_regions)
+		return vfio_pci_read(device_data, buf, count, ppos);
+	else if (index >= VFIO_PCI_NUM_REGIONS + num_vdev_regions +
+			num_vendor_region)
+		return -EINVAL;
+
+	index -= VFIO_PCI_NUM_REGIONS + num_vdev_regions;
+
+	region = &i40e_vf_dev->regions[index];
+	if (!region->ops->rw)
+		return -EINVAL;
+
+	return region->ops->rw(i40e_vf_dev, buf, count, ppos, false);
 }
 
 static ssize_t i40e_vf_write(void *device_data, const char __user *buf,
 			     size_t count, loff_t *ppos)
 {
-	return vfio_pci_write(device_data, buf, count, ppos);
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct i40e_vf_migration *i40e_vf_dev =
+		vfio_pci_vendor_data(device_data);
+	struct i40e_vf_region *region;
+	int num_vdev_regions = vfio_pci_num_regions(device_data);
+	int num_vendor_region = i40e_vf_dev->num_regions;
+
+	if (index == VFIO_PCI_BAR0_REGION_INDEX)
+		;// scan dirty pages
+
+	if (index < VFIO_PCI_NUM_REGIONS + num_vdev_regions)
+		return vfio_pci_write(device_data, buf, count, ppos);
+	else if (index >= VFIO_PCI_NUM_REGIONS + num_vdev_regions +
+			num_vendor_region)
+		return -EINVAL;
+
+	index -= VFIO_PCI_NUM_REGIONS + num_vdev_regions;
+
+	region = &i40e_vf_dev->regions[index];
+
+	if (!region->ops->rw)
+		return -EINVAL;
+
+	return region->ops->rw(i40e_vf_dev, (char __user *)buf,
+			       count, ppos, true);
 }
 
 static int i40e_vf_mmap(void *device_data, struct vm_area_struct *vma)
drivers/net/ethernet/intel/i40e/i40e_vf_migration.h +34 -0
index 696d40601ec3..918ba275d5b5 100644
@@ -17,6 +17,8 @@
 #define VFIO_PCI_INDEX_TO_OFFSET(index)	((u64)(index) << VFIO_PCI_OFFSET_SHIFT)
 #define VFIO_PCI_OFFSET_MASK    (((u64)(1) << VFIO_PCI_OFFSET_SHIFT) - 1)
 
+#define MIGRATION_REGION_SZ (sizeof(struct vfio_device_migration_info))
+
 /* Single Root I/O Virtualization */
 struct pci_sriov {
 	int		pos;		/* Capability position */
@@ -53,6 +55,38 @@ struct i40e_vf_migration {
 	int				vf_id;
 	int				refcnt;
 	struct				mutex reflock; /*mutex protect refcnt */
+
+	struct vfio_device_migration_info *mig_ctl;
+	bool				in_dirty_track;
+
+	struct i40e_vf_region		*regions;
+	int				num_regions;
+	struct notifier_block		iommu_notifier;
+	struct vfio_group		*vfio_group;
+
+};
+
+struct i40e_vf_region_ops {
+	ssize_t	(*rw)(struct i40e_vf_migration *i40e_vf_dev,
+		      char __user *buf, size_t count,
+		      loff_t *ppos, bool iswrite);
+	void	(*release)(struct i40e_vf_migration *i40e_vf_dev,
+			   struct i40e_vf_region *region);
+	int	(*mmap)(struct i40e_vf_migration *i40e_vf_dev,
+			struct i40e_vf_region *region,
+			struct vm_area_struct *vma);
+	int	(*add_cap)(struct i40e_vf_migration *i40e_vf_dev,
+			   struct i40e_vf_region *region,
+			   struct vfio_info_cap *caps);
+};
+
+struct i40e_vf_region {
+	u32				type;
+	u32				subtype;
+	size_t				size;
+	u32				flags;
+	const struct i40e_vf_region_ops	*ops;
+	void				*data;
 };
 
 #endif /* I40E_MIG_H */
Signature · 1 line
2.17.1
#40 lore raw

Re: [RFC PATCH v4 09/10] i40e/vf_migration: register a migration vendor region

[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Yan,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on vfio/next]
[also build test ERROR on jkirsher-next-queue/dev-queue linus/master v5.7-rc6 next-20200515] [cannot apply to linux/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Yan-Zhao/Introduce-vendor-ops-in-vfio-pci/20200518-110542
base:   https://github.com/awilliam/linux-vfio.git next
config: i386-allyesconfig (attached as .config)
compiler: gcc-7 (Ubuntu 7.5.0-6ubuntu2) 7.5.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

If you fix the issue, kindly add following tag as appropriate

All error/warnings (new ones prefixed by >>, old ones prefixed by <<):

drivers/net/ethernet/intel/i40e/i40e_vf_migration.c: In function 'i40e_vf_set_device_state':

>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:132:22: error: dereferencing pointer to incomplete type 'struct vfio_device_migration_info'
if (state == mig_ctl->device_state)
^~
>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:136:7: error: 'VFIO_DEVICE_STATE_RUNNING' undeclared (first use in this function); did you mean 'VFIO_EEH_PE_STATE_UNAVAIL'?

case VFIO_DEVICE_STATE_RUNNING:
^~~~~~~~~~~~~~~~~~~~~~~~~
VFIO_EEH_PE_STATE_UNAVAIL
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:136:7: note: each undeclared identifier is reported only once for each function it appears in

>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:138:7: error: 'VFIO_DEVICE_STATE_SAVING' undeclared (first use in this function); did you mean 'VFIO_DEVICE_STATE_RUNNING'?
case VFIO_DEVICE_STATE_SAVING | VFIO_DEVICE_STATE_RUNNING:
^~~~~~~~~~~~~~~~~~~~~~~~
VFIO_DEVICE_STATE_RUNNING
>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:144:7: error: 'VFIO_DEVICE_STATE_STOP' undeclared (first use in this function); did you mean 'VFIO_DEVICE_STATE_SAVING'?
case VFIO_DEVICE_STATE_STOP:
^~~~~~~~~~~~~~~~~~~~~~
VFIO_DEVICE_STATE_SAVING
>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:149:7: error: 'VFIO_DEVICE_STATE_RESUMING' undeclared (first use in this function); did you mean 'VFIO_DEVICE_STATE_SAVING'?

case VFIO_DEVICE_STATE_RESUMING:
^~~~~~~~~~~~~~~~~~~~~~~~~~
VFIO_DEVICE_STATE_SAVING
In file included from <command-line>:0:0:
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c: In function 'i40e_vf_region_migration_rw':

>> include/linux/compiler_types.h:129:35: error: invalid use of undefined type 'struct vfio_device_migration_info'

#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof' #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER) ^~~~~~~~~~~~~~~~~~~

>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:166:23: note: in expansion of macro 'offsetof'
#define VDM_OFFSET(x) offsetof(struct vfio_device_migration_info, x)
^~~~~~~~
>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:172:7: note: in expansion of macro 'VDM_OFFSET'
case VDM_OFFSET(device_state):
^~~~~~~~~~
>> include/linux/compiler_types.h:129:35: error: invalid use of undefined type 'struct vfio_device_migration_info'

#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof' #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER) ^~~~~~~~~~~~~~~~~~~

>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:166:23: note: in expansion of macro 'offsetof'

#define VDM_OFFSET(x) offsetof(struct vfio_device_migration_info, x)
^~~~~~~~
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:195:7: note: in expansion of macro 'VDM_OFFSET' case VDM_OFFSET(reserved):
^~~~~~~~~~

>> include/linux/compiler_types.h:129:35: error: invalid use of undefined type 'struct vfio_device_migration_info'

#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof' #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER) ^~~~~~~~~~~~~~~~~~~

>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:166:23: note: in expansion of macro 'offsetof'

#define VDM_OFFSET(x) offsetof(struct vfio_device_migration_info, x)
^~~~~~~~
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:199:7: note: in expansion of macro 'VDM_OFFSET' case VDM_OFFSET(pending_bytes):
^~~~~~~~~~

>> include/linux/compiler_types.h:129:35: error: invalid use of undefined type 'struct vfio_device_migration_info'

#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof' #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER) ^~~~~~~~~~~~~~~~~~~

>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:166:23: note: in expansion of macro 'offsetof'

#define VDM_OFFSET(x) offsetof(struct vfio_device_migration_info, x)
^~~~~~~~
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:216:7: note: in expansion of macro 'VDM_OFFSET' case VDM_OFFSET(data_offset):
^~~~~~~~~~

>> include/linux/compiler_types.h:129:35: error: invalid use of undefined type 'struct vfio_device_migration_info'

#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof' #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER) ^~~~~~~~~~~~~~~~~~~

>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:166:23: note: in expansion of macro 'offsetof'

#define VDM_OFFSET(x) offsetof(struct vfio_device_migration_info, x)
^~~~~~~~
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:225:7: note: in expansion of macro 'VDM_OFFSET' case VDM_OFFSET(data_size):
^~~~~~~~~~
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c: In function 'i40e_vf_open':

>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:385:12: error: 'VFIO_REGION_TYPE_MIGRATION' undeclared (first use in this function); did you mean 'VFIO_REGION_TYPE_GFX'?
VFIO_REGION_TYPE_MIGRATION,
^~~~~~~~~~~~~~~~~~~~~~~~~~
VFIO_REGION_TYPE_GFX
>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:386:12: error: 'VFIO_REGION_SUBTYPE_MIGRATION' undeclared (first use in this function); did you mean 'VFIO_REGION_TYPE_MIGRATION'?

VFIO_REGION_SUBTYPE_MIGRATION,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
VFIO_REGION_TYPE_MIGRATION
In file included from drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:16:0:

>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.h:20:37: error: invalid application of 'sizeof' to incomplete type 'struct vfio_device_migration_info'
#define MIGRATION_REGION_SZ (sizeof(struct vfio_device_migration_info))
^
>> drivers/net/ethernet/intel/i40e/i40e_vf_migration.c:388:12: note: in expansion of macro 'MIGRATION_REGION_SZ'

MIGRATION_REGION_SZ,
^~~~~~~~~~~~~~~~~~~

vim +132 drivers/net/ethernet/intel/i40e/i40e_vf_migration.c

125
126 static size_t i40e_vf_set_device_state(struct i40e_vf_migration *i40e_vf_dev,
127 u32 state)
128 {
129 int ret = 0;
130 struct vfio_device_migration_info *mig_ctl = i40e_vf_dev->mig_ctl;
131

 > 132		if (state == mig_ctl->device_state)
   133			return 0;
   134	
   135		switch (state) {
 > 136		case VFIO_DEVICE_STATE_RUNNING:
   137			break;
 > 138		case VFIO_DEVICE_STATE_SAVING | VFIO_DEVICE_STATE_RUNNING:
   139			ret = i40e_vf_prepare_dirty_track(i40e_vf_dev);
   140			break;
   141		case VFIO_DEVICE_STATE_SAVING:
   142			// do the last round of dirty page scanning
   143			break;
 > 144		case VFIO_DEVICE_STATE_STOP:
   145			// release dirty page tracking resources
   146			if (mig_ctl->device_state == VFIO_DEVICE_STATE_SAVING)
   147				i40e_vf_stop_dirty_track(i40e_vf_dev);
   148			break;
 > 149		case VFIO_DEVICE_STATE_RESUMING:
   150			break;
   151		default:
   152			ret = -EFAULT;
   153		}
   154	
   155		if (!ret)
   156			mig_ctl->device_state = state;
   157	
   158		return ret;
   159	}
   160	
   161	static
   162	ssize_t i40e_vf_region_migration_rw(struct i40e_vf_migration *i40e_vf_dev,
   163					    char __user *buf, size_t count,
   164					    loff_t *ppos, bool iswrite)
   165	{
 > 166	#define VDM_OFFSET(x) offsetof(struct vfio_device_migration_info, x)
   167		struct vfio_device_migration_info *mig_ctl = i40e_vf_dev->mig_ctl;
   168		u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
   169		int ret = 0;
   170	
   171		switch (pos) {
 > 172		case VDM_OFFSET(device_state):

173 if (count != sizeof(mig_ctl->device_state)) {
174 ret = -EINVAL;
175 break;
176 }
177
178 if (iswrite) {
179 u32 device_state;
180
181 if (copy_from_user(&device_state, buf, count)) {
182 ret = -EFAULT;
183 break;
184 }
185
186 ret = i40e_vf_set_device_state(i40e_vf_dev,
187 device_state) ?
188 ret : count;
189 } else {
190 ret = copy_to_user(buf, &mig_ctl->device_state,
191 count) ? -EFAULT : count;
192 }
193 break;
194
195 case VDM_OFFSET(reserved):
196 ret = -EFAULT;
197 break;
198
199 case VDM_OFFSET(pending_bytes):
200 {
201 if (count != sizeof(mig_ctl->pending_bytes)) {
202 ret = -EINVAL;
203 break;
204 }
205
206 if (iswrite)
207 ret = -EFAULT;
208 else
209 ret = copy_to_user(buf,
210 &mig_ctl->pending_bytes,
211 count) ? -EFAULT : count;
212
213 break;
214 }
215
216 case VDM_OFFSET(data_offset):
217 {
218 /* as we don't support device internal dirty data
219 * and our pending_bytes is always 0,
220 * return error here.
221 */
222 ret = -EFAULT;
223 break;
224 }
225 case VDM_OFFSET(data_size):
226 if (count != sizeof(mig_ctl->data_size)) {
227 ret = -EINVAL;
228 break;
229 }
230
231 if (iswrite)
232 ret = copy_from_user(&mig_ctl->data_size, buf, count) ?
233 -EFAULT : count;
234 else
235 ret = copy_to_user(buf, &mig_ctl->data_size, count) ?
236 -EFAULT : count;
237 break;
238
239 default:
240 ret = -EFAULT;
241 break;
242 }
243 return ret;
244 }
245

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

author
#41 lore raw

RFC PATCH v4 10/10 i40e/vf_migration: vendor defined irq_type to support dynamic bar map

In reply to Yan Zhao

This patch gives an example implementation to support vendor defined irq_type.

- on this vendor driver open, it registers an irq of type VFIO_IRQ_TYPE_REMAP_BAR_REGION, and reports to driver vfio-pci there's 1 vendor irq.

- after userspace detects and enables the irq of type VFIO_IRQ_TYPE_REMAP_BAR_REGION, this vendor driver will setup a virqfd to monitor file write to the fd of this irq.

(1) when migration starts
(the device state is set to _SAVING & _RUNNING),
a. this vendor driver will signal the irq VFIO_IRQ_TYPE_REMAP_BAR_REGION to ask userspace to remap pci bars. It packs the target bar number in the ctx count. i.e. 1 << bar_number. if there are multiple bars to remap, the numbers are or'ed.

b. on receiving this eventfd signal, userspace will read the bar number, re-query the bar flags (like READ/WRITE/MMAP/SPARSE ranges), and remap the bar's subregions.

c. vendor driver reports bar 0 to be trapped (not MMAP'd).

d. after remapping completion, it writes 0 to the eventfd so that the vendor driver waiting for it would complete too.

(2) as the bar 0 is remapped to be trapped, vendor driver is able to start tracking dirty pages in software way.

(3) when migration stops, similar to what's done in migration start, the vendor driver would signal to remap the bar back to un-trapped (MMAP'd), but it would not wait for the userspace writing back for remapping completion.

- on releasing this vendor driver, it frees resources to vendor defined irqs.

Cc Kevin Tian <[email protected]>
Cc Shaopeng He <[email protected]>
---
 drivers/net/ethernet/intel/Kconfig            |   2 +-
 .../ethernet/intel/i40e/i40e_vf_migration.c   | 322 +++++++++++++++++-
 .../ethernet/intel/i40e/i40e_vf_migration.h   |  26 ++
 3 files changed, 346 insertions(+), 4 deletions(-)
drivers/net/ethernet/intel/Kconfig +1 -1
index 31780d9a59f1..6a52a197c4d8 100644
@@ -266,7 +266,7 @@ config I40E_DCB
 
 config I40E_VF_MIGRATION
 	tristate "XL710 Family VF live migration support -- loadable modules only"
-	depends on I40E && VFIO_PCI && m
+	depends on I40E && VFIO_PCI && VFIO_VIRQFD && m
 	help
 	  Say m if you want to enable live migration of
 	  Virtual Functions of Intel(R) Ethernet Controller XL710
drivers/net/ethernet/intel/i40e/i40e_vf_migration.c +319 -3

387 lines not on this page · raw message

drivers/net/ethernet/intel/i40e/i40e_vf_migration.h +26 -0
index 918ba275d5b5..2c4d9ebee4ac 100644
@@ -46,6 +46,14 @@ struct pci_sriov {
 	bool		drivers_autoprobe; /* Auto probing of VFs by driver */
 };
 
+struct i40e_vf_remap_irq_ctx {
+	struct eventfd_ctx	*trigger;
+	struct virqfd		*sync;
+	atomic_t		cnt;
+	wait_queue_head_t	waitq;
+	bool			init;
+};
+
 struct i40e_vf_migration {
 	__u32				vf_vendor;
 	__u32				vf_device;
@@ -58,11 +66,14 @@ struct i40e_vf_migration {
 
 	struct vfio_device_migration_info *mig_ctl;
 	bool				in_dirty_track;
+	struct i40e_vf_remap_irq_ctx  remap_irq_ctx;
 
 	struct i40e_vf_region		*regions;
 	int				num_regions;
 	struct notifier_block		iommu_notifier;
 	struct vfio_group		*vfio_group;
+	struct i40e_vf_irq		*irqs;
+	int				num_irqs;
 
 };
 
@@ -89,5 +100,20 @@ struct i40e_vf_region {
 	void				*data;
 };
 
+struct i40e_vf_irqops {
+	int (*set_irqs)(struct i40e_vf_migration *i40e_vf_dev,
+			u32 flags, unsigned int index,
+			unsigned int start, unsigned int count,
+			void *data);
+};
+
+struct i40e_vf_irq {
+	u32	type;
+	u32	subtype;
+	u32	flags;
+	u32	count;
+	const struct i40e_vf_irqops *ops;
+};
+
 #endif /* I40E_MIG_H */
 
Signature · 1 line
2.17.1