Skip to contentSkip to conversation

Enhance vfio PCI hot reset for vfio cdev device

PATCH Superseded Yi Liu 4 participants 22 messages May 22, 2023 – Jun 1, 2023 1 message not in the archive

Subsystems (5)
KVM s390 IOMMU VFIO driver +1 more subsystems

v6 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Yi Liu

author
#0 lore raw

v6 cover Enhance vfio PCI hot reset for vfio cdev device

VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. While for cdev devices, user can use an iommufd-based ownership checking model and invoke VFIO_DEVICE_PCI_HOT_RESET with a zero-length fd array.

This series first creates iommufd_access for noiommu devices to fill the gap for adding iommufd-based ownership checking model, then extends VFIO_DEVICE_GET_PCI_HOT_RESET_INFO to check ownership and return the check result and the devid of affected devices to user. In the end, extends the VFIO_DEVICE_PCI_HOT_RESET to accept zero-length fd array for hot-reset with cdev devices.

The new hot reset method and updated _INFO ioctl are tested with the below qemu:

https://github.com/yiliu1765/qemu/tree/iommufd_rfcv4.mig.reset.v4_var3
(requires to test with the cdev kernel)

Change log:

v6:
- Remove noiommu_access, reuse iommufd_access instead (Alex)
- vfio_iommufd_physical_ictx -> vfio_iommufd_device_ictx
- vfio_iommufd_physical_devid -> vfio_iommufd_device_hot_reset_devid
- Refine logic in patch 9 and 10 of v5, no uapi change. (Alex)
- Remove lockdep asset in vfio_pci_is_device_in_set (Cédric)
- Add t-b from Terrence (Tested GVT-g / GVT-d VFIO legacy mode / compat mode / cdev mode, including negative tests. No regression be introduced.)

v5: https://lore.kernel.org/kvm/[email protected]/
- Drop patch 01 of v4 (Alex)
- Create noiommu_access for noiommu devices (Jason)
- Reserve all negative iommufd IDs, hence VFIO can encode negative values (Jason)
- Make vfio_iommufd_physical_devid() return -EINVAL if it's not called with a physical device or a noiommu device.
- Add vfio_find_device_in_devset() in vfio_main.c (Alex)
- Add iommufd_ctx_has_group() to replace vfio_devset_iommufd_has_group().
Reason: vfio_devset_iommufd_has_group() only loops the devices within the given devset to check the iommufd an iommu_group, but an iommu_group can span into multiple devsets. So if failed to find the group in a devset doesn't mean the group is not owned by the iommufd. So here either needs to search all the devsets or add an iommufd API to check it. It appears an iommufd API makes more sense.
- Adopt suggestions from Alex on patch 08 and 09 of v4, refine the hot-reset uapi description and minor tweaks
- Use bitfields for bool members (Alex)

v4: https://lore.kernel.org/kvm/[email protected]/
- Rename the patch series subject
- Patch 01 is moved from the cdev series
- Patch 02, 06 are new per review comments in v3
- Patch 03/04/05/07/08/09 are from v3 with updates

v3: https://lore.kernel.org/kvm/[email protected]/
- Remove the new _INFO ioctl of v2, extend the existing _INFO ioctl to report devid (Alex)
- Add r-b from Jason
- Add t-b from Terrence Xu and Yanting Jiang (mainly regression test)

v2: https://lore.kernel.org/kvm/[email protected]/
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new _INFO ioctl for the usage of device fd passing usage in cdev path (Jason, Alex)

v1: https://lore.kernel.org/kvm/[email protected]/

Regards,
Yi Liu

Yi Liu (10):
vfio-iommufd: Create iommufd_access for noiommu devices vfio/pci: Update comment around group_fd get in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Move the existing hot reset logic to be a helper
iommufd: Reserve all negative IDs in the iommufd xarray
iommufd: Add iommufd_ctx_has_group()
iommufd: Add helper to retrieve iommufd_ctx and devid
vfio: Mark cdev usage in vfio_device
vfio: Add helper to search vfio_device in a dev_set vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device
cdev
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET

 drivers/iommu/iommufd/device.c   |  54 +++++++++++
 drivers/iommu/iommufd/main.c     |   2 +-
 drivers/vfio/iommufd.c           | 101 +++++++++++++++++++-
 drivers/vfio/pci/vfio_pci_core.c | 158 ++++++++++++++++++++++---------
 drivers/vfio/vfio_main.c         |  15 +++
 include/linux/iommufd.h          |  14 +++
 include/linux/vfio.h             |  24 +++++
 include/uapi/linux/vfio.h        |  60 +++++++++++-
 8 files changed, 377 insertions(+), 51 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v6 1/10 vfio-iommufd: Create iommufd_access for noiommu devices

This binds noiommu device to iommufd and creates iommufd_access for this bond. This is useful for adding an iommufd-based device ownership check for VFIO_DEVICE_PCI_HOT_RESET since this model requires all the other affected devices bound to the same iommufd as the device to be reset. For noiommu devices, there is no backend iommu, so create iommufd_access instead of iommufd_device.

---
 drivers/vfio/iommufd.c | 44 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 42 insertions(+), 2 deletions(-)
drivers/vfio/iommufd.c +42 -2
index 88b00c501015..356dd215a8d5 100644
@@ -10,6 +10,43 @@
 MODULE_IMPORT_NS(IOMMUFD);
 MODULE_IMPORT_NS(IOMMUFD_VFIO);
 
+static void vfio_noiommu_access_unmap(void *data, unsigned long iova,
+				      unsigned long length)
+{
+	WARN_ON(1);
+}
+
+static const struct iommufd_access_ops vfio_user_noiommu_ops = {
+	.needs_pin_pages = 1,
+	.unmap = vfio_noiommu_access_unmap,
+};
+
+static int vfio_iommufd_noiommu_bind(struct vfio_device *vdev,
+				     struct iommufd_ctx *ictx,
+				     u32 *out_device_id)
+{
+	struct iommufd_access *user;
+
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	user = iommufd_access_create(ictx, &vfio_user_noiommu_ops,
+				     vdev, out_device_id);
+	if (IS_ERR(user))
+		return PTR_ERR(user);
+	vdev->iommufd_access = user;
+	return 0;
+}
+
+static void vfio_iommufd_noiommu_unbind(struct vfio_device *vdev)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	if (vdev->iommufd_access) {
+		iommufd_access_destroy(vdev->iommufd_access);
+		vdev->iommufd_access = NULL;
+	}
+}
+
 int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
 {
 	u32 ioas_id;
@@ -29,7 +66,8 @@ int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
 		 */
 		if (!iommufd_vfio_compat_ioas_get_id(ictx, &ioas_id))
 			return -EPERM;
-		return 0;
+
+		return vfio_iommufd_noiommu_bind(vdev, ictx, &device_id);
 	}
 
 	ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
@@ -59,8 +97,10 @@ void vfio_iommufd_unbind(struct vfio_device *vdev)
 {
 	lockdep_assert_held(&vdev->dev_set->lock);
 
-	if (vfio_device_is_noiommu(vdev))
+	if (vfio_device_is_noiommu(vdev)) {
+		vfio_iommufd_noiommu_unbind(vdev);
 		return;
+	}
 
 	if (vdev->ops->unbind_iommufd)
 		vdev->ops->unbind_iommufd(vdev);
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH v6 2/10 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset()

In reply to Yi Liu

This suits more on what the code does.

---
 drivers/vfio/pci/vfio_pci_core.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +2 -3
index a5ab416cf476..f824de4dbf27 100644
@@ -1308,9 +1308,8 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	}
 
 	/*
-	 * For each group_fd, get the group through the vfio external user
-	 * interface and store the group and iommu ID.  This ensures the group
-	 * is held across the reset.
+	 * Get the group file for each fd to ensure the group is held across
+	 * the reset
 	 */
 	for (file_idx = 0; file_idx < hdr.count; file_idx++) {
 		struct file *file = fget(group_fds[file_idx]);
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH v6 3/10 vfio/pci: Move the existing hot reset logic to be a helper

In reply to Yi Liu

This prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups().

No functional change is intended.

---
 drivers/vfio/pci/vfio_pci_core.c | 55 +++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 23 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +32 -23
index f824de4dbf27..39e7823088e7 100644
@@ -1255,29 +1255,16 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(
 	return ret;
 }
 
-static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
-					struct vfio_pci_hot_reset __user *arg)
+static int
+vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
+				    int array_count, bool slot,
+				    struct vfio_pci_hot_reset __user *arg)
 {
-	unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count);
-	struct vfio_pci_hot_reset hdr;
 	int32_t *group_fds;
 	struct file **files;
 	struct vfio_pci_group_info info;
-	bool slot = false;
 	int file_idx, count = 0, ret = 0;
 
-	if (copy_from_user(&hdr, arg, minsz))
-		return -EFAULT;
-
-	if (hdr.argsz < minsz || hdr.flags)
-		return -EINVAL;
-
-	/* Can we do a slot or bus reset or neither? */
-	if (!pci_probe_reset_slot(vdev->pdev->slot))
-		slot = true;
-	else if (pci_probe_reset_bus(vdev->pdev->bus))
-		return -ENODEV;
-
 	/*
 	 * We can't let userspace give us an arbitrarily large buffer to copy,
 	 * so verify how many we think there could be.  Note groups can have
@@ -1289,11 +1276,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 		return ret;
 
 	/* Somewhere between 1 and count is OK */
-	if (!hdr.count || hdr.count > count)
+	if (!array_count || array_count > count)
 		return -EINVAL;
 
-	group_fds = kcalloc(hdr.count, sizeof(*group_fds), GFP_KERNEL);
-	files = kcalloc(hdr.count, sizeof(*files), GFP_KERNEL);
+	group_fds = kcalloc(array_count, sizeof(*group_fds), GFP_KERNEL);
+	files = kcalloc(array_count, sizeof(*files), GFP_KERNEL);
 	if (!group_fds || !files) {
 		kfree(group_fds);
 		kfree(files);
@@ -1301,7 +1288,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	}
 
 	if (copy_from_user(group_fds, arg->group_fds,
-			   hdr.count * sizeof(*group_fds))) {
+			   array_count * sizeof(*group_fds))) {
 		kfree(group_fds);
 		kfree(files);
 		return -EFAULT;
@@ -1311,7 +1298,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	 * Get the group file for each fd to ensure the group is held across
 	 * the reset
 	 */
-	for (file_idx = 0; file_idx < hdr.count; file_idx++) {
+	for (file_idx = 0; file_idx < array_count; file_idx++) {
 		struct file *file = fget(group_fds[file_idx]);
 
 		if (!file) {
@@ -1335,7 +1322,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	if (ret)
 		goto hot_reset_release;
 
-	info.count = hdr.count;
+	info.count = array_count;
 	info.files = files;
 
 	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info);
@@ -1348,6 +1335,28 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	return ret;
 }
 
+static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
+					struct vfio_pci_hot_reset __user *arg)
+{
+	unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count);
+	struct vfio_pci_hot_reset hdr;
+	bool slot = false;
+
+	if (copy_from_user(&hdr, arg, minsz))
+		return -EFAULT;
+
+	if (hdr.argsz < minsz || hdr.flags)
+		return -EINVAL;
+
+	/* Can we do a slot or bus reset or neither? */
+	if (!pci_probe_reset_slot(vdev->pdev->slot))
+		slot = true;
+	else if (pci_probe_reset_bus(vdev->pdev->bus))
+		return -ENODEV;
+
+	return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
+}
+
 static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev,
 				    struct vfio_device_ioeventfd __user *arg)
 {
Signature · 1 line
2.34.1
author
#4 lore raw

PATCH v6 4/10 iommufd: Reserve all negative IDs in the iommufd xarray

In reply to Yi Liu

With this reservation, IOMMUFD users can encode the negative IDs for specific purposes. e.g. VFIO needs two reserved values to tell userspace the ID returned is not valid but has other meaning.

---
 drivers/iommu/iommufd/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iommu/iommufd/main.c +1 -1
index 3fbe636c3d8a..32ce7befc8dd 100644
@@ -50,7 +50,7 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx,
 	 * before calling iommufd_object_finalize().
 	 */
 	rc = xa_alloc(&ictx->objects, &obj->id, XA_ZERO_ENTRY,
-		      xa_limit_32b, GFP_KERNEL_ACCOUNT);
+		      xa_limit_31b, GFP_KERNEL_ACCOUNT);
 	if (rc)
 		goto out_free;
 	return obj;
Signature · 1 line
2.34.1
author
#5 lore raw

PATCH v6 5/10 iommufd: Add iommufd_ctx_has_group()

In reply to Yi Liu

This adds the helper to check if any device within the given iommu_group has been bound with the iommufd_ctx. This is helpful for the checking on device ownership for the devices which have not been bound but cannot be bound to any other iommufd_ctx as the iommu_group has been bound.

---
 drivers/iommu/iommufd/device.c | 30 ++++++++++++++++++++++++++++++
 include/linux/iommufd.h        |  8 ++++++++
 2 files changed, 38 insertions(+)
drivers/iommu/iommufd/device.c +30 -0
index 4f9b2142274c..4571344c8508 100644
@@ -98,6 +98,36 @@ struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx,
 }
 EXPORT_SYMBOL_NS_GPL(iommufd_device_bind, IOMMUFD);
 
+/**
+ * iommufd_ctx_has_group - True if any device within the group is bound
+ *                         to the ictx
+ * @ictx: iommufd file descriptor
+ * @group: Pointer to a physical iommu_group struct
+ *
+ * True if any device within the group has been bound to this ictx, ex. via
+ * iommufd_device_bind(), therefore implying ictx ownership of the group.
+ */
+bool iommufd_ctx_has_group(struct iommufd_ctx *ictx, struct iommu_group *group)
+{
+	struct iommufd_object *obj;
+	unsigned long index;
+
+	if (!ictx || !group)
+		return false;
+
+	xa_lock(&ictx->objects);
+	xa_for_each(&ictx->objects, index, obj) {
+		if (obj->type == IOMMUFD_OBJ_DEVICE &&
+		    container_of(obj, struct iommufd_device, obj)->group == group) {
+			xa_unlock(&ictx->objects);
+			return true;
+		}
+	}
+	xa_unlock(&ictx->objects);
+	return false;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_ctx_has_group, IOMMUFD);
+
 /**
  * iommufd_device_unbind - Undo iommufd_device_bind()
  * @idev: Device returned by iommufd_device_bind()
include/linux/iommufd.h +8 -0
index 1129a36a74c4..33fe57e95e42 100644
@@ -16,6 +16,7 @@ struct page;
 struct iommufd_ctx;
 struct iommufd_access;
 struct file;
+struct iommu_group;
 
 struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx,
 					   struct device *dev, u32 *id);
@@ -50,6 +51,7 @@ void iommufd_ctx_get(struct iommufd_ctx *ictx);
 #if IS_ENABLED(CONFIG_IOMMUFD)
 struct iommufd_ctx *iommufd_ctx_from_file(struct file *file);
 void iommufd_ctx_put(struct iommufd_ctx *ictx);
+bool iommufd_ctx_has_group(struct iommufd_ctx *ictx, struct iommu_group *group);
 
 int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova,
 			     unsigned long length, struct page **out_pages,
@@ -71,6 +73,12 @@ static inline void iommufd_ctx_put(struct iommufd_ctx *ictx)
 {
 }
 
+static inline bool iommufd_ctx_has_group(struct iommufd_ctx *ictx,
+					 struct iommu_group *group)
+{
+	return false;
+}
+
 static inline int iommufd_access_pin_pages(struct iommufd_access *access,
 					   unsigned long iova,
 					   unsigned long length,
Signature · 1 line
2.34.1
author
#6 lore raw

PATCH v6 6/10 iommufd: Add helper to retrieve iommufd_ctx and devid

In reply to Yi Liu

This is needed by the vfio-pci driver to report affected devices in the hot-reset for a given device.

---
 drivers/iommu/iommufd/device.c | 24 ++++++++++++++++++++++++
 include/linux/iommufd.h        |  6 ++++++
 2 files changed, 30 insertions(+)
drivers/iommu/iommufd/device.c +24 -0
index 4571344c8508..ff3142dca96f 100644
@@ -146,6 +146,18 @@ void iommufd_device_unbind(struct iommufd_device *idev)
 }
 EXPORT_SYMBOL_NS_GPL(iommufd_device_unbind, IOMMUFD);
 
+struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev)
+{
+	return idev->ictx;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_device_to_ictx, IOMMUFD);
+
+u32 iommufd_device_to_id(struct iommufd_device *idev)
+{
+	return idev->obj.id;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, IOMMUFD);
+
 static int iommufd_device_setup_msi(struct iommufd_device *idev,
 				    struct iommufd_hw_pagetable *hwpt,
 				    phys_addr_t sw_msi_start)
@@ -493,6 +505,18 @@ void iommufd_access_destroy(struct iommufd_access *access)
 }
 EXPORT_SYMBOL_NS_GPL(iommufd_access_destroy, IOMMUFD);
 
+struct iommufd_ctx *iommufd_access_to_ictx(struct iommufd_access *access)
+{
+	return access->ictx;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_access_to_ictx, IOMMUFD);
+
+u32 iommufd_access_to_id(struct iommufd_access *access)
+{
+	return access->obj.id;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_access_to_id, IOMMUFD);
+
 int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id)
 {
 	struct iommufd_ioas *new_ioas;
include/linux/iommufd.h +6 -0
index 33fe57e95e42..e49c16cd6831 100644
@@ -25,6 +25,9 @@ void iommufd_device_unbind(struct iommufd_device *idev);
 int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id);
 void iommufd_device_detach(struct iommufd_device *idev);
 
+struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev);
+u32 iommufd_device_to_id(struct iommufd_device *idev);
+
 struct iommufd_access_ops {
 	u8 needs_pin_pages : 1;
 	void (*unmap)(void *data, unsigned long iova, unsigned long length);
@@ -46,6 +49,9 @@ iommufd_access_create(struct iommufd_ctx *ictx,
 void iommufd_access_destroy(struct iommufd_access *access);
 int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id);
 
+struct iommufd_ctx *iommufd_access_to_ictx(struct iommufd_access *access);
+u32 iommufd_access_to_id(struct iommufd_access *access);
+
 void iommufd_ctx_get(struct iommufd_ctx *ictx);
 
 #if IS_ENABLED(CONFIG_IOMMUFD)
Signature · 1 line
2.34.1
author
#7 lore raw

PATCH v6 7/10 vfio: Mark cdev usage in vfio_device

In reply to Yi Liu

This can be used to differentiate whether to report group_id or devid in the revised VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl. At this moment, no cdev path yet, so the vfio_device_cdev_opened() helper always returns false.

---
 include/linux/vfio.h | 5 +++++
 1 file changed, 5 insertions(+)
include/linux/vfio.h +5 -0
index 2c137ea94a3e..2a45853773a6 100644
@@ -139,6 +139,11 @@ int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
 	((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
 #endif
 
+static inline bool vfio_device_cdev_opened(struct vfio_device *device)
+{
+	return false;
+}
+
 /**
  * struct vfio_migration_ops - VFIO bus device driver migration callbacks
  *
Signature · 1 line
2.34.1
author
#8 lore raw

PATCH v6 8/10 vfio: Add helper to search vfio_device in a dev_set

In reply to Yi Liu

There are drivers that need to search vfio_device within a given dev_set. e.g. vfio-pci. So add a helper.

vfio_pci_is_device_in_set() now returns -EBUSY in commit a882c16a2b7e ("vfio/pci: Change vfio_pci_try_bus_reset() to use the dev_set") where it was trying to preserve the return of vfio_pci_try_zap_and_vma_lock_cb(). However, it makes more sense to return -ENODEV.

---
 drivers/vfio/pci/vfio_pci_core.c |  6 +-----
 drivers/vfio/vfio_main.c         | 15 +++++++++++++++
 include/linux/vfio.h             |  3 +++
 3 files changed, 19 insertions(+), 5 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +1 -5
index 39e7823088e7..3a2f67675036 100644
@@ -2335,12 +2335,8 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev,
 static int vfio_pci_is_device_in_set(struct pci_dev *pdev, void *data)
 {
 	struct vfio_device_set *dev_set = data;
-	struct vfio_device *cur;
 
-	list_for_each_entry(cur, &dev_set->device_list, dev_set_list)
-		if (cur->dev == &pdev->dev)
-			return 0;
-	return -EBUSY;
+	return vfio_find_device_in_devset(dev_set, &pdev->dev) ? 0 : -ENODEV;
 }
 
 /*
drivers/vfio/vfio_main.c +15 -0
index f0ca33b2e1df..ab4f3a794f78 100644
@@ -141,6 +141,21 @@ unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set)
 }
 EXPORT_SYMBOL_GPL(vfio_device_set_open_count);
 
+struct vfio_device *
+vfio_find_device_in_devset(struct vfio_device_set *dev_set,
+			   struct device *dev)
+{
+	struct vfio_device *cur;
+
+	lockdep_assert_held(&dev_set->lock);
+
+	list_for_each_entry(cur, &dev_set->device_list, dev_set_list)
+		if (cur->dev == dev)
+			return cur;
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(vfio_find_device_in_devset);
+
 /*
  * Device objects - create, release, get, put, search
  */
include/linux/vfio.h +3 -0
index 2a45853773a6..ee120d2d530b 100644
@@ -244,6 +244,9 @@ void vfio_unregister_group_dev(struct vfio_device *device);
 
 int vfio_assign_device_set(struct vfio_device *device, void *set_id);
 unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set);
+struct vfio_device *
+vfio_find_device_in_devset(struct vfio_device_set *dev_set,
+			   struct device *dev);
 
 int vfio_mig_get_next_state(struct vfio_device *device,
 			    enum vfio_device_mig_state cur_fsm,
Signature · 1 line
2.34.1
author
#9 lore raw

PATCH v6 9/10 vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev

In reply to Yi Liu

This allows VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl use the iommufd_ctx of the cdev device to check the ownership of the other affected devices.

When VFIO_DEVICE_GET_PCI_HOT_RESET_INFO is called on an IOMMUFD managed device, the new flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is reported to indicate the values returned are IOMMUFD devids rather than group IDs as used when accessing vfio devices through the conventional vfio group interface. Additionally the flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED will be reported in this mode if all of the devices affected by the hot-reset are owned by either virtue of being directly bound to the same iommufd context as the calling device, or implicitly owned via a shared IOMMU group.

---
 drivers/vfio/iommufd.c           | 57 ++++++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_core.c | 40 ++++++++++++++++++----
 include/linux/vfio.h             | 16 +++++++++
 include/uapi/linux/vfio.h        | 46 +++++++++++++++++++++++++-
 4 files changed, 151 insertions(+), 8 deletions(-)
drivers/vfio/iommufd.c +57 -0
index 356dd215a8d5..4dae9ab94eed 100644
@@ -106,6 +106,63 @@ void vfio_iommufd_unbind(struct vfio_device *vdev)
 		vdev->ops->unbind_iommufd(vdev);
 }
 
+struct iommufd_ctx *vfio_iommufd_device_ictx(struct vfio_device *vdev)
+{
+	if (vdev->iommufd_device)
+		return iommufd_device_to_ictx(vdev->iommufd_device);
+	if (vdev->iommufd_access)
+		return iommufd_access_to_ictx(vdev->iommufd_access);
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_device_ictx);
+
+static int vfio_iommufd_device_id(struct vfio_device *vdev)
+{
+	if (vdev->iommufd_device)
+		return iommufd_device_to_id(vdev->iommufd_device);
+	if (vdev->iommufd_access)
+		return iommufd_access_to_id(vdev->iommufd_access);
+	return -EINVAL;
+}
+
+/*
+ * Return devid for vfio_device if the device is owned by the input
+ * ictx.
+ * - valid devid > 0 for the device that are bound to the input
+ *   iommufd_ctx.
+ * - devid == VFIO_PCI_DEVID_OWNED for the devices that have not
+ *   been opened but but other device within its group has been
+ *   bound to the input iommufd_ctx.
+ * - devid == VFIO_PCI_DEVID_NOT_OWNED for others. e.g. vdev is
+ *   NULL.
+ */
+int vfio_iommufd_device_hot_reset_devid(struct vfio_device *vdev,
+					struct iommufd_ctx *ictx)
+{
+	struct iommu_group *group;
+	int devid;
+
+	if (!vdev)
+		return VFIO_PCI_DEVID_NOT_OWNED;
+
+	if (vfio_iommufd_device_ictx(vdev) == ictx)
+		return vfio_iommufd_device_id(vdev);
+
+	group = iommu_group_get(vdev->dev);
+	if (!group)
+		return VFIO_PCI_DEVID_NOT_OWNED;
+
+	if (iommufd_ctx_has_group(ictx, group))
+		devid = VFIO_PCI_DEVID_OWNED;
+	else
+		devid = VFIO_PCI_DEVID_NOT_OWNED;
+
+	iommu_group_put(group);
+
+	return devid;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_device_hot_reset_devid);
+
 /*
  * The physical standard ops mean that the iommufd_device is bound to the
  * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers
drivers/vfio/pci/vfio_pci_core.c +33 -7
index 3a2f67675036..890065f846e4 100644
@@ -27,6 +27,7 @@
 #include <linux/vgaarb.h>
 #include <linux/nospec.h>
 #include <linux/sched/mm.h>
+#include <linux/iommufd.h>
 #if IS_ENABLED(CONFIG_EEH)
 #include <asm/eeh.h>
 #endif
@@ -776,26 +777,42 @@ struct vfio_pci_fill_info {
 	int max;
 	int cur;
 	struct vfio_pci_dependent_device *devices;
+	struct vfio_device *vdev;
+	u32 flags;
 };
 
 static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)
 {
 	struct vfio_pci_fill_info *fill = data;
-	struct iommu_group *iommu_group;
 
 	if (fill->cur == fill->max)
 		return -EAGAIN; /* Something changed, try again */
 
-	iommu_group = iommu_group_get(&pdev->dev);
-	if (!iommu_group)
-		return -EPERM; /* Cannot reset non-isolated devices */
+	if (fill->flags & VFIO_PCI_HOT_RESET_FLAG_DEV_ID) {
+		struct iommufd_ctx *iommufd = vfio_iommufd_device_ictx(fill->vdev);
+		struct vfio_device_set *dev_set = fill->vdev->dev_set;
+		struct vfio_device *vdev;
+
+		vdev = vfio_find_device_in_devset(dev_set, &pdev->dev);
+		fill->devices[fill->cur].devid =
+			vfio_iommufd_device_hot_reset_devid(vdev, iommufd);
+		/* If devid is VFIO_PCI_DEVID_NOT_OWNED, clear owned flag. */
+		if (fill->devices[fill->cur].devid == VFIO_PCI_DEVID_NOT_OWNED)
+			fill->flags &= ~VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED;
+	} else {
+		struct iommu_group *iommu_group;
+
+		iommu_group = iommu_group_get(&pdev->dev);
+		if (!iommu_group)
+			return -EPERM; /* Cannot reset non-isolated devices */
 
-	fill->devices[fill->cur].group_id = iommu_group_id(iommu_group);
+		fill->devices[fill->cur].group_id = iommu_group_id(iommu_group);
+		iommu_group_put(iommu_group);
+	}
 	fill->devices[fill->cur].segment = pci_domain_nr(pdev->bus);
 	fill->devices[fill->cur].bus = pdev->bus->number;
 	fill->devices[fill->cur].devfn = pdev->devfn;
 	fill->cur++;
-	iommu_group_put(iommu_group);
 	return 0;
 }
 
@@ -1229,17 +1246,26 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(
 		return -ENOMEM;
 
 	fill.devices = devices;
+	fill.vdev = &vdev->vdev;
 
+	if (vfio_device_cdev_opened(&vdev->vdev))
+		fill.flags |= VFIO_PCI_HOT_RESET_FLAG_DEV_ID |
+			     VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED;
+
+	mutex_lock(&vdev->vdev.dev_set->lock);
 	ret = vfio_pci_for_each_slot_or_bus(vdev->pdev, vfio_pci_fill_devs,
 					    &fill, slot);
+	mutex_unlock(&vdev->vdev.dev_set->lock);
 
 	/*
 	 * If a device was removed between counting and filling, we may come up
 	 * short of fill.max.  If a device was added, we'll have a return of
 	 * -EAGAIN above.
 	 */
-	if (!ret)
+	if (!ret) {
 		hdr.count = fill.cur;
+		hdr.flags = fill.flags;
+	}
 
 reset_info_exit:
 	if (copy_to_user(arg, &hdr, minsz))
include/linux/vfio.h +16 -0
index ee120d2d530b..382a7b119c7c 100644
@@ -114,6 +114,9 @@ struct vfio_device_ops {
 };
 
 #if IS_ENABLED(CONFIG_IOMMUFD)
+struct iommufd_ctx *vfio_iommufd_device_ictx(struct vfio_device *vdev);
+int vfio_iommufd_device_hot_reset_devid(struct vfio_device *vdev,
+					struct iommufd_ctx *ictx);
 int vfio_iommufd_physical_bind(struct vfio_device *vdev,
 			       struct iommufd_ctx *ictx, u32 *out_device_id);
 void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
@@ -123,6 +126,19 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
 void vfio_iommufd_emulated_unbind(struct vfio_device *vdev);
 int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
 #else
+static inline struct iommufd_ctx *
+vfio_iommufd_device_ictx(struct vfio_device *vdev)
+{
+	return NULL;
+}
+
+static inline int
+vfio_iommufd_device_hot_reset_devid(struct vfio_device *vdev,
+				    struct iommufd_ctx *ictx)
+{
+	return VFIO_PCI_DEVID_NOT_OWNED;
+}
+
 #define vfio_iommufd_physical_bind                                      \
 	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
 		  u32 *out_device_id)) NULL)
include/uapi/linux/vfio.h +45 -1

62 lines not on this page · raw message

Signature · 1 line
2.34.1
#10 lore raw

Re: [Intel-gfx] [PATCH v6 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev

Quoted 46 lines · On Mon, 22 May 2023 04:57:50 -0700 Yi Liu <[email protected]> wrote:

This allows VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl use the iommufd_ctx of the cdev device to check the ownership of the other affected devices.

When VFIO_DEVICE_GET_PCI_HOT_RESET_INFO is called on an IOMMUFD managed device, the new flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is reported to indicate the values returned are IOMMUFD devids rather than group IDs as used when accessing vfio devices through the conventional vfio group interface. Additionally the flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED will be reported in this mode if all of the devices affected by the hot-reset are owned by either virtue of being directly bound to the same iommufd context as the calling device, or implicitly owned via a shared IOMMU group.

---
 drivers/vfio/iommufd.c           | 57 ++++++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_core.c | 40 ++++++++++++++++++----
 include/linux/vfio.h             | 16 +++++++++
 include/uapi/linux/vfio.h        | 46 +++++++++++++++++++++++++-
 4 files changed, 151 insertions(+), 8 deletions(-)
drivers/vfio/iommufd.c +16 -0
index 356dd215a8d5..4dae9ab94eed 100644
@@ -106,6 +106,63 @@ void vfio_iommufd_unbind(struct vfio_device *vdev)
 		vdev->ops->unbind_iommufd(vdev);
 }
 
+struct iommufd_ctx *vfio_iommufd_device_ictx(struct vfio_device *vdev)
+{
+	if (vdev->iommufd_device)
+		return iommufd_device_to_ictx(vdev->iommufd_device);
+	if (vdev->iommufd_access)
+		return iommufd_access_to_ictx(vdev->iommufd_access);
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_device_ictx);
+
+static int vfio_iommufd_device_id(struct vfio_device *vdev)
+{
+	if (vdev->iommufd_device)
+		return iommufd_device_to_id(vdev->iommufd_device);
+	if (vdev->iommufd_access)
+		return iommufd_access_to_id(vdev->iommufd_access);

drivers/vfio/iommufd.c void vfio_iommufd_unbind(struct vfio_device *vdev) · line 134

Quoted 10 lines
drivers/vfio/iommufd.c +10 -0
+	return -EINVAL;
+}
+
+/*
+ * Return devid for vfio_device if the device is owned by the input
+ * ictx.
+ * - valid devid > 0 for the device that are bound to the input
+ *   iommufd_ctx.
+ * - devid == VFIO_PCI_DEVID_OWNED for the devices that have not
+ *   been opened but but other device within its group has been

"but but"

Quoted 185 lines
drivers/vfio/iommufd.c +31 -0
+ *   bound to the input iommufd_ctx.
+ * - devid == VFIO_PCI_DEVID_NOT_OWNED for others. e.g. vdev is
+ *   NULL.
+ */
+int vfio_iommufd_device_hot_reset_devid(struct vfio_device *vdev,
+					struct iommufd_ctx *ictx)
+{
+	struct iommu_group *group;
+	int devid;
+
+	if (!vdev)
+		return VFIO_PCI_DEVID_NOT_OWNED;
+
+	if (vfio_iommufd_device_ictx(vdev) == ictx)
+		return vfio_iommufd_device_id(vdev);
+
+	group = iommu_group_get(vdev->dev);
+	if (!group)
+		return VFIO_PCI_DEVID_NOT_OWNED;
+
+	if (iommufd_ctx_has_group(ictx, group))
+		devid = VFIO_PCI_DEVID_OWNED;
+	else
+		devid = VFIO_PCI_DEVID_NOT_OWNED;
+
+	iommu_group_put(group);
+
+	return devid;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_device_hot_reset_devid);
+
 /*
  * The physical standard ops mean that the iommufd_device is bound to the
  * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers
drivers/vfio/pci/vfio_pci_core.c +33 -7
index 3a2f67675036..890065f846e4 100644
@@ -27,6 +27,7 @@
 #include <linux/vgaarb.h>
 #include <linux/nospec.h>
 #include <linux/sched/mm.h>
+#include <linux/iommufd.h>
 #if IS_ENABLED(CONFIG_EEH)
 #include <asm/eeh.h>
 #endif
@@ -776,26 +777,42 @@ struct vfio_pci_fill_info {
 	int max;
 	int cur;
 	struct vfio_pci_dependent_device *devices;
+	struct vfio_device *vdev;
+	u32 flags;
 };
 
 static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)
 {
 	struct vfio_pci_fill_info *fill = data;
-	struct iommu_group *iommu_group;
 
 	if (fill->cur == fill->max)
 		return -EAGAIN; /* Something changed, try again */
 
-	iommu_group = iommu_group_get(&pdev->dev);
-	if (!iommu_group)
-		return -EPERM; /* Cannot reset non-isolated devices */
+	if (fill->flags & VFIO_PCI_HOT_RESET_FLAG_DEV_ID) {
+		struct iommufd_ctx *iommufd = vfio_iommufd_device_ictx(fill->vdev);
+		struct vfio_device_set *dev_set = fill->vdev->dev_set;
+		struct vfio_device *vdev;
+
+		vdev = vfio_find_device_in_devset(dev_set, &pdev->dev);
+		fill->devices[fill->cur].devid =
+			vfio_iommufd_device_hot_reset_devid(vdev, iommufd);
+		/* If devid is VFIO_PCI_DEVID_NOT_OWNED, clear owned flag. */
+		if (fill->devices[fill->cur].devid == VFIO_PCI_DEVID_NOT_OWNED)
+			fill->flags &= ~VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED;
+	} else {
+		struct iommu_group *iommu_group;
+
+		iommu_group = iommu_group_get(&pdev->dev);
+		if (!iommu_group)
+			return -EPERM; /* Cannot reset non-isolated devices */
 
-	fill->devices[fill->cur].group_id = iommu_group_id(iommu_group);
+		fill->devices[fill->cur].group_id = iommu_group_id(iommu_group);
+		iommu_group_put(iommu_group);
+	}
 	fill->devices[fill->cur].segment = pci_domain_nr(pdev->bus);
 	fill->devices[fill->cur].bus = pdev->bus->number;
 	fill->devices[fill->cur].devfn = pdev->devfn;
 	fill->cur++;
-	iommu_group_put(iommu_group);
 	return 0;
 }
 
@@ -1229,17 +1246,26 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(
 		return -ENOMEM;
 
 	fill.devices = devices;
+	fill.vdev = &vdev->vdev;
 
+	if (vfio_device_cdev_opened(&vdev->vdev))
+		fill.flags |= VFIO_PCI_HOT_RESET_FLAG_DEV_ID |
+			     VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED;
+
+	mutex_lock(&vdev->vdev.dev_set->lock);
 	ret = vfio_pci_for_each_slot_or_bus(vdev->pdev, vfio_pci_fill_devs,
 					    &fill, slot);
+	mutex_unlock(&vdev->vdev.dev_set->lock);
 
 	/*
 	 * If a device was removed between counting and filling, we may come up
 	 * short of fill.max.  If a device was added, we'll have a return of
 	 * -EAGAIN above.
 	 */
-	if (!ret)
+	if (!ret) {
 		hdr.count = fill.cur;
+		hdr.flags = fill.flags;
+	}
 
 reset_info_exit:
 	if (copy_to_user(arg, &hdr, minsz))
include/linux/vfio.h +16 -0
index ee120d2d530b..382a7b119c7c 100644
@@ -114,6 +114,9 @@ struct vfio_device_ops {
 };
 
 #if IS_ENABLED(CONFIG_IOMMUFD)
+struct iommufd_ctx *vfio_iommufd_device_ictx(struct vfio_device *vdev);
+int vfio_iommufd_device_hot_reset_devid(struct vfio_device *vdev,
+					struct iommufd_ctx *ictx);
 int vfio_iommufd_physical_bind(struct vfio_device *vdev,
 			       struct iommufd_ctx *ictx, u32 *out_device_id);
 void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
@@ -123,6 +126,19 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
 void vfio_iommufd_emulated_unbind(struct vfio_device *vdev);
 int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
 #else
+static inline struct iommufd_ctx *
+vfio_iommufd_device_ictx(struct vfio_device *vdev)
+{
+	return NULL;
+}
+
+static inline int
+vfio_iommufd_device_hot_reset_devid(struct vfio_device *vdev,
+				    struct iommufd_ctx *ictx)
+{
+	return VFIO_PCI_DEVID_NOT_OWNED;
+}
+
 #define vfio_iommufd_physical_bind                                      \
 	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
 		  u32 *out_device_id)) NULL)
include/uapi/linux/vfio.h +20 -0
index 0552e8dcf0cb..01203215251a 100644
@@ -650,11 +650,53 @@ enum {
  * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12,
  *					      struct vfio_pci_hot_reset_info)
  *
+ * This command is used to query the affected devices in the hot reset for
+ * a given device.
+ *
+ * This command always reports the segment, bus, and devfn information for
+ * each affected device, and selectively reports the group_id or devid per
+ * the way how the calling device is opened.
+ *
+ *	- If the calling device is opened via the traditional group/container
+ *	  API, group_id is reported.  User should check if it has owned all
+ *	  the affected devices and provides a set of group fds to prove the
+ *	  ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl.
+ *
+ *	- If the calling device is opened as a cdev, devid is reported.
+ *	  Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set to indicate this
+ *	  data type.  For a given affected device, it is considered owned by
+ *	  this interface if it meets the following conditions:
+ *	  1) Has a valid devid within the iommufd_ctx of the calling device.
+ *	     Ownership cannot be determined across separate iommufd_ctx and the
+ *	     cdev calling conventions do not support a proof-of-ownership model
+ *	     as provided in the legacy group interface.  In this case a valid

include/uapi/linux/vfio.h enum { · line 682

Quoted 10 lines
include/uapi/linux/vfio.h +10 -0
+ *	     devid with value greater than zero is provided in the return
+ *	     structure.
+ *	  2) Does not have a valid devid within the iommufd_ctx of the calling
+ *	     device, but belongs to the same IOMMU group as the calling device
+ *	     or another opened device that has a valid devid within the
+ *	     iommufd_ctx of the calling device.  This provides implicit ownership
+ *	     for devices within the same DMA isolation context.  In this case
+ *	     the invalid devid value of zero is provided in the return structure.
+ *
+ *	  A devid value of -1 is provided in the return structure for devices

s/zero/VFIO_PCI_DEVID_OWNED/

s/-1/VFIO_PCI_DEVID_NOT_OWNED/

2) above and previously in the code comment where I noted the repeated "but" still doesn't actually describe the requirement as I noted in the last review. The user implicitly owns a device if they own another device within the IOMMU group, but we also impose a dev_set requirement in the hot reset path. All affected devices need to be represented in the dev_set, ex. bound to a vfio driver. It's possible that requirement might be relaxed in the new DMA ownership model, but as it is right now, the code enforces that requirement and any new discussion about what makes hot-reset available should note both the ownership and dev_set requirement. Thanks,

Alex

Quoted 30 lines
include/uapi/linux/vfio.h +15 -1
+ *	  where ownership is not available.  Such devices prevent the use of
+ *	  VFIO_DEVICE_PCI_HOT_RESET outside of proof-of-ownership calling
+ *	  conventions (ie. via legacy group accessed devices).
+ *	  Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the
+ *	  affected devices are owned by the user.  This flag is available only
+ *	  when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved.
+ *
  * Return: 0 on success, -errno on failure:
  *	-enospc = insufficient buffer, -enodev = unsupported for device.
  */
 struct vfio_pci_dependent_device {
-	__u32	group_id;
+	union {
+		__u32   group_id;
+		__u32	devid;
+#define VFIO_PCI_DEVID_OWNED		0
+#define VFIO_PCI_DEVID_NOT_OWNED	-1
+	};
 	__u16	segment;
 	__u8	bus;
 	__u8	devfn; /* Use PCI_SLOT/PCI_FUNC */
@@ -663,6 +705,8 @@ struct vfio_pci_dependent_device {
 struct vfio_pci_hot_reset_info {
 	__u32	argsz;
 	__u32	flags;
+#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID		(1 << 0)
+#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED	(1 << 1)
 	__u32	count;
 	struct vfio_pci_dependent_device	devices[];
 };
author
#11 lore raw

Re: [Intel-gfx] [PATCH v6 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev

Quoted 5 lines

From: Alex Williamson <[email protected]>
Sent: Thursday, May 25, 2023 3:56 AM
On Mon, 22 May 2023 04:57:50 -0700
Yi Liu <[email protected]> wrote:

Quoted 10 lines

drivers/vfio/iommufd.c void vfio_iommufd_unbind(struct vfio_device *vdev) · line 134

Quoted 8 lines
drivers/vfio/iommufd.c +8 -0
+
+/*
+ * Return devid for vfio_device if the device is owned by the input
+ * ictx.
+ * - valid devid > 0 for the device that are bound to the input
+ *   iommufd_ctx.
+ * - devid == VFIO_PCI_DEVID_OWNED for the devices that have not
+ *   been opened but but other device within its group has been

"but but"

Thanks for catching it.

Quoted 70 lines
Quoted 57 lines

+ * bound to the input iommufd_ctx.
+ * - devid == VFIO_PCI_DEVID_NOT_OWNED for others. e.g. vdev is
+ * NULL.
+ */

drivers/vfio/iommufd.c +25 -0
+int vfio_iommufd_device_hot_reset_devid(struct vfio_device *vdev,
+					struct iommufd_ctx *ictx)
+{
+	struct iommu_group *group;
+	int devid;
+
+	if (!vdev)
+		return VFIO_PCI_DEVID_NOT_OWNED;
+
+	if (vfio_iommufd_device_ictx(vdev) == ictx)
+		return vfio_iommufd_device_id(vdev);
+
+	group = iommu_group_get(vdev->dev);
+	if (!group)
+		return VFIO_PCI_DEVID_NOT_OWNED;
+
+	if (iommufd_ctx_has_group(ictx, group))
+		devid = VFIO_PCI_DEVID_OWNED;
+	else
+		devid = VFIO_PCI_DEVID_NOT_OWNED;
+
+	iommu_group_put(group);
+
+	return devid;
+}
include/uapi/linux/vfio.h +20 -0
@@ -650,11 +650,53 @@ enum {
  * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12,
  *					      struct vfio_pci_hot_reset_info)
  *
+ * This command is used to query the affected devices in the hot reset for
+ * a given device.
+ *
+ * This command always reports the segment, bus, and devfn information for
+ * each affected device, and selectively reports the group_id or devid per
+ * the way how the calling device is opened.
+ *
+ *	- If the calling device is opened via the traditional group/container
+ *	  API, group_id is reported.  User should check if it has owned all
+ *	  the affected devices and provides a set of group fds to prove the
+ *	  ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl.
+ *
+ *	- If the calling device is opened as a cdev, devid is reported.
+ *	  Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set to indicate this
+ *	  data type.  For a given affected device, it is considered owned by
+ *	  this interface if it meets the following conditions:
+ *	  1) Has a valid devid within the iommufd_ctx of the calling device.
+ *	     Ownership cannot be determined across separate iommufd_ctx and the
+ *	     cdev calling conventions do not support a proof-of-ownership model
+ *	     as provided in the legacy group interface.  In this case a valid

include/uapi/linux/vfio.h enum { · line 682

Quoted 10 lines
include/uapi/linux/vfio.h +10 -0
+ *	     devid with value greater than zero is provided in the return
+ *	     structure.
+ *	  2) Does not have a valid devid within the iommufd_ctx of the calling
+ *	     device, but belongs to the same IOMMU group as the calling device
+ *	     or another opened device that has a valid devid within the
+ *	     iommufd_ctx of the calling device.  This provides implicit ownership
+ *	     for devices within the same DMA isolation context.  In this case
+ *	     the invalid devid value of zero is provided in the return structure.
+ *
+ *	  A devid value of -1 is provided in the return structure for devices

s/zero/VFIO_PCI_DEVID_OWNED/

Quoted 1 line

s/-1/VFIO_PCI_DEVID_NOT_OWNED/

Will do.

Quoted 6 lines

2) above and previously in the code comment where I noted the repeated "but" still doesn't actually describe the requirement as I noted in the last review. The user implicitly owns a device if they own another device within the IOMMU group, but we also impose a dev_set requirement in the hot reset path. All affected devices need to be represented in the dev_set, ex. bound to a vfio driver.

Yes. it is. Btw. dev_set is not visible to user. Is it good to mention it in uapi header especially w.r.t. the below potential relaxing of this requirement?

Quoted 5 lines

It's possible that requirement
might be relaxed in the new DMA ownership model, but as it is right now, the code enforces that requirement and any new discussion about what makes hot-reset available should note both the ownership and dev_set requirement. Thanks,

I think your point is that if an iommufd_ctx has acquired DMA ownerhisp of an iommu_group, it means the device is owned. And it should not matter whether all the devices in the iommu_group is present in the dev_set. It is allowed that some devices are bound to pci-stub or pcieport driver. Is it?

Actually I have a doubt on it. IIUC, the above requirement on dev_set is to ensure the reset to the devices are protected by the dev_set->lock. So that either the reset issued by driver itself or a hot reset request from user, there is no race. But if a device is not in the dev_set, then hot reset request from user might race with the bound driver. DMA ownership only guarantees the drivers won't handle DMA via DMA API which would have conflict with DMA mappings from user. I'm not sure if it is able to guarantee reset is exclusive as well. I see pci-stub and pcieport driver are the only two drivers that set the driver_managed_dma flag besides the vfio drivers. pci-stub may be fine. not sure about pcieport driver.

# line filename / context / line
1 39 drivers/pci/pci-stub.c <<GLOBAL>>
.driver_managed_dma = true,
2 796 drivers/pci/pcie/portdrv.c <<GLOBAL>>
.driver_managed_dma = true,
3 607 drivers/vfio/fsl-mc/vfio_fsl_mc.c <<GLOBAL>>
.driver_managed_dma = true,
4 1459 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c <<GLOBAL>>
.driver_managed_dma = true,
5 1374 drivers/vfio/pci/mlx5/main.c <<GLOBAL>>
.driver_managed_dma = true,
6 203 drivers/vfio/pci/vfio_pci.c <<GLOBAL>>
.driver_managed_dma = true,
7 139 drivers/vfio/platform/vfio_amba.c <<GLOBAL>>
.driver_managed_dma = true,
8 120 drivers/vfio/platform/vfio_platform.c <<GLOBAL>>
.driver_managed_dma = true,

Anyhow, I think this is not a must so far. is it? Even doable, it shall be done in the future. :-)

Regards,
Yi Liu

#12 lore raw

Re: [Intel-gfx] [PATCH v6 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev

Quoted 11 lines · Yi L
Quoted 5 lines

It's possible that requirement
might be relaxed in the new DMA ownership model, but as it is right now, the code enforces that requirement and any new discussion about what makes hot-reset available should note both the ownership and dev_set requirement. Thanks,

I think your point is that if an iommufd_ctx has acquired DMA ownerhisp of an iommu_group, it means the device is owned. And it should not matter whether all the devices in the iommu_group is present in the dev_set. It is allowed that some devices are bound to pci-stub or pcieport driver. Is it?

Quoted 10 lines

Actually I have a doubt on it. IIUC, the above requirement on dev_set is to ensure the reset to the devices are protected by the dev_set->lock. So that either the reset issued by driver itself or a hot reset request from user, there is no race. But if a device is not in the dev_set, then hot reset request from user might race with the bound driver. DMA ownership only guarantees the drivers won't handle DMA via DMA API which would have conflict with DMA mappings from user. I'm not sure if it is able to guarantee reset is exclusive as well. I see pci-stub and pcieport driver are the only two drivers that set the driver_managed_dma flag besides the vfio drivers. pci-stub may be fine. not sure about pcieport driver.

commit c7d469849747 ("PCI: portdrv: Set driver_managed_dma") described the criteria of adding driver_managed_dma to the pcieport driver.

"
We achieve this by setting ".driver_managed_dma = true" in pci_driver structure. It is safe because the portdrv driver meets below criteria:

- This driver doesn't use DMA, as you can't find any related calls like pci_set_master() or any kernel DMA API (dma_map_*() and etc.).
- It doesn't use MMIO as you can't find ioremap() or similar calls. It's tolerant to userspace possibly also touching the same MMIO registers via P2P DMA access.
"

pci_rest_device() definitely shouldn't be done by the kernel drivers that have driver_managed_dma set.

Quoted 19 lines

# line filename / context / line
1 39 drivers/pci/pci-stub.c <<GLOBAL>>
.driver_managed_dma = true,
2 796 drivers/pci/pcie/portdrv.c <<GLOBAL>>
.driver_managed_dma = true,
3 607 drivers/vfio/fsl-mc/vfio_fsl_mc.c <<GLOBAL>>
.driver_managed_dma = true,
4 1459 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c <<GLOBAL>>
.driver_managed_dma = true,
5 1374 drivers/vfio/pci/mlx5/main.c <<GLOBAL>>
.driver_managed_dma = true,
6 203 drivers/vfio/pci/vfio_pci.c <<GLOBAL>>
.driver_managed_dma = true,
7 139 drivers/vfio/platform/vfio_amba.c <<GLOBAL>>
.driver_managed_dma = true,
8 120 drivers/vfio/platform/vfio_platform.c <<GLOBAL>>
.driver_managed_dma = true,

Quoted 2 lines

Anyhow, I think this is not a must so far. is it? Even doable, it shall be done in the future. 😄

Perhaps we can take it in this way: it's a bug if any driver sets its driver_managed_dma but still resets the hardware during it's life cycle?

Best regards,
baolu

A message that is not in the archive (sent off-list or not yet received): ZHeZPPo/[email protected]

author
#14 lore raw

Re: [Intel-gfx] [PATCH v6 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev

Quoted 44 lines

From: Jason Gunthorpe <[email protected]>
Sent: Thursday, June 1, 2023 3:00 AM

Quoted 37 lines · Baolu Lu
Quoted 11 lines · Yi L
Quoted 5 lines

It's possible that requirement
might be relaxed in the new DMA ownership model, but as it is right now, the code enforces that requirement and any new discussion about what makes hot-reset available should note both the ownership and dev_set requirement. Thanks,

I think your point is that if an iommufd_ctx has acquired DMA ownerhisp of an iommu_group, it means the device is owned. And it should not matter whether all the devices in the iommu_group is present in the dev_set. It is allowed that some devices are bound to pci-stub or pcieport driver. Is it?

Quoted 10 lines

Actually I have a doubt on it. IIUC, the above requirement on dev_set is to ensure the reset to the devices are protected by the dev_set->lock. So that either the reset issued by driver itself or a hot reset request from user, there is no race. But if a device is not in the dev_set, then hot reset request from user might race with the bound driver. DMA ownership only guarantees the drivers won't handle DMA via DMA API which would have conflict with DMA mappings from user. I'm not sure if it is able to guarantee reset is exclusive as well. I see pci-stub and pcieport driver are the only two drivers that set the driver_managed_dma flag besides the vfio drivers. pci-stub may be fine. not sure about pcieport driver.

commit c7d469849747 ("PCI: portdrv: Set driver_managed_dma") described the criteria of adding driver_managed_dma to the pcieport driver.

"
We achieve this by setting ".driver_managed_dma = true" in pci_driver structure. It is safe because the portdrv driver meets below criteria:

- This driver doesn't use DMA, as you can't find any related calls like pci_set_master() or any kernel DMA API (dma_map_*() and etc.).
- It doesn't use MMIO as you can't find ioremap() or similar calls. It's tolerant to userspace possibly also touching the same MMIO registers via P2P DMA access.
"

Quoted 2 lines

pci_rest_device() definitely shouldn't be done by the kernel drivers that have driver_managed_dma set.

Quoted 7 lines

Right

The only time it is safe to reset is if you know there is no attached driver or you know VFIO is the attached driver and the caller owns the VFIO too.

We haven't done a no attached driver test due to races.

Ok. @Alex, should we relax the above dev_set requirement now or should be in a separate series?

Regards,
Yi Liu

#15 lore raw

Re: [Intel-gfx] [PATCH v6 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev

Quoted 46 lines · Yi L
Quoted 46 lines

From: Jason Gunthorpe <[email protected]>
Sent: Thursday, June 1, 2023 3:00 AM

Quoted 37 lines · Baolu Lu
Quoted 11 lines · Yi L
Quoted 5 lines

It's possible that requirement
might be relaxed in the new DMA ownership model, but as it is right now, the code enforces that requirement and any new discussion about what makes hot-reset available should note both the ownership and dev_set requirement. Thanks,

I think your point is that if an iommufd_ctx has acquired DMA ownerhisp of an iommu_group, it means the device is owned. And it should not matter whether all the devices in the iommu_group is present in the dev_set. It is allowed that some devices are bound to pci-stub or pcieport driver. Is it?

Quoted 10 lines

Actually I have a doubt on it. IIUC, the above requirement on dev_set is to ensure the reset to the devices are protected by the dev_set->lock. So that either the reset issued by driver itself or a hot reset request from user, there is no race. But if a device is not in the dev_set, then hot reset request from user might race with the bound driver. DMA ownership only guarantees the drivers won't handle DMA via DMA API which would have conflict with DMA mappings from user. I'm not sure if it is able to guarantee reset is exclusive as well. I see pci-stub and pcieport driver are the only two drivers that set the driver_managed_dma flag besides the vfio drivers. pci-stub may be fine. not sure about pcieport driver.

commit c7d469849747 ("PCI: portdrv: Set driver_managed_dma") described the criteria of adding driver_managed_dma to the pcieport driver.

"
We achieve this by setting ".driver_managed_dma = true" in pci_driver structure. It is safe because the portdrv driver meets below criteria:

- This driver doesn't use DMA, as you can't find any related calls like pci_set_master() or any kernel DMA API (dma_map_*() and etc.).
- It doesn't use MMIO as you can't find ioremap() or similar calls. It's tolerant to userspace possibly also touching the same MMIO registers via P2P DMA access.
"

Quoted 2 lines

pci_rest_device() definitely shouldn't be done by the kernel drivers that have driver_managed_dma set.

Right

Quoted 8 lines
Quoted 5 lines

The only time it is safe to reset is if you know there is no attached driver or you know VFIO is the attached driver and the caller owns the VFIO too.

We haven't done a no attached driver test due to races.

Ok. @Alex, should we relax the above dev_set requirement now or should be in a separate series?

Sounds like no, you should be rejecting enhancements that increase scope at this point and I don't see consensus here. My concern was that we're not correctly describing the dev_set restriction which is already in place but needs to be more explicitly described in an implied ownership model vs proof of ownership model. Thanks,

Alex

author
#16 lore raw

PATCH v6 10/10 vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET

In reply to Yi Liu

This is the way user to invoke hot-reset for the devices opened by cdev interface. User should check the flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED in the output of VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl before doing hot-reset for cdev devices.

---
 drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++++++++-------
 include/uapi/linux/vfio.h        | 14 ++++++++
 2 files changed, 59 insertions(+), 11 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +45 -11
index 890065f846e4..67f1cb426505 100644
@@ -181,7 +181,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev)
 struct vfio_pci_group_info;
 static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set);
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups);
+				      struct vfio_pci_group_info *groups,
+				      struct iommufd_ctx *iommufd_ctx);
 
 /*
  * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND
@@ -1301,8 +1302,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
 	if (ret)
 		return ret;
 
-	/* Somewhere between 1 and count is OK */
-	if (!array_count || array_count > count)
+	if (array_count > count || vfio_device_cdev_opened(&vdev->vdev))
 		return -EINVAL;
 
 	group_fds = kcalloc(array_count, sizeof(*group_fds), GFP_KERNEL);
@@ -1351,7 +1351,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
 	info.count = array_count;
 	info.files = files;
 
-	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info);
+	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL);
 
 hot_reset_release:
 	for (file_idx--; file_idx >= 0; file_idx--)
@@ -1380,7 +1380,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	else if (pci_probe_reset_bus(vdev->pdev->bus))
 		return -ENODEV;
 
-	return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
+	if (hdr.count)
+		return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
+
+	return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL,
+					  vfio_iommufd_device_ictx(&vdev->vdev));
 }
 
 static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev,
@@ -2347,13 +2351,16 @@ const struct pci_error_handlers vfio_pci_core_err_handlers = {
 };
 EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers);
 
-static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev,
+static bool vfio_dev_in_groups(struct vfio_device *vdev,
 			       struct vfio_pci_group_info *groups)
 {
 	unsigned int i;
 
+	if (!groups)
+		return false;
+
 	for (i = 0; i < groups->count; i++)
-		if (vfio_file_has_dev(groups->files[i], &vdev->vdev))
+		if (vfio_file_has_dev(groups->files[i], vdev))
 			return true;
 	return false;
 }
@@ -2429,7 +2436,8 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set)
  * get each memory_lock.
  */
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups)
+				      struct vfio_pci_group_info *groups,
+				      struct iommufd_ctx *iommufd_ctx)
 {
 	struct vfio_pci_core_device *cur_mem;
 	struct vfio_pci_core_device *cur_vma;
@@ -2459,11 +2467,37 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 		goto err_unlock;
 
 	list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) {
+		bool owned;
+
 		/*
-		 * Test whether all the affected devices are contained by the
-		 * set of groups provided by the user.
+		 * Test whether all the affected devices can be reset by the
+		 * user.
+		 *
+		 * If the user provides a set of groups, all the devices
+		 * in the dev_set should be contained by the set of groups
+		 * provided by the user.
+		 *
+		 * If the user provides a zero-length group fd array, then
+		 * all the devices in the dev_set must be bound to the same
+		 * iommufd_ctx as the input iommufd_ctx.  If there is any
+		 * device that has not been bound to iommufd_ctx yet, check
+		 * if its iommu_group has any device bound to the input
+		 * iommufd_ctx Such devices can be considered owned by
+		 * the input iommufd_ctx as the device cannot be owned
+		 * by another iommufd_ctx when its iommu_group is owned.
+		 *
+		 * Otherwise, reset is not allowed.
 		 */
-		if (!vfio_dev_in_groups(cur_vma, groups)) {
+		if (iommufd_ctx) {
+			int devid = vfio_iommufd_device_hot_reset_devid(&cur_vma->vdev,
+									iommufd_ctx);
+
+			owned = (devid != VFIO_PCI_DEVID_NOT_OWNED);
+		} else {
+			owned = vfio_dev_in_groups(&cur_vma->vdev, groups);
+		}
+
+		if (!owned) {
 			ret = -EINVAL;
 			goto err_undo;
 		}
include/uapi/linux/vfio.h +14 -0
index 01203215251a..24858b650562 100644
@@ -686,6 +686,9 @@ enum {
  *	  Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the
  *	  affected devices are owned by the user.  This flag is available only
  *	  when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved.
+ *	  When set, user could invoke VFIO_DEVICE_PCI_HOT_RESET with a zero
+ *	  length fd array on the calling device as the ownership is validated
+ *	  by iommufd_ctx.
  *
  * Return: 0 on success, -errno on failure:
  *	-enospc = insufficient buffer, -enodev = unsupported for device.
@@ -717,6 +720,17 @@ struct vfio_pci_hot_reset_info {
  * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13,
  *				    struct vfio_pci_hot_reset)
  *
+ * Userspace requests hot reset for the devices it operates.  Due to the
+ * underlying topology, multiple devices can be affected in the reset
+ * while some might be opened by another user.  To avoid interference
+ * the calling user must ensure all affected devices are owned by itself.
+ *
+ * As the ownership described by VFIO_DEVICE_GET_PCI_HOT_RESET_INFO, the
+ * cdev opened devices must exclusively provide a zero-length fd array and
+ * the group opened devices must exclusively use an array of group fds for
+ * proof of ownership.  Mixed access to devices between cdev and legacy
+ * groups are not supported by this interface.
+ *
  * Return: 0 on success, -errno on failure.
  */
 struct vfio_pci_hot_reset {
Signature · 1 line
2.34.1
#17 lore raw

Re: [Intel-gfx] [PATCH v6 10/10] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET

Quoted 95 lines · On Mon, 22 May 2023 04:57:51 -0700 Yi Liu <[email protected]> wrote:

This is the way user to invoke hot-reset for the devices opened by cdev interface. User should check the flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED in the output of VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl before doing hot-reset for cdev devices.

---
 drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++++++++-------
 include/uapi/linux/vfio.h        | 14 ++++++++
 2 files changed, 59 insertions(+), 11 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +17 -8
index 890065f846e4..67f1cb426505 100644
@@ -181,7 +181,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev)
 struct vfio_pci_group_info;
 static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set);
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups);
+				      struct vfio_pci_group_info *groups,
+				      struct iommufd_ctx *iommufd_ctx);
 
 /*
  * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND
@@ -1301,8 +1302,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
 	if (ret)
 		return ret;
 
-	/* Somewhere between 1 and count is OK */
-	if (!array_count || array_count > count)
+	if (array_count > count || vfio_device_cdev_opened(&vdev->vdev))
 		return -EINVAL;
 
 	group_fds = kcalloc(array_count, sizeof(*group_fds), GFP_KERNEL);
@@ -1351,7 +1351,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
 	info.count = array_count;
 	info.files = files;
 
-	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info);
+	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL);
 
 hot_reset_release:
 	for (file_idx--; file_idx >= 0; file_idx--)
@@ -1380,7 +1380,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	else if (pci_probe_reset_bus(vdev->pdev->bus))
 		return -ENODEV;
 
-	return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
+	if (hdr.count)
+		return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
+
+	return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL,
+					  vfio_iommufd_device_ictx(&vdev->vdev));
 }
 
 static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev,
@@ -2347,13 +2351,16 @@ const struct pci_error_handlers vfio_pci_core_err_handlers = {
 };
 EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers);
 
-static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev,
+static bool vfio_dev_in_groups(struct vfio_device *vdev,
 			       struct vfio_pci_group_info *groups)
 {
 	unsigned int i;
 
+	if (!groups)
+		return false;
+
 	for (i = 0; i < groups->count; i++)
-		if (vfio_file_has_dev(groups->files[i], &vdev->vdev))
+		if (vfio_file_has_dev(groups->files[i], vdev))
 			return true;
 	return false;
 }
@@ -2429,7 +2436,8 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set)
  * get each memory_lock.
  */
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups)
+				      struct vfio_pci_group_info *groups,
+				      struct iommufd_ctx *iommufd_ctx)
 {
 	struct vfio_pci_core_device *cur_mem;
 	struct vfio_pci_core_device *cur_vma;
@@ -2459,11 +2467,37 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 		goto err_unlock;
 
 	list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) {
+		bool owned;

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2478

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +7 -2
+
 		/*
-		 * Test whether all the affected devices are contained by the
-		 * set of groups provided by the user.
+		 * Test whether all the affected devices can be reset by the
+		 * user.
+		 *
+		 * If the user provides a set of groups, all the devices
+		 * in the dev_set should be contained by the set of groups
+		 * provided by the user.

"If called from a group opened device and the user provides a set of groups,..."

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2480

Quoted 2 lines
drivers/vfio/pci/vfio_pci_core.c +2 -0
+		 *
+		 * If the user provides a zero-length group fd array, then

"If called from a cdev opened device and the user provides a zero-length array,..."

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2485

Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +5 -0
+		 * all the devices in the dev_set must be bound to the same
+		 * iommufd_ctx as the input iommufd_ctx.  If there is any
+		 * device that has not been bound to iommufd_ctx yet, check
+		 * if its iommu_group has any device bound to the input
+		 * iommufd_ctx Such devices can be considered owned by

"."...........................^

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2489

Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +4 -0
+		 * the input iommufd_ctx as the device cannot be owned
+		 * by another iommufd_ctx when its iommu_group is owned.
+		 *
+		 * Otherwise, reset is not allowed.

In the case where a non-null array is provided, vfio_pci_ioctl_pci_hot_reset_groups() explicitly tests vfio_device_cdev_opened(), so we exclude cdev devices from providing a group list. However, what prevents a compat opened group device from providing a null array?

I thought it would be that this function is called with groups == NULL and therefore the vfio_dev_in_groups() test below fails, but I don't think that's true for a compat opened device. Thanks,

Alex

Quoted 47 lines
drivers/vfio/pci/vfio_pci_core.c +10 -1
 		 */
-		if (!vfio_dev_in_groups(cur_vma, groups)) {
+		if (iommufd_ctx) {
+			int devid = vfio_iommufd_device_hot_reset_devid(&cur_vma->vdev,
+									iommufd_ctx);
+
+			owned = (devid != VFIO_PCI_DEVID_NOT_OWNED);
+		} else {
+			owned = vfio_dev_in_groups(&cur_vma->vdev, groups);
+		}
+
+		if (!owned) {
 			ret = -EINVAL;
 			goto err_undo;
 		}
include/uapi/linux/vfio.h +14 -0
index 01203215251a..24858b650562 100644
@@ -686,6 +686,9 @@ enum {
  *	  Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the
  *	  affected devices are owned by the user.  This flag is available only
  *	  when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved.
+ *	  When set, user could invoke VFIO_DEVICE_PCI_HOT_RESET with a zero
+ *	  length fd array on the calling device as the ownership is validated
+ *	  by iommufd_ctx.
  *
  * Return: 0 on success, -errno on failure:
  *	-enospc = insufficient buffer, -enodev = unsupported for device.
@@ -717,6 +720,17 @@ struct vfio_pci_hot_reset_info {
  * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13,
  *				    struct vfio_pci_hot_reset)
  *
+ * Userspace requests hot reset for the devices it operates.  Due to the
+ * underlying topology, multiple devices can be affected in the reset
+ * while some might be opened by another user.  To avoid interference
+ * the calling user must ensure all affected devices are owned by itself.
+ *
+ * As the ownership described by VFIO_DEVICE_GET_PCI_HOT_RESET_INFO, the
+ * cdev opened devices must exclusively provide a zero-length fd array and
+ * the group opened devices must exclusively use an array of group fds for
+ * proof of ownership.  Mixed access to devices between cdev and legacy
+ * groups are not supported by this interface.
+ *
  * Return: 0 on success, -errno on failure.
  */
 struct vfio_pci_hot_reset {
author
#18 lore raw

Re: [Intel-gfx] [PATCH v6 10/10] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET

Quoted 129 lines

From: Alex Williamson <[email protected]>
Sent: Thursday, May 25, 2023 4:20 AM

drivers/vfio/pci/vfio_pci_core.c static void vfio_pci_probe_mmaps(struct vfio_pci_core_device

Quoted 20 lines · On Mon, 22 May 2023 04:57:51 -0700 Yi Liu <[email protected]> wrote:

This is the way user to invoke hot-reset for the devices opened by cdev interface. User should check the flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED in the output of VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl before doing hot-reset for cdev devices.

---
 drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++++++++-------
 include/uapi/linux/vfio.h        | 14 ++++++++
 2 files changed, 59 insertions(+), 11 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +0 -0
index 890065f846e4..67f1cb426505 100644
@@ -181,7 +181,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device

*vdev)

drivers/vfio/pci/vfio_pci_core.c vfio_pci_ioctl_pci_hot_reset_groups(struct

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +2 -1
 struct vfio_pci_group_info;
 static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set);
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups);
+				      struct vfio_pci_group_info *groups,
+				      struct iommufd_ctx *iommufd_ctx);
 
 /*
  * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND
@@ -1301,8 +1302,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct

vfio_pci_core_device *vdev,

drivers/vfio/pci/vfio_pci_core.c vfio_pci_ioctl_pci_hot_reset_groups(struct

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +1 -2
 	if (ret)
 		return ret;
 
-	/* Somewhere between 1 and count is OK */
-	if (!array_count || array_count > count)
+	if (array_count > count || vfio_device_cdev_opened(&vdev->vdev))
 		return -EINVAL;
 
 	group_fds = kcalloc(array_count, sizeof(*group_fds), GFP_KERNEL);
@@ -1351,7 +1351,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct

vfio_pci_core_device *vdev,

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_pci_hot_reset(struct

Quoted 9 lines
drivers/vfio/pci/vfio_pci_core.c +1 -1
 	info.count = array_count;
 	info.files = files;
 
-	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info);
+	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL);
 
 hot_reset_release:
 	for (file_idx--; file_idx >= 0; file_idx--)
@@ -1380,7 +1380,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct

vfio_pci_core_device *vdev,

drivers/vfio/pci/vfio_pci_core.c const struct pci_error_handlers

Quoted 13 lines
drivers/vfio/pci/vfio_pci_core.c +5 -1
 	else if (pci_probe_reset_bus(vdev->pdev->bus))
 		return -ENODEV;
 
-	return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
+	if (hdr.count)
+		return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
+
+	return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL,
+					  vfio_iommufd_device_ictx(&vdev->vdev));
 }
 
 static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev,
@@ -2347,13 +2351,16 @@ const struct pci_error_handlers

vfio_pci_core_err_handlers = {

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_pm_runtime_get(struct

Quoted 19 lines
drivers/vfio/pci/vfio_pci_core.c +5 -2
 };
 EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers);
 
-static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev,
+static bool vfio_dev_in_groups(struct vfio_device *vdev,
 			       struct vfio_pci_group_info *groups)
 {
 	unsigned int i;
 
+	if (!groups)
+		return false;
+
 	for (i = 0; i < groups->count; i++)
-		if (vfio_file_has_dev(groups->files[i], &vdev->vdev))
+		if (vfio_file_has_dev(groups->files[i], vdev))
 			return true;
 	return false;
 }
@@ -2429,7 +2436,8 @@ static int vfio_pci_dev_set_pm_runtime_get(struct

vfio_device_set *dev_set)

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +2 -1
  * get each memory_lock.
  */
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups)
+				      struct vfio_pci_group_info *groups,
+				      struct iommufd_ctx *iommufd_ctx)
 {
 	struct vfio_pci_core_device *cur_mem;
 	struct vfio_pci_core_device *cur_vma;
@@ -2459,11 +2467,37 @@ static int vfio_pci_dev_set_hot_reset(struct

vfio_device_set *dev_set,

Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +1 -0
 		goto err_unlock;
 
 	list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) {
+		bool owned;

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2478

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +7 -2
+
 		/*
-		 * Test whether all the affected devices are contained by the
-		 * set of groups provided by the user.
+		 * Test whether all the affected devices can be reset by the
+		 * user.
+		 *
+		 * If the user provides a set of groups, all the devices
+		 * in the dev_set should be contained by the set of groups
+		 * provided by the user.

"If called from a group opened device and the user provides a set of groups,..."

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct · line 2480

Quoted 2 lines
drivers/vfio/pci/vfio_pci_core.c +2 -0
+		 *
+		 * If the user provides a zero-length group fd array, then

"If called from a cdev opened device and the user provides a zero-length array,..."

Quoted 7 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2485

Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +5 -0
+		 * all the devices in the dev_set must be bound to the same
+		 * iommufd_ctx as the input iommufd_ctx.  If there is any
+		 * device that has not been bound to iommufd_ctx yet, check
+		 * if its iommu_group has any device bound to the input
+		 * iommufd_ctx Such devices can be considered owned by

"."...........................^

Thanks, above comments well received.

Quoted 10 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2489

Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +4 -0
+		 * the input iommufd_ctx as the device cannot be owned
+		 * by another iommufd_ctx when its iommu_group is owned.
+		 *
+		 * Otherwise, reset is not allowed.

In the case where a non-null array is provided, vfio_pci_ioctl_pci_hot_reset_groups() explicitly tests vfio_device_cdev_opened(), so we exclude cdev devices from providing a group list.

Yes.

Quoted 2 lines

However, what prevents a compat opened group device from providing a null array?

I think I've asked this question before. What I got is seems no need to prevent it[1].

[1] https://lore.kernel.org/kvm/BN9PR11MB5276ABE919C04E06A0326E8E8CBC9@BN9PR11MB5276.namprd11.prod.outlook.com/

Now, I intend to disallow it. If compat mode user binds the devices to different containers, it shall be able to do hot reset as it can use group fd to prove ownership. But if using the zero-length array, it would be failed. So we may add below logic and remove the vfio_device_cdev_opened() in vfio_pci_ioctl_pci_hot_reset_groups().

vfio_pci_ioctl_pci_hot_reset()
{
...

	if (!!hdr.count == !!vfio_device_cdev_opened(&vdev->vdev))
		return -EINVAL;
	if (hdr.count)
		vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
	return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL,
					     vfio_iommufd_device_ictx(&vdev->vdev));
}
Quoted 4 lines

I thought it would be that this function is called with groups == NULL and therefore the vfio_dev_in_groups() test below fails, but I don't think that's true for a compat opened device. Thanks,

How about above logic?

Regards,
Yi Liu

Quoted 49 lines
Quoted 49 lines
drivers/vfio/pci/vfio_pci_core.c +10 -1
 		 */
-		if (!vfio_dev_in_groups(cur_vma, groups)) {
+		if (iommufd_ctx) {
+			int devid = vfio_iommufd_device_hot_reset_devid(&cur_vma-
 vdev,
+									iommufd_ctx);
+
+			owned = (devid != VFIO_PCI_DEVID_NOT_OWNED);
+		} else {
+			owned = vfio_dev_in_groups(&cur_vma->vdev, groups);
+		}
+
+		if (!owned) {
 			ret = -EINVAL;
 			goto err_undo;
drivers/vfio/pci/vfio_pci_core.c +0 -0
 		}
include/uapi/linux/vfio.h +14 -0
index 01203215251a..24858b650562 100644
@@ -686,6 +686,9 @@ enum {
  *	  Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the
  *	  affected devices are owned by the user.  This flag is available only
  *	  when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved.
+ *	  When set, user could invoke VFIO_DEVICE_PCI_HOT_RESET with a zero
+ *	  length fd array on the calling device as the ownership is validated
+ *	  by iommufd_ctx.
  *
  * Return: 0 on success, -errno on failure:
  *	-enospc = insufficient buffer, -enodev = unsupported for device.
@@ -717,6 +720,17 @@ struct vfio_pci_hot_reset_info {
  * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13,
  *				    struct vfio_pci_hot_reset)
  *
+ * Userspace requests hot reset for the devices it operates.  Due to the
+ * underlying topology, multiple devices can be affected in the reset
+ * while some might be opened by another user.  To avoid interference
+ * the calling user must ensure all affected devices are owned by itself.
+ *
+ * As the ownership described by VFIO_DEVICE_GET_PCI_HOT_RESET_INFO, the
+ * cdev opened devices must exclusively provide a zero-length fd array and
+ * the group opened devices must exclusively use an array of group fds for
+ * proof of ownership.  Mixed access to devices between cdev and legacy
+ * groups are not supported by this interface.
+ *
  * Return: 0 on success, -errno on failure.
  */
 struct vfio_pci_hot_reset {
#19 lore raw

Re: [Intel-gfx] [PATCH v6 10/10] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET

Quoted 178 lines · Yi L
Quoted 129 lines

From: Alex Williamson <[email protected]>
Sent: Thursday, May 25, 2023 4:20 AM

drivers/vfio/pci/vfio_pci_core.c static void vfio_pci_probe_mmaps(struct vfio_pci_core_device

Quoted 20 lines · On Mon, 22 May 2023 04:57:51 -0700 Yi Liu <[email protected]> wrote:

This is the way user to invoke hot-reset for the devices opened by cdev interface. User should check the flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED in the output of VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl before doing hot-reset for cdev devices.

---
 drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++++++++-------
 include/uapi/linux/vfio.h        | 14 ++++++++
 2 files changed, 59 insertions(+), 11 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +0 -0
index 890065f846e4..67f1cb426505 100644
@@ -181,7 +181,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device  

*vdev)

drivers/vfio/pci/vfio_pci_core.c vfio_pci_ioctl_pci_hot_reset_groups(struct

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +2 -1
 struct vfio_pci_group_info;
 static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set);
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups);
+				      struct vfio_pci_group_info *groups,
+				      struct iommufd_ctx *iommufd_ctx);
 
 /*
  * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND
@@ -1301,8 +1302,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct  

vfio_pci_core_device *vdev,

drivers/vfio/pci/vfio_pci_core.c vfio_pci_ioctl_pci_hot_reset_groups(struct

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +1 -2
 	if (ret)
 		return ret;
 
-	/* Somewhere between 1 and count is OK */
-	if (!array_count || array_count > count)
+	if (array_count > count || vfio_device_cdev_opened(&vdev->vdev))
 		return -EINVAL;
 
 	group_fds = kcalloc(array_count, sizeof(*group_fds), GFP_KERNEL);
@@ -1351,7 +1351,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct  

vfio_pci_core_device *vdev,

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_pci_hot_reset(struct

Quoted 9 lines
drivers/vfio/pci/vfio_pci_core.c +1 -1
 	info.count = array_count;
 	info.files = files;
 
-	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info);
+	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL);
 
 hot_reset_release:
 	for (file_idx--; file_idx >= 0; file_idx--)
@@ -1380,7 +1380,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct  

vfio_pci_core_device *vdev,

drivers/vfio/pci/vfio_pci_core.c const struct pci_error_handlers

Quoted 13 lines
drivers/vfio/pci/vfio_pci_core.c +5 -1
 	else if (pci_probe_reset_bus(vdev->pdev->bus))
 		return -ENODEV;
 
-	return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
+	if (hdr.count)
+		return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
+
+	return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL,
+					  vfio_iommufd_device_ictx(&vdev->vdev));
 }
 
 static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev,
@@ -2347,13 +2351,16 @@ const struct pci_error_handlers  

vfio_pci_core_err_handlers = {

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_pm_runtime_get(struct

Quoted 19 lines
drivers/vfio/pci/vfio_pci_core.c +5 -2
 };
 EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers);
 
-static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev,
+static bool vfio_dev_in_groups(struct vfio_device *vdev,
 			       struct vfio_pci_group_info *groups)
 {
 	unsigned int i;
 
+	if (!groups)
+		return false;
+
 	for (i = 0; i < groups->count; i++)
-		if (vfio_file_has_dev(groups->files[i], &vdev->vdev))
+		if (vfio_file_has_dev(groups->files[i], vdev))
 			return true;
 	return false;
 }
@@ -2429,7 +2436,8 @@ static int vfio_pci_dev_set_pm_runtime_get(struct  

vfio_device_set *dev_set)

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +2 -1
  * get each memory_lock.
  */
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups)
+				      struct vfio_pci_group_info *groups,
+				      struct iommufd_ctx *iommufd_ctx)
 {
 	struct vfio_pci_core_device *cur_mem;
 	struct vfio_pci_core_device *cur_vma;
@@ -2459,11 +2467,37 @@ static int vfio_pci_dev_set_hot_reset(struct  

vfio_device_set *dev_set,

Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +1 -0
 		goto err_unlock;
 
 	list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) {
+		bool owned;

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2478

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +7 -2
+
 		/*
-		 * Test whether all the affected devices are contained by the
-		 * set of groups provided by the user.
+		 * Test whether all the affected devices can be reset by the
+		 * user.
+		 *
+		 * If the user provides a set of groups, all the devices
+		 * in the dev_set should be contained by the set of groups
+		 * provided by the user.  

"If called from a group opened device and the user provides a set of groups,..."

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct · line 2480

Quoted 2 lines
drivers/vfio/pci/vfio_pci_core.c +2 -0
+		 *
+		 * If the user provides a zero-length group fd array, then  

"If called from a cdev opened device and the user provides a zero-length array,..."

Quoted 7 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2485

Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +5 -0
+		 * all the devices in the dev_set must be bound to the same
+		 * iommufd_ctx as the input iommufd_ctx.  If there is any
+		 * device that has not been bound to iommufd_ctx yet, check
+		 * if its iommu_group has any device bound to the input
+		 * iommufd_ctx Such devices can be considered owned by  

"."...........................^

Thanks, above comments well received.

Quoted 10 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2489

Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +4 -0
+		 * the input iommufd_ctx as the device cannot be owned
+		 * by another iommufd_ctx when its iommu_group is owned.
+		 *
+		 * Otherwise, reset is not allowed.  

In the case where a non-null array is provided, vfio_pci_ioctl_pci_hot_reset_groups() explicitly tests vfio_device_cdev_opened(), so we exclude cdev devices from providing a group list.

Yes.

Quoted 2 lines

However, what prevents a compat opened group device from providing a null array?

I think I've asked this question before. What I got is seems no need to prevent it[1].

[1] https://lore.kernel.org/kvm/BN9PR11MB5276ABE919C04E06A0326E8E8CBC9@BN9PR11MB5276.namprd11.prod.outlook.com/

Now, I intend to disallow it. If compat mode user binds the devices to different containers, it shall be able to do hot reset as it can use group fd to prove ownership. But if using the zero-length array, it would be failed. So we may add below logic and remove the vfio_device_cdev_opened() in vfio_pci_ioctl_pci_hot_reset_groups().

vfio_pci_ioctl_pci_hot_reset()
{
...

	if (!!hdr.count == !!vfio_device_cdev_opened(&vdev->vdev))
		return -EINVAL;
	if (hdr.count)
		vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
	return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL,
					     vfio_iommufd_device_ictx(&vdev->vdev));
}
Quoted 1 line
Quoted 5 lines
Quoted 3 lines

I thought it would be that this function is called with groups == NULL and therefore the vfio_dev_in_groups() test below fails, but I don't think that's true for a compat opened device. Thanks,

How about above logic?

The double negating a function that already returns bool is a bit excessive. I might also move the test closer to the other parameter checking with a comment noting the null array interface is only for cdev opened devices. Thanks,

Alex

author
#20 lore raw

Re: [Intel-gfx] [PATCH v6 10/10] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET

Quoted 21 lines

From: Alex Williamson <[email protected]>
Sent: Thursday, June 1, 2023 1:22 AM

Quoted 18 lines

Now, I intend to disallow it. If compat mode user binds the devices to different containers, it shall be able to do hot reset as it can use group fd to prove ownership. But if using the zero-length array, it would be failed. So we may add below logic and remove the vfio_device_cdev_opened() in vfio_pci_ioctl_pci_hot_reset_groups().

vfio_pci_ioctl_pci_hot_reset()
{
...

	if (!!hdr.count == !!vfio_device_cdev_opened(&vdev->vdev))
		return -EINVAL;
	if (hdr.count)
		vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg);
	return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL,
					     vfio_iommufd_device_ictx(&vdev->vdev));
}
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

I thought it would be that this function is called with groups == NULL and therefore the vfio_dev_in_groups() test below fails, but I don't think that's true for a compat opened device. Thanks,

How about above logic?

The double negating a function that already returns bool is a bit

Yes.

Quoted 3 lines

excessive. I might also move the test closer to the other parameter checking with a comment noting the null array interface is only for cdev opened devices. Thanks,

Yes.

Regards,
Yi Liu

#22 lore raw

[Intel-gfx] ✗ Fi.CI.BAT: failure for Enhance vfio PCI hot reset for vfio cdev device (rev4)

In reply to Yi Liu
== Series Details ==

== Series Details ==

Series: Enhance vfio PCI hot reset for vfio cdev device (rev4)
URL : https://patchwork.freedesktop.org/series/116991/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_13174 -> Patchwork_116991v4
====================================================

Summary
-------

**FAILURE**

Serious unknown changes coming with Patchwork_116991v4 absolutely need to be verified manually.

If you think the reported changes have nothing to do with the changes introduced in Patchwork_116991v4, please notify your bug team to allow them to document this new failure mode, which will reduce false positives in CI.

External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/index.html

Participating hosts (39 -> 39)
------------------------------

Additional (1): fi-kbl-soraka
Missing (1): fi-snb-2520m

Possible new issues
-------------------

Here are the unknown changes that may have been introduced in Patchwork_116991v4:

### IGT changes ###

#### Possible regressions ####

* igt@i915_selftest@live@gt_heartbeat:
- fi-kbl-soraka: NOTRUN -> [DMESG-FAIL][1]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/fi-kbl-soraka/igt@i915_selftest@live@gt_heartbeat.html

Known issues
------------

Here are the changes found in Patchwork_116991v4 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@gem_huc_copy@huc-copy:
- fi-kbl-soraka: NOTRUN -> [SKIP][2] ([fdo#109271] / [i915#2190])
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/fi-kbl-soraka/igt@[email protected]

* igt@gem_lmem_swapping@basic:
- fi-kbl-soraka: NOTRUN -> [SKIP][3] ([fdo#109271] / [i915#4613]) +3 similar issues
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/fi-kbl-soraka/igt@[email protected]

* igt@i915_selftest@live@gt_pm:
- bat-rpls-2: [PASS][4] -> [DMESG-FAIL][5] ([i915#4258] / [i915#7913])
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13174/bat-rpls-2/igt@i915_selftest@live@gt_pm.html
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/bat-rpls-2/igt@i915_selftest@live@gt_pm.html
- fi-kbl-soraka: NOTRUN -> [ABORT][6] ([i915#7913])
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/fi-kbl-soraka/igt@i915_selftest@live@gt_pm.html

* igt@i915_selftest@live@migrate:
- bat-dg2-11: [PASS][7] -> [DMESG-WARN][8] ([i915#7699])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13174/bat-dg2-11/igt@i915_selftest@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/bat-dg2-11/igt@i915_selftest@[email protected]

* igt@kms_chamelium_frames@hdmi-crc-fast:
- fi-kbl-soraka: NOTRUN -> [SKIP][9] ([fdo#109271]) +14 similar issues
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/fi-kbl-soraka/igt@[email protected]

* igt@kms_flip@basic-flip-vs-wf_vblank@c-hdmi-a2:
- fi-bsw-n3050: [PASS][10] -> [FAIL][11] ([i915#2122])
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13174/fi-bsw-n3050/igt@kms_flip@[email protected]
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/fi-bsw-n3050/igt@kms_flip@[email protected]

* igt@kms_pipe_crc_basic@nonblocking-crc@pipe-d-dp-1:
- bat-dg2-8: [PASS][12] -> [FAIL][13] ([i915#7932])
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13174/bat-dg2-8/igt@kms_pipe_crc_basic@[email protected]
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/bat-dg2-8/igt@kms_pipe_crc_basic@[email protected]

* igt@kms_setmode@basic-clone-single-crtc:
- fi-kbl-soraka: NOTRUN -> [SKIP][14] ([fdo#109271] / [i915#4579])
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/fi-kbl-soraka/igt@[email protected]

#### Possible fixes ####

* igt@i915_selftest@live@slpc:
- {bat-mtlp-6}: [DMESG-WARN][15] ([i915#6367]) -> [PASS][16]
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13174/bat-mtlp-6/igt@i915_selftest@[email protected]
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/bat-mtlp-6/igt@i915_selftest@[email protected]

#### Warnings ####

* igt@i915_selftest@live@reset:
- bat-rpls-2: [INCOMPLETE][17] ([i915#7913] / [i915#8347]) -> [ABORT][18] ([i915#4983] / [i915#7461] / [i915#7913] / [i915#8347])
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13174/bat-rpls-2/igt@i915_selftest@[email protected]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/bat-rpls-2/igt@i915_selftest@[email protected]

{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).

[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[i915#2122]: https://gitlab.freedesktop.org/drm/intel/issues/2122
[i915#2190]: https://gitlab.freedesktop.org/drm/intel/issues/2190
[i915#4258]: https://gitlab.freedesktop.org/drm/intel/issues/4258
[i915#4579]: https://gitlab.freedesktop.org/drm/intel/issues/4579
[i915#4613]: https://gitlab.freedesktop.org/drm/intel/issues/4613
[i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
[i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
[i915#7059]: https://gitlab.freedesktop.org/drm/intel/issues/7059
[i915#7461]: https://gitlab.freedesktop.org/drm/intel/issues/7461
[i915#7699]: https://gitlab.freedesktop.org/drm/intel/issues/7699
[i915#7913]: https://gitlab.freedesktop.org/drm/intel/issues/7913
[i915#7932]: https://gitlab.freedesktop.org/drm/intel/issues/7932
[i915#8347]: https://gitlab.freedesktop.org/drm/intel/issues/8347

Build changes
-------------

* Linux: CI_DRM_13174 -> Patchwork_116991v4

CI-20190529: 20190529
CI_DRM_13174: 00e64f04fac388222f3a8407848e185b3ade4256 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_7299: 3effd4be7f6c867d942532b3fe18d6c54fffbd7a @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
Patchwork_116991v4: 00e64f04fac388222f3a8407848e185b3ade4256 @ git://anongit.freedesktop.org/gfx-ci/linux

### Linux commits

7ddb11bb68b7 vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET dcc344d83e21 vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev 28a11abb03dc vfio: Add helper to search vfio_device in a dev_set a3a17ffef69b vfio: Mark cdev usage in vfio_device
be7e683f4d6b iommufd: Add helper to retrieve iommufd_ctx and devid dc41747520d8 iommufd: Add iommufd_ctx_has_group()
cb6176f9bddd iommufd: Reserve all negative IDs in the iommufd xarray 706be336c5a7 vfio/pci: Move the existing hot reset logic to be a helper 111500027f46 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() 9c360a535264 vfio-iommufd: Create iommufd_access for noiommu devices

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v4/index.html