Skip to contentSkip to conversation

Introduce new methods for verifying ownership in vfio PCI hot reset

PATCH Superseded Yi Liu 7 participants 61 messages Mar 27, 2023 – Apr 3, 2023

Subsystems (5)
KVM s390 IOMMU VFIO driver +1 more subsystems

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Yi Liu

author
#0 lore raw

v2 cover Introduce new methods for verifying ownership in vfio PCI hot reset

VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.

First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]

When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]

However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton dev_set hence no ownership check is required. [3]

For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]

As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new _INFO ioctl to get hot reset scope for given device.

[1] https://lore.kernel.org/kvm/Y%[email protected]/
[2] https://lore.kernel.org/kvm/Y%[email protected]/#t
[3] https://lore.kernel.org/kvm/ZACX+Np%[email protected]/
[4] https://lore.kernel.org/kvm/DS0PR11MB7529BE88460582BD599DC1F7C3B19@DS0PR11MB7529.namprd11.prod.outlook.com/#t
[5] https://lore.kernel.org/kvm/[email protected]/
[6] https://lore.kernel.org/kvm/[email protected]/

Change log:

v2:
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new _INFO ioctl for the usage of device fd passing usage in cdev path (Jason, Alex)

v1: https://lore.kernel.org/kvm/[email protected]/

Regards,
Yi Liu

Yi Liu (10):
vfio/pci: Update comment around group_fd get in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Only check ownership of opened devices in hot reset vfio/pci: Move the existing hot reset logic to be a helper
vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for
vfio_device
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
vfio: Refine vfio file kAPIs for vfio PCI hot reset
vfio: Accpet device file from vfio PCI hot reset path vfio/pci: Renaming for accepting device fd in hot reset path vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

 drivers/iommu/iommufd/device.c   |  12 ++
 drivers/vfio/group.c             |  32 ++--
 drivers/vfio/iommufd.c           |  16 ++
 drivers/vfio/pci/vfio_pci_core.c | 244 ++++++++++++++++++++++++-------
 drivers/vfio/vfio.h              |   2 +
 drivers/vfio/vfio_main.c         |  44 ++++++
 include/linux/iommufd.h          |   3 +
 include/linux/vfio.h             |  14 ++
 include/uapi/linux/vfio.h        |  65 +++++++-
 9 files changed, 364 insertions(+), 68 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v2 1/10 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset()

this suits more on what the code does.

---
 drivers/vfio/pci/vfio_pci_core.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +2 -3
index a5ab416cf476..65bbef562268 100644
@@ -1308,9 +1308,8 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	}
 
 	/*
-	 * For each group_fd, get the group through the vfio external user
-	 * interface and store the group and iommu ID.  This ensures the group
-	 * is held across the reset.
+	 * Get the group file for each fd to ensure the group held across
+	 * the reset
 	 */
 	for (file_idx = 0; file_idx < hdr.count; file_idx++) {
 		struct file *file = fget(group_fds[file_idx]);
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH v2 2/10 vfio/pci: Only check ownership of opened devices in hot reset

In reply to Yi Liu

If the affected device is not opened by any user, it's safe to reset it given it's not in use.

---
 drivers/vfio/pci/vfio_pci_core.c | 14 +++++++++++---
 include/uapi/linux/vfio.h        |  8 ++++++++
 2 files changed, 19 insertions(+), 3 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +11 -3
index 65bbef562268..5d745c9abf05 100644
@@ -2429,10 +2429,18 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 
 	list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) {
 		/*
-		 * Test whether all the affected devices are contained by the
-		 * set of groups provided by the user.
+		 * Test whether all the affected devices can be reset by the
+		 * user.
+		 *
+		 * Resetting an unused device (not opened) is safe, because
+		 * dev_set->lock is held in hot reset path so this device
+		 * cannot race being opened by another user simultaneously.
+		 *
+		 * Otherwise all opened devices in the dev_set must be
+		 * contained by the set of groups provided by the user.
 		 */
-		if (!vfio_dev_in_groups(cur_vma, groups)) {
+		if (cur_vma->vdev.open_count &&
+		    !vfio_dev_in_groups(cur_vma, groups)) {
 			ret = -EINVAL;
 			goto err_undo;
 		}
include/uapi/linux/vfio.h +8 -0
index 0552e8dcf0cb..f96e5689cffc 100644
@@ -673,6 +673,14 @@ struct vfio_pci_hot_reset_info {
  * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13,
  *				    struct vfio_pci_hot_reset)
  *
+ * Userspace requests hot reset for the devices it uses.  Due to the
+ * underlying topology, multiple devices can be affected in the reset
+ * while some might be opened by another user.  To avoid interference
+ * the calling user must ensure all affected devices, if opened, are
+ * owned by itself.
+ *
+ * The ownership is proved by an array of group fds.
+ *
  * Return: 0 on success, -errno on failure.
  */
 struct vfio_pci_hot_reset {
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH v2 3/10 vfio/pci: Move the existing hot reset logic to be a helper

In reply to Yi Liu

This prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups().

No functional change is intended.

---
 drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++-------------
 1 file changed, 33 insertions(+), 23 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +33 -23
index 5d745c9abf05..3696b8e58445 100644
@@ -1255,29 +1255,17 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(
 	return ret;
 }
 
-static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
-					struct vfio_pci_hot_reset __user *arg)
+static int
+vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
+				    struct vfio_pci_hot_reset *hdr,
+				    bool slot,
+				    struct vfio_pci_hot_reset __user *arg)
 {
-	unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count);
-	struct vfio_pci_hot_reset hdr;
 	int32_t *group_fds;
 	struct file **files;
 	struct vfio_pci_group_info info;
-	bool slot = false;
 	int file_idx, count = 0, ret = 0;
 
-	if (copy_from_user(&hdr, arg, minsz))
-		return -EFAULT;
-
-	if (hdr.argsz < minsz || hdr.flags)
-		return -EINVAL;
-
-	/* Can we do a slot or bus reset or neither? */
-	if (!pci_probe_reset_slot(vdev->pdev->slot))
-		slot = true;
-	else if (pci_probe_reset_bus(vdev->pdev->bus))
-		return -ENODEV;
-
 	/*
 	 * We can't let userspace give us an arbitrarily large buffer to copy,
 	 * so verify how many we think there could be.  Note groups can have
@@ -1289,11 +1277,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 		return ret;
 
 	/* Somewhere between 1 and count is OK */
-	if (!hdr.count || hdr.count > count)
+	if (!hdr->count || hdr->count > count)
 		return -EINVAL;
 
-	group_fds = kcalloc(hdr.count, sizeof(*group_fds), GFP_KERNEL);
-	files = kcalloc(hdr.count, sizeof(*files), GFP_KERNEL);
+	group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL);
+	files = kcalloc(hdr->count, sizeof(*files), GFP_KERNEL);
 	if (!group_fds || !files) {
 		kfree(group_fds);
 		kfree(files);
@@ -1301,7 +1289,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	}
 
 	if (copy_from_user(group_fds, arg->group_fds,
-			   hdr.count * sizeof(*group_fds))) {
+			   hdr->count * sizeof(*group_fds))) {
 		kfree(group_fds);
 		kfree(files);
 		return -EFAULT;
@@ -1311,7 +1299,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	 * Get the group file for each fd to ensure the group held across
 	 * the reset
 	 */
-	for (file_idx = 0; file_idx < hdr.count; file_idx++) {
+	for (file_idx = 0; file_idx < hdr->count; file_idx++) {
 		struct file *file = fget(group_fds[file_idx]);
 
 		if (!file) {
@@ -1335,7 +1323,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	if (ret)
 		goto hot_reset_release;
 
-	info.count = hdr.count;
+	info.count = hdr->count;
 	info.files = files;
 
 	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info);
@@ -1348,6 +1336,28 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	return ret;
 }
 
+static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
+					struct vfio_pci_hot_reset __user *arg)
+{
+	unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count);
+	struct vfio_pci_hot_reset hdr;
+	bool slot = false;
+
+	if (copy_from_user(&hdr, arg, minsz))
+		return -EFAULT;
+
+	if (hdr.argsz < minsz || hdr.flags)
+		return -EINVAL;
+
+	/* Can we do a slot or bus reset or neither? */
+	if (!pci_probe_reset_slot(vdev->pdev->slot))
+		slot = true;
+	else if (pci_probe_reset_bus(vdev->pdev->bus))
+		return -ENODEV;
+
+	return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg);
+}
+
 static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev,
 				    struct vfio_device_ioeventfd __user *arg)
 {
Signature · 1 line
2.34.1
maintainer Reviewed-by on v2 3/10
#4 lore raw

Re: [Intel-gfx] [PATCH v2 03/10] vfio/pci: Move the existing hot reset logic to be a helper

Quoted 11 lines · Yi Liu

This prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups().

No functional change is intended.

---
 drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++-------------
 1 file changed, 33 insertions(+), 23 deletions(-)

Jason

maintainer Reviewed-by on v2 3/10
#5 lore raw

Re: [Intel-gfx] [PATCH v2 03/10] vfio/pci: Move the existing hot reset logic to be a helper

In reply to Yi Liu
Quoted 11 lines · Yi Liu

This prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups().

No functional change is intended.

---
 drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++-------------
 1 file changed, 33 insertions(+), 23 deletions(-)

Jason

author
#6 lore raw

PATCH v2 4/10 vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device

In reply to Yi Liu

This is needed by the vfio pci driver to report affected devices in the hot reset for a given device.

---
 drivers/iommu/iommufd/device.c | 12 ++++++++++++
 drivers/vfio/iommufd.c         | 16 ++++++++++++++++
 include/linux/iommufd.h        |  3 +++
 include/linux/vfio.h           | 13 +++++++++++++
 4 files changed, 44 insertions(+)
drivers/iommu/iommufd/device.c +12 -0
index 25115d401d8f..04a57aa1ae2c 100644
@@ -131,6 +131,18 @@ void iommufd_device_unbind(struct iommufd_device *idev)
 }
 EXPORT_SYMBOL_NS_GPL(iommufd_device_unbind, IOMMUFD);
 
+struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev)
+{
+	return idev->ictx;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_device_to_ictx, IOMMUFD);
+
+u32 iommufd_device_to_id(struct iommufd_device *idev)
+{
+	return idev->obj.id;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, IOMMUFD);
+
 static int iommufd_device_setup_msi(struct iommufd_device *idev,
 				    struct iommufd_hw_pagetable *hwpt,
 				    phys_addr_t sw_msi_start)
drivers/vfio/iommufd.c +16 -0
index 88b00c501015..44088049dbb1 100644
@@ -66,6 +66,22 @@ void vfio_iommufd_unbind(struct vfio_device *vdev)
 		vdev->ops->unbind_iommufd(vdev);
 }
 
+struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev)
+{
+	if (!vdev->iommufd_device)
+		return NULL;
+	return iommufd_device_to_ictx(vdev->iommufd_device);
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_ictx);
+
+int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id)
+{
+	if (!vdev->iommufd_device)
+		return -EINVAL;
+	*id = iommufd_device_to_id(vdev->iommufd_device);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_devid);
 /*
  * The physical standard ops mean that the iommufd_device is bound to the
  * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers
include/linux/iommufd.h +3 -0
index 1129a36a74c4..ac96df406833 100644
@@ -24,6 +24,9 @@ void iommufd_device_unbind(struct iommufd_device *idev);
 int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id);
 void iommufd_device_detach(struct iommufd_device *idev);
 
+struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev);
+u32 iommufd_device_to_id(struct iommufd_device *idev);
+
 struct iommufd_access_ops {
 	u8 needs_pin_pages : 1;
 	void (*unmap)(void *data, unsigned long iova, unsigned long length);
include/linux/vfio.h +13 -0
index 3188d8a374bd..e54bef5489a0 100644
@@ -113,6 +113,8 @@ struct vfio_device_ops {
 };
 
 #if IS_ENABLED(CONFIG_IOMMUFD)
+struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev);
+int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id);
 int vfio_iommufd_physical_bind(struct vfio_device *vdev,
 			       struct iommufd_ctx *ictx, u32 *out_device_id);
 void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
@@ -122,6 +124,17 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
 void vfio_iommufd_emulated_unbind(struct vfio_device *vdev);
 int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
 #else
+static inline struct iommufd_ctx *
+vfio_iommufd_physical_ictx(struct vfio_device *vdev)
+{
+	return NULL;
+}
+
+static inline int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id)
+{
+	return -EOPNOTSUPP;
+}
+
 #define vfio_iommufd_physical_bind                                      \
 	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
 		  u32 *out_device_id)) NULL)
Signature · 1 line
2.34.1
maintainer Reviewed-by on v2 4/10
#7 lore raw

Re: [Intel-gfx] [PATCH v2 04/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device

Quoted 10 lines · Yi Liu

This is needed by the vfio pci driver to report affected devices in the hot reset for a given device.

---
 drivers/iommu/iommufd/device.c | 12 ++++++++++++
 drivers/vfio/iommufd.c         | 16 ++++++++++++++++
 include/linux/iommufd.h        |  3 +++
 include/linux/vfio.h           | 13 +++++++++++++
 4 files changed, 44 insertions(+)

Jason

author
#8 lore raw

PATCH v2 5/10 vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET

In reply to Yi Liu

as an alternative method for ownership check when iommufd is used. In this case all opened devices in the affected dev_set are verified to be bound to a same valid iommufd value to allow reset. It's simpler and faster as user does not need to pass a set of fds and kernel no need to search the device within the given fds.

a device in noiommu mode doesn't have a valid iommufd, so this method should not be used in a dev_set which contains multiple devices and one of them is in noiommu. The only allowed noiommu scenario is that the calling device is noiommu and it's in a singleton dev_set.

---
 drivers/vfio/pci/vfio_pci_core.c | 42 +++++++++++++++++++++++++++-----
 include/uapi/linux/vfio.h        |  9 ++++++-
 2 files changed, 44 insertions(+), 7 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +36 -6
index 3696b8e58445..b68fcba67a4b 100644
@@ -180,7 +180,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev)
 struct vfio_pci_group_info;
 static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set);
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups);
+				      struct vfio_pci_group_info *groups,
+				      struct iommufd_ctx *iommufd_ctx);
 
 /*
  * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND
@@ -1277,7 +1278,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
 		return ret;
 
 	/* Somewhere between 1 and count is OK */
-	if (!hdr->count || hdr->count > count)
+	if (hdr->count > count)
 		return -EINVAL;
 
 	group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL);
@@ -1326,7 +1327,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
 	info.count = hdr->count;
 	info.files = files;
 
-	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info);
+	ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL);
 
 hot_reset_release:
 	for (file_idx--; file_idx >= 0; file_idx--)
@@ -1341,6 +1342,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 {
 	unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count);
 	struct vfio_pci_hot_reset hdr;
+	struct iommufd_ctx *iommufd;
 	bool slot = false;
 
 	if (copy_from_user(&hdr, arg, minsz))
@@ -1355,7 +1357,12 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 	else if (pci_probe_reset_bus(vdev->pdev->bus))
 		return -ENODEV;
 
-	return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg);
+	if (hdr.count)
+		return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg);
+
+	iommufd = vfio_iommufd_physical_ictx(&vdev->vdev);
+
+	return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL, iommufd);
 }
 
 static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev,
@@ -2327,6 +2334,9 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev,
 {
 	unsigned int i;
 
+	if (!groups)
+		return false;
+
 	for (i = 0; i < groups->count; i++)
 		if (vfio_file_has_dev(groups->files[i], &vdev->vdev))
 			return true;
@@ -2402,13 +2412,25 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set)
 	return ret;
 }
 
+static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev,
+				    struct iommufd_ctx *iommufd_ctx)
+{
+	struct iommufd_ctx *iommufd = vfio_iommufd_physical_ictx(&vdev->vdev);
+
+	if (!iommufd)
+		return false;
+
+	return iommufd == iommufd_ctx;
+}
+
 /*
  * We need to get memory_lock for each device, but devices can share mmap_lock,
  * therefore we need to zap and hold the vma_lock for each device, and only then
  * get each memory_lock.
  */
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups)
+				      struct vfio_pci_group_info *groups,
+				      struct iommufd_ctx *iommufd_ctx)
 {
 	struct vfio_pci_core_device *cur_mem;
 	struct vfio_pci_core_device *cur_vma;
@@ -2448,9 +2470,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 		 *
 		 * Otherwise all opened devices in the dev_set must be
 		 * contained by the set of groups provided by the user.
+		 *
+		 * If user provides a zero-length array, then all the
+		 * opened devices must be bound to a same iommufd_ctx.
+		 *
+		 * If all above checks are failed, reset is allowed only if
+		 * the calling device is in a singleton dev_set.
 		 */
 		if (cur_vma->vdev.open_count &&
-		    !vfio_dev_in_groups(cur_vma, groups)) {
+		    !vfio_dev_in_groups(cur_vma, groups) &&
+		    !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx) &&
+		    (dev_set->device_count > 1)) {
 			ret = -EINVAL;
 			goto err_undo;
 		}
include/uapi/linux/vfio.h +8 -1
index f96e5689cffc..17aa5d09db41 100644
@@ -679,7 +679,14 @@ struct vfio_pci_hot_reset_info {
  * the calling user must ensure all affected devices, if opened, are
  * owned by itself.
  *
- * The ownership is proved by an array of group fds.
+ * The ownership can be proved by:
+ *   - An array of group fds
+ *   - A zero-length array
+ *
+ * In the last case all affected devices which are opened by this user
+ * must have been bound to a same iommufd. If the calling device is in
+ * noiommu mode (no valid iommufd) then it can be reset only if the reset
+ * doesn't affect other devices.
  *
  * Return: 0 on success, -errno on failure.
  */
Signature · 1 line
2.34.1
maintainer Reviewed-by on v2 5/10
#9 lore raw

Re: [Intel-gfx] [PATCH v2 05/10] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET

Quoted 11 lines · Yi Liu

as an alternative method for ownership check when iommufd is used. In this case all opened devices in the affected dev_set are verified to be bound to a same valid iommufd value to allow reset. It's simpler and faster as user does not need to pass a set of fds and kernel no need to search the device within the given fds.

a device in noiommu mode doesn't have a valid iommufd, so this method should not be used in a dev_set which contains multiple devices and one of them is in noiommu. The only allowed noiommu scenario is that the calling device is noiommu and it's in a singleton dev_set.

Quoted 7 lines
---
 drivers/vfio/pci/vfio_pci_core.c | 42 +++++++++++++++++++++++++++-----
 include/uapi/linux/vfio.h        |  9 ++++++-
 2 files changed, 44 insertions(+), 7 deletions(-)

Jason

author
#10 lore raw

PATCH v2 6/10 vfio: Refine vfio file kAPIs for vfio PCI hot reset

In reply to Yi Liu

This prepares vfio core to accept vfio device file from the vfio PCI hot reset path. vfio_file_is_group() is still kept for KVM usage.

---
 drivers/vfio/group.c             | 32 ++++++++++++++------------------
 drivers/vfio/pci/vfio_pci_core.c |  4 ++--
 drivers/vfio/vfio.h              |  2 ++
 drivers/vfio/vfio_main.c         | 29 +++++++++++++++++++++++++++++
 include/linux/vfio.h             |  1 +
 5 files changed, 48 insertions(+), 20 deletions(-)
drivers/vfio/group.c +14 -18
index 27d5ba7cf9dc..d0c95d033605 100644
@@ -745,6 +745,15 @@ bool vfio_device_has_container(struct vfio_device *device)
 	return device->group->container;
 }
 
+struct vfio_group *vfio_group_from_file(struct file *file)
+{
+	struct vfio_group *group = file->private_data;
+
+	if (file->f_op != &vfio_group_fops)
+		return NULL;
+	return group;
+}
+
 /**
  * vfio_file_iommu_group - Return the struct iommu_group for the vfio group file
  * @file: VFIO group file
@@ -755,13 +764,13 @@ bool vfio_device_has_container(struct vfio_device *device)
  */
 struct iommu_group *vfio_file_iommu_group(struct file *file)
 {
-	struct vfio_group *group = file->private_data;
+	struct vfio_group *group = vfio_group_from_file(file);
 	struct iommu_group *iommu_group = NULL;
 
 	if (!IS_ENABLED(CONFIG_SPAPR_TCE_IOMMU))
 		return NULL;
 
-	if (!vfio_file_is_group(file))
+	if (!group)
 		return NULL;
 
 	mutex_lock(&group->group_lock);
@@ -775,12 +784,12 @@ struct iommu_group *vfio_file_iommu_group(struct file *file)
 EXPORT_SYMBOL_GPL(vfio_file_iommu_group);
 
 /**
- * vfio_file_is_group - True if the file is usable with VFIO aPIS
+ * vfio_file_is_group - True if the file is a vfio group file
  * @file: VFIO group file
  */
 bool vfio_file_is_group(struct file *file)
 {
-	return file->f_op == &vfio_group_fops;
+	return vfio_group_from_file(file);
 }
 EXPORT_SYMBOL_GPL(vfio_file_is_group);
 
@@ -842,23 +851,10 @@ void vfio_file_set_kvm(struct file *file, struct kvm *kvm)
 }
 EXPORT_SYMBOL_GPL(vfio_file_set_kvm);
 
-/**
- * vfio_file_has_dev - True if the VFIO file is a handle for device
- * @file: VFIO file to check
- * @device: Device that must be part of the file
- *
- * Returns true if given file has permission to manipulate the given device.
- */
-bool vfio_file_has_dev(struct file *file, struct vfio_device *device)
+bool vfio_group_has_dev(struct vfio_group *group, struct vfio_device *device)
 {
-	struct vfio_group *group = file->private_data;
-
-	if (!vfio_file_is_group(file))
-		return false;
-
 	return group == device->group;
 }
-EXPORT_SYMBOL_GPL(vfio_file_has_dev);
 
 static char *vfio_devnode(const struct device *dev, umode_t *mode)
 {
drivers/vfio/pci/vfio_pci_core.c +2 -2
index b68fcba67a4b..2a510b71edcb 100644
@@ -1308,8 +1308,8 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
 			break;
 		}
 
-		/* Ensure the FD is a vfio group FD.*/
-		if (!vfio_file_is_group(file)) {
+		/* Ensure the FD is a vfio FD. vfio group or vfio device */
+		if (!vfio_file_is_valid(file)) {
 			fput(file);
 			ret = -EINVAL;
 			break;
drivers/vfio/vfio.h +2 -0
index 7b19c621e0e6..c0aeea24fbd6 100644
@@ -84,6 +84,8 @@ void vfio_device_group_unregister(struct vfio_device *device);
 int vfio_device_group_use_iommu(struct vfio_device *device);
 void vfio_device_group_unuse_iommu(struct vfio_device *device);
 void vfio_device_group_close(struct vfio_device *device);
+struct vfio_group *vfio_group_from_file(struct file *file);
+bool vfio_group_has_dev(struct vfio_group *group, struct vfio_device *device);
 bool vfio_device_has_container(struct vfio_device *device);
 int __init vfio_group_init(void);
 void vfio_group_cleanup(void);
drivers/vfio/vfio_main.c +29 -0
index 89497c933490..fe7446805afd 100644
@@ -1154,6 +1154,35 @@ const struct file_operations vfio_device_fops = {
 	.mmap		= vfio_device_fops_mmap,
 };
 
+/**
+ * vfio_file_is_valid - True if the file is valid vfio file
+ * @file: VFIO group file or VFIO device file
+ */
+bool vfio_file_is_valid(struct file *file)
+{
+	return vfio_group_from_file(file);
+}
+EXPORT_SYMBOL_GPL(vfio_file_is_valid);
+
+/**
+ * vfio_file_has_dev - True if the VFIO file is a handle for device
+ * @file: VFIO file to check
+ * @device: Device that must be part of the file
+ *
+ * Returns true if given file has permission to manipulate the given device.
+ */
+bool vfio_file_has_dev(struct file *file, struct vfio_device *device)
+{
+	struct vfio_group *group;
+
+	group = vfio_group_from_file(file);
+	if (!group)
+		return false;
+
+	return vfio_group_has_dev(group, device);
+}
+EXPORT_SYMBOL_GPL(vfio_file_has_dev);
+
 /*
  * Sub-module support
  */
include/linux/vfio.h +1 -0
index e54bef5489a0..79c47733ae0d 100644
@@ -258,6 +258,7 @@ int vfio_mig_get_next_state(struct vfio_device *device,
  */
 struct iommu_group *vfio_file_iommu_group(struct file *file);
 bool vfio_file_is_group(struct file *file);
+bool vfio_file_is_valid(struct file *file);
 bool vfio_file_enforced_coherent(struct file *file);
 void vfio_file_set_kvm(struct file *file, struct kvm *kvm);
 bool vfio_file_has_dev(struct file *file, struct vfio_device *device);
Signature · 1 line
2.34.1
maintainer Reviewed-by on v2 6/10
#11 lore raw

Re: [Intel-gfx] [PATCH v2 06/10] vfio: Refine vfio file kAPIs for vfio PCI hot reset

Quoted 5 lines · Yi Liu

This prepares vfio core to accept vfio device file from the vfio PCI hot reset path. vfio_file_is_group() is still kept for KVM usage.

Quoted 7 lines
---
 drivers/vfio/group.c             | 32 ++++++++++++++------------------
 drivers/vfio/pci/vfio_pci_core.c |  4 ++--
 drivers/vfio/vfio.h              |  2 ++
 drivers/vfio/vfio_main.c         | 29 +++++++++++++++++++++++++++++
 include/linux/vfio.h             |  1 +
 5 files changed, 48 insertions(+), 20 deletions(-)

Jason

author
#12 lore raw

PATCH v2 7/10 vfio: Accpet device file from vfio PCI hot reset path

In reply to Yi Liu

This extends both vfio_file_is_valid() and vfio_file_has_dev() to accept device file from the vfio PCI hot reset.

---
 drivers/vfio/vfio_main.c | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)
drivers/vfio/vfio_main.c +19 -4
index fe7446805afd..ebbb6b91a498 100644
@@ -1154,13 +1154,23 @@ const struct file_operations vfio_device_fops = {
 	.mmap		= vfio_device_fops_mmap,
 };
 
+static struct vfio_device *vfio_device_from_file(struct file *file)
+{
+	struct vfio_device *device = file->private_data;
+
+	if (file->f_op != &vfio_device_fops)
+		return NULL;
+	return device;
+}
+
 /**
  * vfio_file_is_valid - True if the file is valid vfio file
  * @file: VFIO group file or VFIO device file
  */
 bool vfio_file_is_valid(struct file *file)
 {
-	return vfio_group_from_file(file);
+	return vfio_group_from_file(file) ||
+	       vfio_device_from_file(file);
 }
 EXPORT_SYMBOL_GPL(vfio_file_is_valid);
 
@@ -1174,12 +1184,17 @@ EXPORT_SYMBOL_GPL(vfio_file_is_valid);
 bool vfio_file_has_dev(struct file *file, struct vfio_device *device)
 {
 	struct vfio_group *group;
+	struct vfio_device *vdev;
 
 	group = vfio_group_from_file(file);
-	if (!group)
-		return false;
+	if (group)
+		return vfio_group_has_dev(group, device);
+
+	vdev = vfio_device_from_file(file);
+	if (vdev)
+		return vdev == device;
 
-	return vfio_group_has_dev(group, device);
+	return false;
 }
 EXPORT_SYMBOL_GPL(vfio_file_has_dev);
 
Signature · 1 line
2.34.1
maintainer Reviewed-by on v2 7/10
#13 lore raw

Re: [Intel-gfx] [PATCH v2 07/10] vfio: Accpet device file from vfio PCI hot reset path

Quoted 8 lines · Yi Liu

This extends both vfio_file_is_valid() and vfio_file_has_dev() to accept device file from the vfio PCI hot reset.

---
 drivers/vfio/vfio_main.c | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

Jason

author
#14 lore raw

PATCH v2 8/10 vfio/pci: Renaming for accepting device fd in hot reset path

In reply to Yi Liu

No functional change is intended.

---
 drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++----------------
 1 file changed, 26 insertions(+), 26 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +26 -26
index 2a510b71edcb..da6325008872 100644
@@ -177,10 +177,10 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev)
 	}
 }
 
-struct vfio_pci_group_info;
+struct vfio_pci_file_info;
 static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set);
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups,
+				      struct vfio_pci_file_info *info,
 				      struct iommufd_ctx *iommufd_ctx);
 
 /*
@@ -800,7 +800,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)
 	return 0;
 }
 
-struct vfio_pci_group_info {
+struct vfio_pci_file_info {
 	int count;
 	struct file **files;
 };
@@ -1257,14 +1257,14 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(
 }
 
 static int
-vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
-				    struct vfio_pci_hot_reset *hdr,
-				    bool slot,
-				    struct vfio_pci_hot_reset __user *arg)
+vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev,
+				   struct vfio_pci_hot_reset *hdr,
+				   bool slot,
+				   struct vfio_pci_hot_reset __user *arg)
 {
-	int32_t *group_fds;
+	int32_t *fds;
 	struct file **files;
-	struct vfio_pci_group_info info;
+	struct vfio_pci_file_info info;
 	int file_idx, count = 0, ret = 0;
 
 	/*
@@ -1281,17 +1281,17 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
 	if (hdr->count > count)
 		return -EINVAL;
 
-	group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL);
+	fds = kcalloc(hdr->count, sizeof(*fds), GFP_KERNEL);
 	files = kcalloc(hdr->count, sizeof(*files), GFP_KERNEL);
-	if (!group_fds || !files) {
-		kfree(group_fds);
+	if (!fds || !files) {
+		kfree(fds);
 		kfree(files);
 		return -ENOMEM;
 	}
 
-	if (copy_from_user(group_fds, arg->group_fds,
-			   hdr->count * sizeof(*group_fds))) {
-		kfree(group_fds);
+	if (copy_from_user(fds, arg->group_fds,
+			   hdr->count * sizeof(*fds))) {
+		kfree(fds);
 		kfree(files);
 		return -EFAULT;
 	}
@@ -1301,7 +1301,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
 	 * the reset
 	 */
 	for (file_idx = 0; file_idx < hdr->count; file_idx++) {
-		struct file *file = fget(group_fds[file_idx]);
+		struct file *file = fget(fds[file_idx]);
 
 		if (!file) {
 			ret = -EBADF;
@@ -1318,9 +1318,9 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,
 		files[file_idx] = file;
 	}
 
-	kfree(group_fds);
+	kfree(fds);
 
-	/* release reference to groups on error */
+	/* release reference to fds on error */
 	if (ret)
 		goto hot_reset_release;
 
@@ -1358,7 +1358,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,
 		return -ENODEV;
 
 	if (hdr.count)
-		return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg);
+		return vfio_pci_ioctl_pci_hot_reset_files(vdev, &hdr, slot, arg);
 
 	iommufd = vfio_iommufd_physical_ictx(&vdev->vdev);
 
@@ -2329,16 +2329,16 @@ const struct pci_error_handlers vfio_pci_core_err_handlers = {
 };
 EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers);
 
-static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev,
-			       struct vfio_pci_group_info *groups)
+static bool vfio_dev_in_files(struct vfio_pci_core_device *vdev,
+			      struct vfio_pci_file_info *info)
 {
 	unsigned int i;
 
-	if (!groups)
+	if (!info)
 		return false;
 
-	for (i = 0; i < groups->count; i++)
-		if (vfio_file_has_dev(groups->files[i], &vdev->vdev))
+	for (i = 0; i < info->count; i++)
+		if (vfio_file_has_dev(info->files[i], &vdev->vdev))
 			return true;
 	return false;
 }
@@ -2429,7 +2429,7 @@ static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev,
  * get each memory_lock.
  */
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
-				      struct vfio_pci_group_info *groups,
+				      struct vfio_pci_file_info *info,
 				      struct iommufd_ctx *iommufd_ctx)
 {
 	struct vfio_pci_core_device *cur_mem;
@@ -2478,7 +2478,7 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 		 * the calling device is in a singleton dev_set.
 		 */
 		if (cur_vma->vdev.open_count &&
-		    !vfio_dev_in_groups(cur_vma, groups) &&
+		    !vfio_dev_in_files(cur_vma, info) &&
 		    !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx) &&
 		    (dev_set->device_count > 1)) {
 			ret = -EINVAL;
Signature · 1 line
2.34.1
author
#15 lore raw

PATCH v2 9/10 vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl

In reply to Yi Liu

Now user can also provide an array of device fds as a 3rd method to verify the reset ownership. It's not useful at this point when the device fds are acquired via group fds. But it's necessary when moving to device cdev which allows the user to directly acquire device fds by skipping group. In that case this method can be used as a last resort when the preferred iommufd verification doesn't work, e.g. in noiommu usages.

Clarify it in uAPI.

---
 drivers/vfio/pci/vfio_pci_core.c | 9 +++++----
 include/uapi/linux/vfio.h        | 3 ++-
 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +5 -4
index da6325008872..19f5b075d70a 100644
@@ -1289,7 +1289,7 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev,
 		return -ENOMEM;
 	}
 
-	if (copy_from_user(fds, arg->group_fds,
+	if (copy_from_user(fds, arg->fds,
 			   hdr->count * sizeof(*fds))) {
 		kfree(fds);
 		kfree(files);
@@ -1297,8 +1297,8 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev,
 	}
 
 	/*
-	 * Get the group file for each fd to ensure the group held across
-	 * the reset
+	 * Get the file for each fd to ensure the group/device file
+	 * is held across the reset
 	 */
 	for (file_idx = 0; file_idx < hdr->count; file_idx++) {
 		struct file *file = fget(fds[file_idx]);
@@ -2469,7 +2469,8 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 		 * cannot race being opened by another user simultaneously.
 		 *
 		 * Otherwise all opened devices in the dev_set must be
-		 * contained by the set of groups provided by the user.
+		 * contained by the set of groups/devices provided by
+		 * the user.
 		 *
 		 * If user provides a zero-length array, then all the
 		 * opened devices must be bound to a same iommufd_ctx.
include/uapi/linux/vfio.h +2 -1
index 17aa5d09db41..25432ef213ee 100644
@@ -681,6 +681,7 @@ struct vfio_pci_hot_reset_info {
  *
  * The ownership can be proved by:
  *   - An array of group fds
+ *   - An array of device fds
  *   - A zero-length array
  *
  * In the last case all affected devices which are opened by this user
@@ -694,7 +695,7 @@ struct vfio_pci_hot_reset {
 	__u32	argsz;
 	__u32	flags;
 	__u32	count;
-	__s32	group_fds[];
+	__s32	fds[];
 };
 
 #define VFIO_DEVICE_PCI_HOT_RESET	_IO(VFIO_TYPE, VFIO_BASE + 13)
Signature · 1 line
2.34.1
maintainer Reviewed-by on v2 9/10
#16 lore raw

Re: [Intel-gfx] [PATCH v2 09/10] vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl

Quoted 7 lines · Yi Liu

Now user can also provide an array of device fds as a 3rd method to verify the reset ownership. It's not useful at this point when the device fds are acquired via group fds. But it's necessary when moving to device cdev which allows the user to directly acquire device fds by skipping group. In that case this method can be used as a last resort when the preferred iommufd verification doesn't work, e.g. in noiommu usages.

Quoted 8 lines

Clarify it in uAPI.

---
 drivers/vfio/pci/vfio_pci_core.c | 9 +++++----
 include/uapi/linux/vfio.h        | 3 ++-
 2 files changed, 7 insertions(+), 5 deletions(-)

Jason

author
#17 lore raw

PATCH v2 10/10 vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

In reply to Yi Liu

This is a preparation for vfio device cdev as cdev gives userspace the capability to open device cdev fd and management stack (e.g. libvirt) could pass the device fd to the actual user (e.g. QEMU). As a result, the actual user has no idea about the device's bus:devfn information. This is a problem when user uses VFIO_DEVICE_GET_PCI_HOT_RESET_INFO to know the hot reset affected device scope as this ioctl returns bus:devfn info. For the fd passing usage, the acutal user cannot map the bus:devfn to the devices it has opened via the fd passed from management stack. So a new ioctl is required.

This new ioctl reports the list of iommufd dev_id that is opened by the user. If there is affected device that is not bound to vfio driver or opened by another user, this command shall fail with -EPERM. For the noiommu mode in the vfio device cdev path, this shall fail as no dev_id would be generated, hence nothing to report.

This ioctl is useless to the users that open vfio group as such users have no idea about the iommufd dev_id and it can use the existing VFIO_DEVICE_GET_PCI_HOT_RESET_INFO. The user that uses the traditional mode vfio group/container would be failed if invoking this ioctl. But the user that uses the iommufd compat mode vfio group/container shall succeed. This is harmless as long as user cannot make use of it and should use VFIO_DEVICE_GET_PCI_HOT_RESET_INFO.

---
 drivers/vfio/pci/vfio_pci_core.c | 98 ++++++++++++++++++++++++++++++++
 include/uapi/linux/vfio.h        | 47 +++++++++++++++
 2 files changed, 145 insertions(+)
drivers/vfio/pci/vfio_pci_core.c +98 -0
index 19f5b075d70a..45edf4e9b98b 100644
@@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 	return ret;
 }
 
+static struct pci_dev *
+vfio_pci_dev_set_resettable(struct vfio_device_set *dev_set);
+
+static int vfio_pci_ioctl_get_pci_hot_reset_group_info(
+	struct vfio_pci_core_device *vdev,
+	struct vfio_pci_hot_reset_group_info __user *arg)
+{
+	unsigned long minsz =
+		offsetofend(struct vfio_pci_hot_reset_group_info, count);
+	struct vfio_pci_hot_reset_group_info hdr;
+	struct iommufd_ctx *iommufd, *cur_iommufd;
+	u32 count = 0, index = 0, *devices = NULL;
+	struct vfio_pci_core_device *cur;
+	bool slot = false;
+	int ret = 0;
+
+	if (copy_from_user(&hdr, arg, minsz))
+		return -EFAULT;
+
+	if (hdr.argsz < minsz)
+		return -EINVAL;
+
+	hdr.flags = 0;
+
+	/* Can we do a slot or bus reset or neither? */
+	if (!pci_probe_reset_slot(vdev->pdev->slot))
+		slot = true;
+	else if (pci_probe_reset_bus(vdev->pdev->bus))
+		return -ENODEV;
+
+	mutex_lock(&vdev->vdev.dev_set->lock);
+	if (!vfio_pci_dev_set_resettable(vdev->vdev.dev_set)) {
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	iommufd = vfio_iommufd_physical_ictx(&vdev->vdev);
+	if (!iommufd) {
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	/* How many devices are affected? */
+	ret = vfio_pci_for_each_slot_or_bus(vdev->pdev, vfio_pci_count_devs,
+					    &count, slot);
+	if (ret)
+		goto out_unlock;
+
+	WARN_ON(!count); /* Should always be at least one */
+
+	/*
+	 * If there's enough space, fill it now, otherwise return -ENOSPC and
+	 * the number of devices affected.
+	 */
+	if (hdr.argsz < sizeof(hdr) + (count * sizeof(*devices))) {
+		ret = -ENOSPC;
+		hdr.count = count;
+		goto reset_info_exit;
+	}
+
+	devices = kcalloc(count, sizeof(*devices), GFP_KERNEL);
+	if (!devices) {
+		ret = -ENOMEM;
+		goto reset_info_exit;
+	}
+
+	list_for_each_entry(cur, &vdev->vdev.dev_set->device_list, vdev.dev_set_list) {
+		cur_iommufd = vfio_iommufd_physical_ictx(&cur->vdev);
+		if (cur->vdev.open_count) {
+			if (cur_iommufd != iommufd) {
+				ret = -EPERM;
+				break;
+			}
+			ret = vfio_iommufd_physical_devid(&cur->vdev, &devices[index]);
+			if (ret)
+				break;
+			index++;
+		}
+	}
+
+reset_info_exit:
+	if (copy_to_user(arg, &hdr, minsz))
+		ret = -EFAULT;
+
+	if (!ret) {
+		if (copy_to_user(&arg->devices, devices,
+				 hdr.count * sizeof(*devices)))
+			ret = -EFAULT;
+	}
+
+	kfree(devices);
+out_unlock:
+	mutex_unlock(&vdev->vdev.dev_set->lock);
+	return ret;
+}
+
 static int vfio_pci_ioctl_get_pci_hot_reset_info(
 	struct vfio_pci_core_device *vdev,
 	struct vfio_pci_hot_reset_info __user *arg)
@@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
 		return vfio_pci_ioctl_get_irq_info(vdev, uarg);
 	case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO:
 		return vfio_pci_ioctl_get_pci_hot_reset_info(vdev, uarg);
+	case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO:
+		return vfio_pci_ioctl_get_pci_hot_reset_group_info(vdev, uarg);
 	case VFIO_DEVICE_GET_REGION_INFO:
 		return vfio_pci_ioctl_get_region_info(vdev, uarg);
 	case VFIO_DEVICE_IOEVENTFD:
include/uapi/linux/vfio.h +47 -0

53 lines not on this page · raw message

Signature · 1 line
2.34.1
#18 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 16 lines · On Mon, 27 Mar 2023 02:34:58 -0700 Yi Liu <[email protected]> wrote:

This is a preparation for vfio device cdev as cdev gives userspace the capability to open device cdev fd and management stack (e.g. libvirt) could pass the device fd to the actual user (e.g. QEMU). As a result, the actual user has no idea about the device's bus:devfn information. This is a problem when user uses VFIO_DEVICE_GET_PCI_HOT_RESET_INFO to know the hot reset affected device scope as this ioctl returns bus:devfn info. For the fd passing usage, the acutal user cannot map the bus:devfn to the devices it has opened via the fd passed from management stack. So a new ioctl is required.

This new ioctl reports the list of iommufd dev_id that is opened by the user. If there is affected device that is not bound to vfio driver or opened by another user, this command shall fail with -EPERM. For the noiommu mode in the vfio device cdev path, this shall fail as no dev_id would be generated, hence nothing to report.

Quoted 7 lines

This ioctl is useless to the users that open vfio group as such users have no idea about the iommufd dev_id and it can use the existing VFIO_DEVICE_GET_PCI_HOT_RESET_INFO. The user that uses the traditional mode vfio group/container would be failed if invoking this ioctl. But the user that uses the iommufd compat mode vfio group/container shall succeed. This is harmless as long as user cannot make use of it and should use VFIO_DEVICE_GET_PCI_HOT_RESET_INFO.

So VFIO_DEVICE_GET_PCI_HOT_RESET_INFO reports a group and bdf, but VFIO_DEVICE_GET_PCI_HOT_RESET_*GROUP*_INFO is meant for the non-group, cdev use case and returns a dev_id rather than a group???

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a flags arg that isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID. In fact, we could define vfio_dependent_device as:

struct vfio_pci_dependent_device {
	union {
	        __u32   group_id;
		__u32	dev_id;
	}
        __u16   segment;
        __u8    bus;
        __u8    devfn;
};

If the user calls with the above flag, dev_id is valid, otherwise group_id. Perhaps segment:buus:devfn could still be filled in with a NULL/invalid dev_id if the user doesn't have permissions for the device so that debugging from userspace isn't so opaque. Thanks,

Alex

Quoted 180 lines
---
 drivers/vfio/pci/vfio_pci_core.c | 98 ++++++++++++++++++++++++++++++++
 include/uapi/linux/vfio.h        | 47 +++++++++++++++
 2 files changed, 145 insertions(+)
drivers/vfio/pci/vfio_pci_core.c +98 -0
index 19f5b075d70a..45edf4e9b98b 100644
@@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 	return ret;
 }
 
+static struct pci_dev *
+vfio_pci_dev_set_resettable(struct vfio_device_set *dev_set);
+
+static int vfio_pci_ioctl_get_pci_hot_reset_group_info(
+	struct vfio_pci_core_device *vdev,
+	struct vfio_pci_hot_reset_group_info __user *arg)
+{
+	unsigned long minsz =
+		offsetofend(struct vfio_pci_hot_reset_group_info, count);
+	struct vfio_pci_hot_reset_group_info hdr;
+	struct iommufd_ctx *iommufd, *cur_iommufd;
+	u32 count = 0, index = 0, *devices = NULL;
+	struct vfio_pci_core_device *cur;
+	bool slot = false;
+	int ret = 0;
+
+	if (copy_from_user(&hdr, arg, minsz))
+		return -EFAULT;
+
+	if (hdr.argsz < minsz)
+		return -EINVAL;
+
+	hdr.flags = 0;
+
+	/* Can we do a slot or bus reset or neither? */
+	if (!pci_probe_reset_slot(vdev->pdev->slot))
+		slot = true;
+	else if (pci_probe_reset_bus(vdev->pdev->bus))
+		return -ENODEV;
+
+	mutex_lock(&vdev->vdev.dev_set->lock);
+	if (!vfio_pci_dev_set_resettable(vdev->vdev.dev_set)) {
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	iommufd = vfio_iommufd_physical_ictx(&vdev->vdev);
+	if (!iommufd) {
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	/* How many devices are affected? */
+	ret = vfio_pci_for_each_slot_or_bus(vdev->pdev, vfio_pci_count_devs,
+					    &count, slot);
+	if (ret)
+		goto out_unlock;
+
+	WARN_ON(!count); /* Should always be at least one */
+
+	/*
+	 * If there's enough space, fill it now, otherwise return -ENOSPC and
+	 * the number of devices affected.
+	 */
+	if (hdr.argsz < sizeof(hdr) + (count * sizeof(*devices))) {
+		ret = -ENOSPC;
+		hdr.count = count;
+		goto reset_info_exit;
+	}
+
+	devices = kcalloc(count, sizeof(*devices), GFP_KERNEL);
+	if (!devices) {
+		ret = -ENOMEM;
+		goto reset_info_exit;
+	}
+
+	list_for_each_entry(cur, &vdev->vdev.dev_set->device_list, vdev.dev_set_list) {
+		cur_iommufd = vfio_iommufd_physical_ictx(&cur->vdev);
+		if (cur->vdev.open_count) {
+			if (cur_iommufd != iommufd) {
+				ret = -EPERM;
+				break;
+			}
+			ret = vfio_iommufd_physical_devid(&cur->vdev, &devices[index]);
+			if (ret)
+				break;
+			index++;
+		}
+	}
+
+reset_info_exit:
+	if (copy_to_user(arg, &hdr, minsz))
+		ret = -EFAULT;
+
+	if (!ret) {
+		if (copy_to_user(&arg->devices, devices,
+				 hdr.count * sizeof(*devices)))
+			ret = -EFAULT;
+	}
+
+	kfree(devices);
+out_unlock:
+	mutex_unlock(&vdev->vdev.dev_set->lock);
+	return ret;
+}
+
 static int vfio_pci_ioctl_get_pci_hot_reset_info(
 	struct vfio_pci_core_device *vdev,
 	struct vfio_pci_hot_reset_info __user *arg)
@@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
 		return vfio_pci_ioctl_get_irq_info(vdev, uarg);
 	case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO:
 		return vfio_pci_ioctl_get_pci_hot_reset_info(vdev, uarg);
+	case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO:
+		return vfio_pci_ioctl_get_pci_hot_reset_group_info(vdev, uarg);
 	case VFIO_DEVICE_GET_REGION_INFO:
 		return vfio_pci_ioctl_get_region_info(vdev, uarg);
 	case VFIO_DEVICE_IOEVENTFD:
include/uapi/linux/vfio.h +47 -0
index 25432ef213ee..61b801dfd40b 100644
@@ -669,6 +669,53 @@ struct vfio_pci_hot_reset_info {
 
 #define VFIO_DEVICE_GET_PCI_HOT_RESET_INFO	_IO(VFIO_TYPE, VFIO_BASE + 12)
 
+/**
+ * VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12,
+ *						    struct vfio_pci_hot_reset_group_info)
+ *
+ * This is used in the vfio device cdev mode.  It returns the list of
+ * affected devices (represented by iommufd dev_id) when hot reset is
+ * issued on the current device with which this ioctl is invoked.  It
+ * only includes the devices that are opened by the current user in the
+ * time of this command is invoked.  This list may change when user opens
+ * new device or close opened device, hence user should re-invoke it
+ * after open/close devices.  This command has no guarantee on the result
+ * of VFIO_DEVICE_PCI_HOT_RESET since the not-opened affected device can
+ * be by other users in the window between the two ioctls.  If the affected
+ * devices are opened by multiple users, the VFIO_DEVICE_PCI_HOT_RESET
+ * shall fail, detail can check the description of VFIO_DEVICE_PCI_HOT_RESET.
+ *
+ * For the users that open vfio group/container, this ioctl is useless as
+ * they have no idea about the iommufd dev_id returned by this ioctl.  For
+ * the users of the traditional mode vfio group/container, this ioctl will
+ * fail as this mode does not use iommufd hence no dev_id to report back.
+ * For the users of the iommufd compat mode vfio group/container, this ioctl
+ * would succeed as this mode uses iommufd as container fd.  But such users
+ * still have no idea about the iommufd dev_id as the dev_id is only stored
+ * in kernel in this mode.  For the users of the vfio group/container, the
+ * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO should be used to know the hot reset
+ * affected devices.
+ *
+ * Return: 0 on success, -errno on failure:
+ *	-enospc = insufficient buffer;
+ *	-enodev = unsupported for device;
+ *	-eperm = no permission for device, this error comes:
+ *		 - when there are affected devices that are opened but
+ *		   not bound to the same iommufd with the current device
+ *		   with which this ioctl is invoked,
+ *		 - there are affected devices that are not bound to vfio
+ *		   driver yet.
+ *		 - no valid iommufd is bound (e.g. noiommu mode)
+ */
+struct vfio_pci_hot_reset_group_info {
+	__u32	argsz;
+	__u32	flags;
+	__u32	count;
+	__u32	devices[];
+};
+
+#define VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO	_IO(VFIO_TYPE, VFIO_BASE + 18)
+
 /**
  * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13,
  *				    struct vfio_pci_hot_reset)
#19 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 139 lines · On Mon, 27 Mar 2023 13:26:19 -0600 Alex Williamson…
Quoted 16 lines · On Mon, 27 Mar 2023 02:34:58 -0700 Yi Liu <[email protected]> wrote:

This is a preparation for vfio device cdev as cdev gives userspace the capability to open device cdev fd and management stack (e.g. libvirt) could pass the device fd to the actual user (e.g. QEMU). As a result, the actual user has no idea about the device's bus:devfn information. This is a problem when user uses VFIO_DEVICE_GET_PCI_HOT_RESET_INFO to know the hot reset affected device scope as this ioctl returns bus:devfn info. For the fd passing usage, the acutal user cannot map the bus:devfn to the devices it has opened via the fd passed from management stack. So a new ioctl is required.

This new ioctl reports the list of iommufd dev_id that is opened by the user. If there is affected device that is not bound to vfio driver or opened by another user, this command shall fail with -EPERM. For the noiommu mode in the vfio device cdev path, this shall fail as no dev_id would be generated, hence nothing to report.

Quoted 7 lines

This ioctl is useless to the users that open vfio group as such users have no idea about the iommufd dev_id and it can use the existing VFIO_DEVICE_GET_PCI_HOT_RESET_INFO. The user that uses the traditional mode vfio group/container would be failed if invoking this ioctl. But the user that uses the iommufd compat mode vfio group/container shall succeed. This is harmless as long as user cannot make use of it and should use VFIO_DEVICE_GET_PCI_HOT_RESET_INFO.

So VFIO_DEVICE_GET_PCI_HOT_RESET_INFO reports a group and bdf, but VFIO_DEVICE_GET_PCI_HOT_RESET_*GROUP*_INFO is meant for the non-group, cdev use case and returns a dev_id rather than a group???

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a flags arg that isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID. In fact, we could define vfio_dependent_device as:

struct vfio_pci_dependent_device {
	union {
	        __u32   group_id;
		__u32	dev_id;
	}
        __u16   segment;
        __u8    bus;
        __u8    devfn;
};

If the user calls with the above flag, dev_id is valid, otherwise group_id. Perhaps segment:buus:devfn could still be filled in with a NULL/invalid dev_id if the user doesn't have permissions for the device so that debugging from userspace isn't so opaque. Thanks,

Alex

Quoted 85 lines
---
 drivers/vfio/pci/vfio_pci_core.c | 98 ++++++++++++++++++++++++++++++++
 include/uapi/linux/vfio.h        | 47 +++++++++++++++
 2 files changed, 145 insertions(+)
drivers/vfio/pci/vfio_pci_core.c +71 -0
index 19f5b075d70a..45edf4e9b98b 100644
@@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 	return ret;
 }
 
+static struct pci_dev *
+vfio_pci_dev_set_resettable(struct vfio_device_set *dev_set);
+
+static int vfio_pci_ioctl_get_pci_hot_reset_group_info(
+	struct vfio_pci_core_device *vdev,
+	struct vfio_pci_hot_reset_group_info __user *arg)
+{
+	unsigned long minsz =
+		offsetofend(struct vfio_pci_hot_reset_group_info, count);
+	struct vfio_pci_hot_reset_group_info hdr;
+	struct iommufd_ctx *iommufd, *cur_iommufd;
+	u32 count = 0, index = 0, *devices = NULL;
+	struct vfio_pci_core_device *cur;
+	bool slot = false;
+	int ret = 0;
+
+	if (copy_from_user(&hdr, arg, minsz))
+		return -EFAULT;
+
+	if (hdr.argsz < minsz)
+		return -EINVAL;
+
+	hdr.flags = 0;
+
+	/* Can we do a slot or bus reset or neither? */
+	if (!pci_probe_reset_slot(vdev->pdev->slot))
+		slot = true;
+	else if (pci_probe_reset_bus(vdev->pdev->bus))
+		return -ENODEV;
+
+	mutex_lock(&vdev->vdev.dev_set->lock);
+	if (!vfio_pci_dev_set_resettable(vdev->vdev.dev_set)) {
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	iommufd = vfio_iommufd_physical_ictx(&vdev->vdev);
+	if (!iommufd) {
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	/* How many devices are affected? */
+	ret = vfio_pci_for_each_slot_or_bus(vdev->pdev, vfio_pci_count_devs,
+					    &count, slot);
+	if (ret)
+		goto out_unlock;
+
+	WARN_ON(!count); /* Should always be at least one */
+
+	/*
+	 * If there's enough space, fill it now, otherwise return -ENOSPC and
+	 * the number of devices affected.
+	 */
+	if (hdr.argsz < sizeof(hdr) + (count * sizeof(*devices))) {
+		ret = -ENOSPC;
+		hdr.count = count;
+		goto reset_info_exit;
+	}
+
+	devices = kcalloc(count, sizeof(*devices), GFP_KERNEL);
+	if (!devices) {
+		ret = -ENOMEM;
+		goto reset_info_exit;
+	}
+
+	list_for_each_entry(cur, &vdev->vdev.dev_set->device_list, vdev.dev_set_list) {
+		cur_iommufd = vfio_iommufd_physical_ictx(&cur->vdev);
+		if (cur->vdev.open_count) {
+			if (cur_iommufd != iommufd) {
+				ret = -EPERM;
Quoted 8 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1262

Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +8 -0
+				break;
+			}
+			ret = vfio_iommufd_physical_devid(&cur->vdev, &devices[index]);
+			if (ret)
+				break;
+			index++;
+		}
+	}

This also makes use of vfio_pci_for_each_slot_or_bus() to iterate affected devices, but then still assumes those affected devices are necessarily represented in the dev_set. For example, a two-function device with ACS isolation can have function 0 bound to vfio and function 1 bound to a native host driver. The above code requires the user to pass a buffer large enough for both functions, but only function 0 is part of the dev_set, so function 1 is not reported as affected, nor does it generate an error.

It looks like we also fail to set hdr.count except in the error path above, so the below copy_to_user() copies a user specified range beyond the end the allocated devices buffer out to userspace! Thanks,

Alex

Quoted 88 lines
Quoted 87 lines
drivers/vfio/pci/vfio_pci_core.c +19 -0
+
+reset_info_exit:
+	if (copy_to_user(arg, &hdr, minsz))
+		ret = -EFAULT;
+
+	if (!ret) {
+		if (copy_to_user(&arg->devices, devices,
+				 hdr.count * sizeof(*devices)))
+			ret = -EFAULT;
+	}
+
+	kfree(devices);
+out_unlock:
+	mutex_unlock(&vdev->vdev.dev_set->lock);
+	return ret;
+}
+
 static int vfio_pci_ioctl_get_pci_hot_reset_info(
 	struct vfio_pci_core_device *vdev,
 	struct vfio_pci_hot_reset_info __user *arg)
@@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
 		return vfio_pci_ioctl_get_irq_info(vdev, uarg);
 	case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO:
 		return vfio_pci_ioctl_get_pci_hot_reset_info(vdev, uarg);
+	case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO:
+		return vfio_pci_ioctl_get_pci_hot_reset_group_info(vdev, uarg);
 	case VFIO_DEVICE_GET_REGION_INFO:
 		return vfio_pci_ioctl_get_region_info(vdev, uarg);
 	case VFIO_DEVICE_IOEVENTFD:
include/uapi/linux/vfio.h +47 -0
index 25432ef213ee..61b801dfd40b 100644
@@ -669,6 +669,53 @@ struct vfio_pci_hot_reset_info {
 
 #define VFIO_DEVICE_GET_PCI_HOT_RESET_INFO	_IO(VFIO_TYPE, VFIO_BASE + 12)
 
+/**
+ * VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12,
+ *						    struct vfio_pci_hot_reset_group_info)
+ *
+ * This is used in the vfio device cdev mode.  It returns the list of
+ * affected devices (represented by iommufd dev_id) when hot reset is
+ * issued on the current device with which this ioctl is invoked.  It
+ * only includes the devices that are opened by the current user in the
+ * time of this command is invoked.  This list may change when user opens
+ * new device or close opened device, hence user should re-invoke it
+ * after open/close devices.  This command has no guarantee on the result
+ * of VFIO_DEVICE_PCI_HOT_RESET since the not-opened affected device can
+ * be by other users in the window between the two ioctls.  If the affected
+ * devices are opened by multiple users, the VFIO_DEVICE_PCI_HOT_RESET
+ * shall fail, detail can check the description of VFIO_DEVICE_PCI_HOT_RESET.
+ *
+ * For the users that open vfio group/container, this ioctl is useless as
+ * they have no idea about the iommufd dev_id returned by this ioctl.  For
+ * the users of the traditional mode vfio group/container, this ioctl will
+ * fail as this mode does not use iommufd hence no dev_id to report back.
+ * For the users of the iommufd compat mode vfio group/container, this ioctl
+ * would succeed as this mode uses iommufd as container fd.  But such users
+ * still have no idea about the iommufd dev_id as the dev_id is only stored
+ * in kernel in this mode.  For the users of the vfio group/container, the
+ * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO should be used to know the hot reset
+ * affected devices.
+ *
+ * Return: 0 on success, -errno on failure:
+ *	-enospc = insufficient buffer;
+ *	-enodev = unsupported for device;
+ *	-eperm = no permission for device, this error comes:
+ *		 - when there are affected devices that are opened but
+ *		   not bound to the same iommufd with the current device
+ *		   with which this ioctl is invoked,
+ *		 - there are affected devices that are not bound to vfio
+ *		   driver yet.
+ *		 - no valid iommufd is bound (e.g. noiommu mode)
+ */
+struct vfio_pci_hot_reset_group_info {
+	__u32	argsz;
+	__u32	flags;
+	__u32	count;
+	__u32	devices[];
+};
+
+#define VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO	_IO(VFIO_TYPE, VFIO_BASE + 18)
+
 /**
  * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13,
  *				    struct vfio_pci_hot_reset)  
author
#20 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 6 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 4:41 AM

On Mon, 27 Mar 2023 13:26:19 -0600
Alex Williamson <[email protected]> wrote:

[...]

Quoted 34 lines
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +0 -0

b/drivers/vfio/pci/vfio_pci_core.c

Quoted 4 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct

Quoted 4 lines

index 19f5b075d70a..45edf4e9b98b 100644

drivers/vfio/pci/vfio_pci_core.c +0 -0
@@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct

vfio_pci_core_device *vdev,

Quoted 27 lines
Quoted 27 lines
drivers/vfio/pci/vfio_pci_core.c +24 -0
 	return ret;
 }
 
+static struct pci_dev *
+vfio_pci_dev_set_resettable(struct vfio_device_set *dev_set);
+
+static int vfio_pci_ioctl_get_pci_hot_reset_group_info(
+	struct vfio_pci_core_device *vdev,
+	struct vfio_pci_hot_reset_group_info __user *arg)
+{
+	unsigned long minsz =
+		offsetofend(struct vfio_pci_hot_reset_group_info, count);
+	struct vfio_pci_hot_reset_group_info hdr;
+	struct iommufd_ctx *iommufd, *cur_iommufd;
+	u32 count = 0, index = 0, *devices = NULL;
+	struct vfio_pci_core_device *cur;
+	bool slot = false;
+	int ret = 0;
+
+	if (copy_from_user(&hdr, arg, minsz))
+		return -EFAULT;
+
+	if (hdr.argsz < minsz)
+		return -EINVAL;
+
+	hdr.flags = 0;
+
Quoted 8 lines
Quoted 8 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1215

Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +8 -0
+	/* Can we do a slot or bus reset or neither? */
+	if (!pci_probe_reset_slot(vdev->pdev->slot))
+		slot = true;
+	else if (pci_probe_reset_bus(vdev->pdev->bus))
+		return -ENODEV;
+
+	mutex_lock(&vdev->vdev.dev_set->lock);
+	if (!vfio_pci_dev_set_resettable(vdev->vdev.dev_set)) {

[1]

Quoted 45 lines
Quoted 4 lines
Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +4 -0
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
Quoted 8 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,

Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +8 -0
+	iommufd = vfio_iommufd_physical_ictx(&vdev->vdev);
+	if (!iommufd) {
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	/* How many devices are affected? */
+	ret = vfio_pci_for_each_slot_or_bus(vdev->pdev,

vfio_pci_count_devs,

Quoted 15 lines
Quoted 15 lines
drivers/vfio/pci/vfio_pci_core.c +15 -0
+					    &count, slot);
+	if (ret)
+		goto out_unlock;
+
+	WARN_ON(!count); /* Should always be at least one */
+
+	/*
+	 * If there's enough space, fill it now, otherwise return -ENOSPC and
+	 * the number of devices affected.
+	 */
+	if (hdr.argsz < sizeof(hdr) + (count * sizeof(*devices))) {
+		ret = -ENOSPC;
+		hdr.count = count;
+		goto reset_info_exit;
+	}
Quoted 8 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,

Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +8 -0
+
+	devices = kcalloc(count, sizeof(*devices), GFP_KERNEL);
+	if (!devices) {
+		ret = -ENOMEM;
+		goto reset_info_exit;
+	}
+
+	list_for_each_entry(cur, &vdev->vdev.dev_set->device_list,

vdev.dev_set_list) {

Quoted 7 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,

Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +7 -0
+		cur_iommufd = vfio_iommufd_physical_ictx(&cur->vdev);
+		if (cur->vdev.open_count) {
+			if (cur_iommufd != iommufd) {
+				ret = -EPERM;
+				break;
+			}
+			ret = vfio_iommufd_physical_devid(&cur->vdev,

&devices[index]);

Quoted 14 lines
Quoted 5 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1262

Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +5 -0
+			if (ret)
+				break;
+			index++;
+		}
+	}

This also makes use of vfio_pci_for_each_slot_or_bus() to iterate affected devices, but then still assumes those affected devices are necessarily represented in the dev_set. For example, a two-function device with ACS isolation can have function 0 bound to vfio and function 1 bound to a native host driver. The above code requires the user to pass a buffer large enough for both functions, but only function 0 is part of the dev_set, so function 1 is not reported as affected, nor does it generate an error.

The vfio_pci_dev_set_resettable() is used at [1] to check if all the affected devices are in the dev_set. If not, then it fails at the first place. So in the following code, looping the devices in the dev_set is equivalent to looping devices with vfio_pci_for_each_slot_or_bus(). I need to loop dev_set as dev_set has the vfio_device which is more convenient to get dev_id.

Quoted 4 lines

It looks like we also fail to set hdr.count except in the error path above, so the below copy_to_user() copies a user specified range beyond the end the allocated devices buffer out to userspace! Thanks,

Oops, yes, it is. My test only has one device affected, so it does not hit the problem. ☹

Thanks,
Yi Liu

Quoted 108 lines

Alex

Quoted 13 lines
Quoted 13 lines
drivers/vfio/pci/vfio_pci_core.c +13 -0
+
+reset_info_exit:
+	if (copy_to_user(arg, &hdr, minsz))
+		ret = -EFAULT;
+
+	if (!ret) {
+		if (copy_to_user(&arg->devices, devices,
+				 hdr.count * sizeof(*devices)))
+			ret = -EFAULT;
+	}
+
+	kfree(devices);
+out_unlock:
Quoted 8 lines

drivers/vfio/pci/vfio_pci_core.c long vfio_pci_core_ioctl(struct vfio_device

Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +4 -0
+	mutex_unlock(&vdev->vdev.dev_set->lock);
+	return ret;
+}
+
 static int vfio_pci_ioctl_get_pci_hot_reset_info(
 	struct vfio_pci_core_device *vdev,
 	struct vfio_pci_hot_reset_info __user *arg)
@@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device

*core_vdev, unsigned int cmd,

Quoted 5 lines

drivers/vfio/pci/vfio_pci_core.c long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,

Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +2 -0
 		return vfio_pci_ioctl_get_irq_info(vdev, uarg);
 	case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO:
 		return vfio_pci_ioctl_get_pci_hot_reset_info(vdev, uarg);
+	case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO:
+		return vfio_pci_ioctl_get_pci_hot_reset_group_info(vdev,

uarg);

Quoted 10 lines

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 670

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +0 -0
 	case VFIO_DEVICE_GET_REGION_INFO:
 		return vfio_pci_ioctl_get_region_info(vdev, uarg);
 	case VFIO_DEVICE_IOEVENTFD:
include/uapi/linux/vfio.h +0 -0
index 25432ef213ee..61b801dfd40b 100644
@@ -669,6 +669,53 @@ struct vfio_pci_hot_reset_info {
 
 #define VFIO_DEVICE_GET_PCI_HOT_RESET_INFO	_IO(VFIO_TYPE,

VFIO_BASE + 12)

Quoted 3 lines

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 672

Quoted 3 lines
include/uapi/linux/vfio.h +2 -0
+/**
+ * VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO -

_IOWR(VFIO_TYPE, VFIO_BASE + 12,

Quoted 1 line

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 673

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ *						    struct

vfio_pci_hot_reset_group_info)

Quoted 6 lines

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {

Quoted 6 lines
include/uapi/linux/vfio.h +6 -0
+ *
+ * This is used in the vfio device cdev mode.  It returns the list of
+ * affected devices (represented by iommufd dev_id) when hot reset is
+ * issued on the current device with which this ioctl is invoked.  It
+ * only includes the devices that are opened by the current user in the
+ * time of this command is invoked.  This list may change when user

opens

Quoted 2 lines

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 681

Quoted 2 lines
include/uapi/linux/vfio.h +2 -0
+ * new device or close opened device, hence user should re-invoke it
+ * after open/close devices.  This command has no guarantee on the

result

Quoted 1 line

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 682

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * of VFIO_DEVICE_PCI_HOT_RESET since the not-opened affected

device can

Quoted 1 line

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 683

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * be by other users in the window between the two ioctls.  If the

affected

Quoted 1 line

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 684

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * devices are opened by multiple users, the

VFIO_DEVICE_PCI_HOT_RESET

Quoted 1 line

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 685

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * shall fail, detail can check the description of

VFIO_DEVICE_PCI_HOT_RESET.

Quoted 3 lines

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {

Quoted 3 lines
include/uapi/linux/vfio.h +3 -0
+ *
+ * For the users that open vfio group/container, this ioctl is useless as
+ * they have no idea about the iommufd dev_id returned by this ioctl.

For

Quoted 2 lines

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 690

Quoted 2 lines
include/uapi/linux/vfio.h +2 -0
+ * the users of the traditional mode vfio group/container, this ioctl will
+ * fail as this mode does not use iommufd hence no dev_id to report

back.

Quoted 1 line

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 691

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * For the users of the iommufd compat mode vfio group/container,

this ioctl

Quoted 1 line

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 692

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * would succeed as this mode uses iommufd as container fd.  But such

users

Quoted 1 line

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 693

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * still have no idea about the iommufd dev_id as the dev_id is only

stored

Quoted 2 lines

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 695

Quoted 2 lines
include/uapi/linux/vfio.h +2 -0
+ * in kernel in this mode.  For the users of the vfio group/container, the
+ * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO should be used to know

the hot reset

Quoted 13 lines
Quoted 13 lines
include/uapi/linux/vfio.h +13 -0
+ * affected devices.
+ *
+ * Return: 0 on success, -errno on failure:
+ *	-enospc = insufficient buffer;
+ *	-enodev = unsupported for device;
+ *	-eperm = no permission for device, this error comes:
+ *		 - when there are affected devices that are opened but
+ *		   not bound to the same iommufd with the current device
+ *		   with which this ioctl is invoked,
+ *		 - there are affected devices that are not bound to vfio
+ *		   driver yet.
+ *		 - no valid iommufd is bound (e.g. noiommu mode)
+ */
Quoted 8 lines

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {

Quoted 8 lines
include/uapi/linux/vfio.h +8 -0
+struct vfio_pci_hot_reset_group_info {
+	__u32	argsz;
+	__u32	flags;
+	__u32	count;
+	__u32	devices[];
+};
+
+#define VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
	_IO(VFIO_TYPE, VFIO_BASE + 18)
Quoted 5 lines
Quoted 4 lines
include/uapi/linux/vfio.h +1 -0
+
 /**
  * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13,
  *				    struct vfio_pci_hot_reset)
author
#21 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

In reply to Alex Williamson
Quoted 31 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AM

Quoted 16 lines · On Mon, 27 Mar 2023 02:34:58 -0700 Yi Liu <[email protected]> wrote:

This is a preparation for vfio device cdev as cdev gives userspace the capability to open device cdev fd and management stack (e.g. libvirt) could pass the device fd to the actual user (e.g. QEMU). As a result, the actual user has no idea about the device's bus:devfn information. This is a problem when user uses VFIO_DEVICE_GET_PCI_HOT_RESET_INFO to know the hot reset affected device scope as this ioctl returns bus:devfn info. For the fd passing usage, the acutal user cannot map the bus:devfn to the devices it has opened via the fd passed from management stack. So a new ioctl is required.

This new ioctl reports the list of iommufd dev_id that is opened by the user. If there is affected device that is not bound to vfio driver or opened by another user, this command shall fail with -EPERM. For the noiommu mode in the vfio device cdev path, this shall fail as no dev_id would be generated, hence nothing to report.

Quoted 7 lines

This ioctl is useless to the users that open vfio group as such users have no idea about the iommufd dev_id and it can use the existing VFIO_DEVICE_GET_PCI_HOT_RESET_INFO. The user that uses the traditional mode vfio group/container would be failed if invoking this ioctl. But the user that uses the iommufd compat mode vfio group/container shall succeed. This is harmless as long as user cannot make use of it and should use VFIO_DEVICE_GET_PCI_HOT_RESET_INFO.

Quoted 4 lines

So VFIO_DEVICE_GET_PCI_HOT_RESET_INFO reports a group and bdf, but VFIO_DEVICE_GET_PCI_HOT_RESET_*GROUP*_INFO is meant for the non- group,
cdev use case and returns a dev_id rather than a group???

Yes, this is the meaning, but poor naming here ☹ I also struggled on it. Perhaps your below Suggestion makes more sense. Introducing a flag and reuse the existing _INFO ioctl.

Quoted 3 lines

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a flags arg that isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.

Sure. I can follow this suggestion. BTW. I have a doubt here. This new flag is set by user. What if in the future kernel has new extensions and needs to report something new to the user and add new flags to tell user? Such flag is set by kernel. Then the flags field may have two kinds of flags (some set by user while some set by kernel). Will it mess up the flags space?

Quoted 12 lines

In fact, we could define vfio_dependent_device as:

struct vfio_pci_dependent_device {
	union {
	        __u32   group_id;
		__u32	dev_id;
	}
        __u16   segment;
        __u8    bus;
        __u8    devfn;
};
Quoted 4 lines

If the user calls with the above flag, dev_id is valid, otherwise group_id. Perhaps segment:buus:devfn could still be filled in with a NULL/invalid dev_id if the user doesn't have permissions for the device so that debugging from userspace isn't so opaque. Thanks,

Also, have one question here. Should the invalid dev_id be defined in the vfio uapi or iommufd uapi? Maybe the latter one since dev_id is generated by iommufd subsystem.

Regards,
Yi Liu

Quoted 205 lines
Quoted 4 lines
---
 drivers/vfio/pci/vfio_pci_core.c | 98

++++++++++++++++++++++++++++++++

Review

Quoted 4 lines
 include/uapi/linux/vfio.h        | 47 +++++++++++++++
 2 files changed, 145 insertions(+)
(quoted diff) +0 -0

b/drivers/vfio/pci/vfio_pci_core.c

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct

Quoted 4 lines

index 19f5b075d70a..45edf4e9b98b 100644

drivers/vfio/pci/vfio_pci_core.c +0 -0
@@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct

vfio_pci_core_device *vdev,

Quoted 37 lines
drivers/vfio/pci/vfio_pci_core.c +34 -0
 	return ret;
 }
 
+static struct pci_dev *
+vfio_pci_dev_set_resettable(struct vfio_device_set *dev_set);
+
+static int vfio_pci_ioctl_get_pci_hot_reset_group_info(
+	struct vfio_pci_core_device *vdev,
+	struct vfio_pci_hot_reset_group_info __user *arg)
+{
+	unsigned long minsz =
+		offsetofend(struct vfio_pci_hot_reset_group_info, count);
+	struct vfio_pci_hot_reset_group_info hdr;
+	struct iommufd_ctx *iommufd, *cur_iommufd;
+	u32 count = 0, index = 0, *devices = NULL;
+	struct vfio_pci_core_device *cur;
+	bool slot = false;
+	int ret = 0;
+
+	if (copy_from_user(&hdr, arg, minsz))
+		return -EFAULT;
+
+	if (hdr.argsz < minsz)
+		return -EINVAL;
+
+	hdr.flags = 0;
+
+	/* Can we do a slot or bus reset or neither? */
+	if (!pci_probe_reset_slot(vdev->pdev->slot))
+		slot = true;
+	else if (pci_probe_reset_bus(vdev->pdev->bus))
+		return -ENODEV;
+
+	mutex_lock(&vdev->vdev.dev_set->lock);
+	if (!vfio_pci_dev_set_resettable(vdev->vdev.dev_set)) {
+		ret = -EPERM;
+		goto out_unlock;

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
+	}
+
+	iommufd = vfio_iommufd_physical_ictx(&vdev->vdev);
+	if (!iommufd) {
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	/* How many devices are affected? */
+	ret = vfio_pci_for_each_slot_or_bus(vdev->pdev,

vfio_pci_count_devs,

Quoted 13 lines
drivers/vfio/pci/vfio_pci_core.c +13 -0
+					    &count, slot);
+	if (ret)
+		goto out_unlock;
+
+	WARN_ON(!count); /* Should always be at least one */
+
+	/*
+	 * If there's enough space, fill it now, otherwise return -ENOSPC and
+	 * the number of devices affected.
+	 */
+	if (hdr.argsz < sizeof(hdr) + (count * sizeof(*devices))) {
+		ret = -ENOSPC;
+		hdr.count = count;

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
+		goto reset_info_exit;
+	}
+
+	devices = kcalloc(count, sizeof(*devices), GFP_KERNEL);
+	if (!devices) {
+		ret = -ENOMEM;
+		goto reset_info_exit;
+	}
+
+	list_for_each_entry(cur, &vdev->vdev.dev_set->device_list,

vdev.dev_set_list) {

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,

Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +7 -0
+		cur_iommufd = vfio_iommufd_physical_ictx(&cur->vdev);
+		if (cur->vdev.open_count) {
+			if (cur_iommufd != iommufd) {
+				ret = -EPERM;
+				break;
+			}
+			ret = vfio_iommufd_physical_devid(&cur->vdev,

&devices[index]);

drivers/vfio/pci/vfio_pci_core.c long vfio_pci_core_ioctl(struct vfio_device

Quoted 26 lines
drivers/vfio/pci/vfio_pci_core.c +22 -0
+			if (ret)
+				break;
+			index++;
+		}
+	}
+
+reset_info_exit:
+	if (copy_to_user(arg, &hdr, minsz))
+		ret = -EFAULT;
+
+	if (!ret) {
+		if (copy_to_user(&arg->devices, devices,
+				 hdr.count * sizeof(*devices)))
+			ret = -EFAULT;
+	}
+
+	kfree(devices);
+out_unlock:
+	mutex_unlock(&vdev->vdev.dev_set->lock);
+	return ret;
+}
+
 static int vfio_pci_ioctl_get_pci_hot_reset_info(
 	struct vfio_pci_core_device *vdev,
 	struct vfio_pci_hot_reset_info __user *arg)
@@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device

*core_vdev, unsigned int cmd,

drivers/vfio/pci/vfio_pci_core.c long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,

Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +2 -0
 		return vfio_pci_ioctl_get_irq_info(vdev, uarg);
 	case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO:
 		return vfio_pci_ioctl_get_pci_hot_reset_info(vdev, uarg);
+	case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO:
+		return vfio_pci_ioctl_get_pci_hot_reset_group_info(vdev,

uarg);

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 670

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +0 -0
 	case VFIO_DEVICE_GET_REGION_INFO:
 		return vfio_pci_ioctl_get_region_info(vdev, uarg);
 	case VFIO_DEVICE_IOEVENTFD:
include/uapi/linux/vfio.h +0 -0
index 25432ef213ee..61b801dfd40b 100644
@@ -669,6 +669,53 @@ struct vfio_pci_hot_reset_info {
 
 #define VFIO_DEVICE_GET_PCI_HOT_RESET_INFO	_IO(VFIO_TYPE,

VFIO_BASE + 12)

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 672

Quoted 3 lines
include/uapi/linux/vfio.h +2 -0
+/**
+ * VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO - _IOWR(VFIO_TYPE,

VFIO_BASE + 12,

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 673

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ *						    struct

vfio_pci_hot_reset_group_info)

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {

Quoted 6 lines
include/uapi/linux/vfio.h +6 -0
+ *
+ * This is used in the vfio device cdev mode.  It returns the list of
+ * affected devices (represented by iommufd dev_id) when hot reset is
+ * issued on the current device with which this ioctl is invoked.  It
+ * only includes the devices that are opened by the current user in the
+ * time of this command is invoked.  This list may change when user

opens

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 681

Quoted 2 lines
include/uapi/linux/vfio.h +2 -0
+ * new device or close opened device, hence user should re-invoke it
+ * after open/close devices.  This command has no guarantee on the

result

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 682

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * of VFIO_DEVICE_PCI_HOT_RESET since the not-opened affected device

can

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 683

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * be by other users in the window between the two ioctls.  If the

affected

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 684

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * devices are opened by multiple users, the

VFIO_DEVICE_PCI_HOT_RESET

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 685

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * shall fail, detail can check the description of

VFIO_DEVICE_PCI_HOT_RESET.

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {

Quoted 3 lines
include/uapi/linux/vfio.h +3 -0
+ *
+ * For the users that open vfio group/container, this ioctl is useless as
+ * they have no idea about the iommufd dev_id returned by this ioctl.

For

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 690

Quoted 2 lines
include/uapi/linux/vfio.h +2 -0
+ * the users of the traditional mode vfio group/container, this ioctl will
+ * fail as this mode does not use iommufd hence no dev_id to report

back.

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 691

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * For the users of the iommufd compat mode vfio group/container, this

ioctl

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 692

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * would succeed as this mode uses iommufd as container fd.  But such

users

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 693

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+ * still have no idea about the iommufd dev_id as the dev_id is only

stored

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 695

Quoted 2 lines
include/uapi/linux/vfio.h +2 -0
+ * in kernel in this mode.  For the users of the vfio group/container, the
+ * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO should be used to know the

hot reset

Quoted 11 lines
include/uapi/linux/vfio.h +11 -0
+ * affected devices.
+ *
+ * Return: 0 on success, -errno on failure:
+ *	-enospc = insufficient buffer;
+ *	-enodev = unsupported for device;
+ *	-eperm = no permission for device, this error comes:
+ *		 - when there are affected devices that are opened but
+ *		   not bound to the same iommufd with the current device
+ *		   with which this ioctl is invoked,
+ *		 - there are affected devices that are not bound to vfio
+ *		   driver yet.

include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {

Quoted 10 lines
include/uapi/linux/vfio.h +10 -0
+ *		 - no valid iommufd is bound (e.g. noiommu mode)
+ */
+struct vfio_pci_hot_reset_group_info {
+	__u32	argsz;
+	__u32	flags;
+	__u32	count;
+	__u32	devices[];
+};
+
+#define VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
	_IO(VFIO_TYPE, VFIO_BASE + 18)
Quoted 4 lines
include/uapi/linux/vfio.h +1 -0
+
 /**
  * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13,
  *				    struct vfio_pci_hot_reset)
maintainer
#22 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 10 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AM

Quoted 6 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AM

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a flags arg that isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.

Quoted 6 lines

Sure. I can follow this suggestion. BTW. I have a doubt here. This new flag is set by user. What if in the future kernel has new extensions and needs to report something new to the user and add new flags to tell user? Such flag is set by kernel. Then the flags field may have two kinds of flags (some set by user while some set by kernel). Will it mess up the flags space?

flags in a GET_INFO ioctl is for output.

if user needs to use flags as input to select different type of info then it should be split into multiple GET_INFO cmds.

#23 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 17 lines · Kevin
Quoted 10 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AM

Quoted 6 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AM

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a flags arg that isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.

Quoted 6 lines

Sure. I can follow this suggestion. BTW. I have a doubt here. This new flag is set by user. What if in the future kernel has new extensions and needs to report something new to the user and add new flags to tell user? Such flag is set by kernel. Then the flags field may have two kinds of flags (some set by user while some set by kernel). Will it mess up the flags space?

Quoted 4 lines

flags in a GET_INFO ioctl is for output.

if user needs to use flags as input to select different type of info then it should be split into multiple GET_INFO cmds.

I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.

Alternatively, should VFIO_DEVICE_GET_PCI_HOT_RESET_INFO automatically return the dev_id variant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,

Alex

author
#24 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 32 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PM

Quoted 7 lines · Kevin
Quoted 7 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AM

Quoted 4 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AM

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a flags arg

that

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.

Sure. I can follow this suggestion. BTW. I have a doubt here. This new

flag

Quoted 3 lines
Quoted 3 lines

is set by user. What if in the future kernel has new extensions and needs to report something new to the user and add new flags to tell user? Such flag is set by kernel. Then the flags field may have two kinds of flags

(some

Quoted 6 lines
Quoted 2 lines

set by user while some set by kernel). Will it mess up the flags space?

flags in a GET_INFO ioctl is for output.

if user needs to use flags as input to select different type of info then it

should

Quoted 1 line

be split into multiple GET_INFO cmds.

Quoted 8 lines

I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.

Alternatively, should VFIO_DEVICE_GET_PCI_HOT_RESET_INFO automatically
return the dev_id variant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,

Personally I prefer that user asks for dev_id info explicitly. The major reason that we return dev_id is that the group/bdf info is not enough for the device fd passing case. But if qemu opens device by itself, the group/bdf info is still enough. So a device opened as a cdev doesn't mean it should return dev_id, it depends on if user has the bdf knowledge.

Regards,
Yi Liu

#25 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 41 lines · Yi L
Quoted 32 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PM

Quoted 7 lines · Kevin
Quoted 7 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AM

Quoted 4 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AM

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a flags arg

that

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.

Sure. I can follow this suggestion. BTW. I have a doubt here. This new

flag

Quoted 3 lines
Quoted 3 lines

is set by user. What if in the future kernel has new extensions and needs to report something new to the user and add new flags to tell user? Such flag is set by kernel. Then the flags field may have two kinds of flags

(some

Quoted 6 lines
Quoted 2 lines

set by user while some set by kernel). Will it mess up the flags space?

flags in a GET_INFO ioctl is for output.

if user needs to use flags as input to select different type of info then it

should

Quoted 1 line

be split into multiple GET_INFO cmds.

Quoted 8 lines

I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.

Alternatively, should VFIO_DEVICE_GET_PCI_HOT_RESET_INFO automatically
return the dev_id variant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,

Quoted 5 lines

Personally I prefer that user asks for dev_id info explicitly. The major reason that we return dev_id is that the group/bdf info is not enough for the device fd passing case. But if qemu opens device by itself, the group/bdf info is still enough. So a device opened as a cdev doesn't mean it should return dev_id, it depends on if user has the bdf knowledge.

But if QEMU opens the cdev, vs getting it from the group, does it make any sense to return a set of group-ids + bdf in the host-reset info? I'm inclined to think the answer is no.

Per my previous suggestion, I think we should always return the bdf. We can't know if the user is accessing through an fd they opened themselves or were passed, but it allows an actually useful debugging report if userspace can know "I can't perform a hot reset of this device because my iommufd context doesn't know about device <bdf>", vs an opaque -EPERM. Therefore I think we're only discussing the data conveyed in the u32, a group-id or dev_id. Thanks,

Alex

author
#26 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 61 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:46 PM

Quoted 6 lines · Yi L
Quoted 6 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PM

On Tue, 28 Mar 2023 06:19:06 +0000
"Tian, Kevin" <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AM

Quoted 4 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AM

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a flags

arg

Quoted 5 lines
Quoted 5 lines

that

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.

Sure. I can follow this suggestion. BTW. I have a doubt here. This

new

Quoted 2 lines
Quoted 2 lines

flag

Quoted 1 line
Quoted 1 line

is set by user. What if in the future kernel has new extensions and

needs

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to report something new to the user and add new flags to tell user?

Such

Quoted 15 lines
Quoted 15 lines
Quoted 1 line
Quoted 1 line

flag is set by kernel. Then the flags field may have two kinds of flags

(some

Quoted 6 lines
Quoted 2 lines

set by user while some set by kernel). Will it mess up the flags space?

flags in a GET_INFO ioctl is for output.

if user needs to use flags as input to select different type of info then it

should

Quoted 1 line

be split into multiple GET_INFO cmds.

I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.

Quoted 6 lines
Quoted 4 lines

Alternatively, should VFIO_DEVICE_GET_PCI_HOT_RESET_INFO automatically
return the dev_id variant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,

Personally I prefer that user asks for dev_id info explicitly. The major

reason

Quoted 1 line

that we return dev_id is that the group/bdf info is not enough for the

device

Quoted 1 line

fd passing case. But if qemu opens device by itself, the group/bdf info is

still

Quoted 1 line

enough. So a device opened as a cdev doesn't mean it should return

dev_id,

Quoted 1 line

it depends on if user has the bdf knowledge.

Quoted 7 lines

But if QEMU opens the cdev, vs getting it from the group, does it make any sense to return a set of group-ids + bdf in the host-reset info? I'm inclined to think the answer is no.

Per my previous suggestion, I think we should always return the bdf. We can't know if the user is accessing through an fd they opened themselves or were passed,

Oh, yes. I'm convinced by this reason since only cdev mode supports device fd passing. So I'll reuse the existing _INFO and let kernel set a flag to mark the returned info is dev_id+bdf.

A check. If the device that the _INFIO is invoked is opened via cdev, but there are devices in the dev_set that are got via VFIO_GROUP_GET_DEVICE_FD, should I fail it or allow it?

Quoted 5 lines

but it allows an actually useful debugging
report if userspace can know "I can't perform a hot reset of this device because my iommufd context doesn't know about device <bdf>", vs an opaque -EPERM. Therefore I think we're only discussing the data conveyed in the u32, a group-id or dev_id. Thanks,

Sure.

Regards,
Yi Liu

#27 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 69 lines · Yi L
Quoted 61 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:46 PM

Quoted 6 lines · Yi L
Quoted 6 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PM

On Tue, 28 Mar 2023 06:19:06 +0000
"Tian, Kevin" <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AM

Quoted 4 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AM

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a flags

arg

Quoted 5 lines
Quoted 5 lines

that

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.

Sure. I can follow this suggestion. BTW. I have a doubt here. This

new

Quoted 2 lines
Quoted 2 lines

flag

Quoted 1 line
Quoted 1 line

is set by user. What if in the future kernel has new extensions and

needs

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to report something new to the user and add new flags to tell user?

Such

Quoted 15 lines
Quoted 15 lines
Quoted 1 line
Quoted 1 line

flag is set by kernel. Then the flags field may have two kinds of flags

(some

Quoted 6 lines
Quoted 2 lines

set by user while some set by kernel). Will it mess up the flags space?

flags in a GET_INFO ioctl is for output.

if user needs to use flags as input to select different type of info then it

should

Quoted 1 line

be split into multiple GET_INFO cmds.

I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.

Quoted 6 lines
Quoted 4 lines

Alternatively, should VFIO_DEVICE_GET_PCI_HOT_RESET_INFO automatically
return the dev_id variant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,

Personally I prefer that user asks for dev_id info explicitly. The major

reason

Quoted 1 line

that we return dev_id is that the group/bdf info is not enough for the

device

Quoted 1 line

fd passing case. But if qemu opens device by itself, the group/bdf info is

still

Quoted 1 line

enough. So a device opened as a cdev doesn't mean it should return

dev_id,

Quoted 1 line

it depends on if user has the bdf knowledge.

Quoted 7 lines

But if QEMU opens the cdev, vs getting it from the group, does it make any sense to return a set of group-ids + bdf in the host-reset info? I'm inclined to think the answer is no.

Per my previous suggestion, I think we should always return the bdf. We can't know if the user is accessing through an fd they opened themselves or were passed,

Quoted 7 lines

Oh, yes. I'm convinced by this reason since only cdev mode supports device fd passing. So I'll reuse the existing _INFO and let kernel set a flag to mark the returned info is dev_id+bdf.

A check. If the device that the _INFIO is invoked is opened via cdev, but there are devices in the dev_set that are got via VFIO_GROUP_GET_DEVICE_FD, should I fail it or allow it?

It's a niche case, but I think it needs to be allowed. We'd still report the bdf for those devices, but make use of the invalid/null dev-id. I think this empowers userspace that they could make the same call on a group opened fd if necessary. An alternative would be to redefine the returned data structure entirely with a flag per entry describing the output, but then I think we need to invent a kernel policy of which gets reported, which seems overly complicated if our goal is to phase out group usage. Make sense, or will this bite us? Thanks,

Alex

author
#28 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 87 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 11:18 PM

Quoted 12 lines · Yi L
Quoted 12 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:46 PM

Quoted 6 lines · Yi L
Quoted 6 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PM

On Tue, 28 Mar 2023 06:19:06 +0000
"Tian, Kevin" <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AM

Quoted 4 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AM

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a

flags

Quoted 11 lines
Quoted 11 lines

arg

Quoted 5 lines
Quoted 5 lines

that

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.

Sure. I can follow this suggestion. BTW. I have a doubt here. This

new

Quoted 2 lines
Quoted 2 lines

flag

Quoted 1 line
Quoted 1 line

is set by user. What if in the future kernel has new extensions and

needs

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to report something new to the user and add new flags to tell

user?

Quoted 2 lines
Quoted 2 lines

Such

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

flag is set by kernel. Then the flags field may have two kinds of

flags

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

(some

Quoted 1 line
Quoted 1 line

set by user while some set by kernel). Will it mess up the flags

space?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

flags in a GET_INFO ioctl is for output.

if user needs to use flags as input to select different type of info

then it

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

should

Quoted 1 line

be split into multiple GET_INFO cmds.

I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.

Alternatively, should VFIO_DEVICE_GET_PCI_HOT_RESET_INFO automatically
return the dev_id variant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Personally I prefer that user asks for dev_id info explicitly. The major

reason

Quoted 1 line

that we return dev_id is that the group/bdf info is not enough for the

device

Quoted 1 line

fd passing case. But if qemu opens device by itself, the group/bdf info

is

Quoted 9 lines
Quoted 9 lines

still

Quoted 1 line

enough. So a device opened as a cdev doesn't mean it should return

dev_id,

Quoted 1 line

it depends on if user has the bdf knowledge.

But if QEMU opens the cdev, vs getting it from the group, does it make any sense to return a set of group-ids + bdf in the host-reset info? I'm inclined to think the answer is no.

Quoted 5 lines
Quoted 3 lines

Per my previous suggestion, I think we should always return the bdf. We can't know if the user is accessing through an fd they opened themselves or were passed,

Oh, yes. I'm convinced by this reason since only cdev mode supports

device fd

Quoted 1 line

passing. So I'll reuse the existing _INFO and let kernel set a flag to mark

the returned

Quoted 2 lines

info is dev_id+bdf.

Quoted 7 lines
Quoted 1 line

A check. If the device that the _INFIO is invoked is opened via cdev, but

there

Quoted 1 line

are devices in the dev_set that are got via VFIO_GROUP_GET_DEVICE_FD,

should

Quoted 1 line

I fail it or allow it?

It's a niche case, but I think it needs to be allowed.

I'm also wondering if it is common in the future. Actually, a user should be preferred to either use the group or cdev, but not both. Otherwise, it looks to be bad programming.:-)

Also, as an earlier remark from Jason. If there are affected devices that are opened by other users, the new _INFO should fail with -EPERM. I know this remark was for the new _INFO ioctl. But now, we are going to reuse the existing _INFO, so I'd also want to check if we still need this policy? If yes, then it is a problem to check the owner of the devices that are opened by the group path.

https://lore.kernel.org/kvm/[email protected]/

Quoted 4 lines

We'd still
report the bdf for those devices, but make use of the invalid/null dev-id. I think this empowers userspace that they could make the same call on a group opened fd if necessary.

For the devices opened via group path, it should have an entry that includes invalid_dev_id+bdf. So user can map it to the device. But there is no group_id, this may be fine since group is just a shortcut to find the device. Is it?

Quoted 5 lines

An alternative would be to
redefine the returned data structure entirely with a flag per entry describing the output, but then I think we need to invent a kernel policy of which gets reported, which seems overly complicated if our goal is to phase out group usage. Make sense, or will this bite us?

This does smell complicated. 😊 maybe minimum change is better as the group is going to be phased out.

Regards,
Yi Liu

#29 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 99 lines · Yi L
Quoted 87 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 11:18 PM

Quoted 12 lines · Yi L
Quoted 12 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:46 PM

Quoted 6 lines · Yi L
Quoted 6 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PM

On Tue, 28 Mar 2023 06:19:06 +0000
"Tian, Kevin" <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AM

Quoted 4 lines

From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AM

Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a

flags

Quoted 11 lines
Quoted 11 lines

arg

Quoted 5 lines
Quoted 5 lines

that

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.

Sure. I can follow this suggestion. BTW. I have a doubt here. This

new

Quoted 2 lines
Quoted 2 lines

flag

Quoted 1 line
Quoted 1 line

is set by user. What if in the future kernel has new extensions and

needs

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to report something new to the user and add new flags to tell

user?

Quoted 2 lines
Quoted 2 lines

Such

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

flag is set by kernel. Then the flags field may have two kinds of

flags

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

(some

Quoted 1 line
Quoted 1 line

set by user while some set by kernel). Will it mess up the flags

space?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

flags in a GET_INFO ioctl is for output.

if user needs to use flags as input to select different type of info

then it

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

should

Quoted 1 line

be split into multiple GET_INFO cmds.

I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.

Alternatively, should VFIO_DEVICE_GET_PCI_HOT_RESET_INFO automatically
return the dev_id variant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Personally I prefer that user asks for dev_id info explicitly. The major

reason

Quoted 1 line

that we return dev_id is that the group/bdf info is not enough for the

device

Quoted 1 line

fd passing case. But if qemu opens device by itself, the group/bdf info

is

Quoted 9 lines
Quoted 9 lines

still

Quoted 1 line

enough. So a device opened as a cdev doesn't mean it should return

dev_id,

Quoted 1 line

it depends on if user has the bdf knowledge.

But if QEMU opens the cdev, vs getting it from the group, does it make any sense to return a set of group-ids + bdf in the host-reset info? I'm inclined to think the answer is no.

Quoted 5 lines
Quoted 3 lines

Per my previous suggestion, I think we should always return the bdf. We can't know if the user is accessing through an fd they opened themselves or were passed,

Oh, yes. I'm convinced by this reason since only cdev mode supports

device fd

Quoted 1 line

passing. So I'll reuse the existing _INFO and let kernel set a flag to mark

the returned

Quoted 2 lines

info is dev_id+bdf.

Quoted 7 lines
Quoted 1 line

A check. If the device that the _INFIO is invoked is opened via cdev, but

there

Quoted 1 line

are devices in the dev_set that are got via VFIO_GROUP_GET_DEVICE_FD,

should

Quoted 1 line

I fail it or allow it?

It's a niche case, but I think it needs to be allowed.

I'm also wondering if it is common in the future. Actually, a user should be preferred to either use the group or cdev, but not both. Otherwise, it looks to be bad programming.:-)

Quoted 8 lines

Also, as an earlier remark from Jason. If there are affected devices that are opened by other users, the new _INFO should fail with -EPERM. I know this remark was for the new _INFO ioctl. But now, we are going to reuse the existing _INFO, so I'd also want to check if we still need this policy? If yes, then it is a problem to check the owner of the devices that are opened by the group path.

https://lore.kernel.org/kvm/[email protected]/

Personally I don't like the suggestion to fail with -EPERM if the user doesn't own all the affected devices. This isn't a "probe if I can do a reset" ioctl, it's a "provide information about the devices affected by a reset to know how to call the hot-reset ioctl". We're returning the bdf to the cdev version of this ioctl for exactly this debugging purpose when the devices are not owned, that becomes useless if we give up an return -EPERM if ownership doesn't align.

Quoted 9 lines
Quoted 4 lines

We'd still
report the bdf for those devices, but make use of the invalid/null dev-id. I think this empowers userspace that they could make the same call on a group opened fd if necessary.

For the devices opened via group path, it should have an entry that includes invalid_dev_id+bdf. So user can map it to the device. But there is no group_id, this may be fine since group is just a shortcut to find the device. Is it?

Yes, it could be argued that the group-id itself is superfluous, the user can determine the group via the bdf, but it also aligns with the hot-reset ioctl, which currently requires the group fd. Thanks,

Alex

author
#30 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 3 lines

From: Alex Williamson <[email protected]>
Sent: Wednesday, March 29, 2023 12:00 AM

[...]

Quoted 26 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

A check. If the device that the _INFIO is invoked is opened via cdev,

but

Quoted 2 lines
Quoted 2 lines

there

Quoted 1 line

are devices in the dev_set that are got via

VFIO_GROUP_GET_DEVICE_FD,

Quoted 7 lines
Quoted 4 lines

should

Quoted 1 line

I fail it or allow it?

It's a niche case, but I think it needs to be allowed.

I'm also wondering if it is common in the future. Actually, a user should be preferred to either use the group or cdev, but not both. Otherwise, it

looks

Quoted 3 lines

to be bad programming.:-)

Also, as an earlier remark from Jason. If there are affected devices that

are

Quoted 1 line

opened by other users, the new _INFO should fail with -EPERM. I know

this

Quoted 6 lines

remark was for the new _INFO ioctl. But now, we are going to reuse the existing _INFO, so I'd also want to check if we still need this policy? If yes, then it is a problem to check the owner of the devices that are opened by the group path.

https://lore.kernel.org/kvm/[email protected]/

Quoted 7 lines

Personally I don't like the suggestion to fail with -EPERM if the user doesn't own all the affected devices. This isn't a "probe if I can do a reset" ioctl, it's a "provide information about the devices affected by a reset to know how to call the hot-reset ioctl". We're returning the bdf to the cdev version of this ioctl for exactly this debugging purpose when the devices are not owned, that becomes useless if we give up an return -EPERM if ownership doesn't align.

Jason's suggestion makes sense for returning the case of returning dev_id as dev_id is local to iommufd. If there are devices in the same dev_set are opened by multiple users, multiple iommufd would be used. Then the dev_id would have overlap. e.g. a dev_set has three devices. Device A and B are opened by the current user as cdev, dev_id #1 and #2 are generated. While device C opened by another user as cdev, dev_id #n is generated for it. If dev_id #n happens to be #1, then user gets two info entries that have the same dev_id.

I know the existing _INFO does not have such policy since the group/bdf info are unique in the system. But for the dev_id case, seems really necessary to fail if the dev_set is not only opened by one user. From this angle, will it be good to have two ioctls. Sorry for the back-forth here. ☹

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

We'd still
report the bdf for those devices, but make use of the invalid/null dev-id. I think this empowers userspace that they could make the same call on a group opened fd if necessary.

Quoted 8 lines
Quoted 4 lines

For the devices opened via group path, it should have an entry that includes invalid_dev_id+bdf. So user can map it to the device. But there is no group_id, this may be fine since group is just a shortcut to find the device. Is it?

Yes, it could be argued that the group-id itself is superfluous, the user can determine the group via the bdf, but it also aligns with the hot-reset ioctl, which currently requires the group fd. Thanks,

I see. For the existing _INFO and HOT_RESET ioctl, I have no doubt. Both of them use the group as a short-cut.

Regards,
Yi Liu

maintainer
#31 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 15 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, March 29, 2023 11:14 AM

Quoted 11 lines

From: Alex Williamson <[email protected]>
Sent: Wednesday, March 29, 2023 12:00 AM

Personally I don't like the suggestion to fail with -EPERM if the user doesn't own all the affected devices. This isn't a "probe if I can do a reset" ioctl, it's a "provide information about the devices affected by a reset to know how to call the hot-reset ioctl". We're returning the bdf to the cdev version of this ioctl for exactly this debugging purpose when the devices are not owned, that becomes useless if we give up an return -EPERM if ownership doesn't align.

Quoted 9 lines

Jason's suggestion makes sense for returning the case of returning dev_id as dev_id is local to iommufd. If there are devices in the same dev_set are opened by multiple users, multiple iommufd would be used. Then the dev_id would have overlap. e.g. a dev_set has three devices. Device A and B are opened by the current user as cdev, dev_id #1 and #2 are generated. While device C opened by another user as cdev, dev_id #n is generated for it. If dev_id #n happens to be #1, then user gets two info entries that have the same dev_id.

In Alex's proposal you'll set a invalid dev_id for device C so the user can still get the info for diagnostic purpose instead of seeing an -EPERM error.

btw I found an open about fd pass scheme which may affect the choice here.

In concept even with cdev we still expect the userspace to maintain the group knowledge so it won't inadvertently attempt to assign devices in the same group to different IOAS's. It also needs such knowledge when constructing guest topology.

with fd passed in Qemu has no way to associate the fd to a group.

We could extend bind_iommufd to return the group id or introduce a new ioctl to query it per dev_id.

Once that is in place looks we don't need a new _INFO ioctl?

Thanks
Kevin

#32 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 25 lines · Kevin
Quoted 15 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, March 29, 2023 11:14 AM

Quoted 11 lines

From: Alex Williamson <[email protected]>
Sent: Wednesday, March 29, 2023 12:00 AM

Personally I don't like the suggestion to fail with -EPERM if the user doesn't own all the affected devices. This isn't a "probe if I can do a reset" ioctl, it's a "provide information about the devices affected by a reset to know how to call the hot-reset ioctl". We're returning the bdf to the cdev version of this ioctl for exactly this debugging purpose when the devices are not owned, that becomes useless if we give up an return -EPERM if ownership doesn't align.

Quoted 9 lines

Jason's suggestion makes sense for returning the case of returning dev_id as dev_id is local to iommufd. If there are devices in the same dev_set are opened by multiple users, multiple iommufd would be used. Then the dev_id would have overlap. e.g. a dev_set has three devices. Device A and B are opened by the current user as cdev, dev_id #1 and #2 are generated. While device C opened by another user as cdev, dev_id #n is generated for it. If dev_id #n happens to be #1, then user gets two info entries that have the same dev_id.

Quoted 2 lines

In Alex's proposal you'll set a invalid dev_id for device C so the user can still get the info for diagnostic purpose instead of seeing an -EPERM error.

Yes, we shouldn't be reporting dev_ids outside of the user's iommufd context.

Quoted 8 lines

btw I found an open about fd pass scheme which may affect the choice here.

In concept even with cdev we still expect the userspace to maintain the group knowledge so it won't inadvertently attempt to assign devices in the same group to different IOAS's. It also needs such knowledge when constructing guest topology.

with fd passed in Qemu has no way to associate the fd to a group.

Hmm, QEMU tries to get the group for the device address space in the guest, so finding an existing group with a different address space indeed allows QEMU to know of this conflict since the group is the fundamental unit IOMMU context in the legacy vfio model.

Quoted 2 lines

We could extend bind_iommufd to return the group id or introduce a new ioctl to query it per dev_id.

That would be ironic to go to all this trouble to remove groups from the API only to have them show up here. But with a cdev interface, don't we break that model of conflating isolation and address-ability?

For example, devices within a group cannot be bound to separate iommufds due to lack of isolation, which is handled via DMA ownership, but barring DMA aliasing issues, due to conventional PCI buses or quirks, cdev could allow devices within the same group to be managed by separate IOAS's. So the group information really isn't enough for userspace to infer address space restrictions with cdev anyway.

Therefore aren't we expecting this to be denied at attach_ioas() and QEMU shouldn't be making these sorts of assumptions for cdev anyway? Thanks,

Alex

maintainer
#33 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 5 lines · Alex Williamson
Quoted 2 lines

We could extend bind_iommufd to return the group id or introduce a new ioctl to query it per dev_id.

That would be ironic to go to all this trouble to remove groups from the API only to have them show up here.

Groups always had to be part of the API for advanced cases like qemu - the point was to make them a small side bit of information not front and center in control of everything.

Quoted 5 lines

For example, devices within a group cannot be bound to separate iommufds due to lack of isolation, which is handled via DMA ownership, but barring DMA aliasing issues, due to conventional PCI buses or quirks, cdev could allow devices within the same group to be managed by separate IOAS's.

Maybe some future kernel could do this, the API allows it at least..

Quoted 5 lines

So the group information really isn't enough for userspace to infer address space restrictions with cdev anyway.

Therefore aren't we expecting this to be denied at attach_ioas() and QEMU shouldn't be making these sorts of assumptions for cdev anyway?

I guess we could make an API specifically to report same-iommu_domina information?

I was assuming qemu would use the group for now as I don't see a likely future when we would relax that restriction.. So I was keeping a "add it when we need it" attitude here.

Jason

maintainer
#34 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 8 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, March 29, 2023 11:58 PM

Quoted 3 lines · Alex Williamson
Quoted 2 lines

We could extend bind_iommufd to return the group id or introduce a new ioctl to query it per dev_id.

Quoted 6 lines
Quoted 2 lines

That would be ironic to go to all this trouble to remove groups from the API only to have them show up here.

Groups always had to be part of the API for advanced cases like qemu - the point was to make them a small side bit of information not front and center in control of everything.

Agree.

Quoted 15 lines
Quoted 6 lines

For example, devices within a group cannot be bound to separate iommufds due to lack of isolation, which is handled via DMA ownership, but barring DMA aliasing issues, due to conventional PCI buses or quirks, cdev could allow devices within the same group to be managed by separate IOAS's.

Maybe some future kernel could do this, the API allows it at least..

Quoted 5 lines

So the group information really isn't enough for userspace to infer address space restrictions with cdev anyway.

Therefore aren't we expecting this to be denied at attach_ioas() and QEMU shouldn't be making these sorts of assumptions for cdev anyway?

Quoted 7 lines

I guess we could make an API specifically to report same-iommu_domina information?

I was assuming qemu would use the group for now as I don't see a likely future when we would relax that restriction.. So I was keeping a "add it when we need it" attitude here.

IIRC we discussed this subgroup concept in the thread of reviewing my high level design proposal 2yrs ago. The consensus at the moment was that subgroup is architecturally allowed w/o DMA aliasing issues but we're yet to see a real demand of relaxing current group restriction to support it. Also with time moving newer platforms should have less multi-devices group so the need of subgroup is further decreased.

So I'm also inclined to laying the existing group restriction with cdev for now.

Then can we make a decision how this group_id might be reported?

In nesting series we'll have a GET_INFO ioctl per dev_id. It could be extended to report group_id too.

Or alternatively just return it in BIND_IOMMUFD together with dev_id.

maintainer
#35 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 2 lines · Kevin

In nesting series we'll have a GET_INFO ioctl per dev_id. It could be extended to report group_id too.

I like the GET_INFO because it would work with VDPA, is it possible?

Jason

author
#36 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

In reply to Alex Williamson
Quoted 31 lines

From: Alex Williamson <[email protected]>
Sent: Wednesday, March 29, 2023 11:50 PM

Quoted 25 lines · Kevin
Quoted 24 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, March 29, 2023 11:14 AM

Quoted 11 lines

From: Alex Williamson <[email protected]>
Sent: Wednesday, March 29, 2023 12:00 AM

Personally I don't like the suggestion to fail with -EPERM if the user doesn't own all the affected devices. This isn't a "probe if I can do a reset" ioctl, it's a "provide information about the devices affected by a reset to know how to call the hot-reset ioctl". We're returning the bdf to the cdev version of this ioctl for exactly this debugging purpose when the devices are not owned, that becomes useless if we give up an return -EPERM if ownership doesn't align.

Jason's suggestion makes sense for returning the case of returning dev_id as dev_id is local to iommufd. If there are devices in the same dev_set are opened by multiple users, multiple iommufd would be used. Then the dev_id would have overlap. e.g. a dev_set has three devices. Device A and B are opened by the current user as cdev, dev_id #1 and #2 are generated. While device C opened by another user as cdev, dev_id #n is generated for it. If dev_id #n happens to be #1, then user gets two info entries that have the same dev_id.

Quoted 5 lines
Quoted 2 lines

In Alex's proposal you'll set a invalid dev_id for device C so the user can still get the info for diagnostic purpose instead of seeing an -EPERM error.

Yes, we shouldn't be reporting dev_ids outside of the user's iommufd context.

Following Alex's suggestion, here are two commits to extend existing _INFO to report dev_id.

From ad5c81366813c5effd707a0b5f5e797f5fdb3115 Mon Sep 17 00:00:00 2001
From: Yi Liu <[email protected]>
Date: Thu, 30 Mar 2023 05:29:36 -0700
Subject: [PATCH] vfio: Mark cdev usage in vfio_device

There are users that needs to check if vfio_device is opened as cdev. e.g. vfio-pci.

---
 drivers/vfio/device_cdev.c |  2 ++
 include/linux/vfio.h       | 14 ++++++++++++++
 2 files changed, 16 insertions(+)
drivers/vfio/device_cdev.c +2 -0
index b5de997bff6d..56f3bbe34680 100644
@@ -66,6 +66,7 @@ void vfio_device_cdev_close(struct vfio_device_file *df)
 		return;
 
 	mutex_lock(&device->dev_set->lock);
+	device->cdev_opened = false;
 	vfio_device_close(df);
 	vfio_device_put_kvm(device);
 	if (df->iommufd)
@@ -180,6 +181,7 @@ long vfio_device_ioctl_bind_iommufd(struct vfio_device_file *df,
 	 * read/write/mmap
 	 */
 	smp_store_release(&df->access_granted, true);
+	device->cdev_opened = true;
 	mutex_unlock(&device->dev_set->lock);
 
 	return 0;
include/linux/vfio.h +14 -0
index 1367605d617c..86efc1640940 100644
@@ -58,6 +58,7 @@ struct vfio_device {
 	struct device device;	/* device.kref covers object life circle */
 #if IS_ENABLED(CONFIG_VFIO_DEVICE_CDEV)
 	struct cdev cdev;
+	bool cdev_opened;
 #endif
 	refcount_t refcount;	/* user count on registered device*/
 	unsigned int open_count;
@@ -167,6 +168,19 @@ static inline int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id)
 	((void (*)(struct vfio_device *vdev)) NULL)
 #endif
 
+#if IS_ENABLED(CONFIG_VFIO_DEVICE_CDEV)
+static inline bool vfio_device_cdev_opened(struct vfio_device *device)
+{
+	lockdep_assert_held(&device->dev_set->lock);
+	return device->cdev_opened;
+}
+#else
+static inline bool vfio_device_cdev_opened(struct vfio_device *device)
+{
+	return false;
+}
+#endif
+
 /**
  * @migration_set_state: Optional callback to change the migration state for
  *         devices that support migration. It's mandatory for
Signature · 15 lines
2.34.1


From f796cafd6c51e49adcf76352dc1daf14712c3a48 Mon Sep 17 00:00:00 2001
From: Yi Liu <[email protected]>
Date: Thu, 30 Mar 2023 05:44:45 -0700
Subject: [PATCH] vfio/pci: Report dev_id in VFIO_DEVICE_GET_PCI_HOT_RESET_INFO

for the devices opened as cdev.

Signed-off-by: Yi Liu <[email protected]>
---
 drivers/vfio/pci/vfio_pci_core.c | 59 ++++++++++++++++++++++++++++----
 include/uapi/linux/vfio.h        |  6 +++-
 2 files changed, 57 insertions(+), 8 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +51 -7

106 lines not on this page · raw message

include/uapi/linux/vfio.h +5 -1
index 53c72e26ecd3..3fcbc84d51ba 100644
@@ -743,7 +743,10 @@ enum {
  *	-enospc = insufficient buffer, -enodev = unsupported for device.
  */
 struct vfio_pci_dependent_device {
-	__u32	group_id;
+	union {
+		__u32   group_id;
+		__u32	dev_id;
+	};
 	__u16	segment;
 	__u8	bus;
 	__u8	devfn; /* Use PCI_SLOT/PCI_FUNC */
@@ -752,6 +755,7 @@ struct vfio_pci_dependent_device {
 struct vfio_pci_hot_reset_info {
 	__u32	argsz;
 	__u32	flags;
+#define VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID	(1 << 0)
 	__u32	count;
 	struct vfio_pci_dependent_device	devices[];
 };
Signature · 4 lines
2.34.1

Regards,
Yi Liu
author
#37 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 12 lines

From: Liu, Yi L <[email protected]>
Sent: Thursday, March 30, 2023 8:48 PM

drivers/vfio/pci/vfio_pci_core.c +2 -1
 	if (fill->cur == fill->max)
 		return -EAGAIN; /* Something changed, try again */
@@ -791,7 +812,24 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)
 	if (!iommu_group)
 		return -EPERM; /* Cannot reset non-isolated devices */
 
-	fill->devices[fill->cur].group_id = iommu_group_id(iommu_group);
+	/*
+	 * If dev_id is needed, fill in the dev_id field, otherwise

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 826

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
+	 * fill in group_id.
+	 */
+	if (fill->require_devid) {
+		/*
+		 * Report the devices that are opened as cdev and have
+		 * the same iommufd with the fill->iommufd.  Otherwise,
+		 * just fill in an IOMMUFD_INVALID_ID.
+		 */
+		vdev = vfio_pci_find_device_in_devset(dev_set, pdev);
+		if (vdev && !vfio_device_cdev_opened(vdev) &&

a typo..it should be

		if (vdev && vfio_device_cdev_opened(vdev) &&
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +6 -0
+		    fill->iommufd == vfio_iommufd_physical_ictx(vdev))
+			vfio_iommufd_physical_devid(vdev, &fill->devices[fill-
 >cur].dev_id);
+		fill->devices[fill->cur].dev_id = IOMMUFD_INVALID_ID;
+	} else {
+		fill->devices[fill->cur].group_id = iommu_group_id(iommu_group);
+	}
 	fill->devices[fill->cur].segment = pci_domain_nr(pdev->bus);
 	fill->devices[fill->cur].bus = pdev->bus->number;

fill->devices[fill->cur].devfn = pdev->devfn;

Regards,
Yi Liu

maintainer
#38 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

In reply to Liu, Yi L
Quoted 5 lines · Yi L
drivers/vfio/pci/vfio_pci_core.c +5 -0
+	/*
+	 * If dev_id is needed, fill in the dev_id field, otherwise
+	 * fill in group_id.
+	 */
+	if (fill->require_devid) {

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 829

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
+		/*
+		 * Report the devices that are opened as cdev and have
+		 * the same iommufd with the fill->iommufd.  Otherwise,
+		 * just fill in an IOMMUFD_INVALID_ID.
+		 */
+		vdev = vfio_pci_find_device_in_devset(dev_set, pdev);
+		if (vdev && !vfio_device_cdev_opened(vdev) &&
+		    fill->iommufd == vfio_iommufd_physical_ictx(vdev))
+			vfio_iommufd_physical_devid(vdev, &fill->devices[fill->cur].dev_id);
+		fill->devices[fill->cur].dev_id = IOMMUFD_INVALID_ID;

This needs an else?

I suggest to check for VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID on input as well. I know the old kernels don't enforce this but at least we could start enforcing it going forward so that the group path would reject it to catch userspace bugs.

May as well fix it up to fully validate the flags

Jason

#39 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 19 lines · On Thu, 30 Mar 2023 19:44:55 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 5 lines · Yi L
drivers/vfio/pci/vfio_pci_core.c +5 -0
+	/*
+	 * If dev_id is needed, fill in the dev_id field, otherwise
+	 * fill in group_id.
+	 */
+	if (fill->require_devid) {

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 829

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
+		/*
+		 * Report the devices that are opened as cdev and have
+		 * the same iommufd with the fill->iommufd.  Otherwise,
+		 * just fill in an IOMMUFD_INVALID_ID.
+		 */
+		vdev = vfio_pci_find_device_in_devset(dev_set, pdev);
+		if (vdev && !vfio_device_cdev_opened(vdev) &&
+		    fill->iommufd == vfio_iommufd_physical_ictx(vdev))
+			vfio_iommufd_physical_devid(vdev, &fill->devices[fill->cur].dev_id);
+		fill->devices[fill->cur].dev_id = IOMMUFD_INVALID_ID;  

This needs an else?

Quoted 6 lines

I suggest to check for VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID on input as well. I know the old kernels don't enforce this but at least we could start enforcing it going forward so that the group path would reject it to catch userspace bugs.

May as well fix it up to fully validate the flags

Is this under the guise of "if nobody complains it's ok, otherwise revert" plan? We report dev-id based on the nature of the device, not the provided flags, so I'm not sure I follow how this protects the group path, unless we've failed to clear the output flags on that path with this change. Thanks,

Alex

maintainer
#40 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 9 lines · Alex Williamson
Quoted 6 lines

I suggest to check for VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID on input as well. I know the old kernels don't enforce this but at least we could start enforcing it going forward so that the group path would reject it to catch userspace bugs.

May as well fix it up to fully validate the flags

Is this under the guise of "if nobody complains it's ok, otherwise revert" plan?

Yah, assuming people don't pass in uninited structs.

Jason

maintainer
#41 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

In reply to Tian, Kevin
Quoted 4 lines · Kevin

We could extend bind_iommufd to return the group id or introduce a new ioctl to query it per dev_id.

Once that is in place looks we don't need a new _INFO ioctl?

The iommu_group and the reset group are different things

The issue is processing the BDF strings, not the group ID.

Probably we should have some way for iommufd to report the group_id from the dev_id?

Jason

maintainer
#42 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 8 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, March 29, 2023 11:50 PM

Quoted 2 lines · Kevin

We could extend bind_iommufd to return the group id or introduce a new ioctl to query it per dev_id.

Quoted 9 lines
Quoted 1 line

Once that is in place looks we don't need a new _INFO ioctl?

The iommu_group and the reset group are different things

The issue is processing the BDF strings, not the group ID.

Probably we should have some way for iommufd to report the group_id from the dev_id?

Yes, that is my thought. Though iommu_group and reset group are different things we could still leverage existing _INFO ioctl once there is a way to associated dev_id to group_id.

maintainer
#43 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 17 lines

From: Tian, Kevin <[email protected]>
Sent: Thursday, March 30, 2023 9:10 AM

Quoted 14 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, March 29, 2023 11:50 PM

Quoted 4 lines · Kevin

We could extend bind_iommufd to return the group id or introduce a new ioctl to query it per dev_id.

Once that is in place looks we don't need a new _INFO ioctl?

The iommu_group and the reset group are different things

The issue is processing the BDF strings, not the group ID.

Quoted 7 lines
Quoted 3 lines

Probably we should have some way for iommufd to report the group_id from the dev_id?

Yes, that is my thought. Though iommu_group and reset group are different things we could still leverage existing _INFO ioctl once there is a way to associated dev_id to group_id.

Please ignore this comment. Yes they are different things so even if a dev_id is in a group_id reported on a reset BDF string it doesn't mean this dev_id is in the reset group.

Qemu can know that all affected devices are either owned by itself or not used by other processes if dev_id's opened by itself can be associated to all group_id's reported in the BDF strings. But it still lacks of information to tell the reset dependency within those opened devices within Qemu.

So we do need a new _INFO ioctl for cdev. :/

maintainer
#44 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

In reply to Alex Williamson
Quoted 3 lines · Alex Williamson

It's a niche case, but I think it needs to be allowed. We'd still report the bdf for those devices, but make use of the invalid/null dev-id.

IDK, it makes the whole implementation much more complicated. Instead of just copying the current dev_set to the output and calling vfio_pci_dev_set_resettable() we need to do something more complex..

Keeping the current ioctl as-is means this IOCTL can be used to do any debugging by getting the actual BDF list.

It means we can make the a new ioctl simple and just return the dev_id array without these edge complications. I don't think merging two different ioctls is helping make things simple..

It seems like it does what qemu wants: call the new IOCTL, if it fails, call the old IOCTL and print out the BDF list to help debug and then exit.

On success use the data in the new ioctl to generate the machine configuration to pass the reset grouping into the VM.

When reset actually comes in just trigger it.

Jason

#45 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 10 lines · On Tue, 28 Mar 2023 13:29:23 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 3 lines · Alex Williamson

It's a niche case, but I think it needs to be allowed. We'd still report the bdf for those devices, but make use of the invalid/null dev-id.

IDK, it makes the whole implementation much more complicated. Instead of just copying the current dev_set to the output and calling vfio_pci_dev_set_resettable() we need to do something more complex..

Quoted 6 lines

Keeping the current ioctl as-is means this IOCTL can be used to do any debugging by getting the actual BDF list.

It means we can make the a new ioctl simple and just return the dev_id array without these edge complications. I don't think merging two different ioctls is helping make things simple..

OTOH, we already have an ioctl that essentially "does the right thing", we just want to swap out one return field for another. So which is more complicated, adding another ioctl that does not quite the same thing but still needing to maintain the old ioctl for detailed information, or making the old ioctl bi-modal to return the appropriate information for the type of device used to access it?

Quoted 3 lines

It seems like it does what qemu wants: call the new IOCTL, if it fails, call the old IOCTL and print out the BDF list to help debug and then exit.

Userspace is already dealing with a variable length array for the return value, why would it ever want to repeat that process to get debugging info. Besides, wouldn't QEMU prefer the similarity of making the same call for groups and cdev and simply keying on the data type of one field?

Quoted 4 lines

On success use the data in the new ioctl to generate the machine configuration to pass the reset grouping into the VM.

When reset actually comes in just trigger it.

"Just trigger it" is the same in either case. It seems bold to play the complexity argument when we already have a function that does 90% the correct thing where we can share much of the implementation and userspace code without duplicating, but still relying on a legacy interface for debugging. Thanks,

Alex

maintainer
#46 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 5 lines · Alex Williamson

"Just trigger it" is the same in either case. It seems bold to play the complexity argument when we already have a function that does 90% the correct thing where we can share much of the implementation and userspace code without duplicating, but still relying on a legacy interface for debugging.

It just doesn't seem like we are sharing a lot, this patch is a whole new function and you are saying the unique implementation needs to be more complex still.

Maybe the next version will be able to share more??

Like can we just patch the existing code that sets the group_id/dev_id or somethiing???

Still, I'm not sure that qemu will share really share much here, because if it runs in group mode then it has to parse the result in a totally different way than if it runs in dev_id mode. The ioctl call is only a line or two. I imagine qemu will end up with two different functions that both return some kind of internal list of qemu devices in the reset group.

Jason

maintainer
#47 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

In reply to Yi Liu

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1248

Quoted 5 lines · Yi Liu
drivers/vfio/pci/vfio_pci_core.c +5 -0
+	devices = kcalloc(count, sizeof(*devices), GFP_KERNEL);
+	if (!devices) {
+		ret = -ENOMEM;
+		goto reset_info_exit;
+	}

This doesn't need to be so complicated

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1257

Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +8 -0
+	list_for_each_entry(cur, &vdev->vdev.dev_set->device_list, vdev.dev_set_list) {
+		cur_iommufd = vfio_iommufd_physical_ictx(&cur->vdev);
+		if (cur->vdev.open_count) {
+			if (cur_iommufd != iommufd) {
+				ret = -EPERM;
+				break;
+			}
+			ret = vfio_iommufd_physical_devid(&cur->vdev, &devices[index]);

u32 device;

if (index >= hdr.count)
return -ENOSPC;

ret = vfio_iommufd_physical_devid(&cur->vdev, &devices);
...

if (put_user(&arg->devices[index], device))
-EFAULT

index++;

Jason

author
#48 lore raw

Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 25 lines

From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 28, 2023 8:40 PM

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1248

Quoted 5 lines · Yi Liu
drivers/vfio/pci/vfio_pci_core.c +5 -0
+	devices = kcalloc(count, sizeof(*devices), GFP_KERNEL);
+	if (!devices) {
+		ret = -ENOMEM;
+		goto reset_info_exit;
+	}

This doesn't need to be so complicated

Review

Quoted 1 line
(quoted diff) +1 -0
+	list_for_each_entry(cur, &vdev->vdev.dev_set->device_list,

vdev.dev_set_list) {

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,

Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +7 -0
+		cur_iommufd = vfio_iommufd_physical_ictx(&cur->vdev);
+		if (cur->vdev.open_count) {
+			if (cur_iommufd != iommufd) {
+				ret = -EPERM;
+				break;
+			}
+			ret = vfio_iommufd_physical_devid(&cur->vdev,

&devices[index]);

Quoted 6 lines

u32 device;

if (index >= hdr.count)
return -ENOSPC;

ret = vfio_iommufd_physical_devid(&cur->vdev, &devices);

Ok, so the whole point is that if cur->vdev->iommufd_ctx==null, then the vfio_iommufd_physical_devid() shall fail as well.

Quoted 3 lines

...

if (put_user(&arg->devices[index], device))

Will modify it. let's close the "separate ioctl" v.s. "reuse ioctl + new flag" open with Alex first.

Thanks,
Yi Liu

Quoted 5 lines

-EFAULT

index++;

Jason

#49 lore raw

[Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)

In reply to Yi Liu
== Series Details ==

== Series Details ==

Series: Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)
URL : https://patchwork.freedesktop.org/series/115264/
State : warning

== Summary ==

Error: dim checkpatch failed
498fb1274097 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() 09b95d011880 vfio/pci: Only check ownership of opened devices in hot reset
a93e0e52bde9 vfio/pci: Move the existing hot reset logic to be a helper
-:6: WARNING:COMMIT_LOG_LONG_LINE: Possible unwrapped commit description (prefer a maximum 75 chars per line) #6:
This prepares to add another method for hot reset. The major hot reset logic

total: 0 errors, 1 warnings, 0 checks, 100 lines checked
6bed441dc040 vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device 617ce05d9e70 vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET 4302e450c3d2 vfio: Refine vfio file kAPIs for vfio PCI hot reset b87361e1e7b8 vfio: Accpet device file from vfio PCI hot reset path b86ec489e81d vfio/pci: Renaming for accepting device fd in hot reset path -:52: CHECK:PREFER_KERNEL_TYPES: Prefer kernel type 's32' over 'int32_t' #52: FILE: drivers/vfio/pci/vfio_pci_core.c:1265:
+ int32_t *fds;

total: 0 errors, 0 warnings, 1 checks, 127 lines checked 1dbe4a480fac vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl fb6e32ec2ecf vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO -:43: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #43: FILE: drivers/vfio/pci/vfio_pci_core.c:1187:
+static int vfio_pci_ioctl_get_pci_hot_reset_group_info(

total: 0 errors, 0 warnings, 1 checks, 163 lines checked

#50 lore raw

[Intel-gfx] ✓ Fi.CI.BAT: success for Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)

In reply to Yi Liu
== Series Details ==

== Series Details ==

Series: Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)
URL : https://patchwork.freedesktop.org/series/115264/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_12918 -> Patchwork_115264v2
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/index.html

Participating hosts (37 -> 36)
------------------------------

Missing (1): fi-kbl-soraka

Known issues
------------

Here are the changes found in Patchwork_115264v2 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@i915_pm_rpm@module-reload:
- bat-dg1-7: [PASS][1] -> [SKIP][2] ([i915#7855])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/bat-dg1-7/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/bat-dg1-7/igt@[email protected]

* igt@i915_selftest@live@reset:
- bat-rpls-1: [PASS][3] -> [ABORT][4] ([i915#4983])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/bat-rpls-1/igt@i915_selftest@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/bat-rpls-1/igt@i915_selftest@[email protected]

* igt@i915_suspend@basic-s2idle-without-i915:
- fi-rkl-11600: [PASS][5] -> [FAIL][6] ([fdo#103375])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/fi-rkl-11600/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/fi-rkl-11600/igt@[email protected]

* igt@kms_chamelium_hpd@common-hpd-after-suspend:
- fi-kbl-8809g: NOTRUN -> [SKIP][7] ([fdo#109271]) +1 similar issue
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/fi-kbl-8809g/igt@[email protected]

* igt@kms_pipe_crc_basic@read-crc:
- bat-dg2-11: NOTRUN -> [SKIP][8] ([i915#5354])
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/bat-dg2-11/igt@[email protected]

#### Possible fixes ####

* igt@i915_suspend@basic-s2idle-without-i915:
- fi-kbl-8809g: [ABORT][9] -> [PASS][10]
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/fi-kbl-8809g/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/fi-kbl-8809g/igt@[email protected]

#### Warnings ####

* igt@i915_selftest@live@slpc:
- bat-rpls-2: [DMESG-FAIL][11] ([i915#6997] / [i915#7913]) -> [DMESG-FAIL][12] ([i915#6367] / [i915#7913])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/bat-rpls-2/igt@i915_selftest@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/bat-rpls-2/igt@i915_selftest@[email protected]

[fdo#103375]: https://bugs.freedesktop.org/show_bug.cgi?id=103375
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
[i915#5354]: https://gitlab.freedesktop.org/drm/intel/issues/5354
[i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
[i915#6997]: https://gitlab.freedesktop.org/drm/intel/issues/6997
[i915#7855]: https://gitlab.freedesktop.org/drm/intel/issues/7855
[i915#7913]: https://gitlab.freedesktop.org/drm/intel/issues/7913

Build changes
-------------

* Linux: CI_DRM_12918 -> Patchwork_115264v2

CI-20190529: 20190529
CI_DRM_12918: a1cb2211899ba6b8fe078586d0878aa918a5aab3 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_7220: 3eb7beb5c03343b29556025b1ada4b50849b5976 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
Patchwork_115264v2: a1cb2211899ba6b8fe078586d0878aa918a5aab3 @ git://anongit.freedesktop.org/gfx-ci/linux

### Linux commits

777a4de3d017 vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO 2a92707f4cc4 vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl 576175a255f8 vfio/pci: Renaming for accepting device fd in hot reset path 91c8b208c83f vfio: Accpet device file from vfio PCI hot reset path 9fe81a8d19f6 vfio: Refine vfio file kAPIs for vfio PCI hot reset 86cef152607b vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET ee577bf22ada vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device 6fb819b4ba9f vfio/pci: Move the existing hot reset logic to be a helper 6b5f8efac8f9 vfio/pci: Only check ownership of opened devices in hot reset de49aaf53338 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset()

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/index.html

#51 lore raw

[Intel-gfx] ✓ Fi.CI.IGT: success for Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)

In reply to Yi Liu
== Series Details ==

== Series Details ==

Series: Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)
URL : https://patchwork.freedesktop.org/series/115264/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_12918_full -> Patchwork_115264v2_full
====================================================

Summary
-------

**SUCCESS**

No regressions found.

Participating hosts (7 -> 7)
------------------------------

Additional (1): shard-rkl0
Missing (1): shard-rkl

Known issues
------------

Here are the changes found in Patchwork_115264v2_full that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@gem_exec_fair@basic-pace-share@rcs0:
- shard-apl: [PASS][1] -> [FAIL][2] ([i915#2842])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-apl3/igt@gem_exec_fair@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-apl1/igt@gem_exec_fair@[email protected]

* igt@gen9_exec_parse@allowed-single:
- shard-glk: [PASS][3] -> [ABORT][4] ([i915#5566])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-glk9/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-glk9/igt@[email protected]

* igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions:
- shard-glk: [PASS][5] -> [FAIL][6] ([i915#2346])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-glk4/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-glk2/igt@[email protected]

* igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size:
- shard-apl: [PASS][7] -> [FAIL][8] ([i915#2346]) +1 similar issue
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-apl6/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size.html
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-apl6/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size.html

#### Possible fixes ####

* igt@gem_eio@hibernate:
- {shard-tglu}: [ABORT][9] ([i915#7975]) -> [PASS][10]
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-tglu-10/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-tglu-9/igt@[email protected]

* igt@gem_exec_fair@basic-pace-solo@rcs0:
- shard-glk: [FAIL][11] ([i915#2842]) -> [PASS][12]
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-glk6/igt@gem_exec_fair@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-glk9/igt@gem_exec_fair@[email protected]

* igt@i915_pm_rpm@dpms-mode-unset-lpsp:
- {shard-tglu}: [SKIP][13] ([i915#1397]) -> [PASS][14]
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-tglu-9/igt@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-tglu-8/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-gtt:
- {shard-tglu}: [SKIP][15] ([i915#1849]) -> [PASS][16] +7 similar issues
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-tglu-9/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-gtt.html
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-tglu-8/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-gtt.html

* igt@kms_vblank@invalid:
- {shard-tglu}: [SKIP][17] ([i915#1845] / [i915#7651]) -> [PASS][18] +9 similar issues
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-tglu-9/igt@[email protected]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-tglu-8/igt@[email protected]

* igt@kms_vblank@pipe-b-ts-continuation-modeset-rpm:
- {shard-tglu}: [SKIP][19] ([i915#1845]) -> [PASS][20] +18 similar issues
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-tglu-10/igt@[email protected]
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-tglu-5/igt@[email protected]

{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).

[fdo#109274]: https://bugs.freedesktop.org/show_bug.cgi?id=109274
[fdo#109280]: https://bugs.freedesktop.org/show_bug.cgi?id=109280
[fdo#109289]: https://bugs.freedesktop.org/show_bug.cgi?id=109289
[fdo#109314]: https://bugs.freedesktop.org/show_bug.cgi?id=109314
[fdo#109315]: https://bugs.freedesktop.org/show_bug.cgi?id=109315
[fdo#109506]: https://bugs.freedesktop.org/show_bug.cgi?id=109506
[fdo#110189]: https://bugs.freedesktop.org/show_bug.cgi?id=110189
[fdo#111068]: https://bugs.freedesktop.org/show_bug.cgi?id=111068
[fdo#111615]: https://bugs.freedesktop.org/show_bug.cgi?id=111615
[fdo#111825]: https://bugs.freedesktop.org/show_bug.cgi?id=111825
[fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
[fdo#112283]: https://bugs.freedesktop.org/show_bug.cgi?id=112283
[i915#1072]: https://gitlab.freedesktop.org/drm/intel/issues/1072
[i915#1397]: https://gitlab.freedesktop.org/drm/intel/issues/1397
[i915#1755]: https://gitlab.freedesktop.org/drm/intel/issues/1755
[i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
[i915#1849]: https://gitlab.freedesktop.org/drm/intel/issues/1849
[i915#2346]: https://gitlab.freedesktop.org/drm/intel/issues/2346
[i915#2527]: https://gitlab.freedesktop.org/drm/intel/issues/2527
[i915#2575]: https://gitlab.freedesktop.org/drm/intel/issues/2575
[i915#2587]: https://gitlab.freedesktop.org/drm/intel/issues/2587
[i915#2672]: https://gitlab.freedesktop.org/drm/intel/issues/2672
[i915#2681]: https://gitlab.freedesktop.org/drm/intel/issues/2681
[i915#2705]: https://gitlab.freedesktop.org/drm/intel/issues/2705
[i915#2842]: https://gitlab.freedesktop.org/drm/intel/issues/2842
[i915#2856]: https://gitlab.freedesktop.org/drm/intel/issues/2856
[i915#3116]: https://gitlab.freedesktop.org/drm/intel/issues/3116
[i915#3281]: https://gitlab.freedesktop.org/drm/intel/issues/3281
[i915#3282]: https://gitlab.freedesktop.org/drm/intel/issues/3282
[i915#3299]: https://gitlab.freedesktop.org/drm/intel/issues/3299
[i915#3359]: https://gitlab.freedesktop.org/drm/intel/issues/3359
[i915#3458]: https://gitlab.freedesktop.org/drm/intel/issues/3458
[i915#3469]: https://gitlab.freedesktop.org/drm/intel/issues/3469
[i915#3539]: https://gitlab.freedesktop.org/drm/intel/issues/3539
[i915#3555]: https://gitlab.freedesktop.org/drm/intel/issues/3555
[i915#3591]: https://gitlab.freedesktop.org/drm/intel/issues/3591
[i915#3637]: https://gitlab.freedesktop.org/drm/intel/issues/3637
[i915#3638]: https://gitlab.freedesktop.org/drm/intel/issues/3638
[i915#3689]: https://gitlab.freedesktop.org/drm/intel/issues/3689
[i915#3886]: https://gitlab.freedesktop.org/drm/intel/issues/3886
[i915#3938]: https://gitlab.freedesktop.org/drm/intel/issues/3938
[i915#3952]: https://gitlab.freedesktop.org/drm/intel/issues/3952
[i915#4077]: https://gitlab.freedesktop.org/drm/intel/issues/4077
[i915#4078]: https://gitlab.freedesktop.org/drm/intel/issues/4078
[i915#4083]: https://gitlab.freedesktop.org/drm/intel/issues/4083
[i915#4103]: https://gitlab.freedesktop.org/drm/intel/issues/4103
[i915#4212]: https://gitlab.freedesktop.org/drm/intel/issues/4212
[i915#4213]: https://gitlab.freedesktop.org/drm/intel/issues/4213
[i915#4270]: https://gitlab.freedesktop.org/drm/intel/issues/4270
[i915#433]: https://gitlab.freedesktop.org/drm/intel/issues/433
[i915#4538]: https://gitlab.freedesktop.org/drm/intel/issues/4538
[i915#4579]: https://gitlab.freedesktop.org/drm/intel/issues/4579
[i915#4771]: https://gitlab.freedesktop.org/drm/intel/issues/4771
[i915#4812]: https://gitlab.freedesktop.org/drm/intel/issues/4812
[i915#4833]: https://gitlab.freedesktop.org/drm/intel/issues/4833
[i915#4852]: https://gitlab.freedesktop.org/drm/intel/issues/4852
[i915#4859]: https://gitlab.freedesktop.org/drm/intel/issues/4859
[i915#4860]: https://gitlab.freedesktop.org/drm/intel/issues/4860
[i915#4879]: https://gitlab.freedesktop.org/drm/intel/issues/4879
[i915#4881]: https://gitlab.freedesktop.org/drm/intel/issues/4881
[i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
[i915#5176]: https://gitlab.freedesktop.org/drm/intel/issues/5176
[i915#5235]: https://gitlab.freedesktop.org/drm/intel/issues/5235
[i915#5286]: https://gitlab.freedesktop.org/drm/intel/issues/5286
[i915#5325]: https://gitlab.freedesktop.org/drm/intel/issues/5325
[i915#5439]: https://gitlab.freedesktop.org/drm/intel/issues/5439
[i915#5461]: https://gitlab.freedesktop.org/drm/intel/issues/5461
[i915#5563]: https://gitlab.freedesktop.org/drm/intel/issues/5563
[i915#5566]: https://gitlab.freedesktop.org/drm/intel/issues/5566
[i915#5723]: https://gitlab.freedesktop.org/drm/intel/issues/5723
[i915#6095]: https://gitlab.freedesktop.org/drm/intel/issues/6095
[i915#6245]: https://gitlab.freedesktop.org/drm/intel/issues/6245
[i915#6268]: https://gitlab.freedesktop.org/drm/intel/issues/6268
[i915#6524]: https://gitlab.freedesktop.org/drm/intel/issues/6524
[i915#658]: https://gitlab.freedesktop.org/drm/intel/issues/658
[i915#6944]: https://gitlab.freedesktop.org/drm/intel/issues/6944
[i915#7116]: https://gitlab.freedesktop.org/drm/intel/issues/7116
[i915#7118]: https://gitlab.freedesktop.org/drm/intel/issues/7118
[i915#7651]: https://gitlab.freedesktop.org/drm/intel/issues/7651
[i915#7701]: https://gitlab.freedesktop.org/drm/intel/issues/7701
[i915#7711]: https://gitlab.freedesktop.org/drm/intel/issues/7711
[i915#7828]: https://gitlab.freedesktop.org/drm/intel/issues/7828
[i915#7936]: https://gitlab.freedesktop.org/drm/intel/issues/7936
[i915#7975]: https://gitlab.freedesktop.org/drm/intel/issues/7975
[i915#7981]: https://gitlab.freedesktop.org/drm/intel/issues/7981
[i915#8011]: https://gitlab.freedesktop.org/drm/intel/issues/8011
[i915#8211]: https://gitlab.freedesktop.org/drm/intel/issues/8211
[i915#8234]: https://gitlab.freedesktop.org/drm/intel/issues/8234

Build changes
-------------

* Linux: CI_DRM_12918 -> Patchwork_115264v2

CI-20190529: 20190529
CI_DRM_12918: a1cb2211899ba6b8fe078586d0878aa918a5aab3 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_7220: 3eb7beb5c03343b29556025b1ada4b50849b5976 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
Patchwork_115264v2: a1cb2211899ba6b8fe078586d0878aa918a5aab3 @ git://anongit.freedesktop.org/gfx-ci/linux
piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/index.html

#52 lore raw

[Intel-gfx] ✗ Fi.CI.BUILD: failure for Introduce new methods for verifying ownership in vfio PCI hot reset (rev3)

In reply to Yi Liu
== Series Details ==

== Series Details ==

Series: Introduce new methods for verifying ownership in vfio PCI hot reset (rev3)
URL : https://patchwork.freedesktop.org/series/115264/
State : failure

== Summary ==

Error: patch https://patchwork.freedesktop.org/api/1.0/series/115264/revisions/3/mbox/ not applied
Applying: vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset()
Applying: vfio/pci: Only check ownership of opened devices in hot reset
Applying: vfio/pci: Move the existing hot reset logic to be a helper
Applying: vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
Applying: vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET
Applying: vfio: Refine vfio file kAPIs for vfio PCI hot reset
Applying: vfio: Accpet device file from vfio PCI hot reset path
Applying: vfio/pci: Renaming for accepting device fd in hot reset path
Applying: vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl
Patch is empty.
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To record the empty patch as an empty commit, run "git am --allow-empty".
To restore the original branch and stop patching, run "git am --abort".
Build failed, no error log produced

#53 lore raw
In reply to Yi Liu
Quoted 18 lines

VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.

First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]

When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]

However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton dev_set hence no ownership check is required. [3]

Quoted 7 lines

For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]

As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new _INFO ioctl to get hot reset scope for given device.

Tested NIC passthrough on Intel platform. Result looks good hence,
Tested by: Jiang, Yanting <[email protected]>

Thanks,
Yanting

#54 lore raw
Quoted 21 lines · Yanting
Quoted 21 lines

VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.

First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]

When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]

However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton dev_set hence no ownership check is required. [3]

For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]

Quoted 7 lines
Quoted 4 lines

As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new _INFO ioctl to get hot reset scope for given device.

Tested NIC passthrough on Intel platform. Result looks good hence,
Tested by: Jiang, Yanting <[email protected]>

I'm not aware of any userspace that exercises this reset ioctl in cdev mode. Is this regression testing only? Thanks,

Alex

#55 lore raw
Quoted 53 lines

-----Original Message-----
From: Alex Williamson <[email protected]>
Sent: Friday, March 31, 2023 9:25 PM
To: Jiang, Yanting <[email protected]>
Cc: Liu, Yi L <[email protected]>; [email protected]; Tian, Kevin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; intel- [email protected]; [email protected]; Hao, Xudong <[email protected]>; Zhao, Yan Y <[email protected]>; Xu, Terrence <[email protected]>
Subject: Re: [PATCH v2 00/10] Introduce new methods for verifying ownership in vfio PCI hot reset

Quoted 5 lines · Yanting
Quoted 5 lines

VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device

cdev support.

Quoted 7 lines
Quoted 7 lines

First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]

When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd

information.

Quoted 8 lines
Quoted 8 lines

This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]

However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton dev_set hence no ownership check is required. [3]

Quoted 7 lines
Quoted 7 lines

For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership.
[4]

As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new _INFO ioctl to get hot reset scope for given

device.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Tested NIC passthrough on Intel platform. Result looks good hence,
Tested by: Jiang, Yanting <[email protected]>

I'm not aware of any userspace that exercises this reset ioctl in cdev mode. Is this regression testing only? Thanks,

Alex

Hi Alex,

Yes, only regression testing and some negative testing for NIC passthrough with legacy vfio mode, vfio iommufd compat mode, and cdev mode.

Thanks,
Yanting

Tested-by on the v2 series
#56 lore raw
In reply to Yi Liu
Quoted 17 lines

VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.

First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]

When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]

However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton dev_set hence no ownership check is required. [3]

Quoted 7 lines

For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]

As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new _INFO ioctl to get hot reset scope for given device.

Thanks,
Yanting

Tested-by on the v2 series
#57 lore raw
In reply to Yi Liu
Quoted 77 lines

-----Original Message-----
From: Liu, Yi L <[email protected]>
Sent: Monday, March 27, 2023 5:35 PM

VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.

First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]

When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]

However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton dev_set hence no ownership check is required. [3]

For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]

As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new _INFO ioctl to get hot reset scope for given device.

[1] https://lore.kernel.org/kvm/Y%[email protected]/
[2] https://lore.kernel.org/kvm/Y%[email protected]/#t
[3] https://lore.kernel.org/kvm/ZACX+Np%[email protected]/
[4]
https://lore.kernel.org/kvm/DS0PR11MB7529BE88460582BD599DC1F7C3B19
@DS0PR11MB7529.namprd11.prod.outlook.com/#t
[5] https://lore.kernel.org/kvm/[email protected]/
[6] https://lore.kernel.org/kvm/[email protected]/

Change log:

v2:
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new _INFO ioctl for the usage of device fd passing usage in cdev path (Jason, Alex)

v1: https://lore.kernel.org/kvm/[email protected]/

Regards,
Yi Liu

Yi Liu (10):
vfio/pci: Update comment around group_fd get in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Only check ownership of opened devices in hot reset vfio/pci: Move the existing hot reset logic to be a helper
vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for
vfio_device
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
vfio: Refine vfio file kAPIs for vfio PCI hot reset
vfio: Accpet device file from vfio PCI hot reset path vfio/pci: Renaming for accepting device fd in hot reset path vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

 drivers/iommu/iommufd/device.c   |  12 ++
 drivers/vfio/group.c             |  32 ++--
 drivers/vfio/iommufd.c           |  16 ++
 drivers/vfio/pci/vfio_pci_core.c | 244 ++++++++++++++++++++++++-------
 drivers/vfio/vfio.h              |   2 +
 drivers/vfio/vfio_main.c         |  44 ++++++
 include/linux/iommufd.h          |   3 +
 include/linux/vfio.h             |  14 ++
 include/uapi/linux/vfio.h        |  65 +++++++-
 9 files changed, 364 insertions(+), 68 deletions(-)
Quoted 2 lines

--
2.34.1

Verified this series by "Intel GVT-g GPU device mediated passthrough".
Passed VFIO legacy mode / compat mode / cdev mode basic functionality and GPU force reset test.

#58 lore raw
Quoted 76 lines · Terrence
Quoted 76 lines

-----Original Message-----
From: Liu, Yi L <[email protected]>
Sent: Monday, March 27, 2023 5:35 PM

VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.

First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]

When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]

However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton dev_set hence no ownership check is required. [3]

For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]

As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new _INFO ioctl to get hot reset scope for given device.

[1] https://lore.kernel.org/kvm/Y%[email protected]/
[2] https://lore.kernel.org/kvm/Y%[email protected]/#t
[3] https://lore.kernel.org/kvm/ZACX+Np%[email protected]/
[4]
https://lore.kernel.org/kvm/DS0PR11MB7529BE88460582BD599DC1F7C3B19
@DS0PR11MB7529.namprd11.prod.outlook.com/#t
[5] https://lore.kernel.org/kvm/[email protected]/
[6] https://lore.kernel.org/kvm/[email protected]/

Change log:

v2:
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new _INFO ioctl for the usage of device fd passing usage in cdev path (Jason, Alex)

v1: https://lore.kernel.org/kvm/[email protected]/

Regards,
Yi Liu

Yi Liu (10):
vfio/pci: Update comment around group_fd get in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Only check ownership of opened devices in hot reset vfio/pci: Move the existing hot reset logic to be a helper
vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for
vfio_device
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
vfio: Refine vfio file kAPIs for vfio PCI hot reset
vfio: Accpet device file from vfio PCI hot reset path vfio/pci: Renaming for accepting device fd in hot reset path vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

 drivers/iommu/iommufd/device.c   |  12 ++
 drivers/vfio/group.c             |  32 ++--
 drivers/vfio/iommufd.c           |  16 ++
 drivers/vfio/pci/vfio_pci_core.c | 244 ++++++++++++++++++++++++-------
 drivers/vfio/vfio.h              |   2 +
 drivers/vfio/vfio_main.c         |  44 ++++++
 include/linux/iommufd.h          |   3 +
 include/linux/vfio.h             |  14 ++
 include/uapi/linux/vfio.h        |  65 +++++++-
 9 files changed, 364 insertions(+), 68 deletions(-)
Quoted 7 lines
Quoted 2 lines

--
2.34.1

Verified this series by "Intel GVT-g GPU device mediated passthrough".
Passed VFIO legacy mode / compat mode / cdev mode basic functionality and GPU force reset test.

Seems like only this "GPU force reset test" is relevant to the new functionality of this series, GVT-g does not and has no reason to support the HOT_RESET ioctls used here. Can you provide more details of the force-reset test? What userspace driver is being used? Thanks,

Alex

#59 lore raw
Quoted 102 lines

-----Original Message-----
From: intel-gvt-dev <[email protected]> On Behalf Of Alex Williamson
Sent: Saturday, April 1, 2023 1:50 AM

Quoted 8 lines · Terrence
Quoted 8 lines

-----Original Message-----
From: Liu, Yi L <[email protected]>
Sent: Monday, March 27, 2023 5:35 PM

VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device

cdev support.

Quoted 16 lines
Quoted 16 lines

First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]

When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]

However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton dev_set hence no ownership check is required. [3]

Quoted 7 lines
Quoted 7 lines

For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership.
[4]

As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new _INFO ioctl to get hot reset scope for given

device.

Quoted 6 lines
Quoted 6 lines

[1] https://lore.kernel.org/kvm/Y%[email protected]/
[2] https://lore.kernel.org/kvm/Y%[email protected]/#t
[3] https://lore.kernel.org/kvm/ZACX+Np%[email protected]/
[4]

https://lore.kernel.org/kvm/DS0PR11MB7529BE88460582BD599DC1F7C3B19

Quoted 34 lines
Quoted 34 lines

@DS0PR11MB7529.namprd11.prod.outlook.com/#t
[5] https://lore.kernel.org/kvm/[email protected]/
[6] https://lore.kernel.org/kvm/[email protected]/

Change log:

v2:
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new _INFO ioctl for the usage of device fd passing usage in cdev path (Jason, Alex)

v1:
https://lore.kernel.org/kvm/[email protected]
m/

Regards,
Yi Liu

Yi Liu (10):
vfio/pci: Update comment around group_fd get in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Only check ownership of opened devices in hot reset vfio/pci: Move the existing hot reset logic to be a helper
vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for
vfio_device
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
vfio: Refine vfio file kAPIs for vfio PCI hot reset
vfio: Accpet device file from vfio PCI hot reset path vfio/pci: Renaming for accepting device fd in hot reset path vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 4 lines
Quoted 4 lines
 drivers/iommu/iommufd/device.c   |  12 ++
 drivers/vfio/group.c             |  32 ++--
 drivers/vfio/iommufd.c           |  16 ++
 drivers/vfio/pci/vfio_pci_core.c | 244 ++++++++++++++++++++++++----

---

Quoted 7 lines
Quoted 7 lines
 drivers/vfio/vfio.h              |   2 +
 drivers/vfio/vfio_main.c         |  44 ++++++
 include/linux/iommufd.h          |   3 +
 include/linux/vfio.h             |  14 ++
 include/uapi/linux/vfio.h        |  65 +++++++-
 9 files changed, 364 insertions(+), 68 deletions(-)
Quoted 5 lines
Quoted 2 lines

--
2.34.1

Verified this series by "Intel GVT-g GPU device mediated passthrough". Passed VFIO legacy mode / compat mode / cdev mode basic functionality

and GPU force reset test.

Quoted 1 line
Quoted 8 lines
Quoted 1 line

Seems like only this "GPU force reset test" is relevant to the new functionality of this series, GVT-g does not and has no reason to support the HOT_RESET ioctls used here. Can you provide more details of the force-reset test? What userspace driver is being used? Thanks,

Alex

Hi Alex, about the "GPU force reset test", I used the "i915_hangman" test from intel-gpu-tools, it is for GPU force hang / reset. It is an important regression test scenario for this patch series.
To test the HOT_RESET ioctls itself, need to wait the corresponding Qemu changes from Yi.

#60 lore raw
Quoted 108 lines · Terrence
Quoted 108 lines

-----Original Message-----
From: intel-gvt-dev <[email protected]> On Behalf Of Alex Williamson
Sent: Saturday, April 1, 2023 1:50 AM

Quoted 8 lines · Terrence
Quoted 8 lines

-----Original Message-----
From: Liu, Yi L <[email protected]>
Sent: Monday, March 27, 2023 5:35 PM

VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device

cdev support.

Quoted 16 lines
Quoted 16 lines

First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]

When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]

However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton dev_set hence no ownership check is required. [3]

Quoted 7 lines
Quoted 7 lines

For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership.
[4]

As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new _INFO ioctl to get hot reset scope for given

device.

Quoted 6 lines
Quoted 6 lines

[1] https://lore.kernel.org/kvm/Y%[email protected]/
[2] https://lore.kernel.org/kvm/Y%[email protected]/#t
[3] https://lore.kernel.org/kvm/ZACX+Np%[email protected]/
[4]

https://lore.kernel.org/kvm/DS0PR11MB7529BE88460582BD599DC1F7C3B19

Quoted 34 lines
Quoted 34 lines

@DS0PR11MB7529.namprd11.prod.outlook.com/#t
[5] https://lore.kernel.org/kvm/[email protected]/
[6] https://lore.kernel.org/kvm/[email protected]/

Change log:

v2:
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new _INFO ioctl for the usage of device fd passing usage in cdev path (Jason, Alex)

v1:
https://lore.kernel.org/kvm/[email protected]
m/

Regards,
Yi Liu

Yi Liu (10):
vfio/pci: Update comment around group_fd get in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Only check ownership of opened devices in hot reset vfio/pci: Move the existing hot reset logic to be a helper
vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for
vfio_device
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
vfio: Refine vfio file kAPIs for vfio PCI hot reset
vfio: Accpet device file from vfio PCI hot reset path vfio/pci: Renaming for accepting device fd in hot reset path vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO

Quoted 4 lines
Quoted 4 lines
 drivers/iommu/iommufd/device.c   |  12 ++
 drivers/vfio/group.c             |  32 ++--
 drivers/vfio/iommufd.c           |  16 ++
 drivers/vfio/pci/vfio_pci_core.c | 244 ++++++++++++++++++++++++----  

---

Quoted 7 lines
Quoted 7 lines
 drivers/vfio/vfio.h              |   2 +
 drivers/vfio/vfio_main.c         |  44 ++++++
 include/linux/iommufd.h          |   3 +
 include/linux/vfio.h             |  14 ++
 include/uapi/linux/vfio.h        |  65 +++++++-
 9 files changed, 364 insertions(+), 68 deletions(-)
Quoted 5 lines
Quoted 2 lines

--
2.34.1

Verified this series by "Intel GVT-g GPU device mediated passthrough". Passed VFIO legacy mode / compat mode / cdev mode basic functionality

and GPU force reset test.

Quoted 2 lines

Seems like only this "GPU force reset test" is relevant to the new functionality of this series, GVT-g does not and has no reason to support the HOT_RESET ioctls used here. Can you provide more details of the force-reset test? What userspace driver is being used? Thanks,

Quoted 6 lines
Quoted 1 line

Alex

Hi Alex, about the "GPU force reset test", I used the "i915_hangman" test from intel-gpu-tools, it is for GPU force hang / reset. It is an important regression test scenario for this patch series. To test the HOT_RESET ioctls itself, need to wait the corresponding Qemu changes from Yi.

But i915 exists on the host root bus, we fundamentally cannot perform a bus reset of the root bus. So how exactly is testing with GVT-g, which doesn't use the vfio-pci-core hot-reset ioctl, or GVT-d, which can't do a bus reset because it exists on the root bus, relevant to this series? Is this some novel use of a dGPU i915 with out-of-tree drivers?

Obviously any regression testing is fine and appreciated, but if this is intended to express some validation of the new interface, I'm failing to see how. Thanks,

Alex