Introduce new methods for verifying ownership in vfio PCI hot reset
Subsystems (5)
v2 cover Introduce new methods for verifying ownership in vfio PCI hot reset
VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.
First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]
When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]
However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton dev_set hence no ownership check is required. [3]
For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]
As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new _INFO ioctl to get hot reset scope for given device.
[1] https://lore.kernel.org/kvm/Y%[email protected]/
[2] https://lore.kernel.org/kvm/Y%[email protected]/#t
[3] https://lore.kernel.org/kvm/ZACX+Np%[email protected]/
[4] https://lore.kernel.org/kvm/DS0PR11MB7529BE88460582BD599DC1F7C3B19@DS0PR11MB7529.namprd11.prod.outlook.com/#t
[5] https://lore.kernel.org/kvm/[email protected]/
[6] https://lore.kernel.org/kvm/[email protected]/
Change log:
v2:
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new _INFO ioctl for the usage of device fd passing usage in cdev path (Jason, Alex)
v1: https://lore.kernel.org/kvm/[email protected]/
Regards,
Yi Liu
Yi Liu (10):
vfio/pci: Update comment around group_fd get in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Only check ownership of opened devices in hot reset vfio/pci: Move the existing hot reset logic to be a helper
vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for
vfio_device
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
vfio: Refine vfio file kAPIs for vfio PCI hot reset
vfio: Accpet device file from vfio PCI hot reset path vfio/pci: Renaming for accepting device fd in hot reset path vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
drivers/iommu/iommufd/device.c | 12 ++ drivers/vfio/group.c | 32 ++-- drivers/vfio/iommufd.c | 16 ++ drivers/vfio/pci/vfio_pci_core.c | 244 ++++++++++++++++++++++++------- drivers/vfio/vfio.h | 2 + drivers/vfio/vfio_main.c | 44 ++++++ include/linux/iommufd.h | 3 + include/linux/vfio.h | 14 ++ include/uapi/linux/vfio.h | 65 +++++++- 9 files changed, 364 insertions(+), 68 deletions(-)
Signature · 1 line
2.34.1
PATCH v2 1/10 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset()
this suits more on what the code does.
--- drivers/vfio/pci/vfio_pci_core.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +2 -3
@@ -1308,9 +1308,8 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, }/* -* For each group_fd, get the group through the vfio external user -* interface and store the group and iommu ID. This ensures the group -* is held across the reset. +* Get the group file for each fd to ensure the group held across +* the reset */ for (file_idx =0 ; file_idx < hdr.count; file_idx++) {struct file *file =fget (group_fds[file_idx]);
Signature · 1 line
2.34.1
PATCH v2 2/10 vfio/pci: Only check ownership of opened devices in hot reset
In reply to Yi LiuIf the affected device is not opened by any user, it's safe to reset it given it's not in use.
--- drivers/vfio/pci/vfio_pci_core.c | 14 +++++++++++--- include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 19 insertions(+), 3 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +11 -3
@@ -2429,10 +2429,18 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,list_for_each_entry (cur_vma, &dev_set->device_list, vdev.dev_set_list) {/* -* Test whether all the affected devices are contained by the -* set of groups provided by the user. +* Test whether all the affected devices can be reset by the +* user. +* +* Resetting an unused device (not opened) is safe, because +* dev_set->lock is held in hot reset path so this device +* cannot race being opened by another user simultaneously. +* +* Otherwise all opened devices in the dev_set must be +* contained by the set of groups provided by the user. */ -if (!vfio_dev_in_groups (cur_vma, groups)) { +if (cur_vma->vdev.open_count && + !vfio_dev_in_groups (cur_vma, groups)) { ret = -EINVAL ; goto err_undo; }
include/uapi/linux/vfio.h +8 -0
@@ -673,6 +673,14 @@ struct vfio_pci_hot_reset_info {* VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset) * +* Userspace requests hot reset for the devices it uses. Due to the +* underlying topology, multiple devices can be affected in the reset +* while some might be opened by another user. To avoid interference +* the calling user must ensure all affected devices, if opened, are +* owned by itself. +* +* The ownership is proved by an array of group fds. +* * Return: 0 on success, -errno on failure. */ struct vfio_pci_hot_reset {
Signature · 1 line
2.34.1
PATCH v2 3/10 vfio/pci: Move the existing hot reset logic to be a helper
In reply to Yi LiuThis prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups().
No functional change is intended.
--- drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++------------- 1 file changed, 33 insertions(+), 23 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +33 -23
@@ -1255,29 +1255,17 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(return ret; } -static int vfio_pci_ioctl_pci_hot_reset (struct vfio_pci_core_device *vdev, -struct vfio_pci_hot_reset __user *arg) +static int +vfio_pci_ioctl_pci_hot_reset_groups (struct vfio_pci_core_device *vdev, +struct vfio_pci_hot_reset *hdr, +bool slot, +struct vfio_pci_hot_reset __user *arg) { - unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); -struct vfio_pci_hot_reset hdr;int32_t *group_fds;struct file **files;struct vfio_pci_group_info info; -bool slot = false;int file_idx, count =0 , ret =0 ; -if (copy_from_user (&hdr, arg, minsz)) -return -EFAULT ; - -if (hdr.argsz < minsz || hdr.flags) -return -EINVAL ; - -/* Can we do a slot or bus reset or neither? */ -if (!pci_probe_reset_slot (vdev->pdev->slot)) - slot = true; -else if (pci_probe_reset_bus (vdev->pdev->bus)) -return -ENODEV ; -/* * We can't let userspace give us an arbitrarily large buffer to copy, * so verify how many we think there could be. Note groups can have
@@ -1289,11 +1277,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,return ret;/* Somewhere between 1 and count is OK */ -if (!hdr.count || hdr.count > count) +if (!hdr->count || hdr->count > count)return -EINVAL ; - group_fds =kcalloc (hdr.count,sizeof (*group_fds),GFP_KERNEL ); - files =kcalloc (hdr.count,sizeof (*files),GFP_KERNEL ); + group_fds =kcalloc (hdr->count,sizeof (*group_fds),GFP_KERNEL ); + files =kcalloc (hdr->count,sizeof (*files),GFP_KERNEL );if (!group_fds || !files) {kfree (group_fds);kfree (files);
@@ -1301,7 +1289,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, }if (copy_from_user (group_fds, arg->group_fds, - hdr.count *sizeof (*group_fds))) { + hdr->count *sizeof (*group_fds))) {kfree (group_fds);kfree (files);return -EFAULT ;
@@ -1311,7 +1299,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,* Get the group file for each fd to ensure the group held across * the reset */ -for (file_idx =0 ; file_idx < hdr.count; file_idx++) { +for (file_idx =0 ; file_idx < hdr->count; file_idx++) {struct file *file =fget (group_fds[file_idx]);if (!file) {
@@ -1335,7 +1323,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,if (ret) goto hot_reset_release; - info.count = hdr.count; + info.count = hdr->count; info.files = files; ret =vfio_pci_dev_set_hot_reset (vdev->vdev.dev_set, &info);
@@ -1348,6 +1336,28 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,return ret; } +static int vfio_pci_ioctl_pci_hot_reset (struct vfio_pci_core_device *vdev, +struct vfio_pci_hot_reset __user *arg) +{ + unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); +struct vfio_pci_hot_reset hdr; +bool slot = false; + +if (copy_from_user (&hdr, arg, minsz)) +return -EFAULT ; + +if (hdr.argsz < minsz || hdr.flags) +return -EINVAL ; + +/* Can we do a slot or bus reset or neither? */ +if (!pci_probe_reset_slot (vdev->pdev->slot)) + slot = true; +else if (pci_probe_reset_bus (vdev->pdev->bus)) +return -ENODEV ; + +return vfio_pci_ioctl_pci_hot_reset_groups (vdev, &hdr, slot, arg); +} +static int vfio_pci_ioctl_ioeventfd (struct vfio_pci_core_device *vdev,struct vfio_device_ioeventfd __user *arg) {
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v2 03/10] vfio/pci: Move the existing hot reset logic to be a helper
Quoted 11 lines · Yi Liu
This prepares to add another method for hot reset. The major hot reset logic are moved to
vfio_pci_ioctl_pci_hot_reset_groups().No functional change is intended.
Suggested-by Jason Gunthorpe <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++------------- 1 file changed, 33 insertions(+), 23 deletions(-)
Jason
Re: [Intel-gfx] [PATCH v2 03/10] vfio/pci: Move the existing hot reset logic to be a helper
In reply to Yi LiuQuoted 11 lines · Yi Liu
This prepares to add another method for hot reset. The major hot reset logic are moved to
vfio_pci_ioctl_pci_hot_reset_groups().No functional change is intended.
Suggested-by Jason Gunthorpe <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++------------- 1 file changed, 33 insertions(+), 23 deletions(-)
Jason
PATCH v2 4/10 vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
In reply to Yi LiuThis is needed by the vfio pci driver to report affected devices in the hot reset for a given device.
--- drivers/iommu/iommufd/device.c | 12 ++++++++++++ drivers/vfio/iommufd.c | 16 ++++++++++++++++ include/linux/iommufd.h | 3 +++ include/linux/vfio.h | 13 +++++++++++++ 4 files changed, 44 insertions(+)
drivers/iommu/iommufd/device.c +12 -0
@@ -131,6 +131,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) }EXPORT_SYMBOL_NS_GPL (iommufd_device_unbind,IOMMUFD ); +struct iommufd_ctx *iommufd_device_to_ictx (struct iommufd_device *idev) +{ +return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL (iommufd_device_to_ictx,IOMMUFD ); + +u32 iommufd_device_to_id (struct iommufd_device *idev) +{ +return idev->obj.id; +} +EXPORT_SYMBOL_NS_GPL (iommufd_device_to_id,IOMMUFD ); +static int iommufd_device_setup_msi (struct iommufd_device *idev,struct iommufd_hw_pagetable *hwpt,phys_addr_t sw_msi_start)
drivers/vfio/iommufd.c +16 -0
@@ -66,6 +66,22 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd (vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx (struct vfio_device *vdev) +{ +if (!vdev->iommufd_device) +return NULL ; +return iommufd_device_to_ictx (vdev->iommufd_device); +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_ictx); + +int vfio_iommufd_physical_devid (struct vfio_device *vdev,u32 *id) +{ +if (!vdev->iommufd_device) +return -EINVAL ; + *id =iommufd_device_to_id (vdev->iommufd_device); +return 0 ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_devid);/* * The physical standard ops mean that the iommufd_device is bound to the * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers
include/linux/iommufd.h +3 -0
@@ -24,6 +24,9 @@ void iommufd_device_unbind(struct iommufd_device *idev);int iommufd_device_attach (struct iommufd_device *idev,u32 *pt_id);void iommufd_device_detach (struct iommufd_device *idev); +struct iommufd_ctx *iommufd_device_to_ictx (struct iommufd_device *idev); +u32 iommufd_device_to_id (struct iommufd_device *idev); +struct iommufd_access_ops {u8 needs_pin_pages :1 ;void (*unmap)(void *data,unsigned long iova,unsigned long length);
include/linux/vfio.h +13 -0
@@ -113,6 +113,8 @@ struct vfio_device_ops { };#if IS_ENABLED (CONFIG_IOMMUFD ) +struct iommufd_ctx *vfio_iommufd_physical_ictx (struct vfio_device *vdev); +int vfio_iommufd_physical_devid (struct vfio_device *vdev,u32 *id);int vfio_iommufd_physical_bind (struct vfio_device *vdev,struct iommufd_ctx *ictx,u32 *out_device_id);void vfio_iommufd_physical_unbind (struct vfio_device *vdev);
@@ -122,6 +124,17 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev,void vfio_iommufd_emulated_unbind (struct vfio_device *vdev);int vfio_iommufd_emulated_attach_ioas (struct vfio_device *vdev,u32 *pt_id);#else +static inline struct iommufd_ctx * +vfio_iommufd_physical_ictx (struct vfio_device *vdev) +{ +return NULL ; +} + +static inline int vfio_iommufd_physical_devid (struct vfio_device *vdev,u32 *id) +{ +return -EOPNOTSUPP ; +} +#define vfio_iommufd_physical_bind \ ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ u32 *out_device_id)) NULL)
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v2 04/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
Quoted 10 lines · Yi Liu
This is needed by the vfio pci driver to report affected devices in the hot reset for a given device.
Signed-off-by Yi Liu <[email protected]>--- drivers/iommu/iommufd/device.c | 12 ++++++++++++ drivers/vfio/iommufd.c | 16 ++++++++++++++++ include/linux/iommufd.h | 3 +++ include/linux/vfio.h | 13 +++++++++++++ 4 files changed, 44 insertions(+)
Jason
PATCH v2 5/10 vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET
In reply to Yi Liuas an alternative method for ownership check when iommufd is used. In this case all opened devices in the affected dev_set are verified to be bound to a same valid iommufd value to allow reset. It's simpler and faster as user does not need to pass a set of fds and kernel no need to search the device within the given fds.
a device in noiommu mode doesn't have a valid iommufd, so this method should not be used in a dev_set which contains multiple devices and one of them is in noiommu. The only allowed noiommu scenario is that the calling device is noiommu and it's in a singleton dev_set.
--- drivers/vfio/pci/vfio_pci_core.c | 42 +++++++++++++++++++++++++++----- include/uapi/linux/vfio.h | 9 ++++++- 2 files changed, 44 insertions(+), 7 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +36 -6
@@ -180,7 +180,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev)struct vfio_pci_group_info ;static void vfio_pci_dev_set_try_reset (struct vfio_device_set *dev_set);static int vfio_pci_dev_set_hot_reset (struct vfio_device_set *dev_set, -struct vfio_pci_group_info *groups); +struct vfio_pci_group_info *groups, +struct iommufd_ctx *iommufd_ctx);/* * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND
@@ -1277,7 +1278,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,return ret;/* Somewhere between 1 and count is OK */ -if (!hdr->count || hdr->count > count) +if (hdr->count > count)return -EINVAL ; group_fds =kcalloc (hdr->count,sizeof (*group_fds),GFP_KERNEL );
@@ -1326,7 +1327,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, info.count = hdr->count; info.files = files; - ret =vfio_pci_dev_set_hot_reset (vdev->vdev.dev_set, &info); + ret =vfio_pci_dev_set_hot_reset (vdev->vdev.dev_set, &info,NULL ); hot_reset_release:for (file_idx--; file_idx >=0 ; file_idx--)
@@ -1341,6 +1342,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, { unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count);struct vfio_pci_hot_reset hdr; +struct iommufd_ctx *iommufd;bool slot = false;if (copy_from_user (&hdr, arg, minsz))
@@ -1355,7 +1357,12 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,else if (pci_probe_reset_bus (vdev->pdev->bus))return -ENODEV ; -return vfio_pci_ioctl_pci_hot_reset_groups (vdev, &hdr, slot, arg); +if (hdr.count) +return vfio_pci_ioctl_pci_hot_reset_groups (vdev, &hdr, slot, arg); + + iommufd =vfio_iommufd_physical_ictx (&vdev->vdev); + +return vfio_pci_dev_set_hot_reset (vdev->vdev.dev_set,NULL , iommufd); } static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev,
@@ -2327,6 +2334,9 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, {unsigned int i; +if (!groups) +return false; +for (i =0 ; i < groups->count; i++)if (vfio_file_has_dev (groups->files[i], &vdev->vdev))return true;
@@ -2402,13 +2412,25 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set)return ret; } +static bool vfio_dev_in_iommufd_ctx (struct vfio_pci_core_device *vdev, +struct iommufd_ctx *iommufd_ctx) +{ +struct iommufd_ctx *iommufd =vfio_iommufd_physical_ictx (&vdev->vdev); + +if (!iommufd) +return false; + +return iommufd == iommufd_ctx; +} +/* * We need to get memory_lock for each device, but devices can share mmap_lock, * therefore we need to zap and hold the vma_lock for each device, and only then * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset (struct vfio_device_set *dev_set, -struct vfio_pci_group_info *groups) +struct vfio_pci_group_info *groups, +struct iommufd_ctx *iommufd_ctx) {struct vfio_pci_core_device *cur_mem;struct vfio_pci_core_device *cur_vma;
@@ -2448,9 +2470,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,* * Otherwise all opened devices in the dev_set must be * contained by the set of groups provided by the user. +* +* If user provides a zero-length array, then all the +* opened devices must be bound to a same iommufd_ctx. +* +* If all above checks are failed, reset is allowed only if +* the calling device is in a singleton dev_set. */ if (cur_vma->vdev.open_count && - !vfio_dev_in_groups (cur_vma, groups)) { + !vfio_dev_in_groups (cur_vma, groups) && + !vfio_dev_in_iommufd_ctx (cur_vma, iommufd_ctx) && + (dev_set->device_count >1 )) { ret = -EINVAL ; goto err_undo; }
include/uapi/linux/vfio.h +8 -1
@@ -679,7 +679,14 @@ struct vfio_pci_hot_reset_info {* the calling user must ensure all affected devices, if opened, are * owned by itself. * -* The ownership is proved by an array of group fds. +* The ownership can be proved by: +* - An array of group fds +* - A zero-length array +* +* In the last case all affected devices which are opened by this user +* must have been bound to a same iommufd. If the calling device is in +* noiommu mode (no valid iommufd) then it can be reset only if the reset +* doesn't affect other devices. * * Return: 0 on success, -errno on failure. */
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v2 05/10] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET
Quoted 11 lines · Yi Liu
as an alternative method for ownership check when iommufd is used. In this case all opened devices in the affected
dev_setare verified to be bound to a same valid iommufd value to allow reset. It's simpler and faster as user does not need to pass a set of fds and kernel no need to search the device within the given fds.a device in noiommu mode doesn't have a valid iommufd, so this method should not be used in a
dev_setwhich contains multiple devices and one of them is in noiommu. The only allowed noiommu scenario is that the calling device is noiommu and it's in a singletondev_set.
Quoted 7 lines
Suggested-by Jason Gunthorpe <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 42 +++++++++++++++++++++++++++----- include/uapi/linux/vfio.h | 9 ++++++- 2 files changed, 44 insertions(+), 7 deletions(-)
Jason
PATCH v2 6/10 vfio: Refine vfio file kAPIs for vfio PCI hot reset
In reply to Yi LiuThis prepares vfio core to accept vfio device file from the vfio PCI hot reset path. vfio_file_is_group() is still kept for KVM usage.
--- drivers/vfio/group.c | 32 ++++++++++++++------------------ drivers/vfio/pci/vfio_pci_core.c | 4 ++-- drivers/vfio/vfio.h | 2 ++ drivers/vfio/vfio_main.c | 29 +++++++++++++++++++++++++++++ include/linux/vfio.h | 1 + 5 files changed, 48 insertions(+), 20 deletions(-)
drivers/vfio/group.c +14 -18
@@ -745,6 +745,15 @@ bool vfio_device_has_container(struct vfio_device *device)return device->group->container; } +struct vfio_group *vfio_group_from_file (struct file *file) +{ +struct vfio_group *group = file->private_data; + +if (file->f_op != &vfio_group_fops) +return NULL ; +return group; +} +/** * vfio_file_iommu_group - Return the struct iommu_group for the vfio group file * @file: VFIO group file
@@ -755,13 +764,13 @@ bool vfio_device_has_container(struct vfio_device *device)*/ struct iommu_group *vfio_file_iommu_group (struct file *file) { -struct vfio_group *group = file->private_data; +struct vfio_group *group =vfio_group_from_file (file);struct iommu_group *iommu_group =NULL ;if (!IS_ENABLED (CONFIG_SPAPR_TCE_IOMMU ))return NULL ; -if (!vfio_file_is_group (file)) +if (!group)return NULL ;mutex_lock (&group->group_lock);
@@ -775,12 +784,12 @@ struct iommu_group *vfio_file_iommu_group(struct file *file)EXPORT_SYMBOL_GPL (vfio_file_iommu_group);/** -* vfio_file_is_group - True if the file is usable with VFIO aPIS +* vfio_file_is_group - True if the file is a vfio group file * @file: VFIO group file */ bool vfio_file_is_group (struct file *file) { -return file->f_op == &vfio_group_fops; +return vfio_group_from_file (file); }EXPORT_SYMBOL_GPL (vfio_file_is_group);
@@ -842,23 +851,10 @@ void vfio_file_set_kvm(struct file *file, struct kvm *kvm) }EXPORT_SYMBOL_GPL (vfio_file_set_kvm); -/** -* vfio_file_has_dev - True if the VFIO file is a handle for device -* @file: VFIO file to check -* @device: Device that must be part of the file -* -* Returns true if given file has permission to manipulate the given device. -*/ -bool vfio_file_has_dev (struct file *file,struct vfio_device *device) +bool vfio_group_has_dev (struct vfio_group *group,struct vfio_device *device) { -struct vfio_group *group = file->private_data; - -if (!vfio_file_is_group (file)) -return false; -return group == device->group; } -EXPORT_SYMBOL_GPL (vfio_file_has_dev);static char *vfio_devnode (const struct device *dev,umode_t *mode) {
drivers/vfio/pci/vfio_pci_core.c +2 -2
@@ -1308,8 +1308,8 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,break ; } -/* Ensure the FD is a vfio group FD.*/ -if (!vfio_file_is_group (file)) { +/* Ensure the FD is a vfio FD. vfio group or vfio device */ +if (!vfio_file_is_valid (file)) {fput (file); ret = -EINVAL ;break ;
drivers/vfio/vfio.h +2 -0
@@ -84,6 +84,8 @@ void vfio_device_group_unregister(struct vfio_device *device);int vfio_device_group_use_iommu (struct vfio_device *device);void vfio_device_group_unuse_iommu (struct vfio_device *device);void vfio_device_group_close (struct vfio_device *device); +struct vfio_group *vfio_group_from_file (struct file *file); +bool vfio_group_has_dev (struct vfio_group *group,struct vfio_device *device);bool vfio_device_has_container (struct vfio_device *device);int __init vfio_group_init (void );void vfio_group_cleanup (void );
drivers/vfio/vfio_main.c +29 -0
@@ -1154,6 +1154,35 @@ const struct file_operations vfio_device_fops = { .mmap = vfio_device_fops_mmap, }; +/** +* vfio_file_is_valid - True if the file is valid vfio file +* @file: VFIO group file or VFIO device file +*/ +bool vfio_file_is_valid (struct file *file) +{ +return vfio_group_from_file (file); +} +EXPORT_SYMBOL_GPL (vfio_file_is_valid); + +/** +* vfio_file_has_dev - True if the VFIO file is a handle for device +* @file: VFIO file to check +* @device: Device that must be part of the file +* +* Returns true if given file has permission to manipulate the given device. +*/ +bool vfio_file_has_dev (struct file *file,struct vfio_device *device) +{ +struct vfio_group *group; + + group =vfio_group_from_file (file); +if (!group) +return false; + +return vfio_group_has_dev (group, device); +} +EXPORT_SYMBOL_GPL (vfio_file_has_dev); +/* * Sub-module support */
include/linux/vfio.h +1 -0
@@ -258,6 +258,7 @@ int vfio_mig_get_next_state(struct vfio_device *device,*/ struct iommu_group *vfio_file_iommu_group (struct file *file);bool vfio_file_is_group (struct file *file); +bool vfio_file_is_valid (struct file *file);bool vfio_file_enforced_coherent (struct file *file);void vfio_file_set_kvm (struct file *file,struct kvm *kvm);bool vfio_file_has_dev (struct file *file,struct vfio_device *device);
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v2 06/10] vfio: Refine vfio file kAPIs for vfio PCI hot reset
Quoted 5 lines · Yi Liu
This prepares vfio core to accept vfio device file from the vfio PCI hot reset path.
vfio_file_is_group()is still kept for KVM usage.Reviewed-by Kevin Tian <[email protected]>Signed-off-by Yi Liu <[email protected]>
Quoted 7 lines
--- drivers/vfio/group.c | 32 ++++++++++++++------------------ drivers/vfio/pci/vfio_pci_core.c | 4 ++-- drivers/vfio/vfio.h | 2 ++ drivers/vfio/vfio_main.c | 29 +++++++++++++++++++++++++++++ include/linux/vfio.h | 1 + 5 files changed, 48 insertions(+), 20 deletions(-)
Jason
PATCH v2 7/10 vfio: Accpet device file from vfio PCI hot reset path
In reply to Yi LiuThis extends both vfio_file_is_valid() and vfio_file_has_dev() to accept device file from the vfio PCI hot reset.
--- drivers/vfio/vfio_main.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
drivers/vfio/vfio_main.c +19 -4
@@ -1154,13 +1154,23 @@ const struct file_operations vfio_device_fops = { .mmap = vfio_device_fops_mmap, }; +static struct vfio_device *vfio_device_from_file (struct file *file) +{ +struct vfio_device *device = file->private_data; + +if (file->f_op != &vfio_device_fops) +return NULL ; +return device; +} +/** * vfio_file_is_valid - True if the file is valid vfio file * @file: VFIO group file or VFIO device file */ bool vfio_file_is_valid (struct file *file) { -return vfio_group_from_file (file); +return vfio_group_from_file (file) || +vfio_device_from_file (file); }EXPORT_SYMBOL_GPL (vfio_file_is_valid);
@@ -1174,12 +1184,17 @@ EXPORT_SYMBOL_GPL(vfio_file_is_valid);bool vfio_file_has_dev (struct file *file,struct vfio_device *device) {struct vfio_group *group; +struct vfio_device *vdev; group =vfio_group_from_file (file); -if (!group) -return false; +if (group) +return vfio_group_has_dev (group, device); + + vdev =vfio_device_from_file (file); +if (vdev) +return vdev == device; -return vfio_group_has_dev (group, device); +return false; }EXPORT_SYMBOL_GPL (vfio_file_has_dev);
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v2 07/10] vfio: Accpet device file from vfio PCI hot reset path
Quoted 8 lines · Yi Liu
This extends both
vfio_file_is_valid()andvfio_file_has_dev()to accept device file from the vfio PCI hot reset.Reviewed-by Kevin Tian <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/vfio_main.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
Jason
PATCH v2 8/10 vfio/pci: Renaming for accepting device fd in hot reset path
In reply to Yi LiuNo functional change is intended.
--- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++---------------- 1 file changed, 26 insertions(+), 26 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +26 -26
@@ -177,10 +177,10 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) } } -struct vfio_pci_group_info ; +struct vfio_pci_file_info ;static void vfio_pci_dev_set_try_reset (struct vfio_device_set *dev_set);static int vfio_pci_dev_set_hot_reset (struct vfio_device_set *dev_set, -struct vfio_pci_group_info *groups, +struct vfio_pci_file_info *info,struct iommufd_ctx *iommufd_ctx);/*
@@ -800,7 +800,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)return 0 ; } -struct vfio_pci_group_info { +struct vfio_pci_file_info {int count;struct file **files; };
@@ -1257,14 +1257,14 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( }static int -vfio_pci_ioctl_pci_hot_reset_groups (struct vfio_pci_core_device *vdev, -struct vfio_pci_hot_reset *hdr, -bool slot, -struct vfio_pci_hot_reset __user *arg) +vfio_pci_ioctl_pci_hot_reset_files (struct vfio_pci_core_device *vdev, +struct vfio_pci_hot_reset *hdr, +bool slot, +struct vfio_pci_hot_reset __user *arg) { -int32_t *group_fds; +int32_t *fds;struct file **files; -struct vfio_pci_group_info info; +struct vfio_pci_file_info info;int file_idx, count =0 , ret =0 ;/*
@@ -1281,17 +1281,17 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,if (hdr->count > count)return -EINVAL ; - group_fds =kcalloc (hdr->count,sizeof (*group_fds),GFP_KERNEL ); + fds =kcalloc (hdr->count,sizeof (*fds),GFP_KERNEL ); files =kcalloc (hdr->count,sizeof (*files),GFP_KERNEL ); -if (!group_fds || !files) { -kfree (group_fds); +if (!fds || !files) { +kfree (fds);kfree (files);return -ENOMEM ; } -if (copy_from_user (group_fds, arg->group_fds, - hdr->count *sizeof (*group_fds))) { -kfree (group_fds); +if (copy_from_user (fds, arg->group_fds, + hdr->count *sizeof (*fds))) { +kfree (fds);kfree (files);return -EFAULT ; }
@@ -1301,7 +1301,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,* the reset */ for (file_idx =0 ; file_idx < hdr->count; file_idx++) { -struct file *file =fget (group_fds[file_idx]); +struct file *file =fget (fds[file_idx]);if (!file) { ret = -EBADF ;
@@ -1318,9 +1318,9 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, files[file_idx] = file; } -kfree (group_fds); +kfree (fds); -/* release reference to groups on error */ +/* release reference to fds on error */ if (ret) goto hot_reset_release;
@@ -1358,7 +1358,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,return -ENODEV ;if (hdr.count) -return vfio_pci_ioctl_pci_hot_reset_groups (vdev, &hdr, slot, arg); +return vfio_pci_ioctl_pci_hot_reset_files (vdev, &hdr, slot, arg); iommufd =vfio_iommufd_physical_ictx (&vdev->vdev);
@@ -2329,16 +2329,16 @@ const struct pci_error_handlers vfio_pci_core_err_handlers = { };EXPORT_SYMBOL_GPL (vfio_pci_core_err_handlers); -static bool vfio_dev_in_groups (struct vfio_pci_core_device *vdev, -struct vfio_pci_group_info *groups) +static bool vfio_dev_in_files (struct vfio_pci_core_device *vdev, +struct vfio_pci_file_info *info) {unsigned int i; -if (!groups) +if (!info)return false; -for (i =0 ; i < groups->count; i++) -if (vfio_file_has_dev (groups->files[i], &vdev->vdev)) +for (i =0 ; i < info->count; i++) +if (vfio_file_has_dev (info->files[i], &vdev->vdev))return true;return false; }
@@ -2429,7 +2429,7 @@ static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev,* get each memory_lock. */ static int vfio_pci_dev_set_hot_reset (struct vfio_device_set *dev_set, -struct vfio_pci_group_info *groups, +struct vfio_pci_file_info *info,struct iommufd_ctx *iommufd_ctx) {struct vfio_pci_core_device *cur_mem;
@@ -2478,7 +2478,7 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,* the calling device is in a singleton dev_set. */ if (cur_vma->vdev.open_count && - !vfio_dev_in_groups (cur_vma, groups) && + !vfio_dev_in_files (cur_vma, info) && !vfio_dev_in_iommufd_ctx (cur_vma, iommufd_ctx) && (dev_set->device_count >1 )) { ret = -EINVAL ;
Signature · 1 line
2.34.1
PATCH v2 9/10 vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl
In reply to Yi LiuNow user can also provide an array of device fds as a 3rd method to verify the reset ownership. It's not useful at this point when the device fds are acquired via group fds. But it's necessary when moving to device cdev which allows the user to directly acquire device fds by skipping group. In that case this method can be used as a last resort when the preferred iommufd verification doesn't work, e.g. in noiommu usages.
Clarify it in uAPI.
--- drivers/vfio/pci/vfio_pci_core.c | 9 +++++---- include/uapi/linux/vfio.h | 3 ++- 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +5 -4
@@ -1289,7 +1289,7 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev,return -ENOMEM ; } -if (copy_from_user (fds, arg->group_fds, +if (copy_from_user (fds, arg->fds, hdr->count *sizeof (*fds))) {kfree (fds);kfree (files);
@@ -1297,8 +1297,8 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, }/* -* Get the group file for each fd to ensure the group held across -* the reset +* Get the file for each fd to ensure the group/device file +* is held across the reset */ for (file_idx =0 ; file_idx < hdr->count; file_idx++) {struct file *file =fget (fds[file_idx]);
@@ -2469,7 +2469,8 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,* cannot race being opened by another user simultaneously. * * Otherwise all opened devices in the dev_set must be -* contained by the set of groups provided by the user. +* contained by the set of groups/devices provided by +* the user. * * If user provides a zero-length array, then all the * opened devices must be bound to a same iommufd_ctx.
include/uapi/linux/vfio.h +2 -1
@@ -681,6 +681,7 @@ struct vfio_pci_hot_reset_info {* * The ownership can be proved by: * - An array of group fds +* - An array of device fds * - A zero-length array * * In the last case all affected devices which are opened by this user
@@ -694,7 +695,7 @@ struct vfio_pci_hot_reset {__u32 argsz;__u32 flags;__u32 count; -__s32 group_fds[]; +__s32 fds[]; };#define VFIO_DEVICE_PCI_HOT_RESET _IO(VFIO_TYPE, VFIO_BASE + 13)
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v2 09/10] vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl
Quoted 7 lines · Yi Liu
Now user can also provide an array of device fds as a 3rd method to verify the reset ownership. It's not useful at this point when the device fds are acquired via group fds. But it's necessary when moving to device cdev which allows the user to directly acquire device fds by skipping group. In that case this method can be used as a last resort when the preferred iommufd verification doesn't work, e.g. in noiommu usages.
Quoted 8 lines
Clarify it in uAPI.
Reviewed-by Kevin Tian <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 9 +++++---- include/uapi/linux/vfio.h | 3 ++- 2 files changed, 7 insertions(+), 5 deletions(-)
Jason
PATCH v2 10/10 vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
In reply to Yi LiuThis is a preparation for vfio device cdev as cdev gives userspace the capability to open device cdev fd and management stack (e.g. libvirt) could pass the device fd to the actual user (e.g. QEMU). As a result, the actual user has no idea about the device's bus:devfn information. This is a problem when user uses VFIO_DEVICE_GET_PCI_HOT_RESET_INFO to know the hot reset affected device scope as this ioctl returns bus:devfn info. For the fd passing usage, the acutal user cannot map the bus:devfn to the devices it has opened via the fd passed from management stack. So a new ioctl is required.
This new ioctl reports the list of iommufd dev_id that is opened by the user. If there is affected device that is not bound to vfio driver or opened by another user, this command shall fail with -EPERM. For the noiommu mode in the vfio device cdev path, this shall fail as no dev_id would be generated, hence nothing to report.
This ioctl is useless to the users that open vfio group as such users have no idea about the iommufd dev_id and it can use the existing VFIO_DEVICE_GET_PCI_HOT_RESET_INFO. The user that uses the traditional mode vfio group/container would be failed if invoking this ioctl. But the user that uses the iommufd compat mode vfio group/container shall succeed. This is harmless as long as user cannot make use of it and should use VFIO_DEVICE_GET_PCI_HOT_RESET_INFO.
--- drivers/vfio/pci/vfio_pci_core.c | 98 ++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 47 +++++++++++++++ 2 files changed, 145 insertions(+)
drivers/vfio/pci/vfio_pci_core.c +98 -0
@@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,return ret; } +static struct pci_dev * +vfio_pci_dev_set_resettable (struct vfio_device_set *dev_set); + +static int vfio_pci_ioctl_get_pci_hot_reset_group_info ( +struct vfio_pci_core_device *vdev, +struct vfio_pci_hot_reset_group_info __user *arg) +{ +unsigned long minsz = + offsetofend(struct vfio_pci_hot_reset_group_info, count); +struct vfio_pci_hot_reset_group_info hdr; +struct iommufd_ctx *iommufd, *cur_iommufd; +u32 count =0 , index =0 , *devices =NULL ; +struct vfio_pci_core_device *cur; +bool slot = false; +int ret =0 ; + +if (copy_from_user (&hdr, arg, minsz)) +return -EFAULT ; + +if (hdr.argsz < minsz) +return -EINVAL ; + + hdr.flags =0 ; + +/* Can we do a slot or bus reset or neither? */ +if (!pci_probe_reset_slot (vdev->pdev->slot)) + slot = true; +else if (pci_probe_reset_bus (vdev->pdev->bus)) +return -ENODEV ; + +mutex_lock (&vdev->vdev.dev_set->lock); +if (!vfio_pci_dev_set_resettable (vdev->vdev.dev_set)) { + ret = -EPERM ; + goto out_unlock; + } + + iommufd =vfio_iommufd_physical_ictx (&vdev->vdev); +if (!iommufd) { + ret = -EPERM ; + goto out_unlock; + } + +/* How many devices are affected? */ + ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_count_devs, + &count, slot); +if (ret) + goto out_unlock; + +WARN_ON (!count);/* Should always be at least one */ + +/* +* If there's enough space, fill it now, otherwise return -ENOSPC and +* the number of devices affected. +*/ +if (hdr.argsz <sizeof (hdr) + (count *sizeof (*devices))) { + ret = -ENOSPC ; + hdr.count = count; + goto reset_info_exit; + } + + devices =kcalloc (count,sizeof (*devices),GFP_KERNEL ); +if (!devices) { + ret = -ENOMEM ; + goto reset_info_exit; + } + +list_for_each_entry (cur, &vdev->vdev.dev_set->device_list, vdev.dev_set_list) { + cur_iommufd =vfio_iommufd_physical_ictx (&cur->vdev); +if (cur->vdev.open_count) { +if (cur_iommufd != iommufd) { + ret = -EPERM ; +break ; + } + ret =vfio_iommufd_physical_devid (&cur->vdev, &devices[index]); +if (ret) +break ; + index++; + } + } + +reset_info_exit: +if (copy_to_user (arg, &hdr, minsz)) + ret = -EFAULT ; + +if (!ret) { +if (copy_to_user (&arg->devices, devices, + hdr.count *sizeof (*devices))) + ret = -EFAULT ; + } + +kfree (devices); +out_unlock: +mutex_unlock (&vdev->vdev.dev_set->lock); +return ret; +} +static int vfio_pci_ioctl_get_pci_hot_reset_info (struct vfio_pci_core_device *vdev,struct vfio_pci_hot_reset_info __user *arg)
@@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,return vfio_pci_ioctl_get_irq_info (vdev, uarg);case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO :return vfio_pci_ioctl_get_pci_hot_reset_info (vdev, uarg); +case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO : +return vfio_pci_ioctl_get_pci_hot_reset_group_info (vdev, uarg);case VFIO_DEVICE_GET_REGION_INFO :return vfio_pci_ioctl_get_region_info (vdev, uarg);case VFIO_DEVICE_IOEVENTFD :
include/uapi/linux/vfio.h +47 -0
53 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 16 lines · On Mon, 27 Mar 2023 02:34:58 -0700 Yi Liu <[email protected]> wrote:
This is a preparation for vfio device cdev as cdev gives userspace the capability to open device cdev fd and management stack (e.g. libvirt) could pass the device fd to the actual user (e.g. QEMU). As a result, the actual user has no idea about the device's bus:devfn information. This is a problem when user uses
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOto know the hot reset affected device scope as this ioctl returns bus:devfn info. For the fd passing usage, the acutal user cannot map the bus:devfn to the devices it has opened via the fd passed from management stack. So a new ioctl is required.This new ioctl reports the list of iommufd
dev_idthat is opened by the user. If there is affected device that is not bound to vfio driver or opened by another user, this command shall fail with-EPERM. For the noiommu mode in the vfio device cdev path, this shall fail as nodev_idwould be generated, hence nothing to report.
Quoted 7 lines
This ioctl is useless to the users that open vfio group as such users have no idea about the iommufd
dev_idand it can use the existingVFIO_DEVICE_GET_PCI_HOT_RESET_INFO. The user that uses the traditional mode vfio group/container would be failed if invoking this ioctl. But the user that uses the iommufd compat mode vfio group/container shall succeed. This is harmless as long as user cannot make use of it and should useVFIO_DEVICE_GET_PCI_HOT_RESET_INFO.
So VFIO_DEVICE_GET_PCI_HOT_RESET_INFO reports a group and bdf, but VFIO_DEVICE_GET_PCI_HOT_RESET_*GROUP*_INFO is meant for the non-group, cdev use case and returns a dev_id rather than a group???
Additionally, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO has a flags arg that isn't used, why do we need a new ioctl vs defining VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID. In fact, we could define vfio_dependent_device as:
struct vfio_pci_dependent_device {
union {
__u32 group_id;
__u32 dev_id;
}
__u16 segment;
__u8 bus;
__u8 devfn;
};If the user calls with the above flag, dev_id is valid, otherwise group_id. Perhaps segment:buus:devfn could still be filled in with a NULL/invalid dev_id if the user doesn't have permissions for the device so that debugging from userspace isn't so opaque. Thanks,
Alex
Quoted 180 lines
Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 98 ++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 47 +++++++++++++++ 2 files changed, 145 insertions(+)drivers/vfio/pci/vfio_pci_core.c +98
-0@@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,return ret; } +static struct pci_dev * +vfio_pci_dev_set_resettable (struct vfio_device_set *dev_set); + +static int vfio_pci_ioctl_get_pci_hot_reset_group_info ( +struct vfio_pci_core_device *vdev, +struct vfio_pci_hot_reset_group_info __user *arg) +{ +unsigned long minsz = + offsetofend(struct vfio_pci_hot_reset_group_info, count); +struct vfio_pci_hot_reset_group_info hdr; +struct iommufd_ctx *iommufd, *cur_iommufd; +u32 count =0 , index =0 , *devices =NULL ; +struct vfio_pci_core_device *cur; +bool slot = false; +int ret =0 ; + +if (copy_from_user (&hdr, arg, minsz)) +return -EFAULT ; + +if (hdr.argsz < minsz) +return -EINVAL ; + + hdr.flags =0 ; + +/* Can we do a slot or bus reset or neither? */ +if (!pci_probe_reset_slot (vdev->pdev->slot)) + slot = true; +else if (pci_probe_reset_bus (vdev->pdev->bus)) +return -ENODEV ; + +mutex_lock (&vdev->vdev.dev_set->lock); +if (!vfio_pci_dev_set_resettable (vdev->vdev.dev_set)) { + ret = -EPERM ; + goto out_unlock; + } + + iommufd =vfio_iommufd_physical_ictx (&vdev->vdev); +if (!iommufd) { + ret = -EPERM ; + goto out_unlock; + } + +/* How many devices are affected? */ + ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_count_devs, + &count, slot); +if (ret) + goto out_unlock; + +WARN_ON (!count);/* Should always be at least one */ + +/* +* If there's enough space, fill it now, otherwise return -ENOSPC and +* the number of devices affected. +*/ +if (hdr.argsz <sizeof (hdr) + (count *sizeof (*devices))) { + ret = -ENOSPC ; + hdr.count = count; + goto reset_info_exit; + } + + devices =kcalloc (count,sizeof (*devices),GFP_KERNEL ); +if (!devices) { + ret = -ENOMEM ; + goto reset_info_exit; + } + +list_for_each_entry (cur, &vdev->vdev.dev_set->device_list, vdev.dev_set_list) { + cur_iommufd =vfio_iommufd_physical_ictx (&cur->vdev); +if (cur->vdev.open_count) { +if (cur_iommufd != iommufd) { + ret = -EPERM ; +break ; + } + ret =vfio_iommufd_physical_devid (&cur->vdev, &devices[index]); +if (ret) +break ; + index++; + } + } + +reset_info_exit: +if (copy_to_user (arg, &hdr, minsz)) + ret = -EFAULT ; + +if (!ret) { +if (copy_to_user (&arg->devices, devices, + hdr.count *sizeof (*devices))) + ret = -EFAULT ; + } + +kfree (devices); +out_unlock: +mutex_unlock (&vdev->vdev.dev_set->lock); +return ret; +} +static int vfio_pci_ioctl_get_pci_hot_reset_info (struct vfio_pci_core_device *vdev,struct vfio_pci_hot_reset_info __user *arg)@@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,return vfio_pci_ioctl_get_irq_info (vdev, uarg);case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO :return vfio_pci_ioctl_get_pci_hot_reset_info (vdev, uarg); +case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO : +return vfio_pci_ioctl_get_pci_hot_reset_group_info (vdev, uarg);case VFIO_DEVICE_GET_REGION_INFO :return vfio_pci_ioctl_get_region_info (vdev, uarg);case VFIO_DEVICE_IOEVENTFD :include/uapi/linux/vfio.h +47
-0@@ -669,6 +669,53 @@ struct vfio_pci_hot_reset_info {#define VFIO_DEVICE_GET_PCI_HOT_RESET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12) +/** +* VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, +* struct vfio_pci_hot_reset_group_info) +* +* This is used in the vfio device cdev mode. It returns the list of +* affected devices (represented by iommufd dev_id) when hot reset is +* issued on the current device with which this ioctl is invoked. It +* only includes the devices that are opened by the current user in the +* time of this command is invoked. This list may change when user opens +* new device or close opened device, hence user should re-invoke it +* after open/close devices. This command has no guarantee on the result +* of VFIO_DEVICE_PCI_HOT_RESET since the not-opened affected device can +* be by other users in the window between the two ioctls. If the affected +* devices are opened by multiple users, the VFIO_DEVICE_PCI_HOT_RESET +* shall fail, detail can check the description of VFIO_DEVICE_PCI_HOT_RESET. +* +* For the users that open vfio group/container, this ioctl is useless as +* they have no idea about the iommufd dev_id returned by this ioctl. For +* the users of the traditional mode vfio group/container, this ioctl will +* fail as this mode does not use iommufd hence no dev_id to report back. +* For the users of the iommufd compat mode vfio group/container, this ioctl +* would succeed as this mode uses iommufd as container fd. But such users +* still have no idea about the iommufd dev_id as the dev_id is only stored +* in kernel in this mode. For the users of the vfio group/container, the +* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO should be used to know the hot reset +* affected devices. +* +* Return: 0 on success, -errno on failure: +* -enospc = insufficient buffer; +* -enodev = unsupported for device; +* -eperm = no permission for device, this error comes: +* - when there are affected devices that are opened but +* not bound to the same iommufd with the current device +* with which this ioctl is invoked, +* - there are affected devices that are not bound to vfio +* driver yet. +* - no valid iommufd is bound (e.g. noiommu mode) +*/ +struct vfio_pci_hot_reset_group_info { +__u32 argsz; +__u32 flags; +__u32 count; +__u32 devices[]; +}; + +#define VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO _IO(VFIO_TYPE, VFIO_BASE + 18) +/** * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset)
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 139 lines · On Mon, 27 Mar 2023 13:26:19 -0600 Alex Williamson…
Quoted 16 lines · On Mon, 27 Mar 2023 02:34:58 -0700 Yi Liu <[email protected]> wrote:
This is a preparation for vfio device cdev as cdev gives userspace the capability to open device cdev fd and management stack (e.g. libvirt) could pass the device fd to the actual user (e.g. QEMU). As a result, the actual user has no idea about the device's bus:devfn information. This is a problem when user uses
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOto know the hot reset affected device scope as this ioctl returns bus:devfn info. For the fd passing usage, the acutal user cannot map the bus:devfn to the devices it has opened via the fd passed from management stack. So a new ioctl is required.This new ioctl reports the list of iommufd
dev_idthat is opened by the user. If there is affected device that is not bound to vfio driver or opened by another user, this command shall fail with-EPERM. For the noiommu mode in the vfio device cdev path, this shall fail as nodev_idwould be generated, hence nothing to report.Quoted 7 lines
This ioctl is useless to the users that open vfio group as such users have no idea about the iommufd
dev_idand it can use the existingVFIO_DEVICE_GET_PCI_HOT_RESET_INFO. The user that uses the traditional mode vfio group/container would be failed if invoking this ioctl. But the user that uses the iommufd compat mode vfio group/container shall succeed. This is harmless as long as user cannot make use of it and should useVFIO_DEVICE_GET_PCI_HOT_RESET_INFO.So
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOreports a group and bdf, but VFIO_DEVICE_GET_PCI_HOT_RESET_*GROUP*_INFO is meant for the non-group, cdev use case and returns adev_idrather than a group???Additionally,
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOhas a flags arg that isn't used, why do we need a new ioctl vs definingVFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID. In fact, we could definevfio_dependent_deviceas:struct vfio_pci_dependent_device { union { __u32 group_id; __u32 dev_id; } __u16 segment; __u8 bus; __u8 devfn; };If the user calls with the above flag,
dev_idis valid, otherwisegroup_id. Perhaps segment:buus:devfn could still be filled in with a NULL/invaliddev_idif the user doesn't have permissions for the device so that debugging from userspace isn't so opaque. Thanks,Alex
Quoted 85 lines
Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 98 ++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 47 +++++++++++++++ 2 files changed, 145 insertions(+)drivers/vfio/pci/vfio_pci_core.c +71
-0@@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,return ret; } +static struct pci_dev * +vfio_pci_dev_set_resettable (struct vfio_device_set *dev_set); + +static int vfio_pci_ioctl_get_pci_hot_reset_group_info ( +struct vfio_pci_core_device *vdev, +struct vfio_pci_hot_reset_group_info __user *arg) +{ +unsigned long minsz = + offsetofend(struct vfio_pci_hot_reset_group_info, count); +struct vfio_pci_hot_reset_group_info hdr; +struct iommufd_ctx *iommufd, *cur_iommufd; +u32 count =0 , index =0 , *devices =NULL ; +struct vfio_pci_core_device *cur; +bool slot = false; +int ret =0 ; + +if (copy_from_user (&hdr, arg, minsz)) +return -EFAULT ; + +if (hdr.argsz < minsz) +return -EINVAL ; + + hdr.flags =0 ; + +/* Can we do a slot or bus reset or neither? */ +if (!pci_probe_reset_slot (vdev->pdev->slot)) + slot = true; +else if (pci_probe_reset_bus (vdev->pdev->bus)) +return -ENODEV ; + +mutex_lock (&vdev->vdev.dev_set->lock); +if (!vfio_pci_dev_set_resettable (vdev->vdev.dev_set)) { + ret = -EPERM ; + goto out_unlock; + } + + iommufd =vfio_iommufd_physical_ictx (&vdev->vdev); +if (!iommufd) { + ret = -EPERM ; + goto out_unlock; + } + +/* How many devices are affected? */ + ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_count_devs, + &count, slot); +if (ret) + goto out_unlock; + +WARN_ON (!count);/* Should always be at least one */ + +/* +* If there's enough space, fill it now, otherwise return -ENOSPC and +* the number of devices affected. +*/ +if (hdr.argsz <sizeof (hdr) + (count *sizeof (*devices))) { + ret = -ENOSPC ; + hdr.count = count; + goto reset_info_exit; + } + + devices =kcalloc (count,sizeof (*devices),GFP_KERNEL ); +if (!devices) { + ret = -ENOMEM ; + goto reset_info_exit; + } + +list_for_each_entry (cur, &vdev->vdev.dev_set->device_list, vdev.dev_set_list) { + cur_iommufd =vfio_iommufd_physical_ictx (&cur->vdev); +if (cur->vdev.open_count) { +if (cur_iommufd != iommufd) { + ret = -EPERM ;
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1262
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +8
-0+break ; + } + ret =vfio_iommufd_physical_devid (&cur->vdev, &devices[index]); +if (ret) +break ; + index++; + } + }
This also makes use of vfio_pci_for_each_slot_or_bus() to iterate affected devices, but then still assumes those affected devices are necessarily represented in the dev_set. For example, a two-function device with ACS isolation can have function 0 bound to vfio and function 1 bound to a native host driver. The above code requires the user to pass a buffer large enough for both functions, but only function 0 is part of the dev_set, so function 1 is not reported as affected, nor does it generate an error.
It looks like we also fail to set hdr.count except in the error path above, so the below copy_to_user() copies a user specified range beyond the end the allocated devices buffer out to userspace! Thanks,
Alex
Quoted 88 lines
Quoted 87 lines
drivers/vfio/pci/vfio_pci_core.c +19
-0+ +reset_info_exit: +if (copy_to_user (arg, &hdr, minsz)) + ret = -EFAULT ; + +if (!ret) { +if (copy_to_user (&arg->devices, devices, + hdr.count *sizeof (*devices))) + ret = -EFAULT ; + } + +kfree (devices); +out_unlock: +mutex_unlock (&vdev->vdev.dev_set->lock); +return ret; +} +static int vfio_pci_ioctl_get_pci_hot_reset_info (struct vfio_pci_core_device *vdev,struct vfio_pci_hot_reset_info __user *arg)@@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,return vfio_pci_ioctl_get_irq_info (vdev, uarg);case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO :return vfio_pci_ioctl_get_pci_hot_reset_info (vdev, uarg); +case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO : +return vfio_pci_ioctl_get_pci_hot_reset_group_info (vdev, uarg);case VFIO_DEVICE_GET_REGION_INFO :return vfio_pci_ioctl_get_region_info (vdev, uarg);case VFIO_DEVICE_IOEVENTFD :include/uapi/linux/vfio.h +47
-0@@ -669,6 +669,53 @@ struct vfio_pci_hot_reset_info {#define VFIO_DEVICE_GET_PCI_HOT_RESET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12) +/** +* VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, +* struct vfio_pci_hot_reset_group_info) +* +* This is used in the vfio device cdev mode. It returns the list of +* affected devices (represented by iommufd dev_id) when hot reset is +* issued on the current device with which this ioctl is invoked. It +* only includes the devices that are opened by the current user in the +* time of this command is invoked. This list may change when user opens +* new device or close opened device, hence user should re-invoke it +* after open/close devices. This command has no guarantee on the result +* of VFIO_DEVICE_PCI_HOT_RESET since the not-opened affected device can +* be by other users in the window between the two ioctls. If the affected +* devices are opened by multiple users, the VFIO_DEVICE_PCI_HOT_RESET +* shall fail, detail can check the description of VFIO_DEVICE_PCI_HOT_RESET. +* +* For the users that open vfio group/container, this ioctl is useless as +* they have no idea about the iommufd dev_id returned by this ioctl. For +* the users of the traditional mode vfio group/container, this ioctl will +* fail as this mode does not use iommufd hence no dev_id to report back. +* For the users of the iommufd compat mode vfio group/container, this ioctl +* would succeed as this mode uses iommufd as container fd. But such users +* still have no idea about the iommufd dev_id as the dev_id is only stored +* in kernel in this mode. For the users of the vfio group/container, the +* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO should be used to know the hot reset +* affected devices. +* +* Return: 0 on success, -errno on failure: +* -enospc = insufficient buffer; +* -enodev = unsupported for device; +* -eperm = no permission for device, this error comes: +* - when there are affected devices that are opened but +* not bound to the same iommufd with the current device +* with which this ioctl is invoked, +* - there are affected devices that are not bound to vfio +* driver yet. +* - no valid iommufd is bound (e.g. noiommu mode) +*/ +struct vfio_pci_hot_reset_group_info { +__u32 argsz; +__u32 flags; +__u32 count; +__u32 devices[]; +}; + +#define VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO _IO(VFIO_TYPE, VFIO_BASE + 18) +/** * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset)
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 6 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 4:41 AMOn Mon, 27 Mar 2023 13:26:19 -0600
Alex Williamson <[email protected]> wrote:
[...]
Quoted 34 lines
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/vfio/pci/vfio_pci_core.cQuoted 4 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct
Quoted 4 lines
index
19f5b075d70a..45edf4e9b98b100644drivers/vfio/pci/vfio_pci_core.c +0
-0@@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct
vfio_pci_core_device*vdev,Quoted 27 lines
Quoted 27 lines
drivers/vfio/pci/vfio_pci_core.c +24
-0return ret; } +static struct pci_dev * +vfio_pci_dev_set_resettable (struct vfio_device_set *dev_set); + +static int vfio_pci_ioctl_get_pci_hot_reset_group_info ( +struct vfio_pci_core_device *vdev, +struct vfio_pci_hot_reset_group_info __user *arg) +{ +unsigned long minsz = + offsetofend(struct vfio_pci_hot_reset_group_info, count); +struct vfio_pci_hot_reset_group_info hdr; +struct iommufd_ctx *iommufd, *cur_iommufd; +u32 count =0 , index =0 , *devices =NULL ; +struct vfio_pci_core_device *cur; +bool slot = false; +int ret =0 ; + +if (copy_from_user (&hdr, arg, minsz)) +return -EFAULT ; + +if (hdr.argsz < minsz) +return -EINVAL ; + + hdr.flags =0 ; +
Quoted 8 lines
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1215
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +8
-0+/* Can we do a slot or bus reset or neither? */ +if (!pci_probe_reset_slot (vdev->pdev->slot)) + slot = true; +else if (pci_probe_reset_bus (vdev->pdev->bus)) +return -ENODEV ; + +mutex_lock (&vdev->vdev.dev_set->lock); +if (!vfio_pci_dev_set_resettable (vdev->vdev.dev_set)) {
[1]
Quoted 45 lines
Quoted 4 lines
Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +4
-0+ ret = -EPERM ; + goto out_unlock; + } +Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +8
-0+ iommufd =vfio_iommufd_physical_ictx (&vdev->vdev); +if (!iommufd) { + ret = -EPERM ; + goto out_unlock; + } + +/* How many devices are affected? */ + ret = vfio_pci_for_each_slot_or_bus(vdev->pdev,
vfio_pci_count_devs,Quoted 15 lines
Quoted 15 lines
drivers/vfio/pci/vfio_pci_core.c +15
-0+ &count, slot); +if (ret) + goto out_unlock; + +WARN_ON (!count);/* Should always be at least one */ + +/* +* If there's enough space, fill it now, otherwise return -ENOSPC and +* the number of devices affected. +*/ +if (hdr.argsz <sizeof (hdr) + (count *sizeof (*devices))) { + ret = -ENOSPC ; + hdr.count = count; + goto reset_info_exit; + }Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +8
-0+ + devices =kcalloc (count,sizeof (*devices),GFP_KERNEL ); +if (!devices) { + ret = -ENOMEM ; + goto reset_info_exit; + } + + list_for_each_entry(cur, &vdev->vdev.dev_set->device_list,vdev.dev_set_list) {
Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +7
-0+ cur_iommufd =vfio_iommufd_physical_ictx (&cur->vdev); +if (cur->vdev.open_count) { +if (cur_iommufd != iommufd) { + ret = -EPERM ; +break ; + } + ret = vfio_iommufd_physical_devid(&cur->vdev,&devices[index]);
Quoted 14 lines
Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1262
Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +5
-0+if (ret) +break ; + index++; + } + }This also makes use of
vfio_pci_for_each_slot_or_bus()to iterate affected devices, but then still assumes those affected devices are necessarily represented in thedev_set. For example, a two-function device with ACS isolation can have function 0 bound to vfio and function 1 bound to a native host driver. The above code requires the user to pass a buffer large enough for both functions, but only function 0 is part of thedev_set, so function 1 is not reported as affected, nor does it generate an error.
The vfio_pci_dev_set_resettable() is used at [1] to check if all the affected devices are in the dev_set. If not, then it fails at the first place. So in the following code, looping the devices in the dev_set is equivalent to looping devices with vfio_pci_for_each_slot_or_bus(). I need to loop dev_set as dev_set has the vfio_device which is more convenient to get dev_id.
Quoted 4 lines
It looks like we also fail to set hdr.count except in the error path above, so the below
copy_to_user()copies a user specified range beyond the end the allocated devices buffer out to userspace! Thanks,
Oops, yes, it is. My test only has one device affected, so it does not hit the problem. ☹
Thanks,
Yi Liu
Quoted 108 lines
Alex
Quoted 13 lines
Quoted 13 lines
drivers/vfio/pci/vfio_pci_core.c +13
-0+ +reset_info_exit: +if (copy_to_user (arg, &hdr, minsz)) + ret = -EFAULT ; + +if (!ret) { +if (copy_to_user (&arg->devices, devices, + hdr.count *sizeof (*devices))) + ret = -EFAULT ; + } + +kfree (devices); +out_unlock:Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c long vfio_pci_core_ioctl(struct vfio_device
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +4
-0+mutex_unlock (&vdev->vdev.dev_set->lock); +return ret; +} +static int vfio_pci_ioctl_get_pci_hot_reset_info (struct vfio_pci_core_device *vdev,struct vfio_pci_hot_reset_info __user *arg)@@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device
*core_vdev, unsigned int cmd,Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +2
-0return vfio_pci_ioctl_get_irq_info (vdev, uarg);case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO :return vfio_pci_ioctl_get_pci_hot_reset_info (vdev, uarg); +case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO : + return vfio_pci_ioctl_get_pci_hot_reset_group_info(vdev,uarg);
Quoted 10 lines
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 670
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +0
-0case VFIO_DEVICE_GET_REGION_INFO :return vfio_pci_ioctl_get_region_info (vdev, uarg);case VFIO_DEVICE_IOEVENTFD :include/uapi/linux/vfio.h +0
-0@@ -669,6 +669,53 @@ struct vfio_pci_hot_reset_info {#define VFIO_DEVICE_GET_PCI_HOT_RESET_INFO _IO(VFIO_TYPE,
VFIO_BASE+ 12)Quoted 3 lines
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 672
Quoted 3 lines
include/uapi/linux/vfio.h +2
-0+/** +* VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO - _IOWR(VFIO_TYPE,
VFIO_BASE+ 12,Quoted 1 line
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 673
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* struct
vfio_pci_hot_reset_group_info)Quoted 6 lines
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {
Quoted 6 lines
include/uapi/linux/vfio.h +6
-0+* +* This is used in the vfio device cdev mode. It returns the list of +* affected devices (represented by iommufd dev_id) when hot reset is +* issued on the current device with which this ioctl is invoked. It +* only includes the devices that are opened by the current user in the +* time of this command is invoked. This list may change when user opens
Quoted 2 lines
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 681
Quoted 2 lines
include/uapi/linux/vfio.h +2
-0+* new device or close opened device, hence user should re-invoke it +* after open/close devices. This command has no guarantee on the result
Quoted 1 line
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 682
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* of VFIO_DEVICE_PCI_HOT_RESET since the not-opened affected device can
Quoted 1 line
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 683
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* be by other users in the window between the two ioctls. If the affected
Quoted 1 line
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 684
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* devices are opened by multiple users, the
VFIO_DEVICE_PCI_HOT_RESETQuoted 1 line
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 685
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* shall fail, detail can check the description of
VFIO_DEVICE_PCI_HOT_RESET.Quoted 3 lines
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {
Quoted 3 lines
include/uapi/linux/vfio.h +3
-0+* +* For the users that open vfio group/container, this ioctl is useless as +* they have no idea about the iommufd dev_id returned by this ioctl. For
Quoted 2 lines
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 690
Quoted 2 lines
include/uapi/linux/vfio.h +2
-0+* the users of the traditional mode vfio group/container, this ioctl will +* fail as this mode does not use iommufd hence no dev_id to report back.
Quoted 1 line
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 691
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* For the users of the iommufd compat mode vfio group/container, this ioctl
Quoted 1 line
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 692
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* would succeed as this mode uses iommufd as container fd. But such users
Quoted 1 line
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 693
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* still have no idea about the iommufd dev_id as the dev_id is only stored
Quoted 2 lines
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 695
Quoted 2 lines
include/uapi/linux/vfio.h +2
-0+* in kernel in this mode. For the users of the vfio group/container, the +* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO should be used to know the hot reset
Quoted 13 lines
Quoted 13 lines
include/uapi/linux/vfio.h +13
-0+* affected devices. +* +* Return: 0 on success, -errno on failure: +* -enospc = insufficient buffer; +* -enodev = unsupported for device; +* -eperm = no permission for device, this error comes: +* - when there are affected devices that are opened but +* not bound to the same iommufd with the current device +* with which this ioctl is invoked, +* - there are affected devices that are not bound to vfio +* driver yet. +* - no valid iommufd is bound (e.g. noiommu mode) +*/ Quoted 8 lines
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {
Quoted 8 lines
include/uapi/linux/vfio.h +8
-0+struct vfio_pci_hot_reset_group_info { +__u32 argsz; +__u32 flags; +__u32 count; +__u32 devices[]; +}; + +#define VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO _IO(VFIO_TYPE, VFIO_BASE + 18)Quoted 5 lines
Quoted 4 lines
include/uapi/linux/vfio.h +1
-0+/** * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset)
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
In reply to Alex WilliamsonQuoted 31 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AMQuoted 16 lines · On Mon, 27 Mar 2023 02:34:58 -0700 Yi Liu <[email protected]> wrote:
This is a preparation for vfio device cdev as cdev gives userspace the capability to open device cdev fd and management stack (e.g. libvirt) could pass the device fd to the actual user (e.g. QEMU). As a result, the actual user has no idea about the device's bus:devfn information. This is a problem when user uses
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOto know the hot reset affected device scope as this ioctl returns bus:devfn info. For the fd passing usage, the acutal user cannot map the bus:devfn to the devices it has opened via the fd passed from management stack. So a new ioctl is required.This new ioctl reports the list of iommufd
dev_idthat is opened by the user. If there is affected device that is not bound to vfio driver or opened by another user, this command shall fail with-EPERM. For the noiommu mode in the vfio device cdev path, this shall fail as nodev_idwould be generated, hence nothing to report.Quoted 7 lines
This ioctl is useless to the users that open vfio group as such users have no idea about the iommufd
dev_idand it can use the existingVFIO_DEVICE_GET_PCI_HOT_RESET_INFO. The user that uses the traditional mode vfio group/container would be failed if invoking this ioctl. But the user that uses the iommufd compat mode vfio group/container shall succeed. This is harmless as long as user cannot make use of it and should useVFIO_DEVICE_GET_PCI_HOT_RESET_INFO.
Quoted 4 lines
So
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOreports a group and bdf, but VFIO_DEVICE_GET_PCI_HOT_RESET_*GROUP*_INFO is meant for the non- group,
cdev use case and returns adev_idrather than a group???
Yes, this is the meaning, but poor naming here ☹ I also struggled on it. Perhaps your below Suggestion makes more sense. Introducing a flag and reuse the existing _INFO ioctl.
Quoted 3 lines
Additionally,
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOhas a flags arg that isn't used, why do we need a new ioctl vs definingVFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.
Sure. I can follow this suggestion. BTW. I have a doubt here. This new flag is set by user. What if in the future kernel has new extensions and needs to report something new to the user and add new flags to tell user? Such flag is set by kernel. Then the flags field may have two kinds of flags (some set by user while some set by kernel). Will it mess up the flags space?
Quoted 12 lines
In fact, we could define
vfio_dependent_deviceas:struct vfio_pci_dependent_device { union { __u32 group_id; __u32 dev_id; } __u16 segment; __u8 bus; __u8 devfn; };
Quoted 4 lines
If the user calls with the above flag,
dev_idis valid, otherwisegroup_id. Perhaps segment:buus:devfn could still be filled in with a NULL/invaliddev_idif the user doesn't have permissions for the device so that debugging from userspace isn't so opaque. Thanks,
Also, have one question here. Should the invalid dev_id be defined in the vfio uapi or iommufd uapi? Maybe the latter one since dev_id is generated by iommufd subsystem.
Regards,
Yi Liu
Quoted 205 lines
Quoted 4 lines
Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 98++++++++++++++++++++++++++++++++
Review
Quoted 4 lines
include/uapi/linux/vfio.h | 47 +++++++++++++++ 2 files changed, 145 insertions(+)(quoted diff) +0
-0
b/drivers/vfio/pci/vfio_pci_core.cdrivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct
Quoted 4 lines
index
19f5b075d70a..45edf4e9b98b100644drivers/vfio/pci/vfio_pci_core.c +0
-0@@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct
vfio_pci_core_device*vdev,Quoted 37 lines
drivers/vfio/pci/vfio_pci_core.c +34
-0return ret; } +static struct pci_dev * +vfio_pci_dev_set_resettable (struct vfio_device_set *dev_set); + +static int vfio_pci_ioctl_get_pci_hot_reset_group_info ( +struct vfio_pci_core_device *vdev, +struct vfio_pci_hot_reset_group_info __user *arg) +{ +unsigned long minsz = + offsetofend(struct vfio_pci_hot_reset_group_info, count); +struct vfio_pci_hot_reset_group_info hdr; +struct iommufd_ctx *iommufd, *cur_iommufd; +u32 count =0 , index =0 , *devices =NULL ; +struct vfio_pci_core_device *cur; +bool slot = false; +int ret =0 ; + +if (copy_from_user (&hdr, arg, minsz)) +return -EFAULT ; + +if (hdr.argsz < minsz) +return -EINVAL ; + + hdr.flags =0 ; + +/* Can we do a slot or bus reset or neither? */ +if (!pci_probe_reset_slot (vdev->pdev->slot)) + slot = true; +else if (pci_probe_reset_bus (vdev->pdev->bus)) +return -ENODEV ; + +mutex_lock (&vdev->vdev.dev_set->lock); +if (!vfio_pci_dev_set_resettable (vdev->vdev.dev_set)) { + ret = -EPERM ; + goto out_unlock;drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10
-0+ } + + iommufd =vfio_iommufd_physical_ictx (&vdev->vdev); +if (!iommufd) { + ret = -EPERM ; + goto out_unlock; + } + +/* How many devices are affected? */ + ret = vfio_pci_for_each_slot_or_bus(vdev->pdev,
vfio_pci_count_devs,Quoted 13 lines
drivers/vfio/pci/vfio_pci_core.c +13
-0+ &count, slot); +if (ret) + goto out_unlock; + +WARN_ON (!count);/* Should always be at least one */ + +/* +* If there's enough space, fill it now, otherwise return -ENOSPC and +* the number of devices affected. +*/ +if (hdr.argsz <sizeof (hdr) + (count *sizeof (*devices))) { + ret = -ENOSPC ; + hdr.count = count;drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10
-0+ goto reset_info_exit; + } + + devices =kcalloc (count,sizeof (*devices),GFP_KERNEL ); +if (!devices) { + ret = -ENOMEM ; + goto reset_info_exit; + } + + list_for_each_entry(cur, &vdev->vdev.dev_set->device_list,vdev.dev_set_list) {
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +7
-0+ cur_iommufd =vfio_iommufd_physical_ictx (&cur->vdev); +if (cur->vdev.open_count) { +if (cur_iommufd != iommufd) { + ret = -EPERM ; +break ; + } + ret = vfio_iommufd_physical_devid(&cur->vdev,&devices[index]);
drivers/vfio/pci/vfio_pci_core.c long vfio_pci_core_ioctl(struct vfio_device
Quoted 26 lines
drivers/vfio/pci/vfio_pci_core.c +22
-0+if (ret) +break ; + index++; + } + } + +reset_info_exit: +if (copy_to_user (arg, &hdr, minsz)) + ret = -EFAULT ; + +if (!ret) { +if (copy_to_user (&arg->devices, devices, + hdr.count *sizeof (*devices))) + ret = -EFAULT ; + } + +kfree (devices); +out_unlock: +mutex_unlock (&vdev->vdev.dev_set->lock); +return ret; +} +static int vfio_pci_ioctl_get_pci_hot_reset_info (struct vfio_pci_core_device *vdev,struct vfio_pci_hot_reset_info __user *arg)@@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device
*core_vdev, unsigned int cmd,drivers/vfio/pci/vfio_pci_core.c long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +2
-0return vfio_pci_ioctl_get_irq_info (vdev, uarg);case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO :return vfio_pci_ioctl_get_pci_hot_reset_info (vdev, uarg); +case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO : + return vfio_pci_ioctl_get_pci_hot_reset_group_info(vdev,uarg);
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 670
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +0
-0case VFIO_DEVICE_GET_REGION_INFO :return vfio_pci_ioctl_get_region_info (vdev, uarg);case VFIO_DEVICE_IOEVENTFD :include/uapi/linux/vfio.h +0
-0@@ -669,6 +669,53 @@ struct vfio_pci_hot_reset_info {#define VFIO_DEVICE_GET_PCI_HOT_RESET_INFO _IO(VFIO_TYPE,
VFIO_BASE+ 12)include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 672
Quoted 3 lines
include/uapi/linux/vfio.h +2
-0+/** +* VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO - _IOWR(VFIO_TYPE,
VFIO_BASE+ 12,include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 673
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* struct
vfio_pci_hot_reset_group_info)include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {
Quoted 6 lines
include/uapi/linux/vfio.h +6
-0+* +* This is used in the vfio device cdev mode. It returns the list of +* affected devices (represented by iommufd dev_id) when hot reset is +* issued on the current device with which this ioctl is invoked. It +* only includes the devices that are opened by the current user in the +* time of this command is invoked. This list may change when user opens
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 681
Quoted 2 lines
include/uapi/linux/vfio.h +2
-0+* new device or close opened device, hence user should re-invoke it +* after open/close devices. This command has no guarantee on the result
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 682
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* of VFIO_DEVICE_PCI_HOT_RESET since the not-opened affected device can
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 683
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* be by other users in the window between the two ioctls. If the affected
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 684
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* devices are opened by multiple users, the
VFIO_DEVICE_PCI_HOT_RESETinclude/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 685
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* shall fail, detail can check the description of
VFIO_DEVICE_PCI_HOT_RESET.include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {
Quoted 3 lines
include/uapi/linux/vfio.h +3
-0+* +* For the users that open vfio group/container, this ioctl is useless as +* they have no idea about the iommufd dev_id returned by this ioctl. For
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 690
Quoted 2 lines
include/uapi/linux/vfio.h +2
-0+* the users of the traditional mode vfio group/container, this ioctl will +* fail as this mode does not use iommufd hence no dev_id to report back.
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 691
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* For the users of the iommufd compat mode vfio group/container, this ioctl
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 692
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* would succeed as this mode uses iommufd as container fd. But such users
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 693
Quoted 1 line
include/uapi/linux/vfio.h +1
-0+* still have no idea about the iommufd dev_id as the dev_id is only stored
include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info { · line 695
Quoted 2 lines
include/uapi/linux/vfio.h +2
-0+* in kernel in this mode. For the users of the vfio group/container, the +* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO should be used to know the hot reset
Quoted 11 lines
include/uapi/linux/vfio.h +11
-0+* affected devices. +* +* Return: 0 on success, -errno on failure: +* -enospc = insufficient buffer; +* -enodev = unsupported for device; +* -eperm = no permission for device, this error comes: +* - when there are affected devices that are opened but +* not bound to the same iommufd with the current device +* with which this ioctl is invoked, +* - there are affected devices that are not bound to vfio +* driver yet. include/uapi/linux/vfio.h struct vfio_pci_hot_reset_info {
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+* - no valid iommufd is bound (e.g. noiommu mode) +*/ +struct vfio_pci_hot_reset_group_info { +__u32 argsz; +__u32 flags; +__u32 count; +__u32 devices[]; +}; + +#define VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO _IO(VFIO_TYPE, VFIO_BASE + 18)Quoted 4 lines
include/uapi/linux/vfio.h +1
-0+/** * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset)
Tian, Kevin
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 10 lines
From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AMQuoted 6 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AMAdditionally,
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOhas a flags arg that isn't used, why do we need a new ioctl vs definingVFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.
Quoted 6 lines
Sure. I can follow this suggestion. BTW. I have a doubt here. This new flag is set by user. What if in the future kernel has new extensions and needs to report something new to the user and add new flags to tell user? Such flag is set by kernel. Then the flags field may have two kinds of flags (some set by user while some set by kernel). Will it mess up the flags space?
flags in a GET_INFO ioctl is for output.
if user needs to use flags as input to select different type of info then it should be split into multiple GET_INFO cmds.
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 17 lines · Kevin
Quoted 10 lines
From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AMQuoted 6 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AMAdditionally,
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOhas a flags arg that isn't used, why do we need a new ioctl vs definingVFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.Quoted 6 lines
Sure. I can follow this suggestion. BTW. I have a doubt here. This new flag is set by user. What if in the future kernel has new extensions and needs to report something new to the user and add new flags to tell user? Such flag is set by kernel. Then the flags field may have two kinds of flags (some set by user while some set by kernel). Will it mess up the flags space?
Quoted 4 lines
flags in a
GET_INFOioctl is for output.if user needs to use flags as input to select different type of info then it should be split into multiple
GET_INFOcmds.
I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.
Alternatively, should VFIO_DEVICE_GET_PCI_HOT_RESET_INFO automatically return the dev_id variant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,
Alex
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 32 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PMQuoted 7 lines · Kevin
Quoted 7 lines
From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AMQuoted 4 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AMAdditionally,
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOhas a flags argthat
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
isn't used, why do we need a new ioctl vs defining
VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.Sure. I can follow this suggestion. BTW. I have a doubt here. This new
flag
Quoted 3 lines
Quoted 3 lines
is set by user. What if in the future kernel has new extensions and needs to report something new to the user and add new flags to tell user? Such flag is set by kernel. Then the flags field may have two kinds of flags
(some
Quoted 6 lines
Quoted 2 lines
set by user while some set by kernel). Will it mess up the flags space?
flags in a
GET_INFOioctl is for output.if user needs to use flags as input to select different type of info then it
should
Quoted 1 line
be split into multiple
GET_INFOcmds.
Quoted 8 lines
I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.
Alternatively, should
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOautomatically
return thedev_idvariant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,
Personally I prefer that user asks for dev_id info explicitly. The major reason that we return dev_id is that the group/bdf info is not enough for the device fd passing case. But if qemu opens device by itself, the group/bdf info is still enough. So a device opened as a cdev doesn't mean it should return dev_id, it depends on if user has the bdf knowledge.
Regards,
Yi Liu
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 41 lines · Yi L
Quoted 32 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PMQuoted 7 lines · Kevin
Quoted 7 lines
From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AMQuoted 4 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AMAdditionally,
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOhas a flags argthat
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
isn't used, why do we need a new ioctl vs defining
VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.Sure. I can follow this suggestion. BTW. I have a doubt here. This new
flag
Quoted 3 lines
Quoted 3 lines
is set by user. What if in the future kernel has new extensions and needs to report something new to the user and add new flags to tell user? Such flag is set by kernel. Then the flags field may have two kinds of flags
(some
Quoted 6 lines
Quoted 2 lines
set by user while some set by kernel). Will it mess up the flags space?
flags in a
GET_INFOioctl is for output.if user needs to use flags as input to select different type of info then it
should
Quoted 1 line
be split into multiple
GET_INFOcmds.Quoted 8 lines
I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.
Alternatively, should
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOautomatically
return thedev_idvariant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,
Quoted 5 lines
Personally I prefer that user asks for
dev_idinfo explicitly. The major reason that we returndev_idis that the group/bdf info is not enough for the device fd passing case. But if qemu opens device by itself, the group/bdf info is still enough. So a device opened as a cdev doesn't mean it should returndev_id, it depends on if user has the bdf knowledge.
But if QEMU opens the cdev, vs getting it from the group, does it make any sense to return a set of group-ids + bdf in the host-reset info? I'm inclined to think the answer is no.
Per my previous suggestion, I think we should always return the bdf. We can't know if the user is accessing through an fd they opened themselves or were passed, but it allows an actually useful debugging report if userspace can know "I can't perform a hot reset of this device because my iommufd context doesn't know about device <bdf>", vs an opaque -EPERM. Therefore I think we're only discussing the data conveyed in the u32, a group-id or dev_id. Thanks,
Alex
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 61 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:46 PMQuoted 6 lines · Yi L
Quoted 6 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PMOn Tue, 28 Mar 2023 06:19:06 +0000
"Tian, Kevin" <[email protected]> wrote:Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AMQuoted 4 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AMAdditionally,
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOhas a flagsarg
Quoted 5 lines
Quoted 5 lines
that
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
isn't used, why do we need a new ioctl vs defining
VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.Sure. I can follow this suggestion. BTW. I have a doubt here. This
new
Quoted 2 lines
Quoted 2 lines
flag
Quoted 1 line
Quoted 1 line
is set by user. What if in the future kernel has new extensions and
needs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to report something new to the user and add new flags to tell user?
Such
Quoted 15 lines
Quoted 15 lines
Quoted 1 line
Quoted 1 line
flag is set by kernel. Then the flags field may have two kinds of flags
(some
Quoted 6 lines
Quoted 2 lines
set by user while some set by kernel). Will it mess up the flags space?
flags in a
GET_INFOioctl is for output.if user needs to use flags as input to select different type of info then it
should
Quoted 1 line
be split into multiple
GET_INFOcmds.I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.
Quoted 6 lines
Quoted 4 lines
Alternatively, should
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOautomatically
return thedev_idvariant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,Personally I prefer that user asks for
dev_idinfo explicitly. The majorreason
Quoted 1 line
that we return
dev_idis that the group/bdf info is not enough for thedevice
Quoted 1 line
fd passing case. But if qemu opens device by itself, the group/bdf info is
still
Quoted 1 line
enough. So a device opened as a cdev doesn't mean it should return
dev_id,Quoted 1 line
it depends on if user has the bdf knowledge.
Quoted 7 lines
But if QEMU opens the cdev, vs getting it from the group, does it make any sense to return a set of group-ids + bdf in the host-reset info? I'm inclined to think the answer is no.
Per my previous suggestion, I think we should always return the bdf. We can't know if the user is accessing through an fd they opened themselves or were passed,
Oh, yes. I'm convinced by this reason since only cdev mode supports device fd passing. So I'll reuse the existing _INFO and let kernel set a flag to mark the returned info is dev_id+bdf.
A check. If the device that the _INFIO is invoked is opened via cdev, but there are devices in the dev_set that are got via VFIO_GROUP_GET_DEVICE_FD, should I fail it or allow it?
Quoted 5 lines
but it allows an actually useful debugging
report if userspace can know "I can't perform a hot reset of this device because my iommufd context doesn't know about device <bdf>", vs an opaque-EPERM. Therefore I think we're only discussing the data conveyed in the u32, a group-id ordev_id. Thanks,
Sure.
Regards,
Yi Liu
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 69 lines · Yi L
Quoted 61 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:46 PMQuoted 6 lines · Yi L
Quoted 6 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PMOn Tue, 28 Mar 2023 06:19:06 +0000
"Tian, Kevin" <[email protected]> wrote:Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AMQuoted 4 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AMAdditionally,
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOhas a flagsarg
Quoted 5 lines
Quoted 5 lines
that
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
isn't used, why do we need a new ioctl vs defining
VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.Sure. I can follow this suggestion. BTW. I have a doubt here. This
new
Quoted 2 lines
Quoted 2 lines
flag
Quoted 1 line
Quoted 1 line
is set by user. What if in the future kernel has new extensions and
needs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to report something new to the user and add new flags to tell user?
Such
Quoted 15 lines
Quoted 15 lines
Quoted 1 line
Quoted 1 line
flag is set by kernel. Then the flags field may have two kinds of flags
(some
Quoted 6 lines
Quoted 2 lines
set by user while some set by kernel). Will it mess up the flags space?
flags in a
GET_INFOioctl is for output.if user needs to use flags as input to select different type of info then it
should
Quoted 1 line
be split into multiple
GET_INFOcmds.I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.
Quoted 6 lines
Quoted 4 lines
Alternatively, should
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOautomatically
return thedev_idvariant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,Personally I prefer that user asks for
dev_idinfo explicitly. The majorreason
Quoted 1 line
that we return
dev_idis that the group/bdf info is not enough for thedevice
Quoted 1 line
fd passing case. But if qemu opens device by itself, the group/bdf info is
still
Quoted 1 line
enough. So a device opened as a cdev doesn't mean it should return
dev_id,Quoted 1 line
it depends on if user has the bdf knowledge.
Quoted 7 lines
But if QEMU opens the cdev, vs getting it from the group, does it make any sense to return a set of group-ids + bdf in the host-reset info? I'm inclined to think the answer is no.
Per my previous suggestion, I think we should always return the bdf. We can't know if the user is accessing through an fd they opened themselves or were passed,
Quoted 7 lines
Oh, yes. I'm convinced by this reason since only cdev mode supports device fd passing. So I'll reuse the existing
_INFOand let kernel set a flag to mark the returned info is dev_id+bdf.A check. If the device that the
_INFIOis invoked is opened via cdev, but there are devices in thedev_setthat are got viaVFIO_GROUP_GET_DEVICE_FD, should I fail it or allow it?
It's a niche case, but I think it needs to be allowed. We'd still report the bdf for those devices, but make use of the invalid/null dev-id. I think this empowers userspace that they could make the same call on a group opened fd if necessary. An alternative would be to redefine the returned data structure entirely with a flag per entry describing the output, but then I think we need to invent a kernel policy of which gets reported, which seems overly complicated if our goal is to phase out group usage. Make sense, or will this bite us? Thanks,
Alex
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 87 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 11:18 PMQuoted 12 lines · Yi L
Quoted 12 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:46 PMQuoted 6 lines · Yi L
Quoted 6 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PMOn Tue, 28 Mar 2023 06:19:06 +0000
"Tian, Kevin" <[email protected]> wrote:Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AMQuoted 4 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AMAdditionally,
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOhas aflags
Quoted 11 lines
Quoted 11 lines
arg
Quoted 5 lines
Quoted 5 lines
that
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
isn't used, why do we need a new ioctl vs defining
VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.Sure. I can follow this suggestion. BTW. I have a doubt here. This
new
Quoted 2 lines
Quoted 2 lines
flag
Quoted 1 line
Quoted 1 line
is set by user. What if in the future kernel has new extensions and
needs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to report something new to the user and add new flags to tell
user?
Quoted 2 lines
Quoted 2 lines
Such
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
flag is set by kernel. Then the flags field may have two kinds of
flags
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(some
Quoted 1 line
Quoted 1 line
set by user while some set by kernel). Will it mess up the flags
space?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
flags in a
GET_INFOioctl is for output.if user needs to use flags as input to select different type of info
then it
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
should
Quoted 1 line
be split into multiple
GET_INFOcmds.I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.
Alternatively, should
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOautomatically
return thedev_idvariant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Personally I prefer that user asks for
dev_idinfo explicitly. The majorreason
Quoted 1 line
that we return
dev_idis that the group/bdf info is not enough for thedevice
Quoted 1 line
fd passing case. But if qemu opens device by itself, the group/bdf info
is
Quoted 9 lines
Quoted 9 lines
still
Quoted 1 line
enough. So a device opened as a cdev doesn't mean it should return
dev_id,Quoted 1 line
it depends on if user has the bdf knowledge.
But if QEMU opens the cdev, vs getting it from the group, does it make any sense to return a set of group-ids + bdf in the host-reset info? I'm inclined to think the answer is no.
Quoted 5 lines
Quoted 3 lines
Per my previous suggestion, I think we should always return the bdf. We can't know if the user is accessing through an fd they opened themselves or were passed,
Oh, yes. I'm convinced by this reason since only cdev mode supports
device fd
Quoted 1 line
passing. So I'll reuse the existing
_INFOand let kernel set a flag to markthe returned
Quoted 2 lines
info is dev_id+bdf.
Quoted 7 lines
Quoted 1 line
A check. If the device that the
_INFIOis invoked is opened via cdev, butthere
Quoted 1 line
are devices in the
dev_setthat are got viaVFIO_GROUP_GET_DEVICE_FD,should
Quoted 1 line
I fail it or allow it?
It's a niche case, but I think it needs to be allowed.
I'm also wondering if it is common in the future. Actually, a user should be preferred to either use the group or cdev, but not both. Otherwise, it looks to be bad programming.:-)
Also, as an earlier remark from Jason. If there are affected devices that are opened by other users, the new _INFO should fail with -EPERM. I know this remark was for the new _INFO ioctl. But now, we are going to reuse the existing _INFO, so I'd also want to check if we still need this policy? If yes, then it is a problem to check the owner of the devices that are opened by the group path.
https://lore.kernel.org/kvm/[email protected]/
Quoted 4 lines
We'd still
report the bdf for those devices, but make use of the invalid/null dev-id. I think this empowers userspace that they could make the same call on a group opened fd if necessary.
For the devices opened via group path, it should have an entry that includes invalid_dev_id+bdf. So user can map it to the device. But there is no group_id, this may be fine since group is just a shortcut to find the device. Is it?
Quoted 5 lines
An alternative would be to
redefine the returned data structure entirely with a flag per entry describing the output, but then I think we need to invent a kernel policy of which gets reported, which seems overly complicated if our goal is to phase out group usage. Make sense, or will this bite us?
This does smell complicated. 😊 maybe minimum change is better as the group is going to be phased out.
Regards,
Yi Liu
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 99 lines · Yi L
Quoted 87 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 11:18 PMQuoted 12 lines · Yi L
Quoted 12 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:46 PMQuoted 6 lines · Yi L
Quoted 6 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 10:26 PMOn Tue, 28 Mar 2023 06:19:06 +0000
"Tian, Kevin" <[email protected]> wrote:Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
From: Liu, Yi L <[email protected]>
Sent: Tuesday, March 28, 2023 11:32 AMQuoted 4 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, March 28, 2023 3:26 AMAdditionally,
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOhas aflags
Quoted 11 lines
Quoted 11 lines
arg
Quoted 5 lines
Quoted 5 lines
that
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
isn't used, why do we need a new ioctl vs defining
VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID.Sure. I can follow this suggestion. BTW. I have a doubt here. This
new
Quoted 2 lines
Quoted 2 lines
flag
Quoted 1 line
Quoted 1 line
is set by user. What if in the future kernel has new extensions and
needs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to report something new to the user and add new flags to tell
user?
Quoted 2 lines
Quoted 2 lines
Such
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
flag is set by kernel. Then the flags field may have two kinds of
flags
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(some
Quoted 1 line
Quoted 1 line
set by user while some set by kernel). Will it mess up the flags
space?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
flags in a
GET_INFOioctl is for output.if user needs to use flags as input to select different type of info
then it
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
should
Quoted 1 line
be split into multiple
GET_INFOcmds.I don't know that that's actually a rule, however we don't currently test flags is zero for input, so in this case I think we are stuck with it only being for output.
Alternatively, should
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOautomatically
return thedev_idvariant of the output and set a flag to indicate this is the case when called on a device fd opened as a cdev? Thanks,Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Personally I prefer that user asks for
dev_idinfo explicitly. The majorreason
Quoted 1 line
that we return
dev_idis that the group/bdf info is not enough for thedevice
Quoted 1 line
fd passing case. But if qemu opens device by itself, the group/bdf info
is
Quoted 9 lines
Quoted 9 lines
still
Quoted 1 line
enough. So a device opened as a cdev doesn't mean it should return
dev_id,Quoted 1 line
it depends on if user has the bdf knowledge.
But if QEMU opens the cdev, vs getting it from the group, does it make any sense to return a set of group-ids + bdf in the host-reset info? I'm inclined to think the answer is no.
Quoted 5 lines
Quoted 3 lines
Per my previous suggestion, I think we should always return the bdf. We can't know if the user is accessing through an fd they opened themselves or were passed,
Oh, yes. I'm convinced by this reason since only cdev mode supports
device fd
Quoted 1 line
passing. So I'll reuse the existing
_INFOand let kernel set a flag to markthe returned
Quoted 2 lines
info is dev_id+bdf.
Quoted 7 lines
Quoted 1 line
A check. If the device that the
_INFIOis invoked is opened via cdev, butthere
Quoted 1 line
are devices in the
dev_setthat are got viaVFIO_GROUP_GET_DEVICE_FD,should
Quoted 1 line
I fail it or allow it?
It's a niche case, but I think it needs to be allowed.
I'm also wondering if it is common in the future. Actually, a user should be preferred to either use the group or cdev, but not both. Otherwise, it looks to be bad programming.:-)
Quoted 8 lines
Also, as an earlier remark from Jason. If there are affected devices that are opened by other users, the new
_INFOshould fail with-EPERM. I know this remark was for the new_INFOioctl. But now, we are going to reuse the existing_INFO, so I'd also want to check if we still need this policy? If yes, then it is a problem to check the owner of the devices that are opened by the group path.
Personally I don't like the suggestion to fail with -EPERM if the user doesn't own all the affected devices. This isn't a "probe if I can do a reset" ioctl, it's a "provide information about the devices affected by a reset to know how to call the hot-reset ioctl". We're returning the bdf to the cdev version of this ioctl for exactly this debugging purpose when the devices are not owned, that becomes useless if we give up an return -EPERM if ownership doesn't align.
Quoted 9 lines
Quoted 4 lines
We'd still
report the bdf for those devices, but make use of the invalid/null dev-id. I think this empowers userspace that they could make the same call on a group opened fd if necessary.For the devices opened via group path, it should have an entry that includes invalid_dev_id+bdf. So user can map it to the device. But there is no
group_id, this may be fine since group is just a shortcut to find the device. Is it?
Yes, it could be argued that the group-id itself is superfluous, the user can determine the group via the bdf, but it also aligns with the hot-reset ioctl, which currently requires the group fd. Thanks,
Alex
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 3 lines
From: Alex Williamson <[email protected]>
Sent: Wednesday, March 29, 2023 12:00 AM
[...]
Quoted 26 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
A check. If the device that the
_INFIOis invoked is opened via cdev,but
Quoted 2 lines
Quoted 2 lines
there
Quoted 1 line
are devices in the
dev_setthat are got via
VFIO_GROUP_GET_DEVICE_FD,Quoted 7 lines
Quoted 4 lines
should
Quoted 1 line
I fail it or allow it?
It's a niche case, but I think it needs to be allowed.
I'm also wondering if it is common in the future. Actually, a user should be preferred to either use the group or cdev, but not both. Otherwise, it
looks
Quoted 3 lines
to be bad programming.:-)
Also, as an earlier remark from Jason. If there are affected devices that
are
Quoted 1 line
opened by other users, the new
_INFOshould fail with-EPERM. I knowthis
Quoted 6 lines
remark was for the new
_INFOioctl. But now, we are going to reuse the existing_INFO, so I'd also want to check if we still need this policy? If yes, then it is a problem to check the owner of the devices that are opened by the group path.
Quoted 7 lines
Personally I don't like the suggestion to fail with
-EPERMif the user doesn't own all the affected devices. This isn't a "probe if I can do a reset" ioctl, it's a "provide information about the devices affected by a reset to know how to call the hot-reset ioctl". We're returning the bdf to the cdev version of this ioctl for exactly this debugging purpose when the devices are not owned, that becomes useless if we give up an return-EPERMif ownership doesn't align.
Jason's suggestion makes sense for returning the case of returning dev_id as dev_id is local to iommufd. If there are devices in the same dev_set are opened by multiple users, multiple iommufd would be used. Then the dev_id would have overlap. e.g. a dev_set has three devices. Device A and B are opened by the current user as cdev, dev_id #1 and #2 are generated. While device C opened by another user as cdev, dev_id #n is generated for it. If dev_id #n happens to be #1, then user gets two info entries that have the same dev_id.
I know the existing _INFO does not have such policy since the group/bdf info are unique in the system. But for the dev_id case, seems really necessary to fail if the dev_set is not only opened by one user. From this angle, will it be good to have two ioctls. Sorry for the back-forth here. ☹
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
We'd still
report the bdf for those devices, but make use of the invalid/null dev-id. I think this empowers userspace that they could make the same call on a group opened fd if necessary.
Quoted 8 lines
Quoted 4 lines
For the devices opened via group path, it should have an entry that includes invalid_dev_id+bdf. So user can map it to the device. But there is no
group_id, this may be fine since group is just a shortcut to find the device. Is it?Yes, it could be argued that the group-id itself is superfluous, the user can determine the group via the bdf, but it also aligns with the hot-reset ioctl, which currently requires the group fd. Thanks,
I see. For the existing _INFO and HOT_RESET ioctl, I have no doubt. Both of them use the group as a short-cut.
Regards,
Yi Liu
Tian, Kevin
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 15 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, March 29, 2023 11:14 AMQuoted 11 lines
From: Alex Williamson <[email protected]>
Sent: Wednesday, March 29, 2023 12:00 AMPersonally I don't like the suggestion to fail with
-EPERMif the user doesn't own all the affected devices. This isn't a "probe if I can do a reset" ioctl, it's a "provide information about the devices affected by a reset to know how to call the hot-reset ioctl". We're returning the bdf to the cdev version of this ioctl for exactly this debugging purpose when the devices are not owned, that becomes useless if we give up an return-EPERMif ownership doesn't align.
Quoted 9 lines
Jason's suggestion makes sense for returning the case of returning
dev_idasdev_idis local to iommufd. If there are devices in the samedev_setare opened by multiple users, multiple iommufd would be used. Then thedev_idwould have overlap. e.g. adev_sethas three devices. Device A and B are opened by the current user as cdev,dev_id#1 and #2 are generated. While device C opened by another user as cdev,dev_id#n is generated for it. Ifdev_id#n happens to be #1, then user gets two info entries that have the samedev_id.
In Alex's proposal you'll set a invalid dev_id for device C so the user can still get the info for diagnostic purpose instead of seeing an -EPERM error.
btw I found an open about fd pass scheme which may affect the choice here.
In concept even with cdev we still expect the userspace to maintain the group knowledge so it won't inadvertently attempt to assign devices in the same group to different IOAS's. It also needs such knowledge when constructing guest topology.
with fd passed in Qemu has no way to associate the fd to a group.
We could extend bind_iommufd to return the group id or introduce a new ioctl to query it per dev_id.
Once that is in place looks we don't need a new _INFO ioctl?
Thanks
Kevin
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 25 lines · Kevin
Quoted 15 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, March 29, 2023 11:14 AMQuoted 11 lines
From: Alex Williamson <[email protected]>
Sent: Wednesday, March 29, 2023 12:00 AMPersonally I don't like the suggestion to fail with
-EPERMif the user doesn't own all the affected devices. This isn't a "probe if I can do a reset" ioctl, it's a "provide information about the devices affected by a reset to know how to call the hot-reset ioctl". We're returning the bdf to the cdev version of this ioctl for exactly this debugging purpose when the devices are not owned, that becomes useless if we give up an return-EPERMif ownership doesn't align.Quoted 9 lines
Jason's suggestion makes sense for returning the case of returning
dev_idasdev_idis local to iommufd. If there are devices in the samedev_setare opened by multiple users, multiple iommufd would be used. Then thedev_idwould have overlap. e.g. adev_sethas three devices. Device A and B are opened by the current user as cdev,dev_id#1 and #2 are generated. While device C opened by another user as cdev,dev_id#n is generated for it. Ifdev_id#n happens to be #1, then user gets two info entries that have the samedev_id.
Quoted 2 lines
In Alex's proposal you'll set a invalid
dev_idfor device C so the user can still get the info for diagnostic purpose instead of seeing an-EPERMerror.
Yes, we shouldn't be reporting dev_ids outside of the user's iommufd context.
Quoted 8 lines
btw I found an open about fd pass scheme which may affect the choice here.
In concept even with cdev we still expect the userspace to maintain the group knowledge so it won't inadvertently attempt to assign devices in the same group to different IOAS's. It also needs such knowledge when constructing guest topology.
with fd passed in Qemu has no way to associate the fd to a group.
Hmm, QEMU tries to get the group for the device address space in the guest, so finding an existing group with a different address space indeed allows QEMU to know of this conflict since the group is the fundamental unit IOMMU context in the legacy vfio model.
Quoted 2 lines
We could extend
bind_iommufdto return the group id or introduce a new ioctl to query it perdev_id.
That would be ironic to go to all this trouble to remove groups from the API only to have them show up here. But with a cdev interface, don't we break that model of conflating isolation and address-ability?
For example, devices within a group cannot be bound to separate iommufds due to lack of isolation, which is handled via DMA ownership, but barring DMA aliasing issues, due to conventional PCI buses or quirks, cdev could allow devices within the same group to be managed by separate IOAS's. So the group information really isn't enough for userspace to infer address space restrictions with cdev anyway.
Therefore aren't we expecting this to be denied at attach_ioas() and QEMU shouldn't be making these sorts of assumptions for cdev anyway? Thanks,
Alex
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 5 lines · Alex Williamson
Quoted 2 lines
We could extend
bind_iommufdto return the group id or introduce a new ioctl to query it perdev_id.That would be ironic to go to all this trouble to remove groups from the API only to have them show up here.
Groups always had to be part of the API for advanced cases like qemu - the point was to make them a small side bit of information not front and center in control of everything.
Quoted 5 lines
For example, devices within a group cannot be bound to separate iommufds due to lack of isolation, which is handled via DMA ownership, but barring DMA aliasing issues, due to conventional PCI buses or quirks, cdev could allow devices within the same group to be managed by separate IOAS's.
Maybe some future kernel could do this, the API allows it at least..
Quoted 5 lines
So the group information really isn't enough for userspace to infer address space restrictions with cdev anyway.
Therefore aren't we expecting this to be denied at
attach_ioas()and QEMU shouldn't be making these sorts of assumptions for cdev anyway?
I guess we could make an API specifically to report same-iommu_domina information?
I was assuming qemu would use the group for now as I don't see a likely future when we would relax that restriction.. So I was keeping a "add it when we need it" attitude here.
Jason
Tian, Kevin
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 8 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, March 29, 2023 11:58 PMQuoted 3 lines · Alex Williamson
Quoted 2 lines
We could extend
bind_iommufdto return the group id or introduce a new ioctl to query it perdev_id.
Quoted 6 lines
Quoted 2 lines
That would be ironic to go to all this trouble to remove groups from the API only to have them show up here.
Groups always had to be part of the API for advanced cases like qemu - the point was to make them a small side bit of information not front and center in control of everything.
Agree.
Quoted 15 lines
Quoted 6 lines
For example, devices within a group cannot be bound to separate iommufds due to lack of isolation, which is handled via DMA ownership, but barring DMA aliasing issues, due to conventional PCI buses or quirks, cdev could allow devices within the same group to be managed by separate IOAS's.
Maybe some future kernel could do this, the API allows it at least..
Quoted 5 lines
So the group information really isn't enough for userspace to infer address space restrictions with cdev anyway.
Therefore aren't we expecting this to be denied at
attach_ioas()and QEMU shouldn't be making these sorts of assumptions for cdev anyway?
Quoted 7 lines
I guess we could make an API specifically to report same-iommu_domina information?
I was assuming qemu would use the group for now as I don't see a likely future when we would relax that restriction.. So I was keeping a "add it when we need it" attitude here.
IIRC we discussed this subgroup concept in the thread of reviewing my high level design proposal 2yrs ago. The consensus at the moment was that subgroup is architecturally allowed w/o DMA aliasing issues but we're yet to see a real demand of relaxing current group restriction to support it. Also with time moving newer platforms should have less multi-devices group so the need of subgroup is further decreased.
So I'm also inclined to laying the existing group restriction with cdev for now.
Then can we make a decision how this group_id might be reported?
In nesting series we'll have a GET_INFO ioctl per dev_id. It could be extended to report group_id too.
Or alternatively just return it in BIND_IOMMUFD together with dev_id.
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 2 lines · Kevin
In nesting series we'll have a
GET_INFOioctl perdev_id. It could be extended to reportgroup_idtoo.
I like the GET_INFO because it would work with VDPA, is it possible?
Jason
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
In reply to Alex WilliamsonQuoted 31 lines
From: Alex Williamson <[email protected]>
Sent: Wednesday, March 29, 2023 11:50 PMQuoted 25 lines · Kevin
Quoted 24 lines
From: Liu, Yi L <[email protected]>
Sent: Wednesday, March 29, 2023 11:14 AMQuoted 11 lines
From: Alex Williamson <[email protected]>
Sent: Wednesday, March 29, 2023 12:00 AMPersonally I don't like the suggestion to fail with
-EPERMif the user doesn't own all the affected devices. This isn't a "probe if I can do a reset" ioctl, it's a "provide information about the devices affected by a reset to know how to call the hot-reset ioctl". We're returning the bdf to the cdev version of this ioctl for exactly this debugging purpose when the devices are not owned, that becomes useless if we give up an return-EPERMif ownership doesn't align.Jason's suggestion makes sense for returning the case of returning
dev_idasdev_idis local to iommufd. If there are devices in the samedev_setare opened by multiple users, multiple iommufd would be used. Then thedev_idwould have overlap. e.g. adev_sethas three devices. Device A and B are opened by the current user as cdev,dev_id#1 and #2 are generated. While device C opened by another user as cdev,dev_id#n is generated for it. Ifdev_id#n happens to be #1, then user gets two info entries that have the samedev_id.
Quoted 5 lines
Quoted 2 lines
In Alex's proposal you'll set a invalid
dev_idfor device C so the user can still get the info for diagnostic purpose instead of seeing an-EPERMerror.Yes, we shouldn't be reporting
dev_idsoutside of the user's iommufd context.
Following Alex's suggestion, here are two commits to extend existing _INFO to report dev_id.
From ad5c81366813c5effd707a0b5f5e797f5fdb3115 Mon Sep 17 00:00:00 2001
From: Yi Liu <[email protected]>
Date: Thu, 30 Mar 2023 05:29:36 -0700
Subject: [PATCH] vfio: Mark cdev usage in vfio_device
There are users that needs to check if vfio_device is opened as cdev. e.g. vfio-pci.
--- drivers/vfio/device_cdev.c | 2 ++ include/linux/vfio.h | 14 ++++++++++++++ 2 files changed, 16 insertions(+)
drivers/vfio/device_cdev.c +2 -0
@@ -66,6 +66,7 @@ void vfio_device_cdev_close(struct vfio_device_file *df)return ;mutex_lock (&device->dev_set->lock); + device->cdev_opened = false;vfio_device_close (df);vfio_device_put_kvm (device);if (df->iommufd)
@@ -180,6 +181,7 @@ long vfio_device_ioctl_bind_iommufd(struct vfio_device_file *df,* read/write/mmap */ smp_store_release (&df->access_granted, true); + device->cdev_opened = true;mutex_unlock (&device->dev_set->lock);return 0 ;
include/linux/vfio.h +14 -0
@@ -58,6 +58,7 @@ struct vfio_device {struct device device;/* device.kref covers object life circle */ #if IS_ENABLED (CONFIG_VFIO_DEVICE_CDEV )struct cdev cdev; +bool cdev_opened;#endif refcount_t refcount;/* user count on registered device*/ unsigned int open_count;
@@ -167,6 +168,19 @@ static inline int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id) ((void (*)(struct vfio_device *vdev)) NULL)#endif +#if IS_ENABLED (CONFIG_VFIO_DEVICE_CDEV ) +static inline bool vfio_device_cdev_opened (struct vfio_device *device) +{ +lockdep_assert_held (&device->dev_set->lock); +return device->cdev_opened; +} +#else +static inline bool vfio_device_cdev_opened (struct vfio_device *device) +{ +return false; +} +#endif +/** * @migration_set_state: Optional callback to change the migration state for * devices that support migration. It's mandatory for
Signature · 15 lines
2.34.1 From f796cafd6c51e49adcf76352dc1daf14712c3a48 Mon Sep 17 00:00:00 2001 From: Yi Liu <[email protected]> Date: Thu, 30 Mar 2023 05:44:45 -0700 Subject: [PATCH] vfio/pci: Report dev_id in VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for the devices opened as cdev. Signed-off-by: Yi Liu <[email protected]> --- drivers/vfio/pci/vfio_pci_core.c | 59 ++++++++++++++++++++++++++++---- include/uapi/linux/vfio.h | 6 +++- 2 files changed, 57 insertions(+), 8 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +51 -7
106 lines not on this page · raw message
include/uapi/linux/vfio.h +5 -1
@@ -743,7 +743,10 @@ enum {* -enospc = insufficient buffer, -enodev = unsupported for device. */ struct vfio_pci_dependent_device { -__u32 group_id; +union { +__u32 group_id; +__u32 dev_id; + };__u16 segment;__u8 bus;__u8 devfn;/* Use PCI_SLOT/PCI_FUNC */
@@ -752,6 +755,7 @@ struct vfio_pci_dependent_device {struct vfio_pci_hot_reset_info {__u32 argsz;__u32 flags; +#define VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID (1 << 0)__u32 count;struct vfio_pci_dependent_device devices[]; };
Signature · 4 lines
2.34.1 Regards, Yi Liu
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 12 lines
From: Liu, Yi L <[email protected]>
Sent: Thursday, March 30, 2023 8:48 PMdrivers/vfio/pci/vfio_pci_core.c +2
-1if (fill->cur == fill->max)return -EAGAIN ;/* Something changed, try again */ @@ -791,7 +812,24 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)if (!iommu_group)return -EPERM ;/* Cannot reset non-isolated devices */ - fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); +/* +* If dev_id is needed, fill in the dev_id field, otherwise
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 826
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10
-0+* fill in group_id. +*/ +if (fill->require_devid) { +/* +* Report the devices that are opened as cdev and have +* the same iommufd with the fill->iommufd. Otherwise, +* just fill in an IOMMUFD_INVALID_ID. +*/ + vdev =vfio_pci_find_device_in_devset (dev_set, pdev); + if (vdev && !vfio_device_cdev_opened(vdev) &&
a typo..it should be
if (vdev && vfio_device_cdev_opened(vdev) &&
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +6
-0+ fill->iommufd == vfio_iommufd_physical_ictx(vdev)) +vfio_iommufd_physical_devid (vdev, &fill->devices[fill- >cur].dev_id); + fill->devices[fill->cur].dev_id =IOMMUFD_INVALID_ID ; + }else { + fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); + } fill->devices[fill->cur].segment =pci_domain_nr (pdev->bus); fill->devices[fill->cur].bus = pdev->bus->number;fill->devices[fill->cur].devfn =
pdev->devfn;
Regards,
Yi Liu
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
In reply to Liu, Yi LQuoted 5 lines · Yi L
drivers/vfio/pci/vfio_pci_core.c +5
-0+/* +* If dev_id is needed, fill in the dev_id field, otherwise +* fill in group_id. +*/ +if (fill->require_devid) {
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 829
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10
-0+/* +* Report the devices that are opened as cdev and have +* the same iommufd with the fill->iommufd. Otherwise, +* just fill in an IOMMUFD_INVALID_ID. +*/ + vdev =vfio_pci_find_device_in_devset (dev_set, pdev); +if (vdev && !vfio_device_cdev_opened (vdev) && + fill->iommufd ==vfio_iommufd_physical_ictx (vdev)) +vfio_iommufd_physical_devid (vdev, &fill->devices[fill->cur].dev_id); + fill->devices[fill->cur].dev_id =IOMMUFD_INVALID_ID ;
This needs an else?
I suggest to check for VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID on input as well. I know the old kernels don't enforce this but at least we could start enforcing it going forward so that the group path would reject it to catch userspace bugs.
May as well fix it up to fully validate the flags
Jason
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 19 lines · On Thu, 30 Mar 2023 19:44:55 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 5 lines · Yi L
drivers/vfio/pci/vfio_pci_core.c +5
-0+/* +* If dev_id is needed, fill in the dev_id field, otherwise +* fill in group_id. +*/ +if (fill->require_devid) {drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 829
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10
-0+/* +* Report the devices that are opened as cdev and have +* the same iommufd with the fill->iommufd. Otherwise, +* just fill in an IOMMUFD_INVALID_ID. +*/ + vdev =vfio_pci_find_device_in_devset (dev_set, pdev); +if (vdev && !vfio_device_cdev_opened (vdev) && + fill->iommufd ==vfio_iommufd_physical_ictx (vdev)) +vfio_iommufd_physical_devid (vdev, &fill->devices[fill->cur].dev_id); + fill->devices[fill->cur].dev_id =IOMMUFD_INVALID_ID ;This needs an else?
Quoted 6 lines
I suggest to check for
VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_IDon input as well. I know the old kernels don't enforce this but at least we could start enforcing it going forward so that the group path would reject it to catch userspace bugs.May as well fix it up to fully validate the flags
Is this under the guise of "if nobody complains it's ok, otherwise revert" plan? We report dev-id based on the nature of the device, not the provided flags, so I'm not sure I follow how this protects the group path, unless we've failed to clear the output flags on that path with this change. Thanks,
Alex
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 9 lines · Alex Williamson
Quoted 6 lines
I suggest to check for
VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_IDon input as well. I know the old kernels don't enforce this but at least we could start enforcing it going forward so that the group path would reject it to catch userspace bugs.May as well fix it up to fully validate the flags
Is this under the guise of "if nobody complains it's ok, otherwise revert" plan?
Yah, assuming people don't pass in uninited structs.
Jason
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
In reply to Tian, KevinQuoted 4 lines · Kevin
We could extend
bind_iommufdto return the group id or introduce a new ioctl to query it perdev_id.Once that is in place looks we don't need a new
_INFOioctl?
The iommu_group and the reset group are different things
The issue is processing the BDF strings, not the group ID.
Probably we should have some way for iommufd to report the group_id from the dev_id?
Jason
Tian, Kevin
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 8 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, March 29, 2023 11:50 PMQuoted 2 lines · Kevin
We could extend
bind_iommufdto return the group id or introduce a new ioctl to query it perdev_id.
Quoted 9 lines
Quoted 1 line
Once that is in place looks we don't need a new
_INFOioctl?The
iommu_groupand the reset group are different thingsThe issue is processing the BDF strings, not the group ID.
Probably we should have some way for iommufd to report the
group_idfrom thedev_id?
Yes, that is my thought. Though iommu_group and reset group are different things we could still leverage existing _INFO ioctl once there is a way to associated dev_id to group_id.
Tian, Kevin
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 17 lines
From: Tian, Kevin <[email protected]>
Sent: Thursday, March 30, 2023 9:10 AMQuoted 14 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, March 29, 2023 11:50 PMQuoted 4 lines · Kevin
We could extend
bind_iommufdto return the group id or introduce a new ioctl to query it perdev_id.Once that is in place looks we don't need a new
_INFOioctl?The
iommu_groupand the reset group are different thingsThe issue is processing the BDF strings, not the group ID.
Quoted 7 lines
Quoted 3 lines
Probably we should have some way for iommufd to report the
group_idfrom thedev_id?Yes, that is my thought. Though
iommu_groupand reset group are different things we could still leverage existing_INFOioctl once there is a way to associateddev_idtogroup_id.
Please ignore this comment. Yes they are different things so even if a dev_id is in a group_id reported on a reset BDF string it doesn't mean this dev_id is in the reset group.
Qemu can know that all affected devices are either owned by itself or not used by other processes if dev_id's opened by itself can be associated to all group_id's reported in the BDF strings. But it still lacks of information to tell the reset dependency within those opened devices within Qemu.
So we do need a new _INFO ioctl for cdev. :/
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
In reply to Alex WilliamsonQuoted 3 lines · Alex Williamson
It's a niche case, but I think it needs to be allowed. We'd still report the bdf for those devices, but make use of the invalid/null dev-id.
IDK, it makes the whole implementation much more complicated. Instead of just copying the current dev_set to the output and calling vfio_pci_dev_set_resettable() we need to do something more complex..
Keeping the current ioctl as-is means this IOCTL can be used to do any debugging by getting the actual BDF list.
It means we can make the a new ioctl simple and just return the dev_id array without these edge complications. I don't think merging two different ioctls is helping make things simple..
It seems like it does what qemu wants: call the new IOCTL, if it fails, call the old IOCTL and print out the BDF list to help debug and then exit.
On success use the data in the new ioctl to generate the machine configuration to pass the reset grouping into the VM.
When reset actually comes in just trigger it.
Jason
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 10 lines · On Tue, 28 Mar 2023 13:29:23 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 3 lines · Alex Williamson
It's a niche case, but I think it needs to be allowed. We'd still report the bdf for those devices, but make use of the invalid/null dev-id.
IDK, it makes the whole implementation much more complicated. Instead of just copying the current
dev_setto the output and callingvfio_pci_dev_set_resettable()we need to do something more complex..
Quoted 6 lines
Keeping the current ioctl as-is means this IOCTL can be used to do any debugging by getting the actual BDF list.
It means we can make the a new ioctl simple and just return the
dev_idarray without these edge complications. I don't think merging two different ioctls is helping make things simple..
OTOH, we already have an ioctl that essentially "does the right thing", we just want to swap out one return field for another. So which is more complicated, adding another ioctl that does not quite the same thing but still needing to maintain the old ioctl for detailed information, or making the old ioctl bi-modal to return the appropriate information for the type of device used to access it?
Quoted 3 lines
It seems like it does what qemu wants: call the new IOCTL, if it fails, call the old IOCTL and print out the BDF list to help debug and then exit.
Userspace is already dealing with a variable length array for the return value, why would it ever want to repeat that process to get debugging info. Besides, wouldn't QEMU prefer the similarity of making the same call for groups and cdev and simply keying on the data type of one field?
Quoted 4 lines
On success use the data in the new ioctl to generate the machine configuration to pass the reset grouping into the VM.
When reset actually comes in just trigger it.
"Just trigger it" is the same in either case. It seems bold to play the complexity argument when we already have a function that does 90% the correct thing where we can share much of the implementation and userspace code without duplicating, but still relying on a legacy interface for debugging. Thanks,
Alex
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 5 lines · Alex Williamson
"Just trigger it" is the same in either case. It seems bold to play the complexity argument when we already have a function that does 90% the correct thing where we can share much of the implementation and userspace code without duplicating, but still relying on a legacy interface for debugging.
It just doesn't seem like we are sharing a lot, this patch is a whole new function and you are saying the unique implementation needs to be more complex still.
Maybe the next version will be able to share more??
Like can we just patch the existing code that sets the group_id/dev_id or somethiing???
Still, I'm not sure that qemu will share really share much here, because if it runs in group mode then it has to parse the result in a totally different way than if it runs in dev_id mode. The ioctl call is only a line or two. I imagine qemu will end up with two different functions that both return some kind of internal list of qemu devices in the reset group.
Jason
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
In reply to Yi Liudrivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1248
Quoted 5 lines · Yi Liu
drivers/vfio/pci/vfio_pci_core.c +5
-0+ devices =kcalloc (count,sizeof (*devices),GFP_KERNEL ); +if (!devices) { + ret = -ENOMEM ; + goto reset_info_exit; + }
This doesn't need to be so complicated
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1257
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +8
-0+list_for_each_entry (cur, &vdev->vdev.dev_set->device_list, vdev.dev_set_list) { + cur_iommufd =vfio_iommufd_physical_ictx (&cur->vdev); +if (cur->vdev.open_count) { +if (cur_iommufd != iommufd) { + ret = -EPERM ; +break ; + } + ret =vfio_iommufd_physical_devid (&cur->vdev, &devices[index]);
u32 device;
if (index >= hdr.count)
return -ENOSPC;
ret = vfio_iommufd_physical_devid(&cur->vdev, &devices);
...
if (put_user(&arg->devices[index], device))
-EFAULT
index++;
Jason
Re: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO
Quoted 25 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 28, 2023 8:40 PMdrivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · line 1248
Quoted 5 lines · Yi Liu
drivers/vfio/pci/vfio_pci_core.c +5
-0+ devices =kcalloc (count,sizeof (*devices),GFP_KERNEL ); +if (!devices) { + ret = -ENOMEM ; + goto reset_info_exit; + }This doesn't need to be so complicated
Review
Quoted 1 line
(quoted diff) +1
-0+ list_for_each_entry(cur, &vdev->vdev.dev_set->device_list,vdev.dev_set_list) {
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +7
-0+ cur_iommufd =vfio_iommufd_physical_ictx (&cur->vdev); +if (cur->vdev.open_count) { +if (cur_iommufd != iommufd) { + ret = -EPERM ; +break ; + } + ret = vfio_iommufd_physical_devid(&cur->vdev,&devices[index]);
Quoted 6 lines
u32 device;
if (index >= hdr.count)
return-ENOSPC;ret =
vfio_iommufd_physical_devid(&cur->vdev, &devices);
Ok, so the whole point is that if cur->vdev->iommufd_ctx==null, then the vfio_iommufd_physical_devid() shall fail as well.
Quoted 3 lines
...
if (
put_user(&arg->devices[index], device))
Will modify it. let's close the "separate ioctl" v.s. "reuse ioctl + new flag" open with Alex first.
Thanks,
Yi Liu
Quoted 5 lines
-EFAULTindex++;
Jason
Patchwork
CI[Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)
In reply to Yi Liu== Series Details ==
== Series Details ==
Series: Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)
URL : https://patchwork.freedesktop.org/series/115264/
State : warning
== Summary ==
Error: dim checkpatch failed498fb1274097 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() 09b95d011880 vfio/pci: Only check ownership of opened devices in hot reseta93e0e52bde9 vfio/pci: Move the existing hot reset logic to be a helper
-:6: WARNING:COMMIT_LOG_LONG_LINE: Possible unwrapped commit description (prefer a maximum 75 chars per line) #6:
This prepares to add another method for hot reset. The major hot reset logic
total: 0 errors, 1 warnings, 0 checks, 100 lines checked6bed441dc040 vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device 617ce05d9e70 vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET 4302e450c3d2 vfio: Refine vfio file kAPIs for vfio PCI hot reset b87361e1e7b8 vfio: Accpet device file from vfio PCI hot reset path b86ec489e81d vfio/pci: Renaming for accepting device fd in hot reset path -:52: CHECK:PREFER_KERNEL_TYPES: Prefer kernel type 's32' over 'int32_t' #52: FILE: drivers/vfio/pci/vfio_pci_core.c:1265:
+ int32_t *fds;
total: 0 errors, 0 warnings, 1 checks, 127 lines checked 1dbe4a480fac vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl fb6e32ec2ecf vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO -:43: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #43: FILE: drivers/vfio/pci/vfio_pci_core.c:1187:
+static int vfio_pci_ioctl_get_pci_hot_reset_group_info(
total: 0 errors, 0 warnings, 1 checks, 163 lines checked
Patchwork
CI[Intel-gfx] ✓ Fi.CI.BAT: success for Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)
In reply to Yi Liu== Series Details ==
== Series Details ==
Series: Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)
URL : https://patchwork.freedesktop.org/series/115264/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_12918 -> Patchwork_115264v2
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/index.html
Participating hosts (37 -> 36)
------------------------------
Missing (1): fi-kbl-soraka
Known issues
------------
Here are the changes found in Patchwork_115264v2 that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@i915_pm_rpm@module-reload:
- bat-dg1-7: [PASS][1] -> [SKIP][2] ([i915#7855])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/bat-dg1-7/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/bat-dg1-7/igt@[email protected]
* igt@i915_selftest@live@reset:
- bat-rpls-1: [PASS][3] -> [ABORT][4] ([i915#4983])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/bat-rpls-1/igt@i915_selftest@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/bat-rpls-1/igt@i915_selftest@[email protected]
* igt@i915_suspend@basic-s2idle-without-i915:
- fi-rkl-11600: [PASS][5] -> [FAIL][6] ([fdo#103375])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/fi-rkl-11600/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/fi-rkl-11600/igt@[email protected]
* igt@kms_chamelium_hpd@common-hpd-after-suspend:
- fi-kbl-8809g: NOTRUN -> [SKIP][7] ([fdo#109271]) +1 similar issue
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/fi-kbl-8809g/igt@[email protected]
* igt@kms_pipe_crc_basic@read-crc:
- bat-dg2-11: NOTRUN -> [SKIP][8] ([i915#5354])
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/bat-dg2-11/igt@[email protected]
#### Possible fixes ####
* igt@i915_suspend@basic-s2idle-without-i915:
- fi-kbl-8809g: [ABORT][9] -> [PASS][10]
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/fi-kbl-8809g/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/fi-kbl-8809g/igt@[email protected]
#### Warnings ####
* igt@i915_selftest@live@slpc:
- bat-rpls-2: [DMESG-FAIL][11] ([i915#6997] / [i915#7913]) -> [DMESG-FAIL][12] ([i915#6367] / [i915#7913])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/bat-rpls-2/igt@i915_selftest@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/bat-rpls-2/igt@i915_selftest@[email protected]
[fdo#103375]: https://bugs.freedesktop.org/show_bug.cgi?id=103375
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
[i915#5354]: https://gitlab.freedesktop.org/drm/intel/issues/5354
[i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
[i915#6997]: https://gitlab.freedesktop.org/drm/intel/issues/6997
[i915#7855]: https://gitlab.freedesktop.org/drm/intel/issues/7855
[i915#7913]: https://gitlab.freedesktop.org/drm/intel/issues/7913
Build changes
-------------
* Linux: CI_DRM_12918 -> Patchwork_115264v2
CI-20190529: 20190529
CI_DRM_12918: a1cb2211899ba6b8fe078586d0878aa918a5aab3 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_7220: 3eb7beb5c03343b29556025b1ada4b50849b5976 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
Patchwork_115264v2: a1cb2211899ba6b8fe078586d0878aa918a5aab3 @ git://anongit.freedesktop.org/gfx-ci/linux
### Linux commits
777a4de3d017 vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO 2a92707f4cc4 vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl 576175a255f8 vfio/pci: Renaming for accepting device fd in hot reset path 91c8b208c83f vfio: Accpet device file from vfio PCI hot reset path 9fe81a8d19f6 vfio: Refine vfio file kAPIs for vfio PCI hot reset 86cef152607b vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET ee577bf22ada vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device 6fb819b4ba9f vfio/pci: Move the existing hot reset logic to be a helper 6b5f8efac8f9 vfio/pci: Only check ownership of opened devices in hot reset de49aaf53338 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset()
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/index.html
Patchwork
CI[Intel-gfx] ✓ Fi.CI.IGT: success for Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)
In reply to Yi Liu== Series Details ==
== Series Details ==
Series: Introduce new methods for verifying ownership in vfio PCI hot reset (rev2)
URL : https://patchwork.freedesktop.org/series/115264/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_12918_full -> Patchwork_115264v2_full
====================================================
Summary
-------
**SUCCESS**
No regressions found.
Participating hosts (7 -> 7)
------------------------------
Additional (1): shard-rkl0
Missing (1): shard-rkl
Known issues
------------
Here are the changes found in Patchwork_115264v2_full that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@gem_exec_fair@basic-pace-share@rcs0:
- shard-apl: [PASS][1] -> [FAIL][2] ([i915#2842])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-apl3/igt@gem_exec_fair@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-apl1/igt@gem_exec_fair@[email protected]
* igt@gen9_exec_parse@allowed-single:
- shard-glk: [PASS][3] -> [ABORT][4] ([i915#5566])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-glk9/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-glk9/igt@[email protected]
* igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions:
- shard-glk: [PASS][5] -> [FAIL][6] ([i915#2346])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-glk4/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-glk2/igt@[email protected]
* igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size:
- shard-apl: [PASS][7] -> [FAIL][8] ([i915#2346]) +1 similar issue
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-apl6/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size.html
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-apl6/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size.html
#### Possible fixes ####
* igt@gem_eio@hibernate:
- {shard-tglu}: [ABORT][9] ([i915#7975]) -> [PASS][10]
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-tglu-10/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-tglu-9/igt@[email protected]
* igt@gem_exec_fair@basic-pace-solo@rcs0:
- shard-glk: [FAIL][11] ([i915#2842]) -> [PASS][12]
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-glk6/igt@gem_exec_fair@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-glk9/igt@gem_exec_fair@[email protected]
* igt@i915_pm_rpm@dpms-mode-unset-lpsp:
- {shard-tglu}: [SKIP][13] ([i915#1397]) -> [PASS][14]
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-tglu-9/igt@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-tglu-8/igt@[email protected]
* igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-gtt:
- {shard-tglu}: [SKIP][15] ([i915#1849]) -> [PASS][16] +7 similar issues
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-tglu-9/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-gtt.html
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-tglu-8/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-gtt.html
* igt@kms_vblank@invalid:
- {shard-tglu}: [SKIP][17] ([i915#1845] / [i915#7651]) -> [PASS][18] +9 similar issues
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-tglu-9/igt@[email protected]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-tglu-8/igt@[email protected]
* igt@kms_vblank@pipe-b-ts-continuation-modeset-rpm:
- {shard-tglu}: [SKIP][19] ([i915#1845]) -> [PASS][20] +18 similar issues
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12918/shard-tglu-10/igt@[email protected]
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/shard-tglu-5/igt@[email protected]
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
[fdo#109274]: https://bugs.freedesktop.org/show_bug.cgi?id=109274
[fdo#109280]: https://bugs.freedesktop.org/show_bug.cgi?id=109280
[fdo#109289]: https://bugs.freedesktop.org/show_bug.cgi?id=109289
[fdo#109314]: https://bugs.freedesktop.org/show_bug.cgi?id=109314
[fdo#109315]: https://bugs.freedesktop.org/show_bug.cgi?id=109315
[fdo#109506]: https://bugs.freedesktop.org/show_bug.cgi?id=109506
[fdo#110189]: https://bugs.freedesktop.org/show_bug.cgi?id=110189
[fdo#111068]: https://bugs.freedesktop.org/show_bug.cgi?id=111068
[fdo#111615]: https://bugs.freedesktop.org/show_bug.cgi?id=111615
[fdo#111825]: https://bugs.freedesktop.org/show_bug.cgi?id=111825
[fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
[fdo#112283]: https://bugs.freedesktop.org/show_bug.cgi?id=112283
[i915#1072]: https://gitlab.freedesktop.org/drm/intel/issues/1072
[i915#1397]: https://gitlab.freedesktop.org/drm/intel/issues/1397
[i915#1755]: https://gitlab.freedesktop.org/drm/intel/issues/1755
[i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
[i915#1849]: https://gitlab.freedesktop.org/drm/intel/issues/1849
[i915#2346]: https://gitlab.freedesktop.org/drm/intel/issues/2346
[i915#2527]: https://gitlab.freedesktop.org/drm/intel/issues/2527
[i915#2575]: https://gitlab.freedesktop.org/drm/intel/issues/2575
[i915#2587]: https://gitlab.freedesktop.org/drm/intel/issues/2587
[i915#2672]: https://gitlab.freedesktop.org/drm/intel/issues/2672
[i915#2681]: https://gitlab.freedesktop.org/drm/intel/issues/2681
[i915#2705]: https://gitlab.freedesktop.org/drm/intel/issues/2705
[i915#2842]: https://gitlab.freedesktop.org/drm/intel/issues/2842
[i915#2856]: https://gitlab.freedesktop.org/drm/intel/issues/2856
[i915#3116]: https://gitlab.freedesktop.org/drm/intel/issues/3116
[i915#3281]: https://gitlab.freedesktop.org/drm/intel/issues/3281
[i915#3282]: https://gitlab.freedesktop.org/drm/intel/issues/3282
[i915#3299]: https://gitlab.freedesktop.org/drm/intel/issues/3299
[i915#3359]: https://gitlab.freedesktop.org/drm/intel/issues/3359
[i915#3458]: https://gitlab.freedesktop.org/drm/intel/issues/3458
[i915#3469]: https://gitlab.freedesktop.org/drm/intel/issues/3469
[i915#3539]: https://gitlab.freedesktop.org/drm/intel/issues/3539
[i915#3555]: https://gitlab.freedesktop.org/drm/intel/issues/3555
[i915#3591]: https://gitlab.freedesktop.org/drm/intel/issues/3591
[i915#3637]: https://gitlab.freedesktop.org/drm/intel/issues/3637
[i915#3638]: https://gitlab.freedesktop.org/drm/intel/issues/3638
[i915#3689]: https://gitlab.freedesktop.org/drm/intel/issues/3689
[i915#3886]: https://gitlab.freedesktop.org/drm/intel/issues/3886
[i915#3938]: https://gitlab.freedesktop.org/drm/intel/issues/3938
[i915#3952]: https://gitlab.freedesktop.org/drm/intel/issues/3952
[i915#4077]: https://gitlab.freedesktop.org/drm/intel/issues/4077
[i915#4078]: https://gitlab.freedesktop.org/drm/intel/issues/4078
[i915#4083]: https://gitlab.freedesktop.org/drm/intel/issues/4083
[i915#4103]: https://gitlab.freedesktop.org/drm/intel/issues/4103
[i915#4212]: https://gitlab.freedesktop.org/drm/intel/issues/4212
[i915#4213]: https://gitlab.freedesktop.org/drm/intel/issues/4213
[i915#4270]: https://gitlab.freedesktop.org/drm/intel/issues/4270
[i915#433]: https://gitlab.freedesktop.org/drm/intel/issues/433
[i915#4538]: https://gitlab.freedesktop.org/drm/intel/issues/4538
[i915#4579]: https://gitlab.freedesktop.org/drm/intel/issues/4579
[i915#4771]: https://gitlab.freedesktop.org/drm/intel/issues/4771
[i915#4812]: https://gitlab.freedesktop.org/drm/intel/issues/4812
[i915#4833]: https://gitlab.freedesktop.org/drm/intel/issues/4833
[i915#4852]: https://gitlab.freedesktop.org/drm/intel/issues/4852
[i915#4859]: https://gitlab.freedesktop.org/drm/intel/issues/4859
[i915#4860]: https://gitlab.freedesktop.org/drm/intel/issues/4860
[i915#4879]: https://gitlab.freedesktop.org/drm/intel/issues/4879
[i915#4881]: https://gitlab.freedesktop.org/drm/intel/issues/4881
[i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
[i915#5176]: https://gitlab.freedesktop.org/drm/intel/issues/5176
[i915#5235]: https://gitlab.freedesktop.org/drm/intel/issues/5235
[i915#5286]: https://gitlab.freedesktop.org/drm/intel/issues/5286
[i915#5325]: https://gitlab.freedesktop.org/drm/intel/issues/5325
[i915#5439]: https://gitlab.freedesktop.org/drm/intel/issues/5439
[i915#5461]: https://gitlab.freedesktop.org/drm/intel/issues/5461
[i915#5563]: https://gitlab.freedesktop.org/drm/intel/issues/5563
[i915#5566]: https://gitlab.freedesktop.org/drm/intel/issues/5566
[i915#5723]: https://gitlab.freedesktop.org/drm/intel/issues/5723
[i915#6095]: https://gitlab.freedesktop.org/drm/intel/issues/6095
[i915#6245]: https://gitlab.freedesktop.org/drm/intel/issues/6245
[i915#6268]: https://gitlab.freedesktop.org/drm/intel/issues/6268
[i915#6524]: https://gitlab.freedesktop.org/drm/intel/issues/6524
[i915#658]: https://gitlab.freedesktop.org/drm/intel/issues/658
[i915#6944]: https://gitlab.freedesktop.org/drm/intel/issues/6944
[i915#7116]: https://gitlab.freedesktop.org/drm/intel/issues/7116
[i915#7118]: https://gitlab.freedesktop.org/drm/intel/issues/7118
[i915#7651]: https://gitlab.freedesktop.org/drm/intel/issues/7651
[i915#7701]: https://gitlab.freedesktop.org/drm/intel/issues/7701
[i915#7711]: https://gitlab.freedesktop.org/drm/intel/issues/7711
[i915#7828]: https://gitlab.freedesktop.org/drm/intel/issues/7828
[i915#7936]: https://gitlab.freedesktop.org/drm/intel/issues/7936
[i915#7975]: https://gitlab.freedesktop.org/drm/intel/issues/7975
[i915#7981]: https://gitlab.freedesktop.org/drm/intel/issues/7981
[i915#8011]: https://gitlab.freedesktop.org/drm/intel/issues/8011
[i915#8211]: https://gitlab.freedesktop.org/drm/intel/issues/8211
[i915#8234]: https://gitlab.freedesktop.org/drm/intel/issues/8234
Build changes
-------------
* Linux: CI_DRM_12918 -> Patchwork_115264v2
CI-20190529: 20190529
CI_DRM_12918: a1cb2211899ba6b8fe078586d0878aa918a5aab3 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_7220: 3eb7beb5c03343b29556025b1ada4b50849b5976 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
Patchwork_115264v2: a1cb2211899ba6b8fe078586d0878aa918a5aab3 @ git://anongit.freedesktop.org/gfx-ci/linux
piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_115264v2/index.html
Patchwork
CI[Intel-gfx] ✗ Fi.CI.BUILD: failure for Introduce new methods for verifying ownership in vfio PCI hot reset (rev3)
In reply to Yi Liu== Series Details ==
== Series Details ==
Series: Introduce new methods for verifying ownership in vfio PCI hot reset (rev3)
URL : https://patchwork.freedesktop.org/series/115264/
State : failure
== Summary ==
Error: patch https://patchwork.freedesktop.org/api/1.0/series/115264/revisions/3/mbox/ not applied
Applying: vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset()
Applying: vfio/pci: Only check ownership of opened devices in hot reset
Applying: vfio/pci: Move the existing hot reset logic to be a helper
Applying: vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
Applying: vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET
Applying: vfio: Refine vfio file kAPIs for vfio PCI hot reset
Applying: vfio: Accpet device file from vfio PCI hot reset path
Applying: vfio/pci: Renaming for accepting device fd in hot reset path
Applying: vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl
Patch is empty.
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To record the empty patch as an empty commit, run "git am --allow-empty".
To restore the original branch and stop patching, run "git am --abort".
Build failed, no error log produced
Quoted 18 lines
VFIO_DEVICE_PCI_HOT_RESETrequires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]
When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]
However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton
dev_sethence no ownership check is required. [3]
Quoted 7 lines
For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]
As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new
_INFOioctl to get hot reset scope for given device.
Tested NIC passthrough on Intel platform. Result looks good hence,
Tested by: Jiang, Yanting <[email protected]>
Thanks,
Yanting
Quoted 21 lines · Yanting
Quoted 21 lines
VFIO_DEVICE_PCI_HOT_RESETrequires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]
When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]
However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton
dev_sethence no ownership check is required. [3]For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]
Quoted 7 lines
Quoted 4 lines
As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new
_INFOioctl to get hot reset scope for given device.Tested NIC passthrough on Intel platform. Result looks good hence,
Tested by: Jiang, Yanting <[email protected]>
I'm not aware of any userspace that exercises this reset ioctl in cdev mode. Is this regression testing only? Thanks,
Alex
Quoted 53 lines
-----Original Message-----
From: Alex Williamson <[email protected]>
Sent: Friday, March 31, 2023 9:25 PM
To: Jiang, Yanting <[email protected]>
Cc: Liu, Yi L <[email protected]>; [email protected]; Tian, Kevin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; intel- [email protected]; [email protected]; Hao, Xudong <[email protected]>; Zhao, Yan Y <[email protected]>; Xu, Terrence <[email protected]>
Subject: Re: [PATCH v2 00/10] Introduce new methods for verifying ownership in vfio PCI hot resetQuoted 5 lines · Yanting
Quoted 5 lines
VFIO_DEVICE_PCI_HOT_RESETrequires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio devicecdev support.
Quoted 7 lines
Quoted 7 lines
First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]
When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd
information.
Quoted 8 lines
Quoted 8 lines
This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]
However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton
dev_sethence no ownership check is required. [3]Quoted 7 lines
Quoted 7 lines
For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership.
[4]As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new
_INFOioctl to get hot reset scope for givendevice.
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Tested NIC passthrough on Intel platform. Result looks good hence,
Tested by: Jiang, Yanting <[email protected]>I'm not aware of any userspace that exercises this reset ioctl in cdev mode. Is this regression testing only? Thanks,
Alex
Hi Alex,
Yes, only regression testing and some negative testing for NIC passthrough with legacy vfio mode, vfio iommufd compat mode, and cdev mode.
Thanks,
Yanting
Quoted 17 lines
VFIO_DEVICE_PCI_HOT_RESETrequires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]
When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]
However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton
dev_sethence no ownership check is required. [3]
Quoted 7 lines
For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]
As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new
_INFOioctl to get hot reset scope for given device.
Thanks,
Yanting
Quoted 77 lines
-----Original Message-----
From: Liu, Yi L <[email protected]>
Sent: Monday, March 27, 2023 5:35 PM
VFIO_DEVICE_PCI_HOT_RESETrequires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]
When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]
However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton
dev_sethence no ownership check is required. [3]For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]
As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new
_INFOioctl to get hot reset scope for given device.[1] https://lore.kernel.org/kvm/Y%[email protected]/
[2] https://lore.kernel.org/kvm/Y%[email protected]/#t
[3] https://lore.kernel.org/kvm/ZACX+Np%[email protected]/
[4]
https://lore.kernel.org/kvm/DS0PR11MB7529BE88460582BD599DC1F7C3B19
@DS0PR11MB7529.namprd11.prod.outlook.com/#t
[5] https://lore.kernel.org/kvm/[email protected]/
[6] https://lore.kernel.org/kvm/[email protected]/Change log:
v2:
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new_INFOioctl for the usage of device fd passing usage in cdev path (Jason, Alex)v1: https://lore.kernel.org/kvm/[email protected]/
Regards,
Yi LiuYi Liu (10):
vfio/pci: Update comment aroundgroup_fdget in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Only check ownership of opened devices in hot reset vfio/pci: Move the existing hot reset logic to be a helper
vfio-iommufd: Add helper to retrieveiommufd_ctxand devid for
vfio_device
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
vfio: Refine vfio file kAPIs for vfio PCI hot reset
vfio: Accpet device file from vfio PCI hot reset path vfio/pci: Renaming for accepting device fd in hot reset path vfio/pci: Accept device fd inVFIO_DEVICE_PCI_HOT_RESETioctl vfio/pci: AddVFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFOdrivers/iommu/iommufd/device.c | 12 ++ drivers/vfio/group.c | 32 ++-- drivers/vfio/iommufd.c | 16 ++ drivers/vfio/pci/vfio_pci_core.c | 244 ++++++++++++++++++++++++------- drivers/vfio/vfio.h | 2 + drivers/vfio/vfio_main.c | 44 ++++++ include/linux/iommufd.h | 3 + include/linux/vfio.h | 14 ++ include/uapi/linux/vfio.h | 65 +++++++- 9 files changed, 364 insertions(+), 68 deletions(-)
Quoted 2 lines
--
2.34.1
Verified this series by "Intel GVT-g GPU device mediated passthrough".
Passed VFIO legacy mode / compat mode / cdev mode basic functionality and GPU force reset test.
Quoted 76 lines · Terrence
Quoted 76 lines
-----Original Message-----
From: Liu, Yi L <[email protected]>
Sent: Monday, March 27, 2023 5:35 PM
VFIO_DEVICE_PCI_HOT_RESETrequires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio device cdev support.First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]
When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]
However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton
dev_sethence no ownership check is required. [3]For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership. [4]
As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new
_INFOioctl to get hot reset scope for given device.[1] https://lore.kernel.org/kvm/Y%[email protected]/
[2] https://lore.kernel.org/kvm/Y%[email protected]/#t
[3] https://lore.kernel.org/kvm/ZACX+Np%[email protected]/
[4]
https://lore.kernel.org/kvm/DS0PR11MB7529BE88460582BD599DC1F7C3B19
@DS0PR11MB7529.namprd11.prod.outlook.com/#t
[5] https://lore.kernel.org/kvm/[email protected]/
[6] https://lore.kernel.org/kvm/[email protected]/Change log:
v2:
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new_INFOioctl for the usage of device fd passing usage in cdev path (Jason, Alex)v1: https://lore.kernel.org/kvm/[email protected]/
Regards,
Yi LiuYi Liu (10):
vfio/pci: Update comment aroundgroup_fdget in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Only check ownership of opened devices in hot reset vfio/pci: Move the existing hot reset logic to be a helper
vfio-iommufd: Add helper to retrieveiommufd_ctxand devid for
vfio_device
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
vfio: Refine vfio file kAPIs for vfio PCI hot reset
vfio: Accpet device file from vfio PCI hot reset path vfio/pci: Renaming for accepting device fd in hot reset path vfio/pci: Accept device fd inVFIO_DEVICE_PCI_HOT_RESETioctl vfio/pci: AddVFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFOdrivers/iommu/iommufd/device.c | 12 ++ drivers/vfio/group.c | 32 ++-- drivers/vfio/iommufd.c | 16 ++ drivers/vfio/pci/vfio_pci_core.c | 244 ++++++++++++++++++++++++------- drivers/vfio/vfio.h | 2 + drivers/vfio/vfio_main.c | 44 ++++++ include/linux/iommufd.h | 3 + include/linux/vfio.h | 14 ++ include/uapi/linux/vfio.h | 65 +++++++- 9 files changed, 364 insertions(+), 68 deletions(-)
Quoted 7 lines
Quoted 2 lines
--
2.34.1Verified this series by "Intel GVT-g GPU device mediated passthrough".
Passed VFIO legacy mode / compat mode / cdev mode basic functionality and GPU force reset test.Tested-by Terrence Xu <[email protected]>
Seems like only this "GPU force reset test" is relevant to the new functionality of this series, GVT-g does not and has no reason to support the HOT_RESET ioctls used here. Can you provide more details of the force-reset test? What userspace driver is being used? Thanks,
Alex
Quoted 102 lines
-----Original Message-----
From: intel-gvt-dev <[email protected]> On Behalf Of Alex Williamson
Sent: Saturday, April 1, 2023 1:50 AMQuoted 8 lines · Terrence
Quoted 8 lines
-----Original Message-----
From: Liu, Yi L <[email protected]>
Sent: Monday, March 27, 2023 5:35 PM
VFIO_DEVICE_PCI_HOT_RESETrequires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio devicecdev support.
Quoted 16 lines
Quoted 16 lines
First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]
When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]
However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton
dev_sethence no ownership check is required. [3]Quoted 7 lines
Quoted 7 lines
For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership.
[4]As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new
_INFOioctl to get hot reset scope for givendevice.
Quoted 6 lines
https://lore.kernel.org/kvm/DS0PR11MB7529BE88460582BD599DC1F7C3B19
Quoted 34 lines
Quoted 34 lines
@DS0PR11MB7529.namprd11.prod.outlook.com/#t
[5] https://lore.kernel.org/kvm/[email protected]/
[6] https://lore.kernel.org/kvm/[email protected]/Change log:
v2:
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new_INFOioctl for the usage of device fd passing usage in cdev path (Jason, Alex)v1:
https://lore.kernel.org/kvm/[email protected]
m/Regards,
Yi LiuYi Liu (10):
vfio/pci: Update comment aroundgroup_fdget in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Only check ownership of opened devices in hot reset vfio/pci: Move the existing hot reset logic to be a helper
vfio-iommufd: Add helper to retrieveiommufd_ctxand devid for
vfio_device
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
vfio: Refine vfio file kAPIs for vfio PCI hot reset
vfio: Accpet device file from vfio PCI hot reset path vfio/pci: Renaming for accepting device fd in hot reset path vfio/pci: Accept device fd inVFIO_DEVICE_PCI_HOT_RESETioctl vfio/pci: AddVFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFOQuoted 4 lines
Quoted 4 lines
drivers/iommu/iommufd/device.c | 12 ++ drivers/vfio/group.c | 32 ++-- drivers/vfio/iommufd.c | 16 ++ drivers/vfio/pci/vfio_pci_core.c | 244 ++++++++++++++++++++++++-------
Quoted 7 lines
Quoted 7 lines
drivers/vfio/vfio.h | 2 + drivers/vfio/vfio_main.c | 44 ++++++ include/linux/iommufd.h | 3 + include/linux/vfio.h | 14 ++ include/uapi/linux/vfio.h | 65 +++++++- 9 files changed, 364 insertions(+), 68 deletions(-)Quoted 5 lines
Quoted 2 lines
--
2.34.1Verified this series by "Intel GVT-g GPU device mediated passthrough". Passed VFIO legacy mode / compat mode / cdev mode basic functionality
and GPU force reset test.
Quoted 1 line
Quoted 8 lines
Quoted 1 line
Tested-by Terrence Xu <[email protected]>Seems like only this "GPU force reset test" is relevant to the new functionality of this series, GVT-g does not and has no reason to support the
HOT_RESETioctls used here. Can you provide more details of the force-reset test? What userspace driver is being used? Thanks,Alex
Hi Alex, about the "GPU force reset test", I used the "i915_hangman" test from intel-gpu-tools, it is for GPU force hang / reset. It is an important regression test scenario for this patch series.
To test the HOT_RESET ioctls itself, need to wait the corresponding Qemu changes from Yi.
Quoted 108 lines · Terrence
Quoted 108 lines
-----Original Message-----
From: intel-gvt-dev <[email protected]> On Behalf Of Alex Williamson
Sent: Saturday, April 1, 2023 1:50 AMQuoted 8 lines · Terrence
Quoted 8 lines
-----Original Message-----
From: Liu, Yi L <[email protected]>
Sent: Monday, March 27, 2023 5:35 PM
VFIO_DEVICE_PCI_HOT_RESETrequires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. This series introduces several extensions to allow the ownership check better aligned with iommufd and coming vfio devicecdev support.
Quoted 16 lines
Quoted 16 lines
First, resetting an unopened device is always safe given nobody is using it. So relax the check to allow such devices not covered by group fd array. [1]
When iommufd is used we can simply verify that all affected devices are bound to a same iommufd then no need for the user to provide extra fd information. This is enabled by the user passing a zero-length fd array and moving forward this should be the preferred way for hot reset. [2]
However the iommufd method has difficulty working with noiommu devices since those devices don't have a valid iommufd, unless the noiommu device is in a singleton
dev_sethence no ownership check is required. [3]Quoted 7 lines
Quoted 7 lines
For noiommu backward compatibility a 3rd method is introduced by allowing the user to pass an array of device fds to prove ownership.
[4]As suggested by Jason [5], we have this series to introduce the above stuffs to the vfio PCI hot reset. Per the dicussion in [6], this series also adds a new
_INFOioctl to get hot reset scope for givendevice.
Quoted 6 lines
https://lore.kernel.org/kvm/DS0PR11MB7529BE88460582BD599DC1F7C3B19
Quoted 34 lines
Quoted 34 lines
@DS0PR11MB7529.namprd11.prod.outlook.com/#t
[5] https://lore.kernel.org/kvm/[email protected]/
[6] https://lore.kernel.org/kvm/[email protected]/Change log:
v2:
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new_INFOioctl for the usage of device fd passing usage in cdev path (Jason, Alex)v1:
https://lore.kernel.org/kvm/[email protected]
m/Regards,
Yi LiuYi Liu (10):
vfio/pci: Update comment aroundgroup_fdget in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Only check ownership of opened devices in hot reset vfio/pci: Move the existing hot reset logic to be a helper
vfio-iommufd: Add helper to retrieveiommufd_ctxand devid for
vfio_device
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
vfio: Refine vfio file kAPIs for vfio PCI hot reset
vfio: Accpet device file from vfio PCI hot reset path vfio/pci: Renaming for accepting device fd in hot reset path vfio/pci: Accept device fd inVFIO_DEVICE_PCI_HOT_RESETioctl vfio/pci: AddVFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFOQuoted 4 lines
Quoted 4 lines
drivers/iommu/iommufd/device.c | 12 ++ drivers/vfio/group.c | 32 ++-- drivers/vfio/iommufd.c | 16 ++ drivers/vfio/pci/vfio_pci_core.c | 244 ++++++++++++++++++++++++-------
Quoted 7 lines
Quoted 7 lines
drivers/vfio/vfio.h | 2 + drivers/vfio/vfio_main.c | 44 ++++++ include/linux/iommufd.h | 3 + include/linux/vfio.h | 14 ++ include/uapi/linux/vfio.h | 65 +++++++- 9 files changed, 364 insertions(+), 68 deletions(-)Quoted 5 lines
Quoted 2 lines
--
2.34.1Verified this series by "Intel GVT-g GPU device mediated passthrough". Passed VFIO legacy mode / compat mode / cdev mode basic functionality
and GPU force reset test.
Quoted 2 lines
Tested-by Terrence Xu <[email protected]>Seems like only this "GPU force reset test" is relevant to the new functionality of this series, GVT-g does not and has no reason to support the
HOT_RESETioctls used here. Can you provide more details of the force-reset test? What userspace driver is being used? Thanks,
Quoted 6 lines
Quoted 1 line
Alex
Hi Alex, about the "GPU force reset test", I used the "
i915_hangman" test from intel-gpu-tools, it is for GPU force hang / reset. It is an important regression test scenario for this patch series. To test theHOT_RESETioctls itself, need to wait the corresponding Qemu changes from Yi.
But i915 exists on the host root bus, we fundamentally cannot perform a bus reset of the root bus. So how exactly is testing with GVT-g, which doesn't use the vfio-pci-core hot-reset ioctl, or GVT-d, which can't do a bus reset because it exists on the root bus, relevant to this series? Is this some novel use of a dGPU i915 with out-of-tree drivers?
Obviously any regression testing is fine and appreciated, but if this is intended to express some validation of the new interface, I'm failing to see how. Thanks,
Alex