Enhance vfio PCI hot reset for vfio cdev device
Subsystems (5)
v5 cover Enhance vfio PCI hot reset for vfio cdev device
VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. While for cdev devices, user can use an iommufd-based ownership checking model and invoke VFIO_DEVICE_PCI_HOT_RESET with a zero-length fd array.
This series first creates iommufd_access for noiommu devices to fill the gap for adding iommufd-based ownership checking model, then extends VFIO_DEVICE_GET_PCI_HOT_RESET_INFO to check ownership and return the check result and the devid of affected devices to user. In the end, extends the VFIO_DEVICE_PCI_HOT_RESET to accept zero-length fd array for hot-reset with cdev devices.
The new hot reset method and updated _INFO ioctl are tested with the below qemu:
https://github.com/yiliu1765/qemu/tree/iommufd_rfcv4.mig.reset.v4_var3
(requires to test with the cdev kernel)
Change log:
v5:
- Drop patch 01 of v4 (Alex)
- Create noiommu_access for noiommu devices (Jason)
- Reserve all negative iommufd IDs, hence VFIO can encode negative values (Jason)
- Make vfio_iommufd_physical_devid() return -EINVAL if it's not called with a physical device or a noiommu device.
- Add vfio_find_device_in_devset() in vfio_main.c (Alex)
- Add iommufd_ctx_has_group() to replace vfio_devset_iommufd_has_group().
Reason: vfio_devset_iommufd_has_group() only loops the devices within the given devset to check the iommufd an iommu_group, but an iommu_group can span into multiple devsets. So if failed to find the group in a devset doesn't mean the group is not owned by the iommufd. So here either needs to search all the devsets or add an iommufd API to check it. It appears an iommufd API makes more sense.
- Adopt suggestions from Alex on patch 08 and 09 of v4, refine the hot-reset uapi description and minor tweaks
- Use bitfields for bool members (Alex)
v4: https://lore.kernel.org/kvm/[email protected]/
- Rename the patch series subject
- Patch 01 is moved from the cdev series
- Patch 02, 06 are new per review comments in v3
- Patch 03/04/05/07/08/09 are from v3 with updates
v3: https://lore.kernel.org/kvm/[email protected]/
- Remove the new _INFO ioctl of v2, extend the existing _INFO ioctl to report devid (Alex)
- Add r-b from Jason
- Add t-b from Terrence Xu and Yanting Jiang (mainly regression test)
v2: https://lore.kernel.org/kvm/[email protected]/
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new _INFO ioctl for the usage of device fd passing usage in cdev path (Jason, Alex)
v1: https://lore.kernel.org/kvm/[email protected]/
Regards,
Yi Liu
Yi Liu (10):
vfio-iommufd: Create iommufd_access for noiommu devices vfio/pci: Update comment around group_fd get in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Move the existing hot reset logic to be a helper
vfio: Mark cdev usage in vfio_device
iommufd: Reserve all negative IDs in the iommufd xarray
vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for
vfio_device
vfio: Add helper to search vfio_device in a dev_set
iommufd: Add iommufd_ctx_has_group()
vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device
cdev
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESET
drivers/iommu/iommufd/device.c | 53 +++++++++ drivers/iommu/iommufd/main.c | 2 +- drivers/vfio/iommufd.c | 63 ++++++++++- drivers/vfio/pci/vfio_pci_core.c | 184 ++++++++++++++++++++++++------- drivers/vfio/vfio_main.c | 15 +++ include/linux/iommufd.h | 14 +++ include/linux/vfio.h | 23 ++++ include/uapi/linux/vfio.h | 60 +++++++++- 8 files changed, 368 insertions(+), 46 deletions(-)
Signature · 1 line
2.34.1
PATCH v5 1/10 vfio-iommufd: Create iommufd_access for noiommu devices
This binds noiommu device to iommufd and creates iommufd_access for this bond. This is useful for adding an iommufd-based device ownership check for VFIO_DEVICE_PCI_HOT_RESET since this model requires all the other affected devices bound to the same iommufd as the device to be reset. For noiommu devices, there is no backend iommu, so create iommufd_access instead of iommufd_device.
--- drivers/vfio/iommufd.c | 43 ++++++++++++++++++++++++++++++++++++++++-- include/linux/vfio.h | 1 + 2 files changed, 42 insertions(+), 2 deletions(-)
drivers/vfio/iommufd.c +41 -2
@@ -10,6 +10,42 @@MODULE_IMPORT_NS (IOMMUFD );MODULE_IMPORT_NS (IOMMUFD_VFIO ); +static void vfio_noiommu_access_unmap (void *data,unsigned long iova, +unsigned long length) +{ +} + +static const struct iommufd_access_ops vfio_user_noiommu_ops = { + .needs_pin_pages =1 , + .unmap = vfio_noiommu_access_unmap, +}; + +static int vfio_iommufd_noiommu_bind (struct vfio_device *vdev, +struct iommufd_ctx *ictx, +u32 *out_device_id) +{ +struct iommufd_access *user; + +lockdep_assert_held (&vdev->dev_set->lock); + + user =iommufd_access_create (ictx, &vfio_user_noiommu_ops, + vdev, out_device_id); +if (IS_ERR (user)) +return PTR_ERR (user); + vdev->noiommu_access = user; +return 0 ; +} + +static void vfio_iommufd_noiommu_unbind (struct vfio_device *vdev) +{ +lockdep_assert_held (&vdev->dev_set->lock); + +if (vdev->noiommu_access) { +iommufd_access_destroy (vdev->noiommu_access); + vdev->noiommu_access =NULL ; + } +} +int vfio_iommufd_bind (struct vfio_device *vdev,struct iommufd_ctx *ictx) {u32 ioas_id;
@@ -29,7 +65,8 @@ int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)*/ if (!iommufd_vfio_compat_ioas_get_id (ictx, &ioas_id))return -EPERM ; -return 0 ; + +return vfio_iommufd_noiommu_bind (vdev, ictx, &device_id); } ret = vdev->ops->bind_iommufd (vdev, ictx, &device_id);
@@ -59,8 +96,10 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) {lockdep_assert_held (&vdev->dev_set->lock); -if (vfio_device_is_noiommu (vdev)) +if (vfio_device_is_noiommu (vdev)) { +vfio_iommufd_noiommu_unbind (vdev);return ; + }if (vdev->ops->unbind_iommufd) vdev->ops->unbind_iommufd (vdev);
include/linux/vfio.h +1 -0
@@ -57,6 +57,7 @@ struct vfio_device {struct list_head group_next;struct list_head iommu_entry;struct iommufd_access *iommufd_access; +struct iommufd_access *noiommu_access;void (*put_kvm)(struct kvm *kvm);#if IS_ENABLED (CONFIG_IOMMUFD )struct iommufd_device *iommufd_device;
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v5 01/10] vfio-iommufd: Create iommufd_access for noiommu devices
Quoted 19 lines · On Sat, 13 May 2023 06:21:27 -0700 Yi Liu <[email protected]> wrote:
This binds noiommu device to iommufd and creates
iommufd_accessfor this bond. This is useful for adding an iommufd-based device ownership check forVFIO_DEVICE_PCI_HOT_RESETsince this model requires all the other affected devices bound to the same iommufd as the device to be reset. For noiommu devices, there is no backend iommu, so createiommufd_accessinstead ofiommufd_device.Suggested-by Jason Gunthorpe <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/iommufd.c | 43 ++++++++++++++++++++++++++++++++++++++++-- include/linux/vfio.h | 1 + 2 files changed, 42 insertions(+), 2 deletions(-)drivers/vfio/iommufd.c +0
-0
drivers/vfio/iommufd.c line 15
Quoted 6 lines
drivers/vfio/iommufd.c +3
-0@@ -10,6 +10,42 @@MODULE_IMPORT_NS (IOMMUFD );MODULE_IMPORT_NS (IOMMUFD_VFIO ); +static void vfio_noiommu_access_unmap (void *data,unsigned long iova, +unsigned long length) +{
Should this WARN_ON if called?
drivers/vfio/iommufd.c line 19
Quoted 4 lines
drivers/vfio/iommufd.c +4
-0+} + +static const struct iommufd_access_ops vfio_user_noiommu_ops = { + .needs_pin_pages =1 ,
But it doesn't.
Quoted 59 lines
drivers/vfio/iommufd.c +34
-2+ .unmap = vfio_noiommu_access_unmap, +}; + +static int vfio_iommufd_noiommu_bind (struct vfio_device *vdev, +struct iommufd_ctx *ictx, +u32 *out_device_id) +{ +struct iommufd_access *user; + +lockdep_assert_held (&vdev->dev_set->lock); + + user =iommufd_access_create (ictx, &vfio_user_noiommu_ops, + vdev, out_device_id); +if (IS_ERR (user)) +return PTR_ERR (user); + vdev->noiommu_access = user; +return 0 ; +} + +static void vfio_iommufd_noiommu_unbind (struct vfio_device *vdev) +{ +lockdep_assert_held (&vdev->dev_set->lock); + +if (vdev->noiommu_access) { +iommufd_access_destroy (vdev->noiommu_access); + vdev->noiommu_access =NULL ; + } +} +int vfio_iommufd_bind (struct vfio_device *vdev,struct iommufd_ctx *ictx) {u32 ioas_id;@@ -29,7 +65,8 @@ int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)*/ if (!iommufd_vfio_compat_ioas_get_id (ictx, &ioas_id))return -EPERM ; -return 0 ; + +return vfio_iommufd_noiommu_bind (vdev, ictx, &device_id); } ret = vdev->ops->bind_iommufd (vdev, ictx, &device_id);@@ -59,8 +96,10 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) {lockdep_assert_held (&vdev->dev_set->lock); -if (vfio_device_is_noiommu (vdev)) +if (vfio_device_is_noiommu (vdev)) { +vfio_iommufd_noiommu_unbind (vdev);return ; + }if (vdev->ops->unbind_iommufd) vdev->ops->unbind_iommufd (vdev);include/linux/vfio.h +0
-0
include/linux/vfio.h struct vfio_device { · line 60
Quoted 4 lines
include/linux/vfio.h +1
-0@@ -57,6 +57,7 @@ struct vfio_device {struct list_head group_next;struct list_head iommu_entry;struct iommufd_access *iommufd_access; +struct iommufd_access *noiommu_access;
It's not clear to me why we need a separate iommufd_access for noiommu. Can't we add a vfio_device_is_noiommu() check to the vfio_{un}pin_pages() and vfio_dma_rw() interfaces and reuse the existing pointer for both emulated and noiommu cases? Maybe even the iommufd_access* functions should test needs_pin_pages and generate an error/warning if an access that was registered without reporting that it needs page pinning makes use of such an interface. Thanks,
Alex
Quoted 3 lines
include/linux/vfio.h +0
-0void (*put_kvm)(struct kvm *kvm);#if IS_ENABLED (CONFIG_IOMMUFD )struct iommufd_device *iommufd_device;
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v5 01/10] vfio-iommufd: Create iommufd_access for noiommu devices
Quoted 2 lines · Alex Williamson
It's not clear to me why we need a separate
iommufd_accessfor noiommu.
The point was to allocate an ID for the device so we can use that ID with the other interfaces in all cases.
Otherwise it is a too weird special case that is probably going to keep causing trouble down the road...
Jason
Re: [Intel-gfx] [PATCH v5 01/10] vfio-iommufd: Create iommufd_access for noiommu devices
Quoted 10 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, May 18, 2023 2:21 AMQuoted 2 lines · Alex Williamson
It's not clear to me why we need a separate
iommufd_accessfor noiommu.The point was to allocate an ID for the device so we can use that ID with the other interfaces in all cases.
I guess Alex's question is why adding a new pointer named noiommu_access while there is already the iommufd_access pointer named iommufd_access.
Maybe we shall reuse the iommufd_access pointer?
Regards,
Yi Liu
Re: [Intel-gfx] [PATCH v5 01/10] vfio-iommufd: Create iommufd_access for noiommu devices
Quoted 5 lines · Yi L
Quoted 5 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, May 18, 2023 2:21 AMOn Wed, May 17, 2023 at 11:26:09AM -0600, Alex Williamson wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
It's not clear to me why we need a separate
iommufd_accessfor noiommu.The point was to allocate an ID for the device so we can use that ID with the other interfaces in all cases.
I guess Alex's question is why adding a new pointer named
noiommu_accesswhile there is already theiommufd_accesspointer namediommufd_access.
Yes, precisely. Sorry that was confusing, we need the access for noiommu but it's not clear why that access needs to be stored in a separate pointer when we can already differentiate noiommu devices otherwise. Thanks,
Alex
PATCH v5 2/10 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset()
In reply to Yi Liuthis suits more on what the code does.
--- drivers/vfio/pci/vfio_pci_core.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +2 -3
@@ -1308,9 +1308,8 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, }/* -* For each group_fd, get the group through the vfio external user -* interface and store the group and iommu ID. This ensures the group -* is held across the reset. +* Get the group file for each fd to ensure the group is held across +* the reset */ for (file_idx =0 ; file_idx < hdr.count; file_idx++) {struct file *file =fget (group_fds[file_idx]);
Signature · 1 line
2.34.1
PATCH v5 3/10 vfio/pci: Move the existing hot reset logic to be a helper
In reply to Yi LiuThis prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups().
No functional change is intended.
--- drivers/vfio/pci/vfio_pci_core.c | 55 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 23 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +32 -23
@@ -1255,29 +1255,16 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(return ret; } -static int vfio_pci_ioctl_pci_hot_reset (struct vfio_pci_core_device *vdev, -struct vfio_pci_hot_reset __user *arg) +static int +vfio_pci_ioctl_pci_hot_reset_groups (struct vfio_pci_core_device *vdev, +int array_count,bool slot, +struct vfio_pci_hot_reset __user *arg) { - unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); -struct vfio_pci_hot_reset hdr;int32_t *group_fds;struct file **files;struct vfio_pci_group_info info; -bool slot = false;int file_idx, count =0 , ret =0 ; -if (copy_from_user (&hdr, arg, minsz)) -return -EFAULT ; - -if (hdr.argsz < minsz || hdr.flags) -return -EINVAL ; - -/* Can we do a slot or bus reset or neither? */ -if (!pci_probe_reset_slot (vdev->pdev->slot)) - slot = true; -else if (pci_probe_reset_bus (vdev->pdev->bus)) -return -ENODEV ; -/* * We can't let userspace give us an arbitrarily large buffer to copy, * so verify how many we think there could be. Note groups can have
@@ -1289,11 +1276,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,return ret;/* Somewhere between 1 and count is OK */ -if (!hdr.count || hdr.count > count) +if (!array_count || array_count > count)return -EINVAL ; - group_fds =kcalloc (hdr.count,sizeof (*group_fds),GFP_KERNEL ); - files =kcalloc (hdr.count,sizeof (*files),GFP_KERNEL ); + group_fds =kcalloc (array_count,sizeof (*group_fds),GFP_KERNEL ); + files =kcalloc (array_count,sizeof (*files),GFP_KERNEL );if (!group_fds || !files) {kfree (group_fds);kfree (files);
@@ -1301,7 +1288,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, }if (copy_from_user (group_fds, arg->group_fds, - hdr.count *sizeof (*group_fds))) { + array_count *sizeof (*group_fds))) {kfree (group_fds);kfree (files);return -EFAULT ;
@@ -1311,7 +1298,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,* Get the group file for each fd to ensure the group is held across * the reset */ -for (file_idx =0 ; file_idx < hdr.count; file_idx++) { +for (file_idx =0 ; file_idx < array_count; file_idx++) {struct file *file =fget (group_fds[file_idx]);if (!file) {
@@ -1335,7 +1322,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,if (ret) goto hot_reset_release; - info.count = hdr.count; + info.count = array_count; info.files = files; ret =vfio_pci_dev_set_hot_reset (vdev->vdev.dev_set, &info);
@@ -1348,6 +1335,28 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,return ret; } +static int vfio_pci_ioctl_pci_hot_reset (struct vfio_pci_core_device *vdev, +struct vfio_pci_hot_reset __user *arg) +{ + unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); +struct vfio_pci_hot_reset hdr; +bool slot = false; + +if (copy_from_user (&hdr, arg, minsz)) +return -EFAULT ; + +if (hdr.argsz < minsz || hdr.flags) +return -EINVAL ; + +/* Can we do a slot or bus reset or neither? */ +if (!pci_probe_reset_slot (vdev->pdev->slot)) + slot = true; +else if (pci_probe_reset_bus (vdev->pdev->bus)) +return -ENODEV ; + +return vfio_pci_ioctl_pci_hot_reset_groups (vdev, hdr.count, slot, arg); +} +static int vfio_pci_ioctl_ioeventfd (struct vfio_pci_core_device *vdev,struct vfio_device_ioeventfd __user *arg) {
Signature · 1 line
2.34.1
PATCH v5 4/10 vfio: Mark cdev usage in vfio_device
In reply to Yi LiuUse it to differentiate whether to report group_id or devid in the revised VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl. At this moment, no cdev path yet, so the vfio_device_cdev_opened() helper always returns false.
--- include/linux/vfio.h | 5 +++++ 1 file changed, 5 insertions(+)
include/linux/vfio.h +5 -0
@@ -140,6 +140,11 @@ int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id); ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)#endif +static inline bool vfio_device_cdev_opened (struct vfio_device *device) +{ +return false; +} +/** * struct vfio_migration_ops - VFIO bus device driver migration callbacks *
Signature · 1 line
2.34.1
PATCH v5 5/10 iommufd: Reserve all negative IDs in the iommufd xarray
In reply to Yi LiuHence IOMMUFD users can encode the negative IDs for specific purposes. e.g. VFIO needs two reserved values to tell userspace the ID returned is not valid but has other meaning.
--- drivers/iommu/iommufd/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iommu/iommufd/main.c +1 -1
@@ -50,7 +50,7 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx,* before calling iommufd_object_finalize(). */ rc =xa_alloc (&ictx->objects, &obj->id,XA_ZERO_ENTRY , - xa_limit_32b,GFP_KERNEL_ACCOUNT ); + xa_limit_31b,GFP_KERNEL_ACCOUNT );if (rc) goto out_free;return obj;
Signature · 1 line
2.34.1
PATCH v5 6/10 vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
In reply to Yi LiuThis is needed by the vfio-pci driver to report affected devices in the hot reset for a given device.
--- drivers/iommu/iommufd/device.c | 24 ++++++++++++++++++++++++ drivers/vfio/iommufd.c | 20 ++++++++++++++++++++ include/linux/iommufd.h | 6 ++++++ include/linux/vfio.h | 14 ++++++++++++++ 4 files changed, 64 insertions(+)
drivers/iommu/iommufd/device.c +24 -0
@@ -116,6 +116,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) }EXPORT_SYMBOL_NS_GPL (iommufd_device_unbind,IOMMUFD ); +struct iommufd_ctx *iommufd_device_to_ictx (struct iommufd_device *idev) +{ +return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL (iommufd_device_to_ictx,IOMMUFD ); + +u32 iommufd_device_to_id (struct iommufd_device *idev) +{ +return idev->obj.id; +} +EXPORT_SYMBOL_NS_GPL (iommufd_device_to_id,IOMMUFD ); +static int iommufd_device_setup_msi (struct iommufd_device *idev,struct iommufd_hw_pagetable *hwpt,phys_addr_t sw_msi_start)
@@ -463,6 +475,18 @@ void iommufd_access_destroy(struct iommufd_access *access) }EXPORT_SYMBOL_NS_GPL (iommufd_access_destroy,IOMMUFD ); +struct iommufd_ctx *iommufd_access_to_ictx (struct iommufd_access *access) +{ +return access->ictx; +} +EXPORT_SYMBOL_NS_GPL (iommufd_access_to_ictx,IOMMUFD ); + +u32 iommufd_access_to_id (struct iommufd_access *access) +{ +return access->obj.id; +} +EXPORT_SYMBOL_NS_GPL (iommufd_access_to_id,IOMMUFD ); +int iommufd_access_attach (struct iommufd_access *access,u32 ioas_id) {struct iommufd_ioas *new_ioas;
drivers/vfio/iommufd.c +20 -0
@@ -105,6 +105,26 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd (vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_ictx (vdev->iommufd_device); +if (vdev->noiommu_access) +return iommufd_access_to_ictx (vdev->noiommu_access); +return NULL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_ictx); + +int vfio_iommufd_physical_devid (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_id (vdev->iommufd_device); +if (vdev->noiommu_access) +return iommufd_access_to_id (vdev->noiommu_access); +return -EINVAL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_devid); +/* * The physical standard ops mean that the iommufd_device is bound to the * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers
include/linux/iommufd.h +6 -0
@@ -24,6 +24,9 @@ void iommufd_device_unbind(struct iommufd_device *idev);int iommufd_device_attach (struct iommufd_device *idev,u32 *pt_id);void iommufd_device_detach (struct iommufd_device *idev); +struct iommufd_ctx *iommufd_device_to_ictx (struct iommufd_device *idev); +u32 iommufd_device_to_id (struct iommufd_device *idev); +struct iommufd_access_ops {u8 needs_pin_pages :1 ;void (*unmap)(void *data,unsigned long iova,unsigned long length);
@@ -45,6 +48,9 @@ iommufd_access_create(struct iommufd_ctx *ictx,void iommufd_access_destroy (struct iommufd_access *access);int iommufd_access_attach (struct iommufd_access *access,u32 ioas_id); +struct iommufd_ctx *iommufd_access_to_ictx (struct iommufd_access *access); +u32 iommufd_access_to_id (struct iommufd_access *access); +void iommufd_ctx_get (struct iommufd_ctx *ictx);#if IS_ENABLED (CONFIG_IOMMUFD )
include/linux/vfio.h +14 -0
@@ -115,6 +115,8 @@ struct vfio_device_ops { };#if IS_ENABLED (CONFIG_IOMMUFD ) +struct iommufd_ctx *vfio_iommufd_physical_ictx (struct vfio_device *vdev); +int vfio_iommufd_physical_devid (struct vfio_device *vdev);int vfio_iommufd_physical_bind (struct vfio_device *vdev,struct iommufd_ctx *ictx,u32 *out_device_id);void vfio_iommufd_physical_unbind (struct vfio_device *vdev);
@@ -124,6 +126,18 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev,void vfio_iommufd_emulated_unbind (struct vfio_device *vdev);int vfio_iommufd_emulated_attach_ioas (struct vfio_device *vdev,u32 *pt_id);#else +static inline struct iommufd_ctx * +vfio_iommufd_physical_ictx (struct vfio_device *vdev) +{ +return NULL ; +} + +static inline int +vfio_iommufd_physical_devid (struct vfio_device *vdev) +{ +return -EOPNOTSUPP ; +} +#define vfio_iommufd_physical_bind \ ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ u32 *out_device_id)) NULL)
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v5 06/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
Quoted 70 lines · On Sat, 13 May 2023 06:21:32 -0700 Yi Liu <[email protected]> wrote:
This is needed by the vfio-pci driver to report affected devices in the hot reset for a given device.
Signed-off-by Yi Liu <[email protected]>--- drivers/iommu/iommufd/device.c | 24 ++++++++++++++++++++++++ drivers/vfio/iommufd.c | 20 ++++++++++++++++++++ include/linux/iommufd.h | 6 ++++++ include/linux/vfio.h | 14 ++++++++++++++ 4 files changed, 64 insertions(+)drivers/iommu/iommufd/device.c +24
-0@@ -116,6 +116,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) }EXPORT_SYMBOL_NS_GPL (iommufd_device_unbind,IOMMUFD ); +struct iommufd_ctx *iommufd_device_to_ictx (struct iommufd_device *idev) +{ +return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL (iommufd_device_to_ictx,IOMMUFD ); + +u32 iommufd_device_to_id (struct iommufd_device *idev) +{ +return idev->obj.id; +} +EXPORT_SYMBOL_NS_GPL (iommufd_device_to_id,IOMMUFD ); +static int iommufd_device_setup_msi (struct iommufd_device *idev,struct iommufd_hw_pagetable *hwpt,phys_addr_t sw_msi_start)@@ -463,6 +475,18 @@ void iommufd_access_destroy(struct iommufd_access *access) }EXPORT_SYMBOL_NS_GPL (iommufd_access_destroy,IOMMUFD ); +struct iommufd_ctx *iommufd_access_to_ictx (struct iommufd_access *access) +{ +return access->ictx; +} +EXPORT_SYMBOL_NS_GPL (iommufd_access_to_ictx,IOMMUFD ); + +u32 iommufd_access_to_id (struct iommufd_access *access) +{ +return access->obj.id; +} +EXPORT_SYMBOL_NS_GPL (iommufd_access_to_id,IOMMUFD ); +int iommufd_access_attach (struct iommufd_access *access,u32 ioas_id) {struct iommufd_ioas *new_ioas;drivers/vfio/iommufd.c +9
-0@@ -105,6 +105,26 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd (vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_ictx (vdev->iommufd_device); +if (vdev->noiommu_access) +return iommufd_access_to_ictx (vdev->noiommu_access); +return NULL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_ictx);
drivers/vfio/iommufd.c void vfio_iommufd_unbind(struct vfio_device *vdev) · line 126
Quoted 10 lines
drivers/vfio/iommufd.c +10
-0+ +int vfio_iommufd_physical_devid (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_id (vdev->iommufd_device); +if (vdev->noiommu_access) +return iommufd_access_to_id (vdev->noiommu_access); +return -EINVAL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_devid);
I think these exemplify that it would be better if both emulated and noiommu use the same iommufd_access pointer. Thanks,
Alex
Quoted 60 lines
drivers/vfio/iommufd.c +1
-0+/* * The physical standard ops mean that the iommufd_device is bound to the * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers include/linux/iommufd.h +6
-0@@ -24,6 +24,9 @@ void iommufd_device_unbind(struct iommufd_device *idev);int iommufd_device_attach (struct iommufd_device *idev,u32 *pt_id);void iommufd_device_detach (struct iommufd_device *idev); +struct iommufd_ctx *iommufd_device_to_ictx (struct iommufd_device *idev); +u32 iommufd_device_to_id (struct iommufd_device *idev); +struct iommufd_access_ops {u8 needs_pin_pages :1 ;void (*unmap)(void *data,unsigned long iova,unsigned long length);@@ -45,6 +48,9 @@ iommufd_access_create(struct iommufd_ctx *ictx,void iommufd_access_destroy (struct iommufd_access *access);int iommufd_access_attach (struct iommufd_access *access,u32 ioas_id); +struct iommufd_ctx *iommufd_access_to_ictx (struct iommufd_access *access); +u32 iommufd_access_to_id (struct iommufd_access *access); +void iommufd_ctx_get (struct iommufd_ctx *ictx);#if IS_ENABLED (CONFIG_IOMMUFD )include/linux/vfio.h +14
-0@@ -115,6 +115,8 @@ struct vfio_device_ops { };#if IS_ENABLED (CONFIG_IOMMUFD ) +struct iommufd_ctx *vfio_iommufd_physical_ictx (struct vfio_device *vdev); +int vfio_iommufd_physical_devid (struct vfio_device *vdev);int vfio_iommufd_physical_bind (struct vfio_device *vdev,struct iommufd_ctx *ictx,u32 *out_device_id);void vfio_iommufd_physical_unbind (struct vfio_device *vdev);@@ -124,6 +126,18 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev,void vfio_iommufd_emulated_unbind (struct vfio_device *vdev);int vfio_iommufd_emulated_attach_ioas (struct vfio_device *vdev,u32 *pt_id);#else +static inline struct iommufd_ctx * +vfio_iommufd_physical_ictx (struct vfio_device *vdev) +{ +return NULL ; +} + +static inline int +vfio_iommufd_physical_devid (struct vfio_device *vdev) +{ +return -EOPNOTSUPP ; +} +#define vfio_iommufd_physical_bind \ ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ u32 *out_device_id)) NULL)
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v5 06/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
Quoted 4 lines · Alex Williamson
Quoted 4 lines
drivers/vfio/iommufd.c +4
-0+int vfio_iommufd_physical_devid (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_id (vdev->iommufd_device);
Quoted 8 lines
drivers/vfio/iommufd.c void vfio_iommufd_unbind(struct vfio_device *vdev) · line 126
Quoted 5 lines
drivers/vfio/iommufd.c +5
-0+if (vdev->noiommu_access) +return iommufd_access_to_id (vdev->noiommu_access); +return -EINVAL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_devid);I think these exemplify that it would be better if both emulated and noiommu use the same
iommufd_accesspointer. Thanks,
Oh, I mis understood your other remark.. Yeah good question I have to study this also
Jason
Re: [Intel-gfx] [PATCH v5 06/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
Quoted 15 lines · On Wed, 17 May 2023 15:22:27 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 4 lines · Alex Williamson
Quoted 4 lines
drivers/vfio/iommufd.c +4
-0+int vfio_iommufd_physical_devid (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_id (vdev->iommufd_device);Quoted 8 lines
drivers/vfio/iommufd.c void vfio_iommufd_unbind(struct vfio_device *vdev) · line 126
Quoted 5 lines
drivers/vfio/iommufd.c +5
-0+if (vdev->noiommu_access) +return iommufd_access_to_id (vdev->noiommu_access); +return -EINVAL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_devid);I think these exemplify that it would be better if both emulated and noiommu use the same
iommufd_accesspointer. Thanks,
Quoted 2 lines
Oh, I mis understood your other remark.. Yeah good question I have to study this also
I guess I also missed that this wasn't iommufd_access vs noiommu_access, it's device vs access, but shouldn't any iommufd_access pointer provide the devid? I need to go look why we need two different methods to get a devid... Thanks,
Alex
Jason Gunthorpe
maintainerRe: [Intel-gfx] [PATCH v5 06/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
Quoted 18 lines · Alex Williamson
Quoted 15 lines · On Wed, 17 May 2023 15:22:27 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 12 lines · Alex Williamson
drivers/vfio/iommufd.c void vfio_iommufd_unbind(struct vfio_device *vdev) · line 126
Quoted 9 lines
drivers/vfio/iommufd.c +9
-0+int vfio_iommufd_physical_devid (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_id (vdev->iommufd_device); +if (vdev->noiommu_access) +return iommufd_access_to_id (vdev->noiommu_access); +return -EINVAL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_devid);I think these exemplify that it would be better if both emulated and noiommu use the same
iommufd_accesspointer. Thanks,
Quoted 7 lines
Quoted 2 lines
Oh, I mis understood your other remark.. Yeah good question I have to study this also
I guess I also missed that this wasn't
iommufd_accessvsnoiommu_access, it's device vs access, but shouldn't anyiommufd_accesspointer provide the devid? I need to go look why we need two different methods to get a devid...
At least this hunk above makes sense, access and device are two different objects with different types, so having different ID accessors is reasonably logical.
But it is a good point that this should return the dev id of the normal access for a normal mdev too
Ideally I'd like to see that we always return a dev id to userspace for all vfio device types. Then we can rely on it
Jason
Re: [Intel-gfx] [PATCH v5 06/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
In reply to Alex WilliamsonQuoted 82 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, May 18, 2023 2:15 AMdrivers/iommu/iommufd/device.c void iommufd_access_destroy(struct iommufd_access
Quoted 35 lines · On Sat, 13 May 2023 06:21:32 -0700 Yi Liu <[email protected]> wrote:
This is needed by the vfio-pci driver to report affected devices in the hot reset for a given device.
Signed-off-by Yi Liu <[email protected]>--- drivers/iommu/iommufd/device.c | 24 ++++++++++++++++++++++++ drivers/vfio/iommufd.c | 20 ++++++++++++++++++++ include/linux/iommufd.h | 6 ++++++ include/linux/vfio.h | 14 ++++++++++++++ 4 files changed, 64 insertions(+)drivers/iommu/iommufd/device.c +12
-0@@ -116,6 +116,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) }EXPORT_SYMBOL_NS_GPL (iommufd_device_unbind,IOMMUFD ); +struct iommufd_ctx *iommufd_device_to_ictx (struct iommufd_device *idev) +{ +return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL (iommufd_device_to_ictx,IOMMUFD ); + +u32 iommufd_device_to_id (struct iommufd_device *idev) +{ +return idev->obj.id; +} +EXPORT_SYMBOL_NS_GPL (iommufd_device_to_id,IOMMUFD ); +static int iommufd_device_setup_msi (struct iommufd_device *idev,struct iommufd_hw_pagetable *hwpt,phys_addr_t sw_msi_start)@@ -463,6 +475,18 @@ void iommufd_access_destroy(struct iommufd_access*access)
Quoted 40 lines
drivers/iommu/iommufd/device.c +12
-0}EXPORT_SYMBOL_NS_GPL (iommufd_access_destroy,IOMMUFD ); +struct iommufd_ctx *iommufd_access_to_ictx (struct iommufd_access *access) +{ +return access->ictx; +} +EXPORT_SYMBOL_NS_GPL (iommufd_access_to_ictx,IOMMUFD ); + +u32 iommufd_access_to_id (struct iommufd_access *access) +{ +return access->obj.id; +} +EXPORT_SYMBOL_NS_GPL (iommufd_access_to_id,IOMMUFD ); +int iommufd_access_attach (struct iommufd_access *access,u32 ioas_id) {struct iommufd_ioas *new_ioas;drivers/vfio/iommufd.c +14
-0@@ -105,6 +105,26 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd (vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_ictx (vdev->iommufd_device); +if (vdev->noiommu_access) +return iommufd_access_to_ictx (vdev->noiommu_access); +return NULL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_ictx); + +int vfio_iommufd_physical_devid (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_id (vdev->iommufd_device);
Quoted 8 lines
drivers/vfio/iommufd.c void vfio_iommufd_unbind(struct vfio_device *vdev) · line 126
Quoted 5 lines
drivers/vfio/iommufd.c +5
-0+if (vdev->noiommu_access) +return iommufd_access_to_id (vdev->noiommu_access); +return -EINVAL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_devid);I think these exemplify that it would be better if both emulated and noiommu use the same
iommufd_accesspointer. Thanks,
Sure. Then I shall rename this helper. vfio_iommufd_device_devid() What about your opinion?
Regards,
Yi Liu
Re: [Intel-gfx] [PATCH v5 06/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
Quoted 91 lines · Yi L
Quoted 82 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, May 18, 2023 2:15 AMdrivers/iommu/iommufd/device.c void iommufd_access_destroy(struct iommufd_access
Quoted 35 lines · On Sat, 13 May 2023 06:21:32 -0700 Yi Liu <[email protected]> wrote:
This is needed by the vfio-pci driver to report affected devices in the hot reset for a given device.
Signed-off-by Yi Liu <[email protected]>--- drivers/iommu/iommufd/device.c | 24 ++++++++++++++++++++++++ drivers/vfio/iommufd.c | 20 ++++++++++++++++++++ include/linux/iommufd.h | 6 ++++++ include/linux/vfio.h | 14 ++++++++++++++ 4 files changed, 64 insertions(+)drivers/iommu/iommufd/device.c +12
-0@@ -116,6 +116,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) }EXPORT_SYMBOL_NS_GPL (iommufd_device_unbind,IOMMUFD ); +struct iommufd_ctx *iommufd_device_to_ictx (struct iommufd_device *idev) +{ +return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL (iommufd_device_to_ictx,IOMMUFD ); + +u32 iommufd_device_to_id (struct iommufd_device *idev) +{ +return idev->obj.id; +} +EXPORT_SYMBOL_NS_GPL (iommufd_device_to_id,IOMMUFD ); +static int iommufd_device_setup_msi (struct iommufd_device *idev,struct iommufd_hw_pagetable *hwpt,phys_addr_t sw_msi_start)@@ -463,6 +475,18 @@ void iommufd_access_destroy(struct iommufd_access*access)
Quoted 40 lines
drivers/iommu/iommufd/device.c +12
-0}EXPORT_SYMBOL_NS_GPL (iommufd_access_destroy,IOMMUFD ); +struct iommufd_ctx *iommufd_access_to_ictx (struct iommufd_access *access) +{ +return access->ictx; +} +EXPORT_SYMBOL_NS_GPL (iommufd_access_to_ictx,IOMMUFD ); + +u32 iommufd_access_to_id (struct iommufd_access *access) +{ +return access->obj.id; +} +EXPORT_SYMBOL_NS_GPL (iommufd_access_to_id,IOMMUFD ); +int iommufd_access_attach (struct iommufd_access *access,u32 ioas_id) {struct iommufd_ioas *new_ioas;drivers/vfio/iommufd.c +14
-0@@ -105,6 +105,26 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd (vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_ictx (vdev->iommufd_device); +if (vdev->noiommu_access) +return iommufd_access_to_ictx (vdev->noiommu_access); +return NULL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_ictx); + +int vfio_iommufd_physical_devid (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_id (vdev->iommufd_device);Quoted 8 lines
drivers/vfio/iommufd.c void vfio_iommufd_unbind(struct vfio_device *vdev) · line 126
Quoted 5 lines
drivers/vfio/iommufd.c +5
-0+if (vdev->noiommu_access) +return iommufd_access_to_id (vdev->noiommu_access); +return -EINVAL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_devid);I think these exemplify that it would be better if both emulated and noiommu use the same
iommufd_accesspointer. Thanks,
Quoted 2 lines
Sure. Then I shall rename this helper.
vfio_iommufd_device_devid()What about your opinion?
Yes, it really didn't even occur to me that "physical" in the name was meant to suggest this shouldn't be used for emulated mdev devices. It should work for all devices and using a shared iommufd access pointer between noiommu and emulated should simplify that somewhat. Thanks,
Alex
Tian, Kevin
maintainerRe: [Intel-gfx] [PATCH v5 06/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
In reply to Liu, Yi LQuoted 15 lines
From: Liu, Yi L <[email protected]>
Sent: Thursday, May 18, 2023 9:26 PMQuoted 4 lines
Quoted 4 lines
drivers/vfio/iommufd.c +4
-0+int vfio_iommufd_physical_devid (struct vfio_device *vdev) +{ +if (vdev->iommufd_device) +return iommufd_device_to_id (vdev->iommufd_device);Quoted 8 lines
drivers/vfio/iommufd.c void vfio_iommufd_unbind(struct vfio_device *vdev) · line 126
Quoted 5 lines
drivers/vfio/iommufd.c +5
-0+if (vdev->noiommu_access) +return iommufd_access_to_id (vdev->noiommu_access); +return -EINVAL ; +} +EXPORT_SYMBOL_GPL (vfio_iommufd_physical_devid);I think these exemplify that it would be better if both emulated and noiommu use the same
iommufd_accesspointer. Thanks,
Quoted 3 lines
Sure. Then I shall rename this helper.
vfio_iommufd_device_devid()What about your opinion?
Probably just vfio_iommufd_device_id().
PATCH v5 7/10 vfio: Add helper to search vfio_device in a dev_set
In reply to Yi LiuThere are drivers that need to search vfio_device within a given dev_set. e.g. vfio-pci. So add a helper.
--- drivers/vfio/pci/vfio_pci_core.c | 8 +++----- drivers/vfio/vfio_main.c | 15 +++++++++++++++ include/linux/vfio.h | 3 +++ 3 files changed, 21 insertions(+), 5 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +3 -5
@@ -2335,12 +2335,10 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev,static int vfio_pci_is_device_in_set (struct pci_dev *pdev,void *data) {struct vfio_device_set *dev_set = data; -struct vfio_device *cur; -list_for_each_entry (cur, &dev_set->device_list, dev_set_list) -if (cur->dev == &pdev->dev) -return 0 ; -return -EBUSY ; +lockdep_assert_held (&dev_set->lock); + +return vfio_find_device_in_devset (dev_set, &pdev->dev) ?0 : -EBUSY ; }/*
drivers/vfio/vfio_main.c +15 -0
@@ -141,6 +141,21 @@ unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set) }EXPORT_SYMBOL_GPL (vfio_device_set_open_count); +struct vfio_device * +vfio_find_device_in_devset (struct vfio_device_set *dev_set, +struct device *dev) +{ +struct vfio_device *cur; + +lockdep_assert_held (&dev_set->lock); + +list_for_each_entry (cur, &dev_set->device_list, dev_set_list) +if (cur->dev == dev) +return cur; +return NULL ; +} +EXPORT_SYMBOL_GPL (vfio_find_device_in_devset); +/* * Device objects - create, release, get, put, search */
include/linux/vfio.h +3 -0
@@ -259,6 +259,9 @@ void vfio_unregister_group_dev(struct vfio_device *device);int vfio_assign_device_set (struct vfio_device *device,void *set_id);unsigned int vfio_device_set_open_count (struct vfio_device_set *dev_set); +struct vfio_device * +vfio_find_device_in_devset (struct vfio_device_set *dev_set, +struct device *dev); int vfio_mig_get_next_state(struct vfio_device *device, enum vfio_device_mig_state cur_fsm,
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v5 07/10] vfio: Add helper to search vfio_device in a dev_set
Quoted 15 lines · Yi Liu
There are drivers that need to search
vfio_devicewithin a givendev_set. e.g. vfio-pci. So add a helper.Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 8 +++----- drivers/vfio/vfio_main.c | 15 +++++++++++++++ include/linux/vfio.h | 3 +++ 3 files changed, 21 insertions(+), 5 deletions(-)drivers/vfio/pci/vfio_pci_core.c +0
-0
drivers/vfio/pci/vfio_pci_core.c static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, · line 2339
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +1
-5@@ -2335,12 +2335,10 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev,static int vfio_pci_is_device_in_set (struct pci_dev *pdev,void *data) {struct vfio_device_set *dev_set = data; -struct vfio_device *cur; -list_for_each_entry (cur, &dev_set->device_list, dev_set_list) -if (cur->dev == &pdev->dev) -return 0 ; -return -EBUSY ; +lockdep_assert_held (&dev_set->lock);
May be drop the lockdep_assert_held() above since there is one in vfio_find_device_in_devset().
Thanks,
C.
Quoted 45 lines
drivers/vfio/pci/vfio_pci_core.c +2
-0+ +return vfio_find_device_in_devset (dev_set, &pdev->dev) ?0 : -EBUSY ; }/* drivers/vfio/vfio_main.c +15
-0@@ -141,6 +141,21 @@ unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set) }EXPORT_SYMBOL_GPL (vfio_device_set_open_count); +struct vfio_device * +vfio_find_device_in_devset (struct vfio_device_set *dev_set, +struct device *dev) +{ +struct vfio_device *cur; + +lockdep_assert_held (&dev_set->lock); + +list_for_each_entry (cur, &dev_set->device_list, dev_set_list) +if (cur->dev == dev) +return cur; +return NULL ; +} +EXPORT_SYMBOL_GPL (vfio_find_device_in_devset); +/* * Device objects - create, release, get, put, search */ include/linux/vfio.h +3
-0@@ -259,6 +259,9 @@ void vfio_unregister_group_dev(struct vfio_device *device);int vfio_assign_device_set (struct vfio_device *device,void *set_id);unsigned int vfio_device_set_open_count (struct vfio_device_set *dev_set); +struct vfio_device * +vfio_find_device_in_devset (struct vfio_device_set *dev_set, +struct device *dev); int vfio_mig_get_next_state(struct vfio_device *device, enum vfio_device_mig_state cur_fsm,
Re: [Intel-gfx] [PATCH v5 07/10] vfio: Add helper to search vfio_device in a dev_set
In reply to Yi LiuQuoted 17 lines · On Sat, 13 May 2023 06:21:33 -0700 Yi Liu <[email protected]> wrote:
There are drivers that need to search
vfio_devicewithin a givendev_set. e.g. vfio-pci. So add a helper.Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 8 +++----- drivers/vfio/vfio_main.c | 15 +++++++++++++++ include/linux/vfio.h | 3 +++ 3 files changed, 21 insertions(+), 5 deletions(-)drivers/vfio/pci/vfio_pci_core.c +0
-0@@ -2335,12 +2335,10 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev,static int vfio_pci_is_device_in_set (struct pci_dev *pdev,void *data) {
drivers/vfio/pci/vfio_pci_core.c static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, · line 2341
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +3
-5struct vfio_device_set *dev_set = data; -struct vfio_device *cur; -list_for_each_entry (cur, &dev_set->device_list, dev_set_list) -if (cur->dev == &pdev->dev) -return 0 ; -return -EBUSY ; +lockdep_assert_held (&dev_set->lock); + +return vfio_find_device_in_devset (dev_set, &pdev->dev) ?0 : -EBUSY ;
Maybe an opportunity to revisit why this returns -EBUSY rather than something reasonable like -ENODEV. It looks like we picked up the -EBUSY in a882c16a2b7e where I think it was trying to preserve the return of vfio_pci_try_zap_and_vma_lock_cb() but the return value here is not even propagated so this looks like an chance to have it make sense again. Thanks,
Alex
Quoted 43 lines
drivers/vfio/pci/vfio_pci_core.c +0
-0}/* drivers/vfio/vfio_main.c +15
-0@@ -141,6 +141,21 @@ unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set) }EXPORT_SYMBOL_GPL (vfio_device_set_open_count); +struct vfio_device * +vfio_find_device_in_devset (struct vfio_device_set *dev_set, +struct device *dev) +{ +struct vfio_device *cur; + +lockdep_assert_held (&dev_set->lock); + +list_for_each_entry (cur, &dev_set->device_list, dev_set_list) +if (cur->dev == dev) +return cur; +return NULL ; +} +EXPORT_SYMBOL_GPL (vfio_find_device_in_devset); +/* * Device objects - create, release, get, put, search */ include/linux/vfio.h +3
-0@@ -259,6 +259,9 @@ void vfio_unregister_group_dev(struct vfio_device *device);int vfio_assign_device_set (struct vfio_device *device,void *set_id);unsigned int vfio_device_set_open_count (struct vfio_device_set *dev_set); +struct vfio_device * +vfio_find_device_in_devset (struct vfio_device_set *dev_set, +struct device *dev); int vfio_mig_get_next_state(struct vfio_device *device, enum vfio_device_mig_state cur_fsm,
Re: [Intel-gfx] [PATCH v5 07/10] vfio: Add helper to search vfio_device in a dev_set
Quoted 27 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, May 18, 2023 3:13 AMdrivers/vfio/pci/vfio_pci_core.c static bool vfio_dev_in_groups(struct
Quoted 15 lines · On Sat, 13 May 2023 06:21:33 -0700 Yi Liu <[email protected]> wrote:
There are drivers that need to search
vfio_devicewithin a givendev_set. e.g. vfio-pci. So add a helper.Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 8 +++----- drivers/vfio/vfio_main.c | 15 +++++++++++++++ include/linux/vfio.h | 3 +++ 3 files changed, 21 insertions(+), 5 deletions(-)drivers/vfio/pci/vfio_pci_core.c +0
-0@@ -2335,12 +2335,10 @@ static bool vfio_dev_in_groups(struct
vfio_pci_core_device*vdev,Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +0
-1static int vfio_pci_is_device_in_set (struct pci_dev *pdev,void *data) {struct vfio_device_set *dev_set = data; -struct vfio_device *cur;
Quoted 14 lines
drivers/vfio/pci/vfio_pci_core.c static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, · line 2341
Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +3
-4-list_for_each_entry (cur, &dev_set->device_list, dev_set_list) -if (cur->dev == &pdev->dev) -return 0 ; -return -EBUSY ; +lockdep_assert_held (&dev_set->lock); + +return vfio_find_device_in_devset (dev_set, &pdev->dev) ?0 : -EBUSY ;Maybe an opportunity to revisit why this returns
-EBUSYrather than something reasonable like-ENODEV. It looks like we picked up the-EBUSYina882c16a2b7ewhere I think it was trying to preserve the return ofvfio_pci_try_zap_and_vma_lock_cb()but the return value here is not even propagated so this looks like an chance to have it make sense again. Thanks,
From the name of this function, yes, -ENODEV is better. is it Ok to modify it to be -ENODEV in this patch or a separate one?
Regards,
Yi Liu
Quoted 45 lines
drivers/vfio/vfio_main.c unsigned int vfio_device_set_open_count(struct
Quoted 9 lines
drivers/vfio/pci/vfio_pci_core.c +0
-0}/* drivers/vfio/vfio_main.c +0
-0@@ -141,6 +141,21 @@ unsigned int vfio_device_set_open_count(struct
vfio_device_set*dev_set)Quoted 35 lines
drivers/vfio/vfio_main.c +15
-0}EXPORT_SYMBOL_GPL (vfio_device_set_open_count); +struct vfio_device * +vfio_find_device_in_devset (struct vfio_device_set *dev_set, +struct device *dev) +{ +struct vfio_device *cur; + +lockdep_assert_held (&dev_set->lock); + +list_for_each_entry (cur, &dev_set->device_list, dev_set_list) +if (cur->dev == dev) +return cur; +return NULL ; +} +EXPORT_SYMBOL_GPL (vfio_find_device_in_devset); +/* * Device objects - create, release, get, put, search */ include/linux/vfio.h +3
-0@@ -259,6 +259,9 @@ void vfio_unregister_group_dev(struct vfio_device *device);int vfio_assign_device_set (struct vfio_device *device,void *set_id);unsigned int vfio_device_set_open_count (struct vfio_device_set *dev_set); +struct vfio_device * +vfio_find_device_in_devset (struct vfio_device_set *dev_set, +struct device *dev); int vfio_mig_get_next_state(struct vfio_device *device, enum vfio_device_mig_state cur_fsm,
Re: [Intel-gfx] [PATCH v5 07/10] vfio: Add helper to search vfio_device in a dev_set
Quoted 42 lines · Yi L
Quoted 27 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, May 18, 2023 3:13 AMdrivers/vfio/pci/vfio_pci_core.c static bool vfio_dev_in_groups(struct
Quoted 15 lines · On Sat, 13 May 2023 06:21:33 -0700 Yi Liu <[email protected]> wrote:
There are drivers that need to search
vfio_devicewithin a givendev_set. e.g. vfio-pci. So add a helper.Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 8 +++----- drivers/vfio/vfio_main.c | 15 +++++++++++++++ include/linux/vfio.h | 3 +++ 3 files changed, 21 insertions(+), 5 deletions(-)drivers/vfio/pci/vfio_pci_core.c +0
-0@@ -2335,12 +2335,10 @@ static bool vfio_dev_in_groups(struct
vfio_pci_core_device*vdev,Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +0
-1static int vfio_pci_is_device_in_set (struct pci_dev *pdev,void *data) {struct vfio_device_set *dev_set = data; -struct vfio_device *cur;Quoted 14 lines
drivers/vfio/pci/vfio_pci_core.c static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, · line 2341
Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +3
-4-list_for_each_entry (cur, &dev_set->device_list, dev_set_list) -if (cur->dev == &pdev->dev) -return 0 ; -return -EBUSY ; +lockdep_assert_held (&dev_set->lock); + +return vfio_find_device_in_devset (dev_set, &pdev->dev) ?0 : -EBUSY ;Maybe an opportunity to revisit why this returns
-EBUSYrather than something reasonable like-ENODEV. It looks like we picked up the-EBUSYina882c16a2b7ewhere I think it was trying to preserve the return ofvfio_pci_try_zap_and_vma_lock_cb()but the return value here is not even propagated so this looks like an chance to have it make sense again. Thanks,
Quoted 2 lines
From the name of this function, yes,
-ENODEVis better. is it Ok to modify it to be-ENODEVin this patch or a separate one?
This patch is fine so long as it's noted in the commit log. Thanks,
Alex
Quoted 45 lines
Quoted 44 lines
drivers/vfio/vfio_main.c unsigned int vfio_device_set_open_count(struct
Quoted 8 lines
drivers/vfio/pci/vfio_pci_core.c +0
-0}/* drivers/vfio/vfio_main.c +0
-0@@ -141,6 +141,21 @@ unsigned int vfio_device_set_open_count(struct
vfio_device_set*dev_set)Quoted 35 lines
drivers/vfio/vfio_main.c +15
-0}EXPORT_SYMBOL_GPL (vfio_device_set_open_count); +struct vfio_device * +vfio_find_device_in_devset (struct vfio_device_set *dev_set, +struct device *dev) +{ +struct vfio_device *cur; + +lockdep_assert_held (&dev_set->lock); + +list_for_each_entry (cur, &dev_set->device_list, dev_set_list) +if (cur->dev == dev) +return cur; +return NULL ; +} +EXPORT_SYMBOL_GPL (vfio_find_device_in_devset); +/* * Device objects - create, release, get, put, search */ include/linux/vfio.h +3
-0@@ -259,6 +259,9 @@ void vfio_unregister_group_dev(struct vfio_device *device);int vfio_assign_device_set (struct vfio_device *device,void *set_id);unsigned int vfio_device_set_open_count (struct vfio_device_set *dev_set); +struct vfio_device * +vfio_find_device_in_devset (struct vfio_device_set *dev_set, +struct device *dev); int vfio_mig_get_next_state(struct vfio_device *device, enum vfio_device_mig_state cur_fsm,
PATCH v5 8/10 iommufd: Add iommufd_ctx_has_group()
In reply to Yi Liuto check if any device within the given iommu_group has been bound with the iommufd_ctx. This helpful for the checking on device ownership for the devices which have been bound but cannot be bound to any other iommufd as the iommu_group has been bound.
--- drivers/iommu/iommufd/device.c | 29 +++++++++++++++++++++++++++++ include/linux/iommufd.h | 8 ++++++++ 2 files changed, 37 insertions(+)
drivers/iommu/iommufd/device.c +29 -0
@@ -98,6 +98,35 @@ struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, }EXPORT_SYMBOL_NS_GPL (iommufd_device_bind,IOMMUFD ); +/** +* iommufd_ctx_has_group - True if the struct device is bound to this ictx +* @ictx: iommufd file descriptor +* @group: Pointer to a physical iommu_group struct +* +* True if a iommufd_device_bind() is present for any device within the +* group. +*/ +bool iommufd_ctx_has_group (struct iommufd_ctx *ictx,struct iommu_group *group) +{ +struct iommufd_object *obj; +unsigned long index; + +if (!ictx || !group) +return false; + +xa_lock (&ictx->objects); +xa_for_each (&ictx->objects, index, obj) { +if (obj->type ==IOMMUFD_OBJ_DEVICE && + container_of(obj, struct iommufd_device, obj)->group == group) { +xa_unlock (&ictx->objects); +return true; + } + } +xa_unlock (&ictx->objects); +return false; +} +EXPORT_SYMBOL_NS_GPL (iommufd_ctx_has_group,IOMMUFD ); +/** * iommufd_device_unbind - Undo iommufd_device_bind() * @idev: Device returned by iommufd_device_bind()
include/linux/iommufd.h +8 -0
@@ -16,6 +16,7 @@ struct page;struct iommufd_ctx ;struct iommufd_access ;struct file ; +struct iommu_group ;struct iommufd_device *iommufd_device_bind (struct iommufd_ctx *ictx,struct device *dev,u32 *id);
@@ -56,6 +57,7 @@ void iommufd_ctx_get(struct iommufd_ctx *ictx);#if IS_ENABLED (CONFIG_IOMMUFD )struct iommufd_ctx *iommufd_ctx_from_file (struct file *file);void iommufd_ctx_put (struct iommufd_ctx *ictx); +bool iommufd_ctx_has_group (struct iommufd_ctx *ictx,struct iommu_group *group); int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, unsigned long length, struct page **out_pages,
@@ -77,6 +79,12 @@ static inline void iommufd_ctx_put(struct iommufd_ctx *ictx) { } +static inline bool iommufd_ctx_has_group (struct iommufd_ctx *ictx, +struct iommu_group *group) +{ +return false; +} + static inline int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, unsigned long length,
Signature · 1 line
2.34.1
Re: [Intel-gfx] [PATCH v5 08/10] iommufd: Add iommufd_ctx_has_group()
Quoted 1 line · On Sat, 13 May 2023 06:21:34 -0700 Yi Liu <[email protected]> wrote:
to check if any device within the given
iommu_grouphas been bound with
Nit, I find these commit logs where the subject line is intended to flow into the commit log to form a complete sentence difficult to read. I expect complete thoughts within the commit log itself and the subject should be a separate summary of the log. Repeating the subject within the commit log is ok.
Quoted 1 line
the
iommufd_ctx. This helpful for the checking on device ownership for
s/This/This is/
Quoted 1 line
the devices which have been bound but cannot be bound to any other iommufd
s/have been/have not been/?
Quoted 13 lines
as the
iommu_grouphas been bound.Signed-off-by Yi Liu <[email protected]>--- drivers/iommu/iommufd/device.c | 29 +++++++++++++++++++++++++++++ include/linux/iommufd.h | 8 ++++++++ 2 files changed, 37 insertions(+)drivers/iommu/iommufd/device.c +0
-0
drivers/iommu/iommufd/device.c struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, · line 102
Quoted 5 lines
drivers/iommu/iommufd/device.c +2
-0@@ -98,6 +98,35 @@ struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, }EXPORT_SYMBOL_NS_GPL (iommufd_device_bind,IOMMUFD ); +/** +* iommufd_ctx_has_group - True if the struct device is bound to this ictx
What struct device? Isn't this "True if any device within the group is bound to the ictx"?
drivers/iommu/iommufd/device.c struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, · line 107
Quoted 5 lines
drivers/iommu/iommufd/device.c +5
-0+* @ictx: iommufd file descriptor +* @group: Pointer to a physical iommu_group struct +* +* True if a iommufd_device_bind() is present for any device within the +* group.
How can a function be present for a device? Maybe "True if any device within the group has been bound to this ictx, ex. via iommufd_device_bind(), therefore implying ictx ownership of the group." Thanks,
Alex
Quoted 58 lines
drivers/iommu/iommufd/device.c +22
-0+*/ +bool iommufd_ctx_has_group (struct iommufd_ctx *ictx,struct iommu_group *group) +{ +struct iommufd_object *obj; +unsigned long index; + +if (!ictx || !group) +return false; + +xa_lock (&ictx->objects); +xa_for_each (&ictx->objects, index, obj) { +if (obj->type ==IOMMUFD_OBJ_DEVICE && + container_of(obj, struct iommufd_device, obj)->group == group) { +xa_unlock (&ictx->objects); +return true; + } + } +xa_unlock (&ictx->objects); +return false; +} +EXPORT_SYMBOL_NS_GPL (iommufd_ctx_has_group,IOMMUFD ); +/** * iommufd_device_unbind - Undo iommufd_device_bind() * @idev: Device returned by iommufd_device_bind() include/linux/iommufd.h +8
-0@@ -16,6 +16,7 @@ struct page;struct iommufd_ctx ;struct iommufd_access ;struct file ; +struct iommu_group ;struct iommufd_device *iommufd_device_bind (struct iommufd_ctx *ictx,struct device *dev,u32 *id);@@ -56,6 +57,7 @@ void iommufd_ctx_get(struct iommufd_ctx *ictx);#if IS_ENABLED (CONFIG_IOMMUFD )struct iommufd_ctx *iommufd_ctx_from_file (struct file *file);void iommufd_ctx_put (struct iommufd_ctx *ictx); +bool iommufd_ctx_has_group (struct iommufd_ctx *ictx,struct iommu_group *group); int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, unsigned long length, struct page **out_pages,@@ -77,6 +79,12 @@ static inline void iommufd_ctx_put(struct iommufd_ctx *ictx) { } +static inline bool iommufd_ctx_has_group (struct iommufd_ctx *ictx, +struct iommu_group *group) +{ +return false; +} + static inline int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, unsigned long length,
Re: [Intel-gfx] [PATCH v5 08/10] iommufd: Add iommufd_ctx_has_group()
Quoted 6 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, May 18, 2023 3:40 AMOn Sat, 13 May 2023 06:21:34 -0700
Yi Liu <[email protected]> wrote:
Quoted 7 lines
Quoted 1 line
to check if any device within the given
iommu_grouphas been bound withNit, I find these commit logs where the subject line is intended to flow into the commit log to form a complete sentence difficult to read. I expect complete thoughts within the commit log itself and the subject should be a separate summary of the log. Repeating the subject within the commit log is ok.
Sure. I'll go through the commit messages.
Quoted 23 lines
Quoted 2 lines
the
iommufd_ctx. This helpful for the checking on device ownership fors/This/This is/
Quoted 1 line
the devices which have been bound but cannot be bound to any other iommufd
s/have been/have not been/?
drivers/iommu/iommufd/device.c struct iommufd_device *iommufd_device_bind(struct
Quoted 13 lines
as the
iommu_grouphas been bound.Signed-off-by Yi Liu <[email protected]>--- drivers/iommu/iommufd/device.c | 29 +++++++++++++++++++++++++++++ include/linux/iommufd.h | 8 ++++++++ 2 files changed, 37 insertions(+)drivers/iommu/iommufd/device.c +0
-0@@ -98,6 +98,35 @@ struct iommufd_device *iommufd_device_bind(struct
iommufd_ctx*ictx,
Quoted 8 lines
drivers/iommu/iommufd/device.c struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, · line 102
Quoted 5 lines
drivers/iommu/iommufd/device.c +2
-0}EXPORT_SYMBOL_NS_GPL (iommufd_device_bind,IOMMUFD ); +/** +* iommufd_ctx_has_group - True if the struct device is bound to this ictx What struct device? Isn't this "True if any device within the group is bound to the ictx"?
Yes, yes. a poor copy from a prior version..
Quoted 10 lines
drivers/iommu/iommufd/device.c struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, · line 107
Quoted 6 lines
drivers/iommu/iommufd/device.c +5
-0+* @ictx: iommufd file descriptor +* @group: Pointer to a physical iommu_group struct +* +* True if a iommufd_device_bind() is present for any device within the +* group. How can a function be present for a device? Maybe "True if any device within the group has been bound to this ictx, ex. via
iommufd_device_bind(), therefore implying ictx ownership of the group." Thanks,
Yes, this is the meaning of it. will fix it.
Regards,
Yi Liu
Quoted 59 lines
Quoted 59 lines
drivers/iommu/iommufd/device.c +22
-0+*/ +bool iommufd_ctx_has_group (struct iommufd_ctx *ictx,struct iommu_group *group) +{ +struct iommufd_object *obj; +unsigned long index; + +if (!ictx || !group) +return false; + +xa_lock (&ictx->objects); +xa_for_each (&ictx->objects, index, obj) { +if (obj->type ==IOMMUFD_OBJ_DEVICE && + container_of(obj, struct iommufd_device, obj)->group == group) { +xa_unlock (&ictx->objects); +return true; + } + } +xa_unlock (&ictx->objects); +return false; +} +EXPORT_SYMBOL_NS_GPL (iommufd_ctx_has_group,IOMMUFD ); +/** * iommufd_device_unbind - Undo iommufd_device_bind() * @idev: Device returned by iommufd_device_bind() include/linux/iommufd.h +8
-0@@ -16,6 +16,7 @@ struct page;struct iommufd_ctx ;struct iommufd_access ;struct file ; +struct iommu_group ;struct iommufd_device *iommufd_device_bind (struct iommufd_ctx *ictx,struct device *dev,u32 *id);@@ -56,6 +57,7 @@ void iommufd_ctx_get(struct iommufd_ctx *ictx);#if IS_ENABLED (CONFIG_IOMMUFD )struct iommufd_ctx *iommufd_ctx_from_file (struct file *file);void iommufd_ctx_put (struct iommufd_ctx *ictx); +bool iommufd_ctx_has_group (struct iommufd_ctx *ictx,struct iommu_group *group); int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, unsigned long length, struct page **out_pages,@@ -77,6 +79,12 @@ static inline void iommufd_ctx_put(struct iommufd_ctx *ictx) { } +static inline bool iommufd_ctx_has_group (struct iommufd_ctx *ictx, +struct iommu_group *group) +{ +return false; +} + static inline int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, unsigned long length,
PATCH v5 9/10 vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
In reply to Yi LiuThis makes VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl to use the iommufd_ctx of the cdev device to check the ownership of the other affected devices.
This returns devid for each of the affected devices. If it is bound to the iommufd_ctx of the cdev device, _INFO reports a valid devid > 0; If it is not opened by the calling user, but it belongs to the same iommu_group of a device that is bound to the iommufd_ctx of the cdev device, reports devid value of 0; If the device is un-owned device, configured within a different iommufd, or opened outside of the vfio device cdev API, the _INFO ioctl shall report devid value of -1.
devid >=0 doesn't block hot-reset as the affected devices are considered to be owned, while devid == -1 will block the use of VFIO_DEVICE_PCI_HOT_RESET outside of proof-of-ownership calling conventions (ie. via legacy group accessed devices).
This adds flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID to tell the user devid is returned in case of calling user get device fd from other software stack and adds flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED to tell user if all the affected devices are owned, so user can know it without looping all the returned devids.
--- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++++++++++++++++-- include/uapi/linux/vfio.h | 46 +++++++++++++++++++++++++++- 2 files changed, 95 insertions(+), 3 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +50 -2
@@ -27,6 +27,7 @@#include <linux/vgaarb.h> #include <linux/nospec.h> #include <linux/sched/mm.h> +#include <linux/iommufd.h> #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif
@@ -36,6 +37,10 @@#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"#define DRIVER_DESC "core driver for VFIO based PCI devices" +#ifdef CONFIG_IOMMUFD +MODULE_IMPORT_NS (IOMMUFD ); +#endif +static bool nointxmask;static bool disable_vga;static bool disable_idle_d3;
@@ -776,6 +781,9 @@ struct vfio_pci_fill_info {int max;int cur;struct vfio_pci_dependent_device *devices; +struct vfio_device *vdev; + bool devid:1; + bool dev_owned:1; };static int vfio_pci_fill_devs (struct pci_dev *pdev,void *data)
@@ -790,7 +798,37 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)if (!iommu_group)return -EPERM ;/* Cannot reset non-isolated devices */ - fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); +if (fill->devid) { +struct iommufd_ctx *iommufd =vfio_iommufd_physical_ictx (fill->vdev); +struct vfio_device_set *dev_set = fill->vdev->dev_set; +struct vfio_device *vdev; + +/* +* Report devid for the affected devices: +* - valid devid > 0 for the devices that are bound with +* the iommufd of the calling device. +* - devid == 0 for the devices that have not been opened +* but have same group with one of the devices bound to +* the iommufd of the calling device. +* - devid == -1 for others, and clear dev_owned flag. +*/ + vdev =vfio_find_device_in_devset (dev_set, &pdev->dev); +if (vdev && iommufd ==vfio_iommufd_physical_ictx (vdev)) { +int ret; + + ret =vfio_iommufd_physical_devid (vdev); +if (WARN_ON (ret <0 )) +return ret; + fill->devices[fill->cur].devid = ret; + }else if (vdev &&iommufd_ctx_has_group (iommufd, iommu_group)) { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_OWNED ; + }else { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_NOT_OWNED ; + fill->dev_owned = false; + } + }else { + fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); + } fill->devices[fill->cur].segment =pci_domain_nr (pdev->bus); fill->devices[fill->cur].bus = pdev->bus->number; fill->devices[fill->cur].devfn = pdev->devfn;
@@ -1229,17 +1267,27 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(return -ENOMEM ; fill.devices = devices; + fill.vdev = &vdev->vdev; +mutex_lock (&vdev->vdev.dev_set->lock); + fill.devid = fill.dev_owned =vfio_device_cdev_opened (&vdev->vdev); ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_fill_devs, &fill, slot); +mutex_unlock (&vdev->vdev.dev_set->lock);/* * If a device was removed between counting and filling, we may come up * short of fill.max. If a device was added, we'll have a return of * -EAGAIN above. */ -if (!ret) +if (!ret) { hdr.count = fill.cur; +if (fill.devid) { + hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID ; +if (fill.dev_owned) + hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; + } + } reset_info_exit:if (copy_to_user (arg, &hdr, minsz))
include/uapi/linux/vfio.h +45 -1
62 lines not on this page · raw message
Signature · 1 line
2.34.1
kernel test robot
build botRe: [PATCH v5 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
kernel test robot noticed the following build errors:
Hi Yi,
kernel test robot noticed the following build errors:
[auto build test ERROR on linus/master]
[also build test ERROR on v6.4-rc1 next-20230512]
[cannot apply to awilliam-vfio/next awilliam-vfio/for-linus joro-iommu/next] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Yi-Liu/vfio-iommufd-Create-iommufd_access-for-noiommu-devices/20230513-212336 base: linus/master patch link: https://lore.kernel.org/r/20230513132136.15021-10-yi.l.liu%40intel.com patch subject: [PATCH v5 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev config: arc-randconfig-r024-20230511 (https://download.01.org/0day-ci/archive/20230514/[email protected]/config) compiler: arceb-elf-gcc (GCC) 12.1.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/1f9114d23b61451e60915851ae95b7cc3c615c3d
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Yi-Liu/vfio-iommufd-Create-iommufd_access-for-noiommu-devices/20230513-212336 git checkout 1f9114d23b61451e60915851ae95b7cc3c615c3d
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=arc olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=arc SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>, old ones prefixed by <<):
>> ERROR: modpost: module vfio-pci-core uses symbol iommufd_ctx_has_group from namespace IOMMUFD, but does not import it.
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests
Re: [Intel-gfx] [PATCH v5 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
In reply to Yi LiuQuoted 43 lines · Yi Liu
This makes
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOioctl to use theiommufd_ctxof the cdev device to check the ownership of the other affected devices.This returns devid for each of the affected devices. If it is bound to the
iommufd_ctxof the cdev device,_INFOreports a valid devid > 0; If it is not opened by the calling user, but it belongs to the sameiommu_groupof a device that is bound to theiommufd_ctxof the cdev device, reports devid value of 0; If the device is un-owned device, configured within a different iommufd, or opened outside of the vfio device cdev API, the_INFOioctl shall report devid value of -1.devid >=0 doesn't block hot-reset as the affected devices are considered to be owned, while devid == -1 will block the use of
VFIO_DEVICE_PCI_HOT_RESEToutside of proof-of-ownership calling conventions (ie. via legacy group accessed devices).This adds flag
VFIO_PCI_HOT_RESET_FLAG_DEV_IDto tell the user devid is returned in case of calling user get device fd from other software stack and adds flagVFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNEDto tell user if all the affected devices are owned, so user can know it without looping all the returned devids.Suggested-by Jason Gunthorpe <[email protected]>Suggested-by Alex Williamson <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++++++++++++++++-- include/uapi/linux/vfio.h | 46 +++++++++++++++++++++++++++- 2 files changed, 95 insertions(+), 3 deletions(-)drivers/vfio/pci/vfio_pci_core.c +1
-0@@ -27,6 +27,7 @@#include <linux/vgaarb.h> #include <linux/nospec.h> #include <linux/sched/mm.h> +#include <linux/iommufd.h> #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif
drivers/vfio/pci/vfio_pci_core.c line 40
Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +1
-0@@ -36,6 +37,10 @@#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"#define DRIVER_DESC "core driver for VFIO based PCI devices" +#ifdef CONFIG_IOMMUFD
To import the IOMMUFD namespace, I had to use :
#if IS_ENABLED(CONFIG_IOMMUFD)
Thanks,
C.
Quoted 152 lines
drivers/vfio/pci/vfio_pci_core.c +48
-2+MODULE_IMPORT_NS (IOMMUFD ); +#endif +static bool nointxmask;static bool disable_vga;static bool disable_idle_d3;@@ -776,6 +781,9 @@ struct vfio_pci_fill_info {int max;int cur;struct vfio_pci_dependent_device *devices; +struct vfio_device *vdev; + bool devid:1; + bool dev_owned:1; };static int vfio_pci_fill_devs (struct pci_dev *pdev,void *data)@@ -790,7 +798,37 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)if (!iommu_group)return -EPERM ;/* Cannot reset non-isolated devices */ - fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); +if (fill->devid) { +struct iommufd_ctx *iommufd =vfio_iommufd_physical_ictx (fill->vdev); +struct vfio_device_set *dev_set = fill->vdev->dev_set; +struct vfio_device *vdev; + +/* +* Report devid for the affected devices: +* - valid devid > 0 for the devices that are bound with +* the iommufd of the calling device. +* - devid == 0 for the devices that have not been opened +* but have same group with one of the devices bound to +* the iommufd of the calling device. +* - devid == -1 for others, and clear dev_owned flag. +*/ + vdev =vfio_find_device_in_devset (dev_set, &pdev->dev); +if (vdev && iommufd ==vfio_iommufd_physical_ictx (vdev)) { +int ret; + + ret =vfio_iommufd_physical_devid (vdev); +if (WARN_ON (ret <0 )) +return ret; + fill->devices[fill->cur].devid = ret; + }else if (vdev &&iommufd_ctx_has_group (iommufd, iommu_group)) { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_OWNED ; + }else { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_NOT_OWNED ; + fill->dev_owned = false; + } + }else { + fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); + } fill->devices[fill->cur].segment =pci_domain_nr (pdev->bus); fill->devices[fill->cur].bus = pdev->bus->number; fill->devices[fill->cur].devfn = pdev->devfn;@@ -1229,17 +1267,27 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(return -ENOMEM ; fill.devices = devices; + fill.vdev = &vdev->vdev; +mutex_lock (&vdev->vdev.dev_set->lock); + fill.devid = fill.dev_owned =vfio_device_cdev_opened (&vdev->vdev); ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_fill_devs, &fill, slot); +mutex_unlock (&vdev->vdev.dev_set->lock);/* * If a device was removed between counting and filling, we may come up * short of fill.max. If a device was added, we'll have a return of * -EAGAIN above. */ -if (!ret) +if (!ret) { hdr.count = fill.cur; +if (fill.devid) { + hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID ; +if (fill.dev_owned) + hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; + } + } reset_info_exit:if (copy_to_user (arg, &hdr, minsz))include/uapi/linux/vfio.h +45
-1@@ -650,11 +650,53 @@ enum {* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info) * +* This command is used to query the affected devices in the hot reset for +* a given device. +* +* This command always reports the segment, bus, and devfn information for +* each affected device, and selectively reports the group_id or devid per +* the way how the calling device is opened. +* +* - If the calling device is opened via the traditional group/container +* API, group_id is reported. User should check if it has owned all +* the affected devices and provides a set of group fds to prove the +* ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl. +* +* - If the calling device is opened as a cdev, devid is reported. +* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set to indicate this +* data type. For a given affected device, it is considered owned by +* this interface if it meets the following conditions: +* 1) Has a valid devid within the iommufd_ctx of the calling device. +* Ownership cannot be determined across separate iommufd_ctx and the +* cdev calling conventions do not support a proof-of-ownership model +* as provided in the legacy group interface. In this case a valid +* devid with value greater than zero is provided in the return +* structure. +* 2) Does not have a valid devid within the iommufd_ctx of the calling +* device, but belongs to the same IOMMU group as the calling device +* or another opened device that has a valid devid within the +* iommufd_ctx of the calling device. This provides implicit ownership +* for devices within the same DMA isolation context. In this case +* the invalid devid value of zero is provided in the return structure. +* +* A devid value of -1 is provided in the return structure for devices +* where ownership is not available. Such devices prevent the use of +* VFIO_DEVICE_PCI_HOT_RESET outside of proof-of-ownership calling +* conventions (ie. via legacy group accessed devices). +* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the +* affected devices are owned by the user. This flag is available only +* when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved. +* * Return: 0 on success, -errno on failure: * -enospc = insufficient buffer, -enodev = unsupported for device. */ struct vfio_pci_dependent_device { -__u32 group_id; +union { +__u32 group_id; +__u32 devid; +#define VFIO_PCI_DEVID_OWNED 0 +#define VFIO_PCI_DEVID_NOT_OWNED -1 + };__u16 segment;__u8 bus;__u8 devfn;/* Use PCI_SLOT/PCI_FUNC */ @@ -663,6 +705,8 @@ struct vfio_pci_dependent_device {struct vfio_pci_hot_reset_info {__u32 argsz;__u32 flags; +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID (1 << 0) +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED (1 << 1)__u32 count;struct vfio_pci_dependent_device devices[]; };
Re: [Intel-gfx] [PATCH v5 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
Quoted 50 lines
From: Cédric Le Goater <[email protected]>
Sent: Monday, May 15, 2023 3:30 PMQuoted 46 lines · Yi Liu
This makes
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOioctl to use theiommufd_ctxof the cdev device to check the ownership of the other affected devices.This returns devid for each of the affected devices. If it is bound to the
iommufd_ctxof the cdev device,_INFOreports a valid devid > 0; If it is not opened by the calling user, but it belongs to the sameiommu_groupof a device that is bound to theiommufd_ctxof the cdev device, reports devid value of 0; If the device is un-owned device, configured within a different iommufd, or opened outside of the vfio device cdev API, the_INFOioctl shall report devid value of -1.devid >=0 doesn't block hot-reset as the affected devices are considered to be owned, while devid == -1 will block the use of
VFIO_DEVICE_PCI_HOT_RESEToutside of proof-of-ownership calling conventions (ie. via legacy group accessed devices).This adds flag
VFIO_PCI_HOT_RESET_FLAG_DEV_IDto tell the user devid is returned in case of calling user get device fd from other software stack and adds flagVFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNEDto tell user if all the affected devices are owned, so user can know it without looping all the returned devids.Suggested-by Jason Gunthorpe <[email protected]>Suggested-by Alex Williamson <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++++++++++++++++-- include/uapi/linux/vfio.h | 46 +++++++++++++++++++++++++++- 2 files changed, 95 insertions(+), 3 deletions(-)drivers/vfio/pci/vfio_pci_core.c +1
-0@@ -27,6 +27,7 @@#include <linux/vgaarb.h> #include <linux/nospec.h> #include <linux/sched/mm.h> +#include <linux/iommufd.h> #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif @@ -36,6 +37,10 @@#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"#define DRIVER_DESC "core driver for VFIO based PCI devices"
Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c line 40
Quoted 1 line
drivers/vfio/pci/vfio_pci_core.c +1
-0+#ifdef CONFIG_IOMMUFD To import the IOMMUFD namespace, I had to use :
#if
IS_ENABLED(CONFIG_IOMMUFD)
Thanks. Yes, IOMMUFD is tristate now, so needs to test CONFIG_IOMMUFD=m. and "#if IS_ENABLED(CONFIG_IOMMUFD)" fixes the compiling failure.
Regards,
Yi Liu
Quoted 155 lines
Quoted 71 lines
drivers/vfio/pci/vfio_pci_core.c +41
-1+MODULE_IMPORT_NS (IOMMUFD ); +#endif +static bool nointxmask;static bool disable_vga;static bool disable_idle_d3;@@ -776,6 +781,9 @@ struct vfio_pci_fill_info {int max;int cur;struct vfio_pci_dependent_device *devices; +struct vfio_device *vdev; + bool devid:1; + bool dev_owned:1; };static int vfio_pci_fill_devs (struct pci_dev *pdev,void *data)@@ -790,7 +798,37 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)if (!iommu_group)return -EPERM ;/* Cannot reset non-isolated devices */ - fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); +if (fill->devid) { +struct iommufd_ctx *iommufd =vfio_iommufd_physical_ictx (fill->vdev); +struct vfio_device_set *dev_set = fill->vdev->dev_set; +struct vfio_device *vdev; + +/* +* Report devid for the affected devices: +* - valid devid > 0 for the devices that are bound with +* the iommufd of the calling device. +* - devid == 0 for the devices that have not been opened +* but have same group with one of the devices bound to +* the iommufd of the calling device. +* - devid == -1 for others, and clear dev_owned flag. +*/ + vdev =vfio_find_device_in_devset (dev_set, &pdev->dev); +if (vdev && iommufd ==vfio_iommufd_physical_ictx (vdev)) { +int ret; + + ret =vfio_iommufd_physical_devid (vdev); +if (WARN_ON (ret <0 )) +return ret; + fill->devices[fill->cur].devid = ret; + }else if (vdev &&iommufd_ctx_has_group (iommufd, iommu_group)) { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_OWNED ; + }else { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_NOT_OWNED ; + fill->dev_owned = false; + } + }else { + fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); + } fill->devices[fill->cur].segment =pci_domain_nr (pdev->bus); fill->devices[fill->cur].bus = pdev->bus->number; fill->devices[fill->cur].devfn = pdev->devfn;@@ -1229,17 +1267,27 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(return -ENOMEM ; fill.devices = devices; + fill.vdev = &vdev->vdev; +mutex_lock (&vdev->vdev.dev_set->lock); + fill.devid = fill.dev_owned =vfio_device_cdev_opened (&vdev->vdev); ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_fill_devs, &fill, slot); +mutex_unlock (&vdev->vdev.dev_set->lock);/* * If a device was removed between counting and filling, we may come up drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_pci_hot_reset_info( · line 1288
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +5
-1* short of fill.max. If a device was added, we'll have a return of * -EAGAIN above. */ -if (!ret) +if (!ret) { hdr.count = fill.cur; +if (fill.devid) { + hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID ; +if (fill.dev_owned) + hdr.flags |=
VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED;Quoted 73 lines
drivers/vfio/pci/vfio_pci_core.c +2
-0+ } + } reset_info_exit:if (copy_to_user (arg, &hdr, minsz))include/uapi/linux/vfio.h +45
-1@@ -650,11 +650,53 @@ enum {* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info) * +* This command is used to query the affected devices in the hot reset for +* a given device. +* +* This command always reports the segment, bus, and devfn information for +* each affected device, and selectively reports the group_id or devid per +* the way how the calling device is opened. +* +* - If the calling device is opened via the traditional group/container +* API, group_id is reported. User should check if it has owned all +* the affected devices and provides a set of group fds to prove the +* ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl. +* +* - If the calling device is opened as a cdev, devid is reported. +* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set to indicate this +* data type. For a given affected device, it is considered owned by +* this interface if it meets the following conditions: +* 1) Has a valid devid within the iommufd_ctx of the calling device. +* Ownership cannot be determined across separate iommufd_ctx and the +* cdev calling conventions do not support a proof-of-ownership model +* as provided in the legacy group interface. In this case a valid +* devid with value greater than zero is provided in the return +* structure. +* 2) Does not have a valid devid within the iommufd_ctx of the calling +* device, but belongs to the same IOMMU group as the calling device +* or another opened device that has a valid devid within the +* iommufd_ctx of the calling device. This provides implicit ownership +* for devices within the same DMA isolation context. In this case +* the invalid devid value of zero is provided in the return structure. +* +* A devid value of -1 is provided in the return structure for devices +* where ownership is not available. Such devices prevent the use of +* VFIO_DEVICE_PCI_HOT_RESET outside of proof-of-ownership calling +* conventions (ie. via legacy group accessed devices). +* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the +* affected devices are owned by the user. This flag is available only +* when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved. +* * Return: 0 on success, -errno on failure: * -enospc = insufficient buffer, -enodev = unsupported for device. */ struct vfio_pci_dependent_device { -__u32 group_id; +union { +__u32 group_id; +__u32 devid; +#define VFIO_PCI_DEVID_OWNED 0 +#define VFIO_PCI_DEVID_NOT_OWNED -1 + };__u16 segment;__u8 bus;__u8 devfn;/* Use PCI_SLOT/PCI_FUNC */ @@ -663,6 +705,8 @@ struct vfio_pci_dependent_device {struct vfio_pci_hot_reset_info {__u32 argsz;__u32 flags; +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID (1 << 0) +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED (1 << 1)__u32 count;struct vfio_pci_dependent_device devices[]; };
Re: [Intel-gfx] [PATCH v5 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
In reply to Yi LiuQuoted 1 line · On Sat, 13 May 2023 06:21:35 -0700 Yi Liu <[email protected]> wrote:
This makes
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOioctl to use theiommufd_ctx
s/makes/allows/?
s/to//
Quoted 10 lines
of the cdev device to check the ownership of the other affected devices.
This returns devid for each of the affected devices. If it is bound to the
iommufd_ctxof the cdev device,_INFOreports a valid devid > 0; If it is not opened by the calling user, but it belongs to the sameiommu_groupof a device that is bound to theiommufd_ctxof the cdev device, reports devid value of 0; If the device is un-owned device, configured within a different iommufd, or opened outside of the vfio device cdev API, the_INFOioctl shall report devid value of -1.
Quoted 7 lines
devid >=0 doesn't block hot-reset as the affected devices are considered to be owned, while devid == -1 will block the use of
VFIO_DEVICE_PCI_HOT_RESEToutside of proof-of-ownership calling conventions (ie. via legacy group accessed devices).This adds flag
VFIO_PCI_HOT_RESET_FLAG_DEV_IDto tell the user devid is returned in case of calling user get device fd from other software stack
"other software stack"? I think this is trying to say something like:
When VFIO_DEVICE_GET_PCI_HOT_RESET_INFO is called on an IOMMUFD managed device, the new flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is reported to indicate the values returned are IOMMUFD devids rather than group IDs as used when accessing vfio devices through the conventional vfio group interface. Additionally the flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED will be reported in this mode if all of the devices affected by the hot-reset are owned by either virtue of being directly bound to the same iommufd context as the calling device, or implicitly owned via a shared IOMMU group.
Quoted 62 lines
and adds flag
VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNEDto tell user if all the affected devices are owned, so user can know it without looping all the returned devids.Suggested-by Jason Gunthorpe <[email protected]>Suggested-by Alex Williamson <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++++++++++++++++-- include/uapi/linux/vfio.h | 46 +++++++++++++++++++++++++++- 2 files changed, 95 insertions(+), 3 deletions(-)drivers/vfio/pci/vfio_pci_core.c +20
-1@@ -27,6 +27,7 @@#include <linux/vgaarb.h> #include <linux/nospec.h> #include <linux/sched/mm.h> +#include <linux/iommufd.h> #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif @@ -36,6 +37,10 @@#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"#define DRIVER_DESC "core driver for VFIO based PCI devices" +#ifdef CONFIG_IOMMUFD +MODULE_IMPORT_NS (IOMMUFD ); +#endif +static bool nointxmask;static bool disable_vga;static bool disable_idle_d3;@@ -776,6 +781,9 @@ struct vfio_pci_fill_info {int max;int cur;struct vfio_pci_dependent_device *devices; +struct vfio_device *vdev; + bool devid:1; + bool dev_owned:1; };static int vfio_pci_fill_devs (struct pci_dev *pdev,void *data)@@ -790,7 +798,37 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)if (!iommu_group)return -EPERM ;/* Cannot reset non-isolated devices */ - fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); +if (fill->devid) { +struct iommufd_ctx *iommufd =vfio_iommufd_physical_ictx (fill->vdev); +struct vfio_device_set *dev_set = fill->vdev->dev_set; +struct vfio_device *vdev; + +/* +* Report devid for the affected devices: +* - valid devid > 0 for the devices that are bound with +* the iommufd of the calling device. +* - devid == 0 for the devices that have not been opened +* but have same group with one of the devices bound to +* the iommufd of the calling device.
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 822
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10
-0+* - devid == -1 for others, and clear dev_owned flag. +*/ + vdev =vfio_find_device_in_devset (dev_set, &pdev->dev); +if (vdev && iommufd ==vfio_iommufd_physical_ictx (vdev)) { +int ret; + + ret =vfio_iommufd_physical_devid (vdev); +if (WARN_ON (ret <0 )) +return ret; + fill->devices[fill->cur].devid = ret;
Nit, @devid seems like a better variable name here rather than @ret.
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 828
Quoted 6 lines
drivers/vfio/pci/vfio_pci_core.c +6
-0+ }else if (vdev &&iommufd_ctx_has_group (iommufd, iommu_group)) { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_OWNED ; + }else { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_NOT_OWNED ; + fill->dev_owned = false; + }
I think we're not describing the requirements for this middle test correctly. We're essentially only stating the iommufd_ctx_has_group() part of the requirement, but we're also enforcing a vfio_find_device_in_devset() requirement, which means the device is not just unopened within a group shared by the iommufd context, but it must also still be a device registered as a member of the devset, ie. it must be bound to a vfio driver.
It's not a new requirement, it's imposed in the hot-reset ioctl itself, but it's new for the info ioctl given that it's now trying to report that the user can perform the reset for cdev callers.
This also shares too much logic with vfio_device_owned() added in the next patch. I think it might be cleaner to move the iommu_group_get() to the group path below and change vfio_device_owned() to something that can be used here and in the reset path. For example, if we had a function like:
static int vfio_hot_reset_devid(struct vfio_device *vdev,
struct iommufd_ctx *iommufd_ctx)
{
struct iommu_group *group;
int devid;
if (!vdev)
return VFIO_PCI_DEVID_NOT_OWNED;
if (vfio_iommufd_physical_ictx(vdev) == iommufd_ctx)
return vfio_iommufd_physical_devid(vdev);
group = iommu_group_get(vdev->dev);
if (!group)
return VFIO_PCI_DEVID_NOT_OWNED;
if (iommufd_ctx_has_group(iommufd_ctx, group))
devid = VFIO_PCI_DEVID_OWNED;
iommu_group_put(group);
return devid;
} It could be called above as:
vdev = vfio_find_device_in_devset(dev_set, &pdev->dev); fill->devices[fill->cur].devid = vfio_hot_reset_devid(vdev, iommufd);
And from vfio_pci_dev_set_hot_reset() as:
bool owned;
if (iommufd_ctx) {
int devid = vfio_hot_reset_devid(&cur_vma->vdev,
iommufd_ctx);
owned = (devid != VFIO_PCI_DEVID_NOT_OWNED);
} else
owned = vfio_dev_in_groups(&cur_vma->vdev, groups);Any better?
Quoted 22 lines
drivers/vfio/pci/vfio_pci_core.c +7
-0+ }else { + fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); + } fill->devices[fill->cur].segment =pci_domain_nr (pdev->bus); fill->devices[fill->cur].bus = pdev->bus->number; fill->devices[fill->cur].devfn = pdev->devfn;@@ -1229,17 +1267,27 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(return -ENOMEM ; fill.devices = devices; + fill.vdev = &vdev->vdev; +mutex_lock (&vdev->vdev.dev_set->lock); + fill.devid = fill.dev_owned =vfio_device_cdev_opened (&vdev->vdev); ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_fill_devs, &fill, slot); +mutex_unlock (&vdev->vdev.dev_set->lock);/* * If a device was removed between counting and filling, we may come up * short of fill.max. If a device was added, we'll have a return of * -EAGAIN above.
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_pci_hot_reset_info( · line 1290
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +7
-1*/ -if (!ret) +if (!ret) { hdr.count = fill.cur; +if (fill.devid) { + hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID ; +if (fill.dev_owned) + hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; + } + }
Does this clean up the flag and branching a bit?
drivers/vfio/pci/vfio_pci_core.c +9 -11
@@ -786,8 +786,7 @@ struct vfio_pci_fill_info {int cur;struct vfio_pci_dependent_device *devices;struct vfio_device *vdev; - bool devid:1; - bool dev_owned:1; +u32 flags; };static int vfio_pci_fill_devs (struct pci_dev *pdev,void *data)
@@ -802,7 +801,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)if (!iommu_group)return -EPERM ;/* Cannot reset non-isolated devices */ -if (fill->devid) { +if (fill->flags &VFIO_PCI_HOT_RESET_FLAG_DEV_ID ) {struct iommufd_ctx *iommufd =vfio_iommufd_physical_ictx (fill->vdev);struct vfio_device_set *dev_set = fill->vdev->dev_set;struct vfio_device *vdev;
@@ -814,7 +813,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)* - devid == 0 for the devices that have not been opened * but have same group with one of the devices bound to * the iommufd of the calling device. -* - devid == -1 for others, and clear dev_owned flag. +* - devid == -1 for others, and clear owned flag. */ vdev =vfio_find_device_in_devset (dev_set, &pdev->dev);if (vdev && iommufd ==vfio_iommufd_physical_ictx (vdev)) {
@@ -828,7 +827,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) fill->devices[fill->cur].devid =VFIO_PCI_DEVID_OWNED ; }else { fill->devices[fill->cur].devid =VFIO_PCI_DEVID_NOT_OWNED ; - fill->dev_owned = false; + fill->flags &= ~VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; } }else { fill->devices[fill->cur].group_id =iommu_group_id (iommu_group);
@@ -1273,8 +1272,11 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( fill.devices = devices; fill.vdev = &vdev->vdev; +if (vfio_device_cdev_opened (&vdev->vdev)) + fill.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID | +VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; +mutex_lock (&vdev->vdev.dev_set->lock); - fill.devid = fill.dev_owned =vfio_device_cdev_opened (&vdev->vdev); ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_fill_devs, &fill, slot);mutex_unlock (&vdev->vdev.dev_set->lock);
@@ -1286,11 +1288,7 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(*/ if (!ret) { hdr.count = fill.cur; -if (fill.devid) { - hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID ; -if (fill.dev_owned) - hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; - } + hdr.flags = fill.flags; } reset_info_exit:
Thanks,
Alex
Quoted 71 lines
drivers/vfio/pci/vfio_pci_core.c +0
-0reset_info_exit:if (copy_to_user (arg, &hdr, minsz))include/uapi/linux/vfio.h +45
-1@@ -650,11 +650,53 @@ enum {* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info) * +* This command is used to query the affected devices in the hot reset for +* a given device. +* +* This command always reports the segment, bus, and devfn information for +* each affected device, and selectively reports the group_id or devid per +* the way how the calling device is opened. +* +* - If the calling device is opened via the traditional group/container +* API, group_id is reported. User should check if it has owned all +* the affected devices and provides a set of group fds to prove the +* ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl. +* +* - If the calling device is opened as a cdev, devid is reported. +* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set to indicate this +* data type. For a given affected device, it is considered owned by +* this interface if it meets the following conditions: +* 1) Has a valid devid within the iommufd_ctx of the calling device. +* Ownership cannot be determined across separate iommufd_ctx and the +* cdev calling conventions do not support a proof-of-ownership model +* as provided in the legacy group interface. In this case a valid +* devid with value greater than zero is provided in the return +* structure. +* 2) Does not have a valid devid within the iommufd_ctx of the calling +* device, but belongs to the same IOMMU group as the calling device +* or another opened device that has a valid devid within the +* iommufd_ctx of the calling device. This provides implicit ownership +* for devices within the same DMA isolation context. In this case +* the invalid devid value of zero is provided in the return structure. +* +* A devid value of -1 is provided in the return structure for devices +* where ownership is not available. Such devices prevent the use of +* VFIO_DEVICE_PCI_HOT_RESET outside of proof-of-ownership calling +* conventions (ie. via legacy group accessed devices). +* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the +* affected devices are owned by the user. This flag is available only +* when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved. +* * Return: 0 on success, -errno on failure: * -enospc = insufficient buffer, -enodev = unsupported for device. */ struct vfio_pci_dependent_device { -__u32 group_id; +union { +__u32 group_id; +__u32 devid; +#define VFIO_PCI_DEVID_OWNED 0 +#define VFIO_PCI_DEVID_NOT_OWNED -1 + };__u16 segment;__u8 bus;__u8 devfn;/* Use PCI_SLOT/PCI_FUNC */ @@ -663,6 +705,8 @@ struct vfio_pci_dependent_device {struct vfio_pci_hot_reset_info {__u32 argsz;__u32 flags; +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID (1 << 0) +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED (1 << 1)__u32 count;struct vfio_pci_dependent_device devices[]; };
Re: [Intel-gfx] [PATCH v5 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
Quoted 32 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, May 18, 2023 6:02 AMQuoted 1 line · On Sat, 13 May 2023 06:21:35 -0700 Yi Liu <[email protected]> wrote:
This makes
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOioctl to use theiommufd_ctxs/makes/allows/?
s/to//
Quoted 10 lines
of the cdev device to check the ownership of the other affected devices.
This returns devid for each of the affected devices. If it is bound to the
iommufd_ctxof the cdev device,_INFOreports a valid devid > 0; If it is not opened by the calling user, but it belongs to the sameiommu_groupof a device that is bound to theiommufd_ctxof the cdev device, reports devid value of 0; If the device is un-owned device, configured within a different iommufd, or opened outside of the vfio device cdev API, the_INFOioctl shall report devid value of -1.Quoted 7 lines
devid >=0 doesn't block hot-reset as the affected devices are considered to be owned, while devid == -1 will block the use of
VFIO_DEVICE_PCI_HOT_RESEToutside of proof-of-ownership calling conventions (ie. via legacy group accessed devices).This adds flag
VFIO_PCI_HOT_RESET_FLAG_DEV_IDto tell the user devid is returned in case of calling user get device fd from other software stack"other software stack"? I think this is trying to say something like:
Quoted 9 lines
When
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOis called on an IOMMUFD managed device, the new flagVFIO_PCI_HOT_RESET_FLAG_DEV_IDis reported to indicate the values returned are IOMMUFD devids rather than group IDs as used when accessing vfio devices through the conventional vfio group interface. Additionally the flagVFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNEDwill be reported in this mode if all of the devices affected by the hot-reset are owned by either virtue of being directly bound to the same iommufd context as the calling device, or implicitly owned via a shared IOMMU group.
Yes. it is.
Quoted 75 lines
Quoted 62 lines
and adds flag
VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNEDto tell user if all the affected devices are owned, so user can know it without looping all the returned devids.Suggested-by Jason Gunthorpe <[email protected]>Suggested-by Alex Williamson <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++++++++++++++++-- include/uapi/linux/vfio.h | 46 +++++++++++++++++++++++++++- 2 files changed, 95 insertions(+), 3 deletions(-)drivers/vfio/pci/vfio_pci_core.c +20
-1@@ -27,6 +27,7 @@#include <linux/vgaarb.h> #include <linux/nospec.h> #include <linux/sched/mm.h> +#include <linux/iommufd.h> #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif @@ -36,6 +37,10 @@#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"#define DRIVER_DESC "core driver for VFIO based PCI devices" +#ifdef CONFIG_IOMMUFD +MODULE_IMPORT_NS (IOMMUFD ); +#endif +static bool nointxmask;static bool disable_vga;static bool disable_idle_d3;@@ -776,6 +781,9 @@ struct vfio_pci_fill_info {int max;int cur;struct vfio_pci_dependent_device *devices; +struct vfio_device *vdev; + bool devid:1; + bool dev_owned:1; };static int vfio_pci_fill_devs (struct pci_dev *pdev,void *data)@@ -790,7 +798,37 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)if (!iommu_group)return -EPERM ;/* Cannot reset non-isolated devices */ - fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); +if (fill->devid) { +struct iommufd_ctx *iommufd =vfio_iommufd_physical_ictx (fill->vdev); +struct vfio_device_set *dev_set = fill->vdev->dev_set; +struct vfio_device *vdev; + +/* +* Report devid for the affected devices: +* - valid devid > 0 for the devices that are bound with +* the iommufd of the calling device. +* - devid == 0 for the devices that have not been opened +* but have same group with one of the devices bound to +* the iommufd of the calling device. drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 822
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10
-0+* - devid == -1 for others, and clear dev_owned flag. +*/ + vdev =vfio_find_device_in_devset (dev_set, &pdev->dev); +if (vdev && iommufd ==vfio_iommufd_physical_ictx (vdev)) { +int ret; + + ret =vfio_iommufd_physical_devid (vdev); +if (WARN_ON (ret <0 )) +return ret; + fill->devices[fill->cur].devid = ret;Nit, @devid seems like a better variable name here rather than @ret.
Quoted 14 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 828
Quoted 6 lines
drivers/vfio/pci/vfio_pci_core.c +6
-0+ }else if (vdev &&iommufd_ctx_has_group (iommufd, iommu_group)) { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_OWNED ; + }else { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_NOT_OWNED ; + fill->dev_owned = false; + }I think we're not describing the requirements for this middle test correctly. We're essentially only stating the
iommufd_ctx_has_group()part of the requirement, but we're also enforcing avfio_find_device_in_devset()requirement, which means the device is not just unopened within a group shared by the iommufd context, but it must also still be a device registered as a member of the devset, ie. it must be bound to a vfio driver.
Yes. This is to make it be par with hot-reset path. This is needed. Is it?
Quoted 27 lines
It's not a new requirement, it's imposed in the hot-reset ioctl itself, but it's new for the info ioctl given that it's now trying to report that the user can perform the reset for cdev callers.
This also shares too much logic with
vfio_device_owned()added in the next patch. I think it might be cleaner to move theiommu_group_get()to the group path below and changevfio_device_owned()to something that can be used here and in the reset path. For example, if we had a function like:static int vfio_hot_reset_devid(struct
vfio_device*vdev,
structiommufd_ctx*iommufd_ctx){ struct iommu_group *group; int devid; if (!vdev) return VFIO_PCI_DEVID_NOT_OWNED; if (vfio_iommufd_physical_ictx(vdev) == iommufd_ctx) return vfio_iommufd_physical_devid(vdev); group = iommu_group_get(vdev->dev); if (!group) return VFIO_PCI_DEVID_NOT_OWNED;
Quoted 7 lines
if (iommufd_ctx_has_group(iommufd_ctx, group))
devid = VFIO_PCI_DEVID_OWNED;
iommu_group_put(group);
return devid;
}Thanks. This can be placed in vfio/iommufd.c, hence no need to Import the IOMMUFD namespace in vfio_pci_core.c. vfio_iommufd_physical_devid() can ebe static within vfio/iommufd.c
Quoted 19 lines
It could be called above as:
vdev = vfio_find_device_in_devset(dev_set, &pdev->dev); fill->devices[fill->cur].devid = vfio_hot_reset_devid(vdev, iommufd);And from
vfio_pci_dev_set_hot_reset()as:bool owned; if (iommufd_ctx) { int devid = vfio_hot_reset_devid(&cur_vma->vdev, iommufd_ctx); owned = (devid != VFIO_PCI_DEVID_NOT_OWNED); } else owned = vfio_dev_in_groups(&cur_vma->vdev, groups);
Quoted 1 line
Any better?
Above looks good enough.
Quoted 32 lines
Quoted 21 lines
drivers/vfio/pci/vfio_pci_core.c +7
-0+ }else { + fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); + } fill->devices[fill->cur].segment =pci_domain_nr (pdev->bus); fill->devices[fill->cur].bus = pdev->bus->number; fill->devices[fill->cur].devfn = pdev->devfn;@@ -1229,17 +1267,27 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(return -ENOMEM ; fill.devices = devices; + fill.vdev = &vdev->vdev; +mutex_lock (&vdev->vdev.dev_set->lock); + fill.devid = fill.dev_owned =vfio_device_cdev_opened (&vdev->vdev); ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_fill_devs, &fill, slot); +mutex_unlock (&vdev->vdev.dev_set->lock);/* * If a device was removed between counting and filling, we may come up drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_pci_hot_reset_info( · line 1288
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +5
-1* short of fill.max. If a device was added, we'll have a return of * -EAGAIN above. */ -if (!ret) +if (!ret) { hdr.count = fill.cur; +if (fill.devid) { + hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID ; +if (fill.dev_owned) + hdr.flags |=
VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED;
Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_pci_hot_reset_info( · line 1290
Quoted 2 lines
drivers/vfio/pci/vfio_pci_core.c +2
-0+ } + }Does this clean up the flag and branching a bit?
Yes. 😊
Quoted 70 lines
drivers/vfio/pci/vfio_pci_core.c +9
-11@@ -786,8 +786,7 @@ struct vfio_pci_fill_info {int cur;struct vfio_pci_dependent_device *devices;struct vfio_device *vdev; - bool devid:1; - bool dev_owned:1; +u32 flags; };static int vfio_pci_fill_devs (struct pci_dev *pdev,void *data)@@ -802,7 +801,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)if (!iommu_group)return -EPERM ;/* Cannot reset non-isolated devices */ -if (fill->devid) { +if (fill->flags &VFIO_PCI_HOT_RESET_FLAG_DEV_ID ) {struct iommufd_ctx *iommufd =vfio_iommufd_physical_ictx (fill->vdev);struct vfio_device_set *dev_set = fill->vdev->dev_set;struct vfio_device *vdev;@@ -814,7 +813,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)* - devid == 0 for the devices that have not been opened * but have same group with one of the devices bound to * the iommufd of the calling device. -* - devid == -1 for others, and clear dev_owned flag. +* - devid == -1 for others, and clear owned flag. */ vdev =vfio_find_device_in_devset (dev_set, &pdev->dev);if (vdev && iommufd ==vfio_iommufd_physical_ictx (vdev)) {@@ -828,7 +827,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) fill->devices[fill->cur].devid =VFIO_PCI_DEVID_OWNED ; }else { fill->devices[fill->cur].devid =VFIO_PCI_DEVID_NOT_OWNED ; - fill->dev_owned = false; + fill->flags &= ~VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; } }else { fill->devices[fill->cur].group_id =iommu_group_id (iommu_group);@@ -1273,8 +1272,11 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( fill.devices = devices; fill.vdev = &vdev->vdev; +if (vfio_device_cdev_opened (&vdev->vdev)) + fill.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID | +VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; +mutex_lock (&vdev->vdev.dev_set->lock); - fill.devid = fill.dev_owned =vfio_device_cdev_opened (&vdev->vdev); ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_fill_devs, &fill, slot);mutex_unlock (&vdev->vdev.dev_set->lock);@@ -1286,11 +1288,7 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(*/ if (!ret) { hdr.count = fill.cur; -if (fill.devid) { - hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID ; -if (fill.dev_owned) - hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; - } + hdr.flags = fill.flags; }
reset_info_exit:
Quoted 2 lines
Thanks,
Alex
Thanks,
Yi Liu
Quoted 71 lines
Quoted 71 lines
drivers/vfio/pci/vfio_pci_core.c +0
-0reset_info_exit:if (copy_to_user (arg, &hdr, minsz))include/uapi/linux/vfio.h +45
-1@@ -650,11 +650,53 @@ enum {* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info) * +* This command is used to query the affected devices in the hot reset for +* a given device. +* +* This command always reports the segment, bus, and devfn information for +* each affected device, and selectively reports the group_id or devid per +* the way how the calling device is opened. +* +* - If the calling device is opened via the traditional group/container +* API, group_id is reported. User should check if it has owned all +* the affected devices and provides a set of group fds to prove the +* ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl. +* +* - If the calling device is opened as a cdev, devid is reported. +* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set to indicate this +* data type. For a given affected device, it is considered owned by +* this interface if it meets the following conditions: +* 1) Has a valid devid within the iommufd_ctx of the calling device. +* Ownership cannot be determined across separate iommufd_ctx and the +* cdev calling conventions do not support a proof-of-ownership model +* as provided in the legacy group interface. In this case a valid +* devid with value greater than zero is provided in the return +* structure. +* 2) Does not have a valid devid within the iommufd_ctx of the calling +* device, but belongs to the same IOMMU group as the calling device +* or another opened device that has a valid devid within the +* iommufd_ctx of the calling device. This provides implicit ownership +* for devices within the same DMA isolation context. In this case +* the invalid devid value of zero is provided in the return structure. +* +* A devid value of -1 is provided in the return structure for devices +* where ownership is not available. Such devices prevent the use of +* VFIO_DEVICE_PCI_HOT_RESET outside of proof-of-ownership calling +* conventions (ie. via legacy group accessed devices). +* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the +* affected devices are owned by the user. This flag is available only +* when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved. +* * Return: 0 on success, -errno on failure: * -enospc = insufficient buffer, -enodev = unsupported for device. */ struct vfio_pci_dependent_device { -__u32 group_id; +union { +__u32 group_id; +__u32 devid; +#define VFIO_PCI_DEVID_OWNED 0 +#define VFIO_PCI_DEVID_NOT_OWNED -1 + };__u16 segment;__u8 bus;__u8 devfn;/* Use PCI_SLOT/PCI_FUNC */ @@ -663,6 +705,8 @@ struct vfio_pci_dependent_device {struct vfio_pci_hot_reset_info {__u32 argsz;__u32 flags; +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID (1 << 0) +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED (1 << 1)__u32 count;struct vfio_pci_dependent_device devices[]; };
Re: [Intel-gfx] [PATCH v5 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
Quoted 21 lines
From: Liu, Yi L <[email protected]>
Sent: Thursday, May 18, 2023 9:22 PMQuoted 18 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, May 18, 2023 6:02 AMOn Sat, 13 May 2023 06:21:35 -0700
Yi Liu <[email protected]> wrote:static int vfio_hot_reset_devid(struct
vfio_device*vdev,
structiommufd_ctx*iommufd_ctx){ struct iommu_group *group; int devid; if (!vdev) return VFIO_PCI_DEVID_NOT_OWNED; if (vfio_iommufd_physical_ictx(vdev) == iommufd_ctx) return vfio_iommufd_physical_devid(vdev);
Do we need to check the return of this helper? It returns -EINVAL when iommufd_access and iommufd_device are both null. Though not possible in this path. Is a WARN_ON needed or not?
Regards,
Yi Liu
Quoted 12 lines
Quoted 12 lines
group = iommu_group_get(vdev->dev); if (!group) return VFIO_PCI_DEVID_NOT_OWNED; if (iommufd_ctx_has_group(iommufd_ctx, group)) devid = VFIO_PCI_DEVID_OWNED; iommu_group_put(group); return devid; }
Re: [Intel-gfx] [PATCH v5 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
Quoted 19 lines · Yi L
Quoted 19 lines
From: Liu, Yi L <[email protected]>
Sent: Thursday, May 18, 2023 9:22 PMQuoted 16 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, May 18, 2023 6:02 AMOn Sat, 13 May 2023 06:21:35 -0700
Yi Liu <[email protected]> wrote:static int vfio_hot_reset_devid(struct
vfio_device*vdev,
structiommufd_ctx*iommufd_ctx){ struct iommu_group *group; int devid; if (!vdev) return VFIO_PCI_DEVID_NOT_OWNED;
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
if (vfio_iommufd_physical_ictx(vdev) == iommufd_ctx) return vfio_iommufd_physical_devid(vdev);Do we need to check the return of this helper? It returns
-EINVALwheniommufd_accessandiommufd_deviceare both null. Though not possible in this path. Is aWARN_ONneeded or not?
I also came to the conclusion that it wasn't possible while reviewing the code. I wouldn't got to extreme measures to introduce paranoia checks for impossible conditions. Thanks,
Alex
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
group = iommu_group_get(vdev->dev); if (!group) return VFIO_PCI_DEVID_NOT_OWNED; if (iommufd_ctx_has_group(iommufd_ctx, group)) devid = VFIO_PCI_DEVID_OWNED; iommu_group_put(group); return devid; }
Re: [Intel-gfx] [PATCH v5 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
In reply to Liu, Yi LQuoted 134 lines · Yi L
Quoted 32 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, May 18, 2023 6:02 AMQuoted 1 line · On Sat, 13 May 2023 06:21:35 -0700 Yi Liu <[email protected]> wrote:
This makes
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOioctl to use theiommufd_ctxs/makes/allows/?
s/to//
Quoted 10 lines
of the cdev device to check the ownership of the other affected devices.
This returns devid for each of the affected devices. If it is bound to the
iommufd_ctxof the cdev device,_INFOreports a valid devid > 0; If it is not opened by the calling user, but it belongs to the sameiommu_groupof a device that is bound to theiommufd_ctxof the cdev device, reports devid value of 0; If the device is un-owned device, configured within a different iommufd, or opened outside of the vfio device cdev API, the_INFOioctl shall report devid value of -1.Quoted 7 lines
devid >=0 doesn't block hot-reset as the affected devices are considered to be owned, while devid == -1 will block the use of
VFIO_DEVICE_PCI_HOT_RESEToutside of proof-of-ownership calling conventions (ie. via legacy group accessed devices).This adds flag
VFIO_PCI_HOT_RESET_FLAG_DEV_IDto tell the user devid is returned in case of calling user get device fd from other software stack"other software stack"? I think this is trying to say something like:
Quoted 9 lines
When
VFIO_DEVICE_GET_PCI_HOT_RESET_INFOis called on an IOMMUFD managed device, the new flagVFIO_PCI_HOT_RESET_FLAG_DEV_IDis reported to indicate the values returned are IOMMUFD devids rather than group IDs as used when accessing vfio devices through the conventional vfio group interface. Additionally the flagVFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNEDwill be reported in this mode if all of the devices affected by the hot-reset are owned by either virtue of being directly bound to the same iommufd context as the calling device, or implicitly owned via a shared IOMMU group.Yes. it is.
Quoted 75 lines
Quoted 62 lines
and adds flag
VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNEDto tell user if all the affected devices are owned, so user can know it without looping all the returned devids.Suggested-by Jason Gunthorpe <[email protected]>Suggested-by Alex Williamson <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++++++++++++++++-- include/uapi/linux/vfio.h | 46 +++++++++++++++++++++++++++- 2 files changed, 95 insertions(+), 3 deletions(-)drivers/vfio/pci/vfio_pci_core.c +20
-1@@ -27,6 +27,7 @@#include <linux/vgaarb.h> #include <linux/nospec.h> #include <linux/sched/mm.h> +#include <linux/iommufd.h> #if IS_ENABLED (CONFIG_EEH )#include <asm/eeh.h> #endif @@ -36,6 +37,10 @@#define DRIVER_AUTHOR "Alex Williamson <[email protected]>"#define DRIVER_DESC "core driver for VFIO based PCI devices" +#ifdef CONFIG_IOMMUFD +MODULE_IMPORT_NS (IOMMUFD ); +#endif +static bool nointxmask;static bool disable_vga;static bool disable_idle_d3;@@ -776,6 +781,9 @@ struct vfio_pci_fill_info {int max;int cur;struct vfio_pci_dependent_device *devices; +struct vfio_device *vdev; + bool devid:1; + bool dev_owned:1; };static int vfio_pci_fill_devs (struct pci_dev *pdev,void *data)@@ -790,7 +798,37 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)if (!iommu_group)return -EPERM ;/* Cannot reset non-isolated devices */ - fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); +if (fill->devid) { +struct iommufd_ctx *iommufd =vfio_iommufd_physical_ictx (fill->vdev); +struct vfio_device_set *dev_set = fill->vdev->dev_set; +struct vfio_device *vdev; + +/* +* Report devid for the affected devices: +* - valid devid > 0 for the devices that are bound with +* the iommufd of the calling device. +* - devid == 0 for the devices that have not been opened +* but have same group with one of the devices bound to +* the iommufd of the calling device. drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 822
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10
-0+* - devid == -1 for others, and clear dev_owned flag. +*/ + vdev =vfio_find_device_in_devset (dev_set, &pdev->dev); +if (vdev && iommufd ==vfio_iommufd_physical_ictx (vdev)) { +int ret; + + ret =vfio_iommufd_physical_devid (vdev); +if (WARN_ON (ret <0 )) +return ret; + fill->devices[fill->cur].devid = ret;Nit, @devid seems like a better variable name here rather than @ret.
Quoted 14 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) · line 828
Quoted 6 lines
drivers/vfio/pci/vfio_pci_core.c +6
-0+ }else if (vdev &&iommufd_ctx_has_group (iommufd, iommu_group)) { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_OWNED ; + }else { + fill->devices[fill->cur].devid =VFIO_PCI_DEVID_NOT_OWNED ; + fill->dev_owned = false; + }I think we're not describing the requirements for this middle test correctly. We're essentially only stating the
iommufd_ctx_has_group()part of the requirement, but we're also enforcing avfio_find_device_in_devset()requirement, which means the device is not just unopened within a group shared by the iommufd context, but it must also still be a device registered as a member of the devset, ie. it must be bound to a vfio driver.
Quoted 1 line
Yes. This is to make it be par with hot-reset path. This is needed. Is it?
Yes. In supporting this null-array model, the kernel is taking on the responsibility to indicate that the hot-reset is available, which necessarily includes the devset test. I think the logic we have is correct, but the documentation and comments should match the code as well.
Quoted 29 lines
Quoted 29 lines
It's not a new requirement, it's imposed in the hot-reset ioctl itself, but it's new for the info ioctl given that it's now trying to report that the user can perform the reset for cdev callers.
This also shares too much logic with
vfio_device_owned()added in the next patch. I think it might be cleaner to move theiommu_group_get()to the group path below and changevfio_device_owned()to something that can be used here and in the reset path. For example, if we had a function like:static int vfio_hot_reset_devid(struct
vfio_device*vdev,
structiommufd_ctx*iommufd_ctx){ struct iommu_group *group; int devid; if (!vdev) return VFIO_PCI_DEVID_NOT_OWNED; if (vfio_iommufd_physical_ictx(vdev) == iommufd_ctx) return vfio_iommufd_physical_devid(vdev); group = iommu_group_get(vdev->dev); if (!group) return VFIO_PCI_DEVID_NOT_OWNED; if (iommufd_ctx_has_group(iommufd_ctx, group)) devid = VFIO_PCI_DEVID_OWNED;
Quoted 8 lines
Quoted 4 lines
iommu_group_put(group); return devid; }Thanks. This can be placed in
vfio/iommufd.c, hence no need to Import the IOMMUFD namespace invfio_pci_core.c.vfio_iommufd_physical_devid()can ebe static withinvfio/iommufd.c
Good, that's an improvement too. Thanks,
Alex
Quoted 210 lines
Quoted 19 lines
It could be called above as:
vdev = vfio_find_device_in_devset(dev_set, &pdev->dev); fill->devices[fill->cur].devid = vfio_hot_reset_devid(vdev, iommufd);And from
vfio_pci_dev_set_hot_reset()as:bool owned; if (iommufd_ctx) { int devid = vfio_hot_reset_devid(&cur_vma->vdev, iommufd_ctx); owned = (devid != VFIO_PCI_DEVID_NOT_OWNED); } else owned = vfio_dev_in_groups(&cur_vma->vdev, groups);Quoted 1 line
Any better?
Above looks good enough.
Quoted 32 lines
Quoted 21 lines
drivers/vfio/pci/vfio_pci_core.c +7
-0+ }else { + fill->devices[fill->cur].group_id =iommu_group_id (iommu_group); + } fill->devices[fill->cur].segment =pci_domain_nr (pdev->bus); fill->devices[fill->cur].bus = pdev->bus->number; fill->devices[fill->cur].devfn = pdev->devfn;@@ -1229,17 +1267,27 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(return -ENOMEM ; fill.devices = devices; + fill.vdev = &vdev->vdev; +mutex_lock (&vdev->vdev.dev_set->lock); + fill.devid = fill.dev_owned =vfio_device_cdev_opened (&vdev->vdev); ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_fill_devs, &fill, slot); +mutex_unlock (&vdev->vdev.dev_set->lock);/* * If a device was removed between counting and filling, we may come up drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_pci_hot_reset_info( · line 1288
Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +5
-1* short of fill.max. If a device was added, we'll have a return of * -EAGAIN above. */ -if (!ret) +if (!ret) { hdr.count = fill.cur; +if (fill.devid) { + hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID ; +if (fill.dev_owned) + hdr.flags |=
VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED;Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_pci_hot_reset_info( · line 1290
Quoted 2 lines
drivers/vfio/pci/vfio_pci_core.c +2
-0+ } + }Does this clean up the flag and branching a bit?
Yes. 😊
Quoted 70 lines
drivers/vfio/pci/vfio_pci_core.c +9
-11@@ -786,8 +786,7 @@ struct vfio_pci_fill_info {int cur;struct vfio_pci_dependent_device *devices;struct vfio_device *vdev; - bool devid:1; - bool dev_owned:1; +u32 flags; };static int vfio_pci_fill_devs (struct pci_dev *pdev,void *data)@@ -802,7 +801,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)if (!iommu_group)return -EPERM ;/* Cannot reset non-isolated devices */ -if (fill->devid) { +if (fill->flags &VFIO_PCI_HOT_RESET_FLAG_DEV_ID ) {struct iommufd_ctx *iommufd =vfio_iommufd_physical_ictx (fill->vdev);struct vfio_device_set *dev_set = fill->vdev->dev_set;struct vfio_device *vdev;@@ -814,7 +813,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data)* - devid == 0 for the devices that have not been opened * but have same group with one of the devices bound to * the iommufd of the calling device. -* - devid == -1 for others, and clear dev_owned flag. +* - devid == -1 for others, and clear owned flag. */ vdev =vfio_find_device_in_devset (dev_set, &pdev->dev);if (vdev && iommufd ==vfio_iommufd_physical_ictx (vdev)) {@@ -828,7 +827,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) fill->devices[fill->cur].devid =VFIO_PCI_DEVID_OWNED ; }else { fill->devices[fill->cur].devid =VFIO_PCI_DEVID_NOT_OWNED ; - fill->dev_owned = false; + fill->flags &= ~VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; } }else { fill->devices[fill->cur].group_id =iommu_group_id (iommu_group);@@ -1273,8 +1272,11 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( fill.devices = devices; fill.vdev = &vdev->vdev; +if (vfio_device_cdev_opened (&vdev->vdev)) + fill.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID | +VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; +mutex_lock (&vdev->vdev.dev_set->lock); - fill.devid = fill.dev_owned =vfio_device_cdev_opened (&vdev->vdev); ret =vfio_pci_for_each_slot_or_bus (vdev->pdev, vfio_pci_fill_devs, &fill, slot);mutex_unlock (&vdev->vdev.dev_set->lock);@@ -1286,11 +1288,7 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(*/ if (!ret) { hdr.count = fill.cur; -if (fill.devid) { - hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID ; -if (fill.dev_owned) - hdr.flags |=VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED ; - } + hdr.flags = fill.flags; }
reset_info_exit:Quoted 2 lines
Thanks,
AlexThanks,
Yi LiuQuoted 71 lines
Quoted 71 lines
drivers/vfio/pci/vfio_pci_core.c +0
-0reset_info_exit:if (copy_to_user (arg, &hdr, minsz))include/uapi/linux/vfio.h +45
-1@@ -650,11 +650,53 @@ enum {* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info) * +* This command is used to query the affected devices in the hot reset for +* a given device. +* +* This command always reports the segment, bus, and devfn information for +* each affected device, and selectively reports the group_id or devid per +* the way how the calling device is opened. +* +* - If the calling device is opened via the traditional group/container +* API, group_id is reported. User should check if it has owned all +* the affected devices and provides a set of group fds to prove the +* ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl. +* +* - If the calling device is opened as a cdev, devid is reported. +* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set to indicate this +* data type. For a given affected device, it is considered owned by +* this interface if it meets the following conditions: +* 1) Has a valid devid within the iommufd_ctx of the calling device. +* Ownership cannot be determined across separate iommufd_ctx and the +* cdev calling conventions do not support a proof-of-ownership model +* as provided in the legacy group interface. In this case a valid +* devid with value greater than zero is provided in the return +* structure. +* 2) Does not have a valid devid within the iommufd_ctx of the calling +* device, but belongs to the same IOMMU group as the calling device +* or another opened device that has a valid devid within the +* iommufd_ctx of the calling device. This provides implicit ownership +* for devices within the same DMA isolation context. In this case +* the invalid devid value of zero is provided in the return structure. +* +* A devid value of -1 is provided in the return structure for devices +* where ownership is not available. Such devices prevent the use of +* VFIO_DEVICE_PCI_HOT_RESET outside of proof-of-ownership calling +* conventions (ie. via legacy group accessed devices). +* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the +* affected devices are owned by the user. This flag is available only +* when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved. +* * Return: 0 on success, -errno on failure: * -enospc = insufficient buffer, -enodev = unsupported for device. */ struct vfio_pci_dependent_device { -__u32 group_id; +union { +__u32 group_id; +__u32 devid; +#define VFIO_PCI_DEVID_OWNED 0 +#define VFIO_PCI_DEVID_NOT_OWNED -1 + };__u16 segment;__u8 bus;__u8 devfn;/* Use PCI_SLOT/PCI_FUNC */ @@ -663,6 +705,8 @@ struct vfio_pci_dependent_device {struct vfio_pci_hot_reset_info {__u32 argsz;__u32 flags; +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID (1 << 0) +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED (1 << 1)__u32 count;struct vfio_pci_dependent_device devices[]; };
PATCH v5 10/10 vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET
In reply to Yi LiuThis is the way user to invoke hot-reset for the devices opened by cdev interface. User should check the flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED in the output of VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl before doing hot-reset for cdev devices.
--- drivers/vfio/pci/vfio_pci_core.c | 68 ++++++++++++++++++++++++++------ include/uapi/linux/vfio.h | 14 +++++++ 2 files changed, 71 insertions(+), 11 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +57 -11
@@ -185,7 +185,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev)struct vfio_pci_group_info ;static void vfio_pci_dev_set_try_reset (struct vfio_device_set *dev_set);static int vfio_pci_dev_set_hot_reset (struct vfio_device_set *dev_set, -struct vfio_pci_group_info *groups); +struct vfio_pci_group_info *groups, +struct iommufd_ctx *iommufd_ctx);/* * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND
@@ -1323,8 +1324,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev,if (ret)return ret; -/* Somewhere between 1 and count is OK */ -if (!array_count || array_count > count) +if (array_count > count ||vfio_device_cdev_opened (&vdev->vdev))return -EINVAL ; group_fds =kcalloc (array_count,sizeof (*group_fds),GFP_KERNEL );
@@ -1373,7 +1373,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, info.count = array_count; info.files = files; - ret =vfio_pci_dev_set_hot_reset (vdev->vdev.dev_set, &info); + ret =vfio_pci_dev_set_hot_reset (vdev->vdev.dev_set, &info,NULL ); hot_reset_release:for (file_idx--; file_idx >=0 ; file_idx--)
@@ -1402,7 +1402,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev,else if (pci_probe_reset_bus (vdev->pdev->bus))return -ENODEV ; -return vfio_pci_ioctl_pci_hot_reset_groups (vdev, hdr.count, slot, arg); +if (hdr.count) +return vfio_pci_ioctl_pci_hot_reset_groups (vdev, hdr.count, slot, arg); + +return vfio_pci_dev_set_hot_reset (vdev->vdev.dev_set,NULL , +vfio_iommufd_physical_ictx (&vdev->vdev)); } static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev,
@@ -2369,13 +2373,16 @@ const struct pci_error_handlers vfio_pci_core_err_handlers = { };EXPORT_SYMBOL_GPL (vfio_pci_core_err_handlers); -static bool vfio_dev_in_groups (struct vfio_pci_core_device *vdev, +static bool vfio_dev_in_groups (struct vfio_device *vdev,struct vfio_pci_group_info *groups) {unsigned int i; +if (!groups) +return false; +for (i =0 ; i < groups->count; i++) -if (vfio_file_has_dev (groups->files[i], &vdev->vdev)) +if (vfio_file_has_dev (groups->files[i], vdev))return true;return false; }
@@ -2447,13 +2454,37 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set)return ret; } +static bool vfio_device_owned (struct vfio_device *vdev, +struct vfio_pci_group_info *groups, +struct iommufd_ctx *iommufd_ctx) +{ +struct iommu_group *group; + +WARN_ON (!!groups == !!iommufd_ctx); + +if (groups) +return vfio_dev_in_groups (vdev, groups); + +if (vfio_iommufd_physical_ictx (vdev) == iommufd_ctx) +return true; + + group =iommu_group_get (vdev->dev); +if (!group) +return false; + +iommu_group_put (group); + +return iommufd_ctx_has_group (iommufd_ctx, group); +} +/* * We need to get memory_lock for each device, but devices can share mmap_lock, * therefore we need to zap and hold the vma_lock for each device, and only then * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset (struct vfio_device_set *dev_set, -struct vfio_pci_group_info *groups) +struct vfio_pci_group_info *groups, +struct iommufd_ctx *iommufd_ctx) {struct vfio_pci_core_device *cur_mem;struct vfio_pci_core_device *cur_vma;
@@ -2484,10 +2515,25 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,list_for_each_entry (cur_vma, &dev_set->device_list, vdev.dev_set_list) {/* -* Test whether all the affected devices are contained by the -* set of groups provided by the user. +* Test whether all the affected devices can be reset by the +* user. +* +* If the user provides a set of groups, all the devices +* in the dev_set should be contained by the set of groups +* provided by the user. +* +* If the user provides a zero-length group fd array, then +* all the devices in the dev_set must be bound to the same +* iommufd_ctx as the input iommufd_ctx. If there is any +* device that has not been bound to iommufd_ctx yet, check +* if its iommu_group has any device bound to the input +* iommufd_ctx Such devices can be considered owned by +* the input iommufd_ctx as the device cannot be owned +* by another iommufd_ctx when its iommu_group is owned. +* +* Otherwise, reset is not allowed. */ -if (!vfio_dev_in_groups (cur_vma, groups)) { +if (!vfio_device_owned (&cur_vma->vdev, groups, iommufd_ctx)) { ret = -EINVAL ; goto err_undo; }
include/uapi/linux/vfio.h +14 -0
@@ -686,6 +686,9 @@ enum {* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the * affected devices are owned by the user. This flag is available only * when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved. +* When set, user could invoke VFIO_DEVICE_PCI_HOT_RESET with a zero +* length fd array on the calling device as the ownership is validated +* by iommufd_ctx. * * Return: 0 on success, -errno on failure: * -enospc = insufficient buffer, -enodev = unsupported for device.
@@ -717,6 +720,17 @@ struct vfio_pci_hot_reset_info {* VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset) * +* Userspace requests hot reset for the devices it operates. Due to the +* underlying topology, multiple devices can be affected in the reset +* while some might be opened by another user. To avoid interference +* the calling user must ensure all affected devices are owned by itself. +* +* As the ownership described by VFIO_DEVICE_GET_PCI_HOT_RESET_INFO, the +* cdev opened devices must exclusively provide a zero-length fd array and +* the group opened devices must exclusively use an array of group fds for +* proof of ownership. Mixed access to devices between cdev and legacy +* groups are not supported by this interface. +* * Return: 0 on success, -errno on failure. */ struct vfio_pci_hot_reset {
Signature · 1 line
2.34.1
Patchwork
CI[Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for Enhance vfio PCI hot reset for vfio cdev device (rev2)
In reply to Yi Liu== Series Details ==
== Series Details ==
Series: Enhance vfio PCI hot reset for vfio cdev device (rev2)
URL : https://patchwork.freedesktop.org/series/116991/
State : warning
== Summary ==
Error: dim checkpatch faileda7174f490b03 vfio-iommufd: Create iommufd_access for noiommu devicesec30c8fef218 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() a77f9349425e vfio/pci: Move the existing hot reset logic to be a helper
-:6: WARNING:COMMIT_LOG_LONG_LINE: Possible unwrapped commit description (prefer a maximum 75 chars per line) #6:
This prepares to add another method for hot reset. The major hot reset logic
total: 0 errors, 1 warnings, 0 checks, 99 lines checkede61bed084fa4 vfio: Mark cdev usage in vfio_device0bdfe10424d2 iommufd: Reserve all negative IDs in the iommufd xarraya1ce1e605e4c vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device 5d87236894e5 vfio: Add helper to search vfio_device in a dev_sete7c23092383b iommufd: Add iommufd_ctx_has_group()a6a19d8a857f vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
-:15: WARNING:COMMIT_LOG_LONG_LINE: Possible unwrapped commit description (prefer a maximum 75 chars per line) #15:
iommufd, or opened outside of the vfio device cdev API, the _INFO ioctl shall
-:112: CHECK:MULTIPLE_ASSIGNMENTS: multiple assignments should be avoided #112: FILE: drivers/vfio/pci/vfio_pci_core.c:1273:
+ fill.devid = fill.dev_owned = vfio_device_cdev_opened(&vdev->vdev);
total: 0 errors, 1 warnings, 1 checks, 154 lines checked8b87fbee5680 vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET
Patchwork
CI[Intel-gfx] ✓ Fi.CI.BAT: success for Enhance vfio PCI hot reset for vfio cdev device (rev2)
In reply to Yi Liu== Series Details ==
== Series Details ==
Series: Enhance vfio PCI hot reset for vfio cdev device (rev2)
URL : https://patchwork.freedesktop.org/series/116991/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_13143 -> Patchwork_116991v2
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/index.html
Participating hosts (38 -> 36)
------------------------------
Missing (2): fi-snb-2520m bat-mtlp-6
Known issues
------------
Here are the changes found in Patchwork_116991v2 that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@i915_pm_backlight@basic-brightness@edp-1:
- bat-rplp-1: NOTRUN -> [ABORT][1] ([i915#7077])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/bat-rplp-1/igt@i915_pm_backlight@[email protected]
* igt@i915_selftest@live@slpc:
- bat-rpls-2: NOTRUN -> [DMESG-WARN][2] ([i915#6367])
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/bat-rpls-2/igt@i915_selftest@[email protected]
* igt@i915_suspend@basic-s2idle-without-i915:
- bat-rpls-2: NOTRUN -> [ABORT][3] ([i915#6687])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/bat-rpls-2/igt@[email protected]
* igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence@pipe-c-dp-1:
- bat-dg2-8: [PASS][4] -> [FAIL][5] ([i915#7932]) +1 similar issue
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/bat-dg2-8/igt@kms_pipe_crc_basic@[email protected]
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/bat-dg2-8/igt@kms_pipe_crc_basic@[email protected]
#### Possible fixes ####
* igt@i915_selftest@live@gt_heartbeat:
- fi-apl-guc: [DMESG-FAIL][6] ([i915#5334]) -> [PASS][7]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/fi-apl-guc/igt@i915_selftest@live@gt_heartbeat.html
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/fi-apl-guc/igt@i915_selftest@live@gt_heartbeat.html
* igt@i915_selftest@live@reset:
- bat-rpls-2: [ABORT][8] ([i915#4983] / [i915#7461] / [i915#7913] / [i915#8347]) -> [PASS][9]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/bat-rpls-2/igt@i915_selftest@[email protected]
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/bat-rpls-2/igt@i915_selftest@[email protected]
#### Warnings ####
* igt@kms_setmode@basic-clone-single-crtc:
- bat-rplp-1: [ABORT][10] ([i915#4579] / [i915#8260]) -> [SKIP][11] ([i915#3555] / [i915#4579])
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/bat-rplp-1/igt@[email protected]
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/bat-rplp-1/igt@[email protected]
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
[i915#3555]: https://gitlab.freedesktop.org/drm/intel/issues/3555
[i915#4579]: https://gitlab.freedesktop.org/drm/intel/issues/4579
[i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
[i915#5334]: https://gitlab.freedesktop.org/drm/intel/issues/5334
[i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
[i915#6687]: https://gitlab.freedesktop.org/drm/intel/issues/6687
[i915#7077]: https://gitlab.freedesktop.org/drm/intel/issues/7077
[i915#7461]: https://gitlab.freedesktop.org/drm/intel/issues/7461
[i915#7913]: https://gitlab.freedesktop.org/drm/intel/issues/7913
[i915#7932]: https://gitlab.freedesktop.org/drm/intel/issues/7932
[i915#7953]: https://gitlab.freedesktop.org/drm/intel/issues/7953
[i915#8260]: https://gitlab.freedesktop.org/drm/intel/issues/8260
[i915#8347]: https://gitlab.freedesktop.org/drm/intel/issues/8347
Build changes
-------------
* Linux: CI_DRM_13143 -> Patchwork_116991v2
CI-20190529: 20190529
CI_DRM_13143: 222ff19f23b0bd6aca0b52001d69699f78f5a206 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_7286: a482779488f11c432d7ddcb1980691ab1603f356 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
Patchwork_116991v2: 222ff19f23b0bd6aca0b52001d69699f78f5a206 @ git://anongit.freedesktop.org/gfx-ci/linux
### Linux commits
3f541983f698 vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET cef13857cc34 vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev 2e6dd120837f iommufd: Add iommufd_ctx_has_group()593a7da914e9 vfio: Add helper to search vfio_device in a dev_set da2127aa7129 vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device 6c893fcd1273 iommufd: Reserve all negative IDs in the iommufd xarray d102efa91e1b vfio: Mark cdev usage in vfio_device1ea446478dbe vfio/pci: Move the existing hot reset logic to be a helper 89b8b1e95639 vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() c0b566e1b13f vfio-iommufd: Create iommufd_access for noiommu devices
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/index.html
Patchwork
CI[Intel-gfx] ✓ Fi.CI.IGT: success for Enhance vfio PCI hot reset for vfio cdev device (rev2)
In reply to Yi Liu== Series Details ==
== Series Details ==
Series: Enhance vfio PCI hot reset for vfio cdev device (rev2)
URL : https://patchwork.freedesktop.org/series/116991/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_13143_full -> Patchwork_116991v2_full
====================================================
Summary
-------
**SUCCESS**
No regressions found.
Participating hosts (7 -> 7)
------------------------------
No changes in participating hosts
Known issues
------------
Here are the changes found in Patchwork_116991v2_full that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@gem_lmem_swapping@massive-random:
- shard-glk: NOTRUN -> [SKIP][1] ([fdo#109271] / [i915#4613])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-glk5/igt@[email protected]
* igt@kms_chamelium_color@ctm-max:
- shard-glk: NOTRUN -> [SKIP][2] ([fdo#109271]) +26 similar issues
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-glk5/igt@[email protected]
* igt@kms_content_protection@atomic:
- shard-glk: NOTRUN -> [SKIP][3] ([fdo#109271] / [i915#4579])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-glk5/igt@[email protected]
#### Possible fixes ####
* igt@gem_barrier_race@remote-request@rcs0:
- shard-glk: [ABORT][4] ([i915#7461] / [i915#8211]) -> [PASS][5]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/shard-glk2/igt@gem_barrier_race@[email protected]
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-glk3/igt@gem_barrier_race@[email protected]
* igt@gem_ctx_freq@sysfs:
- {shard-dg1}: [FAIL][6] ([i915#6786]) -> [PASS][7]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/shard-dg1-15/igt@[email protected]
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-dg1-18/igt@[email protected]
* igt@gem_eio@in-flight-contexts-10ms:
- {shard-tglu}: [TIMEOUT][8] ([i915#3063] / [i915#7941]) -> [PASS][9]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/shard-tglu-3/igt@[email protected]
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-tglu-4/igt@[email protected]
* igt@gem_eio@reset-stress:
- {shard-dg1}: [FAIL][10] ([i915#5784]) -> [PASS][11]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/shard-dg1-12/igt@[email protected]
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-dg1-16/igt@[email protected]
* igt@gem_exec_fair@basic-pace-share@rcs0:
- shard-apl: [FAIL][12] ([i915#2842]) -> [PASS][13]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/shard-apl2/igt@gem_exec_fair@[email protected]
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-apl1/igt@gem_exec_fair@[email protected]
- {shard-tglu}: [FAIL][14] ([i915#2842]) -> [PASS][15]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/shard-tglu-10/igt@gem_exec_fair@[email protected]
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-tglu-7/igt@gem_exec_fair@[email protected]
* igt@gem_exec_suspend@basic-s4-devices@lmem0:
- {shard-dg1}: [ABORT][16] ([i915#7975] / [i915#8213]) -> [PASS][17]
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/shard-dg1-14/igt@gem_exec_suspend@[email protected]
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-dg1-17/igt@gem_exec_suspend@[email protected]
* igt@gen9_exec_parse@allowed-single:
- shard-glk: [ABORT][18] ([i915#5566]) -> [PASS][19]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/shard-glk1/igt@[email protected]
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-glk5/igt@[email protected]
* igt@kms_cursor_legacy@single-bo@pipe-b:
- {shard-rkl}: [INCOMPLETE][20] ([i915#8011]) -> [PASS][21]
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/shard-rkl-7/igt@kms_cursor_legacy@[email protected]
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-rkl-6/igt@kms_cursor_legacy@[email protected]
* igt@kms_flip@2x-flip-vs-expired-vblank-interruptible@ac-hdmi-a1-hdmi-a2:
- shard-glk: [FAIL][22] ([i915#79]) -> [PASS][23]
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13143/shard-glk2/igt@kms_flip@[email protected]
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/shard-glk3/igt@kms_flip@[email protected]
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[fdo#109289]: https://bugs.freedesktop.org/show_bug.cgi?id=109289
[fdo#109300]: https://bugs.freedesktop.org/show_bug.cgi?id=109300
[fdo#110189]: https://bugs.freedesktop.org/show_bug.cgi?id=110189
[fdo#111825]: https://bugs.freedesktop.org/show_bug.cgi?id=111825
[i915#1072]: https://gitlab.freedesktop.org/drm/intel/issues/1072
[i915#1397]: https://gitlab.freedesktop.org/drm/intel/issues/1397
[i915#2527]: https://gitlab.freedesktop.org/drm/intel/issues/2527
[i915#2587]: https://gitlab.freedesktop.org/drm/intel/issues/2587
[i915#2672]: https://gitlab.freedesktop.org/drm/intel/issues/2672
[i915#2842]: https://gitlab.freedesktop.org/drm/intel/issues/2842
[i915#3063]: https://gitlab.freedesktop.org/drm/intel/issues/3063
[i915#315]: https://gitlab.freedesktop.org/drm/intel/issues/315
[i915#3281]: https://gitlab.freedesktop.org/drm/intel/issues/3281
[i915#3458]: https://gitlab.freedesktop.org/drm/intel/issues/3458
[i915#3539]: https://gitlab.freedesktop.org/drm/intel/issues/3539
[i915#3689]: https://gitlab.freedesktop.org/drm/intel/issues/3689
[i915#3886]: https://gitlab.freedesktop.org/drm/intel/issues/3886
[i915#3952]: https://gitlab.freedesktop.org/drm/intel/issues/3952
[i915#3955]: https://gitlab.freedesktop.org/drm/intel/issues/3955
[i915#4070]: https://gitlab.freedesktop.org/drm/intel/issues/4070
[i915#4077]: https://gitlab.freedesktop.org/drm/intel/issues/4077
[i915#4078]: https://gitlab.freedesktop.org/drm/intel/issues/4078
[i915#4083]: https://gitlab.freedesktop.org/drm/intel/issues/4083
[i915#4270]: https://gitlab.freedesktop.org/drm/intel/issues/4270
[i915#4538]: https://gitlab.freedesktop.org/drm/intel/issues/4538
[i915#4579]: https://gitlab.freedesktop.org/drm/intel/issues/4579
[i915#4613]: https://gitlab.freedesktop.org/drm/intel/issues/4613
[i915#4771]: https://gitlab.freedesktop.org/drm/intel/issues/4771
[i915#4812]: https://gitlab.freedesktop.org/drm/intel/issues/4812
[i915#4816]: https://gitlab.freedesktop.org/drm/intel/issues/4816
[i915#4852]: https://gitlab.freedesktop.org/drm/intel/issues/4852
[i915#4860]: https://gitlab.freedesktop.org/drm/intel/issues/4860
[i915#4936]: https://gitlab.freedesktop.org/drm/intel/issues/4936
[i915#5176]: https://gitlab.freedesktop.org/drm/intel/issues/5176
[i915#5235]: https://gitlab.freedesktop.org/drm/intel/issues/5235
[i915#5354]: https://gitlab.freedesktop.org/drm/intel/issues/5354
[i915#5493]: https://gitlab.freedesktop.org/drm/intel/issues/5493
[i915#5566]: https://gitlab.freedesktop.org/drm/intel/issues/5566
[i915#5784]: https://gitlab.freedesktop.org/drm/intel/issues/5784
[i915#6095]: https://gitlab.freedesktop.org/drm/intel/issues/6095
[i915#6268]: https://gitlab.freedesktop.org/drm/intel/issues/6268
[i915#6786]: https://gitlab.freedesktop.org/drm/intel/issues/6786
[i915#7116]: https://gitlab.freedesktop.org/drm/intel/issues/7116
[i915#7461]: https://gitlab.freedesktop.org/drm/intel/issues/7461
[i915#79]: https://gitlab.freedesktop.org/drm/intel/issues/79
[i915#7941]: https://gitlab.freedesktop.org/drm/intel/issues/7941
[i915#7975]: https://gitlab.freedesktop.org/drm/intel/issues/7975
[i915#8011]: https://gitlab.freedesktop.org/drm/intel/issues/8011
[i915#8211]: https://gitlab.freedesktop.org/drm/intel/issues/8211
[i915#8213]: https://gitlab.freedesktop.org/drm/intel/issues/8213
[i915#8414]: https://gitlab.freedesktop.org/drm/intel/issues/8414
Build changes
-------------
* Linux: CI_DRM_13143 -> Patchwork_116991v2
CI-20190529: 20190529
CI_DRM_13143: 222ff19f23b0bd6aca0b52001d69699f78f5a206 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_7286: a482779488f11c432d7ddcb1980691ab1603f356 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
Patchwork_116991v2: 222ff19f23b0bd6aca0b52001d69699f78f5a206 @ git://anongit.freedesktop.org/gfx-ci/linux
piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_116991v2/index.html
Patchwork
CI[Intel-gfx] ✗ Fi.CI.BUILD: failure for Enhance vfio PCI hot reset for vfio cdev device (rev3)
In reply to Yi Liu== Series Details ==
== Series Details ==
Series: Enhance vfio PCI hot reset for vfio cdev device (rev3)
URL : https://patchwork.freedesktop.org/series/116991/
State : failure
== Summary ==
Error: patch https://patchwork.freedesktop.org/api/1.0/series/116991/revisions/3/mbox/ not applied
Applying: vfio-iommufd: Create iommufd_access for noiommu devices
Applying: vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset()
Applying: vfio/pci: Move the existing hot reset logic to be a helper
Applying: vfio: Mark cdev usage in vfio_device
Applying: iommufd: Reserve all negative IDs in the iommufd xarray
Applying: vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device
Applying: vfio: Add helper to search vfio_device in a dev_set
Applying: iommufd: Add iommufd_ctx_has_group()
Applying: vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
error: sha1 information is lacking or useless (drivers/vfio/pci/vfio_pci_core.c).
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0009 vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Build failed, no error log produced
Quoted 95 lines
-----Original Message-----
From: Liu, Yi L <[email protected]>
Subject: [PATCH v5 00/10] Enhance vfio PCI hot reset for vfio cdev device
VFIO_DEVICE_PCI_HOT_RESETrequires user to pass an array of group fds to prove that it owns all devices affected by resetting the calling device. While for cdev devices, user can use an iommufd-based ownership checking model and invokeVFIO_DEVICE_PCI_HOT_RESETwith a zero-length fd array.This series first creates
iommufd_accessfor noiommu devices to fill the gap for adding iommufd-based ownership checking model, then extendsVFIO_DEVICE_GET_PCI_HOT_RESET_INFOto check ownership and return the check result and the devid of affected devices to user. In the end, extends theVFIO_DEVICE_PCI_HOT_RESETto accept zero-length fd array for hot-reset with cdev devices.The new hot reset method and updated
_INFOioctl are tested with the below qemu:https://github.com/yiliu1765/qemu/tree/iommufd_rfcv4.mig.reset.v4_var3
(requires to test with the cdev kernel)Change log:
v5:
- Drop patch 01 of v4 (Alex)
- Createnoiommu_accessfor noiommu devices (Jason)
- Reserve all negative iommufd IDs, hence VFIO can encode negative values (Jason)
- Makevfio_iommufd_physical_devid()return-EINVALif it's not called with a physical device or a noiommu device.
- Addvfio_find_device_in_devset()invfio_main.c(Alex)
- Addiommufd_ctx_has_group()to replacevfio_devset_iommufd_has_group().
Reason:vfio_devset_iommufd_has_group()only loops the devices within the given devset to check the iommufd aniommu_group, but aniommu_group
can span into multiple devsets. So if failed to find the group in a devset doesn't mean the group is not owned by the iommufd. So here
either
needs to search all the devsets or add an iommufd API to check it. It appears an iommufd API makes more sense.
- Adopt suggestions from Alex on patch 08 and 09 of v4, refine the hot-reset uapi description and minor tweaks
- Use bitfields for bool members (Alex)v4: https://lore.kernel.org/kvm/[email protected]/
- Rename the patch series subject
- Patch 01 is moved from the cdev series
- Patch 02, 06 are new per review comments in v3
- Patch 03/04/05/07/08/09 are from v3 with updatesv3: https://lore.kernel.org/kvm/[email protected]/
- Remove the new_INFOioctl of v2, extend the existing_INFOioctl to report devid (Alex)
- Add r-b from Jason
- Add t-b from Terrence Xu and Yanting Jiang (mainly regression test)v2: https://lore.kernel.org/kvm/[email protected]/
- Split the patch 03 of v1 to be 03, 04 and 05 of v2 (Jaon)
- Add r-b from Kevin and Jason
- Add patch 10 to introduce a new_INFOioctl for the usage of device fd passing usage in cdev path (Jason, Alex)v1: https://lore.kernel.org/kvm/[email protected]/
Regards,
Yi LiuYi Liu (10):
vfio-iommufd: Createiommufd_accessfor noiommu devices vfio/pci: Update comment aroundgroup_fdget in
vfio_pci_ioctl_pci_hot_reset()
vfio/pci: Move the existing hot reset logic to be a helper
vfio: Mark cdev usage invfio_device
iommufd: Reserve all negative IDs in the iommufd xarray
vfio-iommufd: Add helper to retrieveiommufd_ctxand devid for
vfio_device
vfio: Add helper to searchvfio_devicein adev_set
iommufd: Addiommufd_ctx_has_group()
vfio/pci: ExtendVFIO_DEVICE_GET_PCI_HOT_RESET_INFOfor vfio device
cdev
vfio/pci: Allow passing zero-length fd array in
VFIO_DEVICE_PCI_HOT_RESETdrivers/iommu/iommufd/device.c | 53 +++++++++ drivers/iommu/iommufd/main.c | 2 +- drivers/vfio/iommufd.c | 63 ++++++++++- drivers/vfio/pci/vfio_pci_core.c | 184 ++++++++++++++++++++++++------- drivers/vfio/vfio_main.c | 15 +++ include/linux/iommufd.h | 14 +++ include/linux/vfio.h | 23 ++++ include/uapi/linux/vfio.h | 60 +++++++++- 8 files changed, 368 insertions(+), 46 deletions(-)
Quoted 2 lines
--
2.34.1
The new uapi works fine.
Tested GVT-g / GVT-d VFIO legacy mode / compat mode / cdev mode, including negative tests. No regression be introduced.