PATCH v3 0/2] Make mdev driver dma_unmap callback tolerant to unmaps come before device open
Subsystems (7)
Jason's "Connect VFIO to IOMMUFD" introduces vfio iommufd compat mode. Under this mode, vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. This means mdev drivers may receive unmap requests before the mdev is opened. For now, there are only three drivers (gvt, vfio-ap and vfio-ccw) providing dma_unmap(). vfio-ccw is fine with such requests. While gvt-g and vfio-ap may have potential problem with such requests due to internal implementation. This series tries to enhance the two drivers.
This series is based on Jason's below branch.
https://github.com/jgunthorpe/linux/tree/iommufd
(commit: 2079f24ce168f580a30e8eea94e660461d7d0d7a)
Change:
v3:
- Add explicit check on vgpu->nr_cache_entries, explicitly reset gvt cache (Zhenyu)
- Add Tony Krowiak's r-b for patch 0002
v2: https://lore.kernel.org/kvm/[email protected]/
- Refine the cover letter and commit message of patch 0001 (Kevin)
- Rename patch 0001 to better fit the commit message
- Add r-b from Zhi for patch 0001
- Tweak iova range test to assume page-aligned for patch 0002 (Jason)
- Remove break so all queues within range are removed for patch 0002 (Kevin)
v1: https://lore.kernel.org/kvm/[email protected]/
Regards,
Yi Liu
Matthew Rosato (1):
iommufd PATCH v2 2/2] vfio/ap: validate iova during dma_unmap and
trigger irq disable
Yi Liu (1):
i915/gvt: Move gvt mapping cache initialization to
intel_vgpu_init_dev()
drivers/gpu/drm/i915/gvt/kvmgt.c | 18 ++++++++++++++---- drivers/s390/crypto/vfio_ap_ops.c | 18 +++++++++++++++++- 2 files changed, 31 insertions(+), 5 deletions(-)
Signature · 1 line
2.34.1
[[iommufd] PATCH v3 2/2] iommufd PATCH v2 2/2] vfio/ap: validate iova during dma_unmap and trigger irq disable
From: Matthew Rosato <[email protected]>
Currently, each mapped iova is stashed in its associated vfio_ap_queue; when we get an unmap request, validate that it matches with one or more of these stashed values before attempting unpins.
Each stashed iova represents IRQ that was enabled for a queue. Therefore, if a match is found, trigger IRQ disable for this queue to ensure that underlying firmware will no longer try to use the associated pfn after the page is unpinned. IRQ disable will also handle the associated unpin.
--- drivers/s390/crypto/vfio_ap_ops.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +17 -1
@@ -1535,13 +1535,29 @@ static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev,return 0 ; } +static void unmap_iova (struct ap_matrix_mdev *matrix_mdev,u64 iova,u64 length) +{ +struct ap_queue_table *qtable = &matrix_mdev->qtable; +struct vfio_ap_queue *q; +int loop_cursor; + +hash_for_each (qtable->queues, loop_cursor, q, mdev_qnode) { +if (q->saved_iova >= iova && q->saved_iova < iova + length) +vfio_ap_irq_disable (q); + } +} +static void vfio_ap_mdev_dma_unmap (struct vfio_device *vdev,u64 iova,u64 length) {struct ap_matrix_mdev *matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); -vfio_unpin_pages (&matrix_mdev->vdev, iova,1 ); +mutex_lock (&matrix_dev->mdevs_lock); + +unmap_iova (matrix_mdev, iova, length); + +mutex_unlock (&matrix_dev->mdevs_lock); }/**
Signature · 1 line
2.34.1
[[iommufd] PATCH v3 1/2] i915/gvt: Move gvt mapping cache initialization to intel_vgpu_init_dev()
In reply to Yi Liuvfio container registers .dma_unmap() callback after the device is opened. So it's fine for mdev drivers to initialize internal mapping cache in .open_device(). See vfio_device_container_register().
Now with iommufd an access ops with an unmap callback is registered when the device is bound to iommufd which is before .open_device() is called. This implies gvt's .dma_unmap() could be called before its internal mapping cache is initialized.
The fix is moving gvt mapping cache initialization to vGPU init. While at it also move ptable initialization together.
--- drivers/gpu/drm/i915/gvt/kvmgt.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/gpu/drm/i915/gvt/kvmgt.c +14 -4
@@ -671,9 +671,6 @@ static int intel_vgpu_open_device(struct vfio_device *vfio_dev) vgpu->attached = true; -kvmgt_protect_table_init (vgpu); -gvt_cache_init (vgpu); - vgpu->track_node.track_write = kvmgt_page_track_write; vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot; kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
@@ -718,6 +715,11 @@ static void intel_vgpu_close_device(struct vfio_device *vfio_dev)kvmgt_protect_table_destroy (vgpu);gvt_cache_destroy (vgpu); +WARN_ON (vgpu->nr_cache_entries); + + vgpu->gfn_cache =RB_ROOT ; + vgpu->dma_addr_cache =RB_ROOT ; +intel_vgpu_release_msi_eventfd_ctx (vgpu); vgpu->attached = false;
@@ -1451,9 +1453,17 @@ static int intel_vgpu_init_dev(struct vfio_device *vfio_dev)struct intel_vgpu *vgpu =vfio_dev_to_vgpu (vfio_dev);struct intel_vgpu_type *type = container_of(mdev->type, struct intel_vgpu_type, type); +int ret; vgpu->gvt =kdev_to_i915 (mdev->type->parent->dev)->gvt; -return intel_gvt_create_vgpu (vgpu, type->conf); + ret =intel_gvt_create_vgpu (vgpu, type->conf); +if (ret) +return ret; + +kvmgt_protect_table_init (vgpu); +gvt_cache_init (vgpu); + +return 0 ; }static void intel_vgpu_release_dev (struct vfio_device *vfio_dev)
Signature · 1 line
2.34.1
Zhenyu Wang
Reviewed-byRe: [[iommufd] PATCH v3 1/2] i915/gvt: Move gvt mapping cache initialization to intel_vgpu_init_dev()
Quoted 58 lines · Yi Liu
vfio container registers .dma_unmap() callback after the device is opened. So it's fine for mdev drivers to initialize internal mapping cache in .open_device(). See
vfio_device_container_register().Now with iommufd an access ops with an unmap callback is registered when the device is bound to iommufd which is before .open_device() is called. This implies gvt's .dma_unmap() could be called before its internal mapping cache is initialized.
The fix is moving gvt mapping cache initialization to vGPU init. While at it also move ptable initialization together.
Cc Zhenyu Wang <[email protected]>Cc Zhi Wang <[email protected]>Cc Kevin Tian <[email protected]>Reviewed-by Zhi Wang <[email protected]>Signed-off-by Yi Liu <[email protected]>--- drivers/gpu/drm/i915/gvt/kvmgt.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)drivers/gpu/drm/i915/gvt/kvmgt.c +7
-4@@ -671,9 +671,6 @@ static int intel_vgpu_open_device(struct vfio_device *vfio_dev) vgpu->attached = true; -kvmgt_protect_table_init (vgpu); -gvt_cache_init (vgpu); - vgpu->track_node.track_write = kvmgt_page_track_write; vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot; kvm_page_track_register_notifier(vgpu->vfio_device.kvm,@@ -718,6 +715,11 @@ static void intel_vgpu_close_device(struct vfio_device *vfio_dev)kvmgt_protect_table_destroy (vgpu);gvt_cache_destroy (vgpu); +WARN_ON (vgpu->nr_cache_entries); + + vgpu->gfn_cache =RB_ROOT ; + vgpu->dma_addr_cache =RB_ROOT ; +intel_vgpu_release_msi_eventfd_ctx (vgpu); vgpu->attached = false;@@ -1451,9 +1453,17 @@ static int intel_vgpu_init_dev(struct vfio_device *vfio_dev)struct intel_vgpu *vgpu =vfio_dev_to_vgpu (vfio_dev);struct intel_vgpu_type *type = container_of(mdev->type, struct intel_vgpu_type, type); +int ret; vgpu->gvt =kdev_to_i915 (mdev->type->parent->dev)->gvt; -return intel_gvt_create_vgpu (vgpu, type->conf); + ret =intel_gvt_create_vgpu (vgpu, type->conf);
drivers/gpu/drm/i915/gvt/kvmgt.c static int intel_vgpu_init_dev(struct vfio_device *vfio_dev) · line 1469
Quoted 10 lines
drivers/gpu/drm/i915/gvt/kvmgt.c +7
-0+if (ret) +return ret; + +kvmgt_protect_table_init (vgpu); +gvt_cache_init (vgpu); + +return 0 ; }static void intel_vgpu_release_dev (struct vfio_device *vfio_dev)
thanks!
Quoted 7 lines · Yi Liu
Matthew Rosato (1):
iommufd PATCH v2 2/2] vfio/ap: validate iova duringdma_unmapand
trigger irq disableYi Liu (1):
i915/gvt: Move gvt mapping cache initialization to
intel_vgpu_init_dev()
I refreshed the patches with this version
Jason