Skip to contentSkip to conversation

Make mdev driver dma_unmap callback tolerant to unmaps come before device open

PATCH Superseded Yi Liu 6 participants 22 messages Nov 23, 2022 – Nov 29, 2022

Subsystems (7)
DRM drivers KVM Intel i915 s390 +3 more subsystems

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Yi Liu

author
#0 lore raw

cover Make mdev driver dma_unmap callback tolerant to unmaps come before device open

Jason's "Connect VFIO to IOMMUFD" introduces vfio iommufd compat mode. Under this mode, vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. This means mdev drivers may receive unmap requests before the mdev is opened. For now, most dma_unmap() callbacks are tolerant with such unmap requests, except for gvt-g and vfio-ap. This series tries to enhance the two drivers.

This series is based on Jason's below branch.

https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/log/?h=for-next
(commit: 57f62422b6f0477afaddd2fc77a4bb9b94275f42)

Cc Tony Krowiak <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Jason Herne <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Kevin Tian <[email protected]>

Regards,
Yi Liu

Matthew Rosato (1):
vfio/ap: validate iova during dma_unmap and trigger irq disable

Yi Liu (1):
i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into
init

 drivers/gpu/drm/i915/gvt/gvt.h    |  2 ++
 drivers/gpu/drm/i915/gvt/kvmgt.c  |  7 ++-----
 drivers/gpu/drm/i915/gvt/vgpu.c   |  2 ++
 drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++++++++++++-
 4 files changed, 29 insertions(+), 6 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/2 i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into init

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

To achieve above, move the protect_table_init and gvt_cache_init into the init op which is supposed to be triggered prior to the open_device() op.

Cc Zhenyu Wang <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Kevin Tian <[email protected]>
---
 drivers/gpu/drm/i915/gvt/gvt.h   | 2 ++
 drivers/gpu/drm/i915/gvt/kvmgt.c | 7 ++-----
 drivers/gpu/drm/i915/gvt/vgpu.c  | 2 ++
 3 files changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/gvt/gvt.h +2 -0
index dbf8d7470b2c..a3a7e16078ba 100644
@@ -754,6 +754,8 @@ void intel_gvt_debugfs_remove_vgpu(struct intel_vgpu *vgpu);
 void intel_gvt_debugfs_init(struct intel_gvt *gvt);
 void intel_gvt_debugfs_clean(struct intel_gvt *gvt);
 
+void gvt_cache_init(struct intel_vgpu *vgpu);
+void kvmgt_protect_table_init(struct intel_vgpu *info);
 int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn);
 int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn);
 int intel_gvt_dma_pin_guest_page(struct intel_vgpu *vgpu, dma_addr_t dma_addr);
drivers/gpu/drm/i915/gvt/kvmgt.c +2 -5
index 579b230a0f58..cb21b1ba4162 100644
@@ -322,7 +322,7 @@ static void gvt_cache_destroy(struct intel_vgpu *vgpu)
 	}
 }
 
-static void gvt_cache_init(struct intel_vgpu *vgpu)
+void gvt_cache_init(struct intel_vgpu *vgpu)
 {
 	vgpu->gfn_cache = RB_ROOT;
 	vgpu->dma_addr_cache = RB_ROOT;
@@ -330,7 +330,7 @@ static void gvt_cache_init(struct intel_vgpu *vgpu)
 	mutex_init(&vgpu->cache_lock);
 }
 
-static void kvmgt_protect_table_init(struct intel_vgpu *info)
+void kvmgt_protect_table_init(struct intel_vgpu *info)
 {
 	hash_init(info->ptable);
 }
@@ -671,9 +671,6 @@ static int intel_vgpu_open_device(struct vfio_device *vfio_dev)
 
 	vgpu->attached = true;
 
-	kvmgt_protect_table_init(vgpu);
-	gvt_cache_init(vgpu);
-
 	vgpu->track_node.track_write = kvmgt_page_track_write;
 	vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;
 	kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
drivers/gpu/drm/i915/gvt/vgpu.c +2 -0
index 56c71474008a..036e1a72a26b 100644
@@ -382,6 +382,8 @@ int intel_gvt_create_vgpu(struct intel_vgpu *vgpu,
 
 	intel_gvt_update_reg_whitelist(vgpu);
 	mutex_unlock(&gvt->lock);
+	kvmgt_protect_table_init(vgpu);
+	gvt_cache_init(vgpu);
 	return 0;
 
 out_clean_sched_policy:
Signature · 1 line
2.34.1
#2 lore raw

RE: [iommufd 1/2] i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into init

Quoted 6 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

this should first talk about how it works today and then why iommufd changes it.

Quoted 3 lines

To achieve above, move the protect_table_init and gvt_cache_init into the init op which is supposed to be triggered prior to the open_device() op.

what about below?
--
vfio container registers .dma_unmap() callback after the device is opened. So it's fine for mdev drivers to initialize internal mapping cache in .open_device(). See vfio_device_container_register().

Now with iommufd an access ops with an unmap callback is registered when the device is bound to iommufd which is before .open_device() is called. This implies gvt's .dma_unmap() could be called before its internal mapping cache is initialized.

The fix is moving gvt mapping cache initialization to vGPU creation. While at it also move ptable initialization together.

Quoted 32 lines
Cc Zhenyu Wang <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Kevin Tian <[email protected]>
---
 drivers/gpu/drm/i915/gvt/gvt.h   | 2 ++
 drivers/gpu/drm/i915/gvt/kvmgt.c | 7 ++-----
 drivers/gpu/drm/i915/gvt/vgpu.c  | 2 ++
 3 files changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/gvt/gvt.h +2 -0
index dbf8d7470b2c..a3a7e16078ba 100644
@@ -754,6 +754,8 @@ void intel_gvt_debugfs_remove_vgpu(struct
 intel_vgpu *vgpu);
 void intel_gvt_debugfs_init(struct intel_gvt *gvt);
 void intel_gvt_debugfs_clean(struct intel_gvt *gvt);
 
+void gvt_cache_init(struct intel_vgpu *vgpu);
+void kvmgt_protect_table_init(struct intel_vgpu *info);
 int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn);
 int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn);

int intel_gvt_dma_pin_guest_page(struct intel_vgpu *vgpu, dma_addr_t
dma_addr);

drivers/gpu/drm/i915/gvt/kvmgt.c +0 -0
index 579b230a0f58..cb21b1ba4162 100644

drivers/gpu/drm/i915/gvt/kvmgt.c static void gvt_cache_destroy(struct intel_vgpu *vgpu) · line 325

Quoted 5 lines
drivers/gpu/drm/i915/gvt/kvmgt.c +1 -1
@@ -322,7 +322,7 @@ static void gvt_cache_destroy(struct intel_vgpu *vgpu)
 	}
 }
 
-static void gvt_cache_init(struct intel_vgpu *vgpu)
+void gvt_cache_init(struct intel_vgpu *vgpu)

those are local functions. Just move to vgpu.c.

or you can remove the function wrap and directly put the internal lines in intel_gvt_create_vgpu()

Quoted 39 lines
drivers/gpu/drm/i915/gvt/kvmgt.c +1 -4
 {
 	vgpu->gfn_cache = RB_ROOT;
 	vgpu->dma_addr_cache = RB_ROOT;
@@ -330,7 +330,7 @@ static void gvt_cache_init(struct intel_vgpu *vgpu)
 	mutex_init(&vgpu->cache_lock);
 }
 
-static void kvmgt_protect_table_init(struct intel_vgpu *info)
+void kvmgt_protect_table_init(struct intel_vgpu *info)
 {
 	hash_init(info->ptable);
 }
@@ -671,9 +671,6 @@ static int intel_vgpu_open_device(struct vfio_device
 *vfio_dev)
 
 	vgpu->attached = true;
 
-	kvmgt_protect_table_init(vgpu);
-	gvt_cache_init(vgpu);
-
 	vgpu->track_node.track_write = kvmgt_page_track_write;
 	vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;
drivers/gpu/drm/i915/gvt/kvmgt.c +0 -0
 	kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
drivers/gpu/drm/i915/gvt/vgpu.c +2 -1
index 56c71474008a..036e1a72a26b 100644
@@ -382,6 +382,8 @@ int intel_gvt_create_vgpu(struct intel_vgpu *vgpu,
 
 	intel_gvt_update_reg_whitelist(vgpu);
 	mutex_unlock(&gvt->lock);
+	kvmgt_protect_table_init(vgpu);
+	gvt_cache_init(vgpu);
 	return 0;
 
 out_clean_sched_policy:
--

2.34.1

author
#3 lore raw

Re: [iommufd 1/2] i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into init

Quoted 20 lines · Kevin
Quoted 6 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

this should first talk about how it works today and then why iommufd changes it.

Quoted 3 lines

To achieve above, move the protect_table_init and gvt_cache_init into the init op which is supposed to be triggered prior to the open_device() op.

what about below?
--
vfio container registers .dma_unmap() callback after the device is opened. So it's fine for mdev drivers to initialize internal mapping cache in .open_device(). See vfio_device_container_register().

Quoted 7 lines

Now with iommufd an access ops with an unmap callback is registered when the device is bound to iommufd which is before .open_device() is called. This implies gvt's .dma_unmap() could be called before its internal mapping cache is initialized.

The fix is moving gvt mapping cache initialization to vGPU creation. While at it also move ptable initialization together.

much clearer :-)

Quoted 35 lines
Quoted 35 lines
Cc Zhenyu Wang <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Kevin Tian <[email protected]>
---
  drivers/gpu/drm/i915/gvt/gvt.h   | 2 ++
  drivers/gpu/drm/i915/gvt/kvmgt.c | 7 ++-----
  drivers/gpu/drm/i915/gvt/vgpu.c  | 2 ++
  3 files changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/gvt/gvt.h +2 -0
index dbf8d7470b2c..a3a7e16078ba 100644
@@ -754,6 +754,8 @@ void intel_gvt_debugfs_remove_vgpu(struct
 intel_vgpu *vgpu);
  void intel_gvt_debugfs_init(struct intel_gvt *gvt);
  void intel_gvt_debugfs_clean(struct intel_gvt *gvt);
 
+void gvt_cache_init(struct intel_vgpu *vgpu);
+void kvmgt_protect_table_init(struct intel_vgpu *info);
  int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn);
  int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn);

int intel_gvt_dma_pin_guest_page(struct intel_vgpu *vgpu, dma_addr_t
dma_addr);

drivers/gpu/drm/i915/gvt/kvmgt.c +0 -0
index 579b230a0f58..cb21b1ba4162 100644
@@ -322,7 +322,7 @@ static void gvt_cache_destroy(struct intel_vgpu *vgpu)
  	}
  }
 
Quoted 7 lines

drivers/gpu/drm/i915/gvt/kvmgt.c static void gvt_cache_destroy(struct intel_vgpu *vgpu) · line 325

Quoted 2 lines
drivers/gpu/drm/i915/gvt/kvmgt.c +1 -1
-static void gvt_cache_init(struct intel_vgpu *vgpu)
+void gvt_cache_init(struct intel_vgpu *vgpu)

those are local functions. Just move to vgpu.c.

or you can remove the function wrap and directly put the internal lines in intel_gvt_create_vgpu()

yes. maybe see Zhenyu and Zhi's input. which way is preferred by them.

Quoted 40 lines
Quoted 39 lines
drivers/gpu/drm/i915/gvt/kvmgt.c +1 -4
  {
  	vgpu->gfn_cache = RB_ROOT;
  	vgpu->dma_addr_cache = RB_ROOT;
@@ -330,7 +330,7 @@ static void gvt_cache_init(struct intel_vgpu *vgpu)
  	mutex_init(&vgpu->cache_lock);
  }
 
-static void kvmgt_protect_table_init(struct intel_vgpu *info)
+void kvmgt_protect_table_init(struct intel_vgpu *info)
  {
  	hash_init(info->ptable);
  }
@@ -671,9 +671,6 @@ static int intel_vgpu_open_device(struct vfio_device
 *vfio_dev)
 
  	vgpu->attached = true;
 
-	kvmgt_protect_table_init(vgpu);
-	gvt_cache_init(vgpu);
-
  	vgpu->track_node.track_write = kvmgt_page_track_write;
  	vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;

kvm_page_track_register_notifier(vgpu->vfio_device.kvm,

drivers/gpu/drm/i915/gvt/vgpu.c +2 -1
index 56c71474008a..036e1a72a26b 100644
@@ -382,6 +382,8 @@ int intel_gvt_create_vgpu(struct intel_vgpu *vgpu,
 
  	intel_gvt_update_reg_whitelist(vgpu);
  	mutex_unlock(&gvt->lock);
+	kvmgt_protect_table_init(vgpu);
+	gvt_cache_init(vgpu);
  	return 0;
 
  out_clean_sched_policy:
--

2.34.1

Signature · 2 lines
Regards,
Yi Liu
#4 lore raw

Re: [iommufd 1/2] i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into init

Quoted 26 lines · Yi Liu
Quoted 25 lines · Kevin
Quoted 6 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

this should first talk about how it works today and then why iommufd changes it.

Quoted 3 lines

To achieve above, move the protect_table_init and gvt_cache_init into the init op which is supposed to be triggered prior to the open_device() op.

what about below?
--
vfio container registers .dma_unmap() callback after the device is opened. So it's fine for mdev drivers to initialize internal mapping cache in .open_device(). See vfio_device_container_register().

Now with iommufd an access ops with an unmap callback is registered when the device is bound to iommufd which is before .open_device() is called. This implies gvt's .dma_unmap() could be called before its internal mapping cache is initialized.

Quoted 5 lines
Quoted 2 lines

The fix is moving gvt mapping cache initialization to vGPU creation. While at it also move ptable initialization together.

much clearer :-)

Current gvt internal cache is handled with .open_device() and .close_device() pair, so those internal cache is now re-initialized for each device session, how is that handled for iommufd? Looks that's missed in this patch..

Quoted 89 lines
Quoted 35 lines
Quoted 35 lines
Cc Zhenyu Wang <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Kevin Tian <[email protected]>
---
  drivers/gpu/drm/i915/gvt/gvt.h   | 2 ++
  drivers/gpu/drm/i915/gvt/kvmgt.c | 7 ++-----
  drivers/gpu/drm/i915/gvt/vgpu.c  | 2 ++
  3 files changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/gvt/gvt.h +2 -0
index dbf8d7470b2c..a3a7e16078ba 100644
@@ -754,6 +754,8 @@ void intel_gvt_debugfs_remove_vgpu(struct
 intel_vgpu *vgpu);
  void intel_gvt_debugfs_init(struct intel_gvt *gvt);
  void intel_gvt_debugfs_clean(struct intel_gvt *gvt);
 
+void gvt_cache_init(struct intel_vgpu *vgpu);
+void kvmgt_protect_table_init(struct intel_vgpu *info);
  int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn);
  int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn);

int intel_gvt_dma_pin_guest_page(struct intel_vgpu *vgpu, dma_addr_t
dma_addr);

drivers/gpu/drm/i915/gvt/kvmgt.c +0 -0
index 579b230a0f58..cb21b1ba4162 100644
@@ -322,7 +322,7 @@ static void gvt_cache_destroy(struct intel_vgpu *vgpu)
  	}
  }
 
Quoted 7 lines

drivers/gpu/drm/i915/gvt/kvmgt.c static void gvt_cache_destroy(struct intel_vgpu *vgpu) · line 325

Quoted 2 lines
drivers/gpu/drm/i915/gvt/kvmgt.c +1 -1
-static void gvt_cache_init(struct intel_vgpu *vgpu)
+void gvt_cache_init(struct intel_vgpu *vgpu)

those are local functions. Just move to vgpu.c.

or you can remove the function wrap and directly put the internal lines in intel_gvt_create_vgpu()

yes. maybe see Zhenyu and Zhi's input. which way is preferred by them.

Quoted 40 lines
Quoted 39 lines
drivers/gpu/drm/i915/gvt/kvmgt.c +1 -4
  {
  	vgpu->gfn_cache = RB_ROOT;
  	vgpu->dma_addr_cache = RB_ROOT;
@@ -330,7 +330,7 @@ static void gvt_cache_init(struct intel_vgpu *vgpu)
  	mutex_init(&vgpu->cache_lock);
  }
 
-static void kvmgt_protect_table_init(struct intel_vgpu *info)
+void kvmgt_protect_table_init(struct intel_vgpu *info)
  {
  	hash_init(info->ptable);
  }
@@ -671,9 +671,6 @@ static int intel_vgpu_open_device(struct vfio_device
 *vfio_dev)
 
  	vgpu->attached = true;
 
-	kvmgt_protect_table_init(vgpu);
-	gvt_cache_init(vgpu);
-
  	vgpu->track_node.track_write = kvmgt_page_track_write;
  	vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;

kvm_page_track_register_notifier(vgpu->vfio_device.kvm,

drivers/gpu/drm/i915/gvt/vgpu.c +2 -1
index 56c71474008a..036e1a72a26b 100644
@@ -382,6 +382,8 @@ int intel_gvt_create_vgpu(struct intel_vgpu *vgpu,
 
  	intel_gvt_update_reg_whitelist(vgpu);
  	mutex_unlock(&gvt->lock);
+	kvmgt_protect_table_init(vgpu);
+	gvt_cache_init(vgpu);
  	return 0;
 
  out_clean_sched_policy:
--

2.34.1

Signature · 2 lines
Regards,
Yi Liu
author
#5 lore raw

Re: [iommufd 1/2] i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into init

Quoted 30 lines · Zhenyu Wang
Quoted 29 lines · Yi Liu
Quoted 27 lines · Kevin
Quoted 6 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

this should first talk about how it works today and then why iommufd changes it.

Quoted 3 lines

To achieve above, move the protect_table_init and gvt_cache_init into the init op which is supposed to be triggered prior to the open_device() op.

what about below?
--
vfio container registers .dma_unmap() callback after the device is opened. So it's fine for mdev drivers to initialize internal mapping cache in .open_device(). See vfio_device_container_register().

Now with iommufd an access ops with an unmap callback is registered when the device is bound to iommufd which is before .open_device() is called. This implies gvt's .dma_unmap() could be called before its internal mapping cache is initialized.

The fix is moving gvt mapping cache initialization to vGPU creation. While at it also move ptable initialization together.

Quoted 6 lines
Quoted 2 lines

much clearer :-)

Current gvt internal cache is handled with .open_device() and .close_device() pair, so those internal cache is now re-initialized for each device session, how is that handled for iommufd? Looks that's missed in this patch..

you are right. I noticed below two helpers are used to destroy. However, the code seems to be more clear the internal cache. So seems no need to re-initialize. I'm no expert here. :)

gvt_cache_destroy()
kvmgt_protect_table_destroy()

Quoted 89 lines
Quoted 89 lines
Quoted 35 lines
Quoted 35 lines
Cc Zhenyu Wang <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Kevin Tian <[email protected]>
---
   drivers/gpu/drm/i915/gvt/gvt.h   | 2 ++
   drivers/gpu/drm/i915/gvt/kvmgt.c | 7 ++-----
   drivers/gpu/drm/i915/gvt/vgpu.c  | 2 ++
   3 files changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/gvt/gvt.h +2 -0
index dbf8d7470b2c..a3a7e16078ba 100644
@@ -754,6 +754,8 @@ void intel_gvt_debugfs_remove_vgpu(struct
 intel_vgpu *vgpu);
   void intel_gvt_debugfs_init(struct intel_gvt *gvt);
   void intel_gvt_debugfs_clean(struct intel_gvt *gvt);
 
+void gvt_cache_init(struct intel_vgpu *vgpu);
+void kvmgt_protect_table_init(struct intel_vgpu *info);
   int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn);
   int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn);

int intel_gvt_dma_pin_guest_page(struct intel_vgpu *vgpu, dma_addr_t
dma_addr);

drivers/gpu/drm/i915/gvt/kvmgt.c +0 -0
index 579b230a0f58..cb21b1ba4162 100644
@@ -322,7 +322,7 @@ static void gvt_cache_destroy(struct intel_vgpu *vgpu)
   	}
   }
 
Quoted 7 lines

drivers/gpu/drm/i915/gvt/kvmgt.c static void gvt_cache_destroy(struct intel_vgpu *vgpu) · line 325

Quoted 2 lines
drivers/gpu/drm/i915/gvt/kvmgt.c +1 -1
-static void gvt_cache_init(struct intel_vgpu *vgpu)
+void gvt_cache_init(struct intel_vgpu *vgpu)

those are local functions. Just move to vgpu.c.

or you can remove the function wrap and directly put the internal lines in intel_gvt_create_vgpu()

yes. maybe see Zhenyu and Zhi's input. which way is preferred by them.

Quoted 40 lines
Quoted 39 lines
drivers/gpu/drm/i915/gvt/kvmgt.c +1 -4
   {
   	vgpu->gfn_cache = RB_ROOT;
   	vgpu->dma_addr_cache = RB_ROOT;
@@ -330,7 +330,7 @@ static void gvt_cache_init(struct intel_vgpu *vgpu)
   	mutex_init(&vgpu->cache_lock);
   }
 
-static void kvmgt_protect_table_init(struct intel_vgpu *info)
+void kvmgt_protect_table_init(struct intel_vgpu *info)
   {
   	hash_init(info->ptable);
   }
@@ -671,9 +671,6 @@ static int intel_vgpu_open_device(struct vfio_device
 *vfio_dev)
 
   	vgpu->attached = true;
 
-	kvmgt_protect_table_init(vgpu);
-	gvt_cache_init(vgpu);
-
   	vgpu->track_node.track_write = kvmgt_page_track_write;
   	vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;
   	kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
drivers/gpu/drm/i915/gvt/vgpu.c +2 -1
index 56c71474008a..036e1a72a26b 100644
@@ -382,6 +382,8 @@ int intel_gvt_create_vgpu(struct intel_vgpu *vgpu,
 
   	intel_gvt_update_reg_whitelist(vgpu);
   	mutex_unlock(&gvt->lock);
+	kvmgt_protect_table_init(vgpu);
+	gvt_cache_init(vgpu);
   	return 0;
 
   out_clean_sched_policy:
--

2.34.1

Signature · 2 lines
Regards,
Yi Liu
Signature · 2 lines
Regards,
Yi Liu
#6 lore raw

RE: [iommufd 1/2] i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into init

Quoted 7 lines

From: Liu, Yi L <[email protected]>
Sent: Friday, November 25, 2022 5:07 PM

Quoted 2 lines

Current gvt internal cache is handled with .open_device()

and .close_device() pair,

Quoted 1 line

so those internal cache is now re-initialized for each device session, how is

that

Quoted 9 lines
Quoted 1 line

handled for iommufd? Looks that's missed in this patch..

you are right. I noticed below two helpers are used to destroy. However, the code seems to be more clear the internal cache. So seems no need to re-initialize. I'm no expert here. :)

gvt_cache_destroy()
kvmgt_protect_table_destroy()

It's common practice to have init in the probe path and then do the housekeeping in the close path.

Re-init in every open is unnecessary.

#7 lore raw

Re: [iommufd 1/2] i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into init

In reply to Yi Liu
Quoted 3 lines · Yi L

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

After reading code on Jason's tree, I think it would be nice to document

the requirements for a vfio driver somewhere, like what they must do and

must not do in the callbacks of vfio_device_ops. maybe in include/linux/vfio.h?

Quoted 63 lines

To achieve above, move the protect_table_init and gvt_cache_init into the init op which is supposed to be triggered prior to the open_device() op.

Cc Zhenyu Wang <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Kevin Tian <[email protected]>
---
 drivers/gpu/drm/i915/gvt/gvt.h   | 2 ++
 drivers/gpu/drm/i915/gvt/kvmgt.c | 7 ++-----
 drivers/gpu/drm/i915/gvt/vgpu.c  | 2 ++
 3 files changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/gvt/gvt.h +2 -0
index dbf8d7470b2c..a3a7e16078ba 100644
@@ -754,6 +754,8 @@ void intel_gvt_debugfs_remove_vgpu(struct intel_vgpu *vgpu);
 void intel_gvt_debugfs_init(struct intel_gvt *gvt);
 void intel_gvt_debugfs_clean(struct intel_gvt *gvt);
 
+void gvt_cache_init(struct intel_vgpu *vgpu);
+void kvmgt_protect_table_init(struct intel_vgpu *info);
 int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn);
 int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn);
 int intel_gvt_dma_pin_guest_page(struct intel_vgpu *vgpu, dma_addr_t dma_addr);
drivers/gpu/drm/i915/gvt/kvmgt.c +2 -5
index 579b230a0f58..cb21b1ba4162 100644
@@ -322,7 +322,7 @@ static void gvt_cache_destroy(struct intel_vgpu *vgpu)
 	}
 }
 
-static void gvt_cache_init(struct intel_vgpu *vgpu)
+void gvt_cache_init(struct intel_vgpu *vgpu)
 {
 	vgpu->gfn_cache = RB_ROOT;
 	vgpu->dma_addr_cache = RB_ROOT;
@@ -330,7 +330,7 @@ static void gvt_cache_init(struct intel_vgpu *vgpu)
 	mutex_init(&vgpu->cache_lock);
 }
 
-static void kvmgt_protect_table_init(struct intel_vgpu *info)
+void kvmgt_protect_table_init(struct intel_vgpu *info)
 {
 	hash_init(info->ptable);
 }
@@ -671,9 +671,6 @@ static int intel_vgpu_open_device(struct vfio_device *vfio_dev)
 
 	vgpu->attached = true;
 
-	kvmgt_protect_table_init(vgpu);
-	gvt_cache_init(vgpu);
-
 	vgpu->track_node.track_write = kvmgt_page_track_write;
 	vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;
 	kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
drivers/gpu/drm/i915/gvt/vgpu.c +0 -0
index 56c71474008a..036e1a72a26b 100644

drivers/gpu/drm/i915/gvt/vgpu.c int intel_gvt_create_vgpu(struct intel_vgpu *vgpu, · line 386

Quoted 5 lines
drivers/gpu/drm/i915/gvt/vgpu.c +2 -0
@@ -382,6 +382,8 @@ int intel_gvt_create_vgpu(struct intel_vgpu *vgpu,
 
 	intel_gvt_update_reg_whitelist(vgpu);
 	mutex_unlock(&gvt->lock);
+	kvmgt_protect_table_init(vgpu);
+	gvt_cache_init(vgpu);

It would be nice that you can still keep the initialization in the kvmgt.c as they are

kvmgt related.

With the changes above: Reviewed-by: Zhi Wang <[email protected]>

Quoted 3 lines
drivers/gpu/drm/i915/gvt/vgpu.c +0 -0
 	return 0;
 
 out_clean_sched_policy:
author
#8 lore raw

Re: [iommufd 1/2] i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into init

Quoted 10 lines · Zhi A
Quoted 3 lines · Yi L

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

After reading code on Jason's tree, I think it would be nice to document

the requirements for a vfio driver somewhere, like what they must do and

must not do in the callbacks of vfio_device_ops. maybe in include/linux/vfio.h?

yes. maybe vfio.rst. Altough I found it is out of date. I've got one fix and may send out later.

Quoted 64 lines
Quoted 64 lines

To achieve above, move the protect_table_init and gvt_cache_init into the init op which is supposed to be triggered prior to the open_device() op.

Cc Zhenyu Wang <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Kevin Tian <[email protected]>
---
  drivers/gpu/drm/i915/gvt/gvt.h   | 2 ++
  drivers/gpu/drm/i915/gvt/kvmgt.c | 7 ++-----
  drivers/gpu/drm/i915/gvt/vgpu.c  | 2 ++
  3 files changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/gvt/gvt.h +2 -0
index dbf8d7470b2c..a3a7e16078ba 100644
@@ -754,6 +754,8 @@ void intel_gvt_debugfs_remove_vgpu(struct intel_vgpu *vgpu);
  void intel_gvt_debugfs_init(struct intel_gvt *gvt);
  void intel_gvt_debugfs_clean(struct intel_gvt *gvt);
 
+void gvt_cache_init(struct intel_vgpu *vgpu);
+void kvmgt_protect_table_init(struct intel_vgpu *info);
  int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn);
  int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn);
  int intel_gvt_dma_pin_guest_page(struct intel_vgpu *vgpu, dma_addr_t dma_addr);
drivers/gpu/drm/i915/gvt/kvmgt.c +2 -5
index 579b230a0f58..cb21b1ba4162 100644
@@ -322,7 +322,7 @@ static void gvt_cache_destroy(struct intel_vgpu *vgpu)
  	}
  }
 
-static void gvt_cache_init(struct intel_vgpu *vgpu)
+void gvt_cache_init(struct intel_vgpu *vgpu)
  {
  	vgpu->gfn_cache = RB_ROOT;
  	vgpu->dma_addr_cache = RB_ROOT;
@@ -330,7 +330,7 @@ static void gvt_cache_init(struct intel_vgpu *vgpu)
  	mutex_init(&vgpu->cache_lock);
  }
 
-static void kvmgt_protect_table_init(struct intel_vgpu *info)
+void kvmgt_protect_table_init(struct intel_vgpu *info)
  {
  	hash_init(info->ptable);
  }
@@ -671,9 +671,6 @@ static int intel_vgpu_open_device(struct vfio_device *vfio_dev)
 
  	vgpu->attached = true;
 
-	kvmgt_protect_table_init(vgpu);
-	gvt_cache_init(vgpu);
-
  	vgpu->track_node.track_write = kvmgt_page_track_write;
  	vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot;
  	kvm_page_track_register_notifier(vgpu->vfio_device.kvm,
drivers/gpu/drm/i915/gvt/vgpu.c +0 -0
index 56c71474008a..036e1a72a26b 100644
@@ -382,6 +382,8 @@ int intel_gvt_create_vgpu(struct intel_vgpu *vgpu,
 
Quoted 6 lines

drivers/gpu/drm/i915/gvt/vgpu.c int intel_gvt_create_vgpu(struct intel_vgpu *vgpu, · line 386

Quoted 4 lines
drivers/gpu/drm/i915/gvt/vgpu.c +2 -0
  	intel_gvt_update_reg_whitelist(vgpu);
  	mutex_unlock(&gvt->lock);
+	kvmgt_protect_table_init(vgpu);
+	gvt_cache_init(vgpu);

It would be nice that you can still keep the initialization in the kvmgt.c as they are

ok, so I'll leave the function implementation in kvmgt.c.

Quoted 3 lines

kvmgt related.

With the changes above: Reviewed-by: Zhi Wang <[email protected]>

thanks. :-)

Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/i915/gvt/vgpu.c +0 -0
  	return 0;
 
  out_clean_sched_policy:
Signature · 2 lines
Regards,
Yi Liu
author
#9 lore raw

PATCH 2/2 vfio/ap: validate iova during dma_unmap and trigger irq disable

In reply to Yi Liu

From: Matthew Rosato <[email protected]>

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

To achieve above, vfio_ap_mdev_dma_unmap() needs to validate that unmap request matches with one or more of these stashed values before attempting unpins.

Currently, each mapped iova is stashed in its associated vfio_ap_queue; Each stashed iova represents IRQ that was enabled for a queue. Therefore, if a match is found, trigger IRQ disable for this queue to ensure that underlying firmware will no longer try to use the associated pfn after the page is unpinned. IRQ disable will also handle the associated unpin.

Cc Tony Krowiak <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Jason Herne <[email protected]>
---
 drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +23 -1
index bb7776d20792..62bfca2bbe6d 100644
@@ -1535,13 +1535,35 @@ static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev,
 	return 0;
 }
 
+static void unmap_iova(struct ap_matrix_mdev *matrix_mdev, u64 iova, u64 length)
+{
+	struct ap_queue_table *qtable = &matrix_mdev->qtable;
+	u64 iova_pfn_end = (iova + length - 1) >> PAGE_SHIFT;
+	u64 iova_pfn_start = iova >> PAGE_SHIFT;
+	struct vfio_ap_queue *q;
+	int loop_cursor;
+	u64 pfn;
+
+	hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) {
+		pfn = q->saved_iova >> PAGE_SHIFT;
+		if (pfn >= iova_pfn_start && pfn <= iova_pfn_end) {
+			vfio_ap_irq_disable(q);
+			break;
+		}
+	}
+}
+
 static void vfio_ap_mdev_dma_unmap(struct vfio_device *vdev, u64 iova,
 				   u64 length)
 {
 	struct ap_matrix_mdev *matrix_mdev =
 		container_of(vdev, struct ap_matrix_mdev, vdev);
 
-	vfio_unpin_pages(&matrix_mdev->vdev, iova, 1);
+	mutex_lock(&matrix_dev->mdevs_lock);
+
+	unmap_iova(matrix_mdev, iova, length);
+
+	mutex_unlock(&matrix_dev->mdevs_lock);
 }
 
 /**
Signature · 1 line
2.34.1
#10 lore raw

RE: [iommufd 2/2] vfio/ap: validate iova during dma_unmap and trigger irq disable

Quoted 44 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM

From: Matthew Rosato <[email protected]>

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

To achieve above, vfio_ap_mdev_dma_unmap() needs to validate that unmap
request matches with one or more of these stashed values before attempting unpins.

Currently, each mapped iova is stashed in its associated vfio_ap_queue; Each stashed iova represents IRQ that was enabled for a queue. Therefore, if a match is found, trigger IRQ disable for this queue to ensure that underlying firmware will no longer try to use the associated pfn after the page is unpinned. IRQ disable will also handle the associated unpin.

Cc Tony Krowiak <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Jason Herne <[email protected]>
---
 drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +4 -0
index bb7776d20792..62bfca2bbe6d 100644
@@ -1535,13 +1535,35 @@ static int vfio_ap_mdev_set_kvm(struct
 ap_matrix_mdev *matrix_mdev,
 	return 0;
 }
 
+static void unmap_iova(struct ap_matrix_mdev *matrix_mdev, u64 iova,
 u64 length)
+{
+	struct ap_queue_table *qtable = &matrix_mdev->qtable;
+	u64 iova_pfn_end = (iova + length - 1) >> PAGE_SHIFT;

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_set_kvm(struct · line 1553

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +10 -0
+	u64 iova_pfn_start = iova >> PAGE_SHIFT;
+	struct vfio_ap_queue *q;
+	int loop_cursor;
+	u64 pfn;
+
+	hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) {
+		pfn = q->saved_iova >> PAGE_SHIFT;
+		if (pfn >= iova_pfn_start && pfn <= iova_pfn_end) {
+			vfio_ap_irq_disable(q);
+			break;

does this need a WARN_ON if the length is more than one page?

Quoted 21 lines
drivers/s390/crypto/vfio_ap_ops.c +9 -1
+		}
+	}
+}
+
 static void vfio_ap_mdev_dma_unmap(struct vfio_device *vdev, u64 iova,
 				   u64 length)
 {
 	struct ap_matrix_mdev *matrix_mdev =
 		container_of(vdev, struct ap_matrix_mdev, vdev);
 
-	vfio_unpin_pages(&matrix_mdev->vdev, iova, 1);
+	mutex_lock(&matrix_dev->mdevs_lock);
+
+	unmap_iova(matrix_mdev, iova, length);
+
+	mutex_unlock(&matrix_dev->mdevs_lock);
 }

/**
--
2.34.1

#11 lore raw

Re: [iommufd 2/2] vfio/ap: validate iova during dma_unmap and trigger irq disable

Quoted 48 lines · Kevin
Quoted 48 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM

From: Matthew Rosato <[email protected]>

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

To achieve above, vfio_ap_mdev_dma_unmap() needs to validate that unmap
request matches with one or more of these stashed values before attempting unpins.

Currently, each mapped iova is stashed in its associated vfio_ap_queue; Each stashed iova represents IRQ that was enabled for a queue. Therefore, if a match is found, trigger IRQ disable for this queue to ensure that underlying firmware will no longer try to use the associated pfn after the page is unpinned. IRQ disable will also handle the associated unpin.

Cc Tony Krowiak <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Jason Herne <[email protected]>
---
 drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +8 -0
index bb7776d20792..62bfca2bbe6d 100644
@@ -1535,13 +1535,35 @@ static int vfio_ap_mdev_set_kvm(struct
 ap_matrix_mdev *matrix_mdev,
 	return 0;
 }
 
+static void unmap_iova(struct ap_matrix_mdev *matrix_mdev, u64 iova,
 u64 length)
+{
+	struct ap_queue_table *qtable = &matrix_mdev->qtable;
+	u64 iova_pfn_end = (iova + length - 1) >> PAGE_SHIFT;
+	u64 iova_pfn_start = iova >> PAGE_SHIFT;
+	struct vfio_ap_queue *q;
+	int loop_cursor;
+	u64 pfn;
Quoted 8 lines

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev, · line 1551

Quoted 6 lines
drivers/s390/crypto/vfio_ap_ops.c +6 -0
+
+	hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) {
+		pfn = q->saved_iova >> PAGE_SHIFT;
+		if (pfn >= iova_pfn_start && pfn <= iova_pfn_end) {
+			vfio_ap_irq_disable(q);
+			break;

does this need a WARN_ON if the length is more than one page?

The iova and length are the range being invalidated, the driver has no control over them and length is probably multiple pages.

But this test doesn't look right?

   if (iova > q->saved_iova && q->saved_iova < iova + length)

Since the page was pinned we can assume iova and length are already PAGE_SIZE aligned.

Jason

#12 lore raw

RE: [iommufd 2/2] vfio/ap: validate iova during dma_unmap and trigger irq disable

Quoted 24 lines

From: Jason Gunthorpe <[email protected]>
Sent: Thursday, November 24, 2022 8:59 PM

Quoted 11 lines · Kevin
Quoted 11 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM
+static void unmap_iova(struct ap_matrix_mdev *matrix_mdev, u64 iova, u64 length)

drivers/s390/crypto/vfio_ap_ops.c +7 -0
+{
+	struct ap_queue_table *qtable = &matrix_mdev->qtable;
+	u64 iova_pfn_end = (iova + length - 1) >> PAGE_SHIFT;
+	u64 iova_pfn_start = iova >> PAGE_SHIFT;
+	struct vfio_ap_queue *q;
+	int loop_cursor;
+	u64 pfn;
Quoted 8 lines

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev, · line 1551

Quoted 6 lines
drivers/s390/crypto/vfio_ap_ops.c +6 -0
+
+	hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) {
+		pfn = q->saved_iova >> PAGE_SHIFT;
+		if (pfn >= iova_pfn_start && pfn <= iova_pfn_end) {
+			vfio_ap_irq_disable(q);
+			break;

does this need a WARN_ON if the length is more than one page?

Quoted 2 lines

The iova and length are the range being invalidated, the driver has no control over them and length is probably multiple pages.

Yes. I'm misled by the 'break'. Presumably all queues covered by the unmapped range should have interrupt disabled while above only disables interrupt for the first covered queue.

#13 lore raw

Re: [iommufd 2/2] vfio/ap: validate iova during dma_unmap and trigger irq disable

Quoted 24 lines · Kevin
Quoted 24 lines

From: Jason Gunthorpe <[email protected]>
Sent: Thursday, November 24, 2022 8:59 PM

Quoted 19 lines · Kevin
Quoted 7 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM
+static void unmap_iova(struct ap_matrix_mdev *matrix_mdev, u64 iova, u64 length)

drivers/s390/crypto/vfio_ap_ops.c +3 -0
+{
+	struct ap_queue_table *qtable = &matrix_mdev->qtable;
+	u64 iova_pfn_end = (iova + length - 1) >> PAGE_SHIFT;

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev, · line 1551

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +10 -0
+	u64 iova_pfn_start = iova >> PAGE_SHIFT;
+	struct vfio_ap_queue *q;
+	int loop_cursor;
+	u64 pfn;
+
+	hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) {
+		pfn = q->saved_iova >> PAGE_SHIFT;
+		if (pfn >= iova_pfn_start && pfn <= iova_pfn_end) {
+			vfio_ap_irq_disable(q);
+			break;

does this need a WARN_ON if the length is more than one page?

Quoted 6 lines
Quoted 2 lines

The iova and length are the range being invalidated, the driver has no control over them and length is probably multiple pages.

Yes. I'm misled by the 'break'. Presumably all queues covered by the unmapped range should have interrupt disabled while above only disables interrupt for the first covered queue.

Oops, yeah the break shouldn't be there; we want to disable any queue in the table that falls within the iova range.

#14 lore raw

Re: [iommufd 2/2] vfio/ap: validate iova during dma_unmap and trigger irq disable

In reply to Jason Gunthorpe
Quoted 58 lines · Jason Gunthorpe
Quoted 48 lines · Kevin
Quoted 48 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM

From: Matthew Rosato <[email protected]>

vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. So dma_unmap() callback should tolerate the unmaps that come before the emulated device is opened.

To achieve above, vfio_ap_mdev_dma_unmap() needs to validate that unmap
request matches with one or more of these stashed values before attempting unpins.

Currently, each mapped iova is stashed in its associated vfio_ap_queue; Each stashed iova represents IRQ that was enabled for a queue. Therefore, if a match is found, trigger IRQ disable for this queue to ensure that underlying firmware will no longer try to use the associated pfn after the page is unpinned. IRQ disable will also handle the associated unpin.

Cc Tony Krowiak <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Jason Herne <[email protected]>
---
 drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +8 -0
index bb7776d20792..62bfca2bbe6d 100644
@@ -1535,13 +1535,35 @@ static int vfio_ap_mdev_set_kvm(struct
 ap_matrix_mdev *matrix_mdev,
 	return 0;
 }
 
+static void unmap_iova(struct ap_matrix_mdev *matrix_mdev, u64 iova,
 u64 length)
+{
+	struct ap_queue_table *qtable = &matrix_mdev->qtable;
+	u64 iova_pfn_end = (iova + length - 1) >> PAGE_SHIFT;
+	u64 iova_pfn_start = iova >> PAGE_SHIFT;
+	struct vfio_ap_queue *q;
+	int loop_cursor;
+	u64 pfn;
Quoted 8 lines

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev, · line 1551

Quoted 6 lines
drivers/s390/crypto/vfio_ap_ops.c +6 -0
+
+	hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) {
+		pfn = q->saved_iova >> PAGE_SHIFT;
+		if (pfn >= iova_pfn_start && pfn <= iova_pfn_end) {
+			vfio_ap_irq_disable(q);
+			break;

does this need a WARN_ON if the length is more than one page?

Quoted 8 lines

The iova and length are the range being invalidated, the driver has no control over them and length is probably multiple pages.

But this test doesn't look right?

   if (iova > q->saved_iova && q->saved_iova < iova + length)> 

Since the page was pinned we can assume iova and length are already PAGE_SIZE aligned.

Yeah, I think that would be fine with a minor tweak to pick up q->saved_iova at the very start of the iova range:

   if (iova >= q->saved_iova && q->saved_iova < iova + length)
#15 lore raw

Re: [iommufd 2/2] vfio/ap: validate iova during dma_unmap and trigger irq disable

Quoted 6 lines · Matthew Rosato
Quoted 5 lines · Jason Gunthorpe

The iova and length are the range being invalidated, the driver has no control over them and length is probably multiple pages.

But this test doesn't look right?

Quoted 8 lines
Quoted 3 lines
   if (iova > q->saved_iova && q->saved_iova < iova + length)> 

Since the page was pinned we can assume iova and length are already PAGE_SIZE aligned.

Yeah, I think that would be fine with a minor tweak to pick up q->saved_iova at the very start of the iova range:

   if (iova >= q->saved_iova && q->saved_iova < iova + length)

Yi can you update and repost this please?

I don't know if we will get a rc8, but we must be prepared with a final branch by Friday in case not.

Thanks,
Jason

#16 lore raw

RE: [iommufd 2/2] vfio/ap: validate iova during dma_unmap and trigger irq disable

Quoted 17 lines

From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, November 29, 2022 4:28 AM

Quoted 13 lines · Matthew Rosato
Quoted 8 lines · Jason Gunthorpe

The iova and length are the range being invalidated, the driver has no control over them and length is probably multiple pages.

But this test doesn't look right?

   if (iova > q->saved_iova && q->saved_iova < iova + length)>

Since the page was pinned we can assume iova and length are already PAGE_SIZE aligned.

Yeah, I think that would be fine with a minor tweak to pick up q- saved_iova at the very start of the iova range:

Quoted 8 lines
Quoted 2 lines
   if (iova >= q->saved_iova && q->saved_iova < iova + length)

Yi can you update and repost this please?

I don't know if we will get a rc8, but we must be prepared with a final branch by Friday in case not.

Just in case, Linus already said:
--
As a result, I'm now pretty sure that this is going to be one of those "we'll have an extra week and I'll make an rc8" releases. --

author
#17 lore raw

Re: [iommufd 2/2] vfio/ap: validate iova during dma_unmap and trigger irq disable

In reply to Jason Gunthorpe
Quoted 13 lines · Jason Gunthorpe
Quoted 12 lines · Matthew Rosato
Quoted 8 lines · Jason Gunthorpe

The iova and length are the range being invalidated, the driver has no control over them and length is probably multiple pages.

But this test doesn't look right?

    if (iova > q->saved_iova && q->saved_iova < iova + length)>

Since the page was pinned we can assume iova and length are already PAGE_SIZE aligned.

Yeah, I think that would be fine with a minor tweak to pick up q->saved_iova at the very start of the iova range:

Quoted 7 lines
Quoted 2 lines
    if (iova >= q->saved_iova && q->saved_iova < iova + length)

Yi can you update and repost this please?

I don't know if we will get a rc8, but we must be prepared with a final branch by Friday in case not.

done.

https://lore.kernel.org/kvm/[email protected]/T/#m261c22a7d9c9b2fe4258c5156f53ea6da97a27c9

Signature · 2 lines
Regards,
Yi Liu
#18 lore raw
In reply to Yi Liu
Quoted 14 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM

Jason's "Connect VFIO to IOMMUFD" introduces vfio iommufd compat mode. Under
this mode, vfio_iommufd_bind() creates an access which has an unmap callback,
which can be called immediately. This means mdev drivers may receive unmap
requests before the mdev is opened. For now, most dma_unmap() callbacks are
tolerant with such unmap requests, except for gvt-g and vfio-ap. This series tries to enhance the two drivers.

there are only three drivers (gvt, vfio-ap and vfio-ccw) providing dma_unmap().

I'd not call the situation where only one driver is OK as "most dma_unmap() callbacks are tolerant".

author
#19 lore raw
Quoted 15 lines · Kevin
Quoted 14 lines

From: Liu, Yi L <[email protected]>
Sent: Wednesday, November 23, 2022 9:49 PM

Jason's "Connect VFIO to IOMMUFD" introduces vfio iommufd compat mode. Under
this mode, vfio_iommufd_bind() creates an access which has an unmap callback,
which can be called immediately. This means mdev drivers may receive unmap
requests before the mdev is opened. For now, most dma_unmap() callbacks are
tolerant with such unmap requests, except for gvt-g and vfio-ap. This series tries to enhance the two drivers.

Quoted 5 lines

there are only three drivers (gvt, vfio-ap and vfio-ccw) providing dma_unmap().

I'd not call the situation where only one driver is OK as "most dma_unmap() callbacks are tolerant".

oh, yes. will refine it.

Signature · 2 lines
Regards,
Yi Liu
#20 lore raw
In reply to Yi Liu
Quoted 22 lines · Yi Liu

Jason's "Connect VFIO to IOMMUFD" introduces vfio iommufd compat mode. Under this mode, vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. This means mdev drivers may receive unmap requests before the mdev is opened. For now, most dma_unmap() callbacks are tolerant with such unmap requests, except for gvt-g and vfio-ap. This series tries to enhance the two drivers.

This series is based on Jason's below branch.

https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/log/?h=for-next
(commit: 57f62422b6f0477afaddd2fc77a4bb9b94275f42)

Cc Tony Krowiak <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Jason Herne <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Kevin Tian <[email protected]>

Regards,
Yi Liu

Quoted 6 lines

Matthew Rosato (1):
vfio/ap: validate iova during dma_unmap and trigger irq disable

Yi Liu (1):
i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into
init

btw, for gvt change, pls at least add [email protected] in cc.

thanks

Quoted 10 lines
 drivers/gpu/drm/i915/gvt/gvt.h    |  2 ++
 drivers/gpu/drm/i915/gvt/kvmgt.c  |  7 ++-----
 drivers/gpu/drm/i915/gvt/vgpu.c   |  2 ++
 drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++++++++++++-
 4 files changed, 29 insertions(+), 6 deletions(-)
Signature · 1 line
2.34.1
author
#21 lore raw
Quoted 23 lines · Zhenyu Wang
Quoted 22 lines · Yi Liu

Jason's "Connect VFIO to IOMMUFD" introduces vfio iommufd compat mode. Under this mode, vfio_iommufd_bind() creates an access which has an unmap callback, which can be called immediately. This means mdev drivers may receive unmap requests before the mdev is opened. For now, most dma_unmap() callbacks are tolerant with such unmap requests, except for gvt-g and vfio-ap. This series tries to enhance the two drivers.

This series is based on Jason's below branch.

https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/log/?h=for-next
(commit: 57f62422b6f0477afaddd2fc77a4bb9b94275f42)

Cc Tony Krowiak <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Jason Herne <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Kevin Tian <[email protected]>

Regards,
Yi Liu

Quoted 8 lines
Quoted 6 lines

Matthew Rosato (1):
vfio/ap: validate iova during dma_unmap and trigger irq disable

Yi Liu (1):
i915/gvt: Move kvmgt_protect_table_init() and gvt_cache_init() into
init

btw, for gvt change, pls at least add [email protected] in cc.

sure. :-)

Signature · 2 lines
Regards,
Yi Liu