Skip to contentSkip to conversation

cover-letter: Allow MMIO regions to be exported through dmabuf

PATCH In review Wei Lin Guay 6 participants 18 messages Dec 16, 2024 – Mar 4, 2025

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover cover-letter: Allow MMIO regions to be exported through dmabuf

From: Wei Lin Guay <[email protected]>

This is another attempt to revive the patches posted by Jason Gunthorpe and Vivek Kasireddy, at
https://patchwork.kernel.org/project/linux-media/cover/[email protected]/
https://lwn.net/Articles/970751/

In addition to the initial proposal by Jason, another promising application is exposing memory from an AI accelerator (bound to VFIO) to an RDMA device. This would allow the RDMA device to directly access the accelerator's memory, thereby facilitating direct data transactions between the RDMA device and the accelerator.

Below is from the text/motivation from the orginal cover letter.

dma-buf has become a way to safely acquire a handle to non-struct page memory that can still have lifetime controlled by the exporter. Notably RDMA can now import dma-buf FDs and build them into MRs which allows for PCI P2P operations. Extend this to allow vfio-pci to export MMIO memory from PCI device BARs.

This series supports a use case for SPDK where a NVMe device will be owned by SPDK through VFIO but interacting with a RDMA device. The RDMA device may directly access the NVMe CMB or directly manipulate the NVMe device's doorbell using PCI P2P.

However, as a general mechanism, it can support many other scenarios with VFIO. I imagine this dmabuf approach to be usable by iommufd as well for generic and safe P2P mappings.

This series goes after the "Break up ioctl dispatch functions to one function per ioctl" series.

v2:
- Name the new file dma_buf.c
- Restore orig_nents before freeing
- Fix reversed logic around priv->revoked
- Set priv->index
- Rebased on v2 "Break up ioctl dispatch functions"
v1: https://lore.kernel.org/r/[email protected]

Cc Oded Gabbay <[email protected]>
Cc Christian König <[email protected]>
Cc Daniel Vetter <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Maor Gottlieb <[email protected]>

Jason Gunthorpe (3):
vfio: Add vfio_device_get()
dma-buf: Add dma_buf_try_get()
vfio/pci: Allow MMIO regions to be exported through dma-buf

Wei Lin Guay (1):
vfio/pci: Allow export dmabuf without move_notify from importer

 drivers/vfio/pci/Makefile          |   1 +
 drivers/vfio/pci/dma_buf.c         | 291 +++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_config.c |   8 +-
 drivers/vfio/pci/vfio_pci_core.c   |  44 ++++-
 drivers/vfio/pci/vfio_pci_priv.h   |  30 +++
 drivers/vfio/vfio_main.c           |   1 +
 include/linux/dma-buf.h            |  13 ++
 include/linux/vfio.h               |   6 +
 include/linux/vfio_pci_core.h      |   1 +
 include/uapi/linux/vfio.h          |  18 ++
 10 files changed, 405 insertions(+), 8 deletions(-)
 create mode 100644 drivers/vfio/pci/dma_buf.c

--
2.43.5

#1 lore raw

PATCH 1/4 vfio: Add vfio_device_get()

From: Jason Gunthorpe <[email protected]>

Summary:
To increment a reference the caller already holds. Export vfio_device_put() to pair with it.

---
 drivers/vfio/vfio_main.c | 1 +
 include/linux/vfio.h     | 6 ++++++
 2 files changed, 7 insertions(+)
drivers/vfio/vfio_main.c +1 -0
index a5a62d9d963f..7e318e15abd5 100644
@@ -171,6 +171,7 @@ void vfio_device_put_registration(struct vfio_device *device)
 	if (refcount_dec_and_test(&device->refcount))
 		complete(&device->comp);
 }
+EXPORT_SYMBOL_GPL(vfio_device_put_registration);
 
 bool vfio_device_try_get_registration(struct vfio_device *device)
 {
include/linux/vfio.h +6 -0
index 000a6cab2d31..d7c790be4bbc 100644
@@ -279,6 +279,12 @@ static inline void vfio_put_device(struct vfio_device *device)
 int vfio_register_group_dev(struct vfio_device *device);
 int vfio_register_emulated_iommu_dev(struct vfio_device *device);
 void vfio_unregister_group_dev(struct vfio_device *device);
+void vfio_device_put_registration(struct vfio_device *device);
+
+static inline void vfio_device_get(struct vfio_device *device)
+{
+	refcount_inc(&device->refcount);
+}
 
 int vfio_assign_device_set(struct vfio_device *device, void *set_id);
 unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set);

--
2.43.5

#2 lore raw

PATCH 2/4 dma-buf: Add dma_buf_try_get()

In reply to Wei Lin Guay

From: Jason Gunthorpe <[email protected]>

Summary:
Used to increment the refcount of the dma buf's struct file, only if the refcount is not zero. Useful to allow the struct file's lifetime to control the lifetime of the dmabuf while still letting the driver to keep track of created dmabufs.

---
 include/linux/dma-buf.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)
include/linux/dma-buf.h +13 -0
index 36216d28d8bd..9854578afecd 100644
@@ -614,6 +614,19 @@ int dma_buf_fd(struct dma_buf *dmabuf, int flags);
 struct dma_buf *dma_buf_get(int fd);
 void dma_buf_put(struct dma_buf *dmabuf);
 
+/**
+ * dma_buf_try_get - try to get a reference on a dmabuf
+ * @dmabuf - the dmabuf to get
+ *
+ * Returns true if a reference was successfully obtained. The caller must
+ * interlock with the dmabuf's release function in some way, such as RCU, to
+ * ensure that this is not called on freed memory.
+ */
+static inline bool dma_buf_try_get(struct dma_buf *dmabuf)
+{
+	return get_file_rcu(&dmabuf->file);
+}
+
 struct sg_table *dma_buf_map_attachment(struct dma_buf_attachment *,
 					enum dma_data_direction);
 void dma_buf_unmap_attachment(struct dma_buf_attachment *, struct sg_table *,

--
2.43.5

#3 lore raw

PATCH 3/4 vfio/pci: Allow MMIO regions to be exported through dma-buf

In reply to Wei Lin Guay

From: Jason Gunthorpe <[email protected]>

Summary:
dma-buf has become a way to safely acquire a handle to non-struct page memory that can still have lifetime controlled by the exporter. Notably RDMA can now import dma-buf FDs and build them into MRs which allows for PCI P2P operations. Extend this to allow vfio-pci to export MMIO memory from PCI device BARs.

The patch design loosely follows the pattern in commit db1a8dd916aa ("habanalabs: add support for dma-buf exporter") except this does not support pinning.

Instead, this implements what, in the past, we've called a revocable attachment using move. In normal situations the attachment is pinned, as a BAR does not change physical address. However when the VFIO device is closed, or a PCI reset is issued, access to the MMIO memory is revoked.

Revoked means that move occurs, but an attempt to immediately re-map the memory will fail. In the reset case a future move will be triggered when MMIO access returns. As both close and reset are under userspace control it is expected that userspace will suspend use of the dma-buf before doing these operations, the revoke is purely for kernel self-defense against a hostile userspace.

---
 drivers/vfio/pci/Makefile          |   1 +
 drivers/vfio/pci/dma_buf.c         | 269 +++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_config.c |   8 +-
 drivers/vfio/pci/vfio_pci_core.c   |  28 ++-
 drivers/vfio/pci/vfio_pci_priv.h   |  23 +++
 include/linux/vfio_pci_core.h      |   1 +
 include/uapi/linux/vfio.h          |  18 ++
 7 files changed, 340 insertions(+), 8 deletions(-)
 create mode 100644 drivers/vfio/pci/dma_buf.c
drivers/vfio/pci/Makefile +1 -0
index cf00c0a7e55c..0cfdc9ede82f 100644
@@ -2,6 +2,7 @@
 
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
+vfio-pci-core-$(CONFIG_DMA_SHARED_BUFFER) += dma_buf.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
 
 vfio-pci-y := vfio_pci.o
drivers/vfio/pci/dma_buf.c new file +269 -0

269 lines not on this page · raw message

drivers/vfio/pci/vfio_pci_config.c +6 -2
index 97422aafaa7b..c605c5cb0078 100644
@@ -585,10 +585,12 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
 		virt_mem = !!(le16_to_cpu(*virt_cmd) & PCI_COMMAND_MEMORY);
 		new_mem = !!(new_cmd & PCI_COMMAND_MEMORY);
 
-		if (!new_mem)
+		if (!new_mem) {
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
-		else
+			vfio_pci_dma_buf_move(vdev, true);
+		} else {
 			down_write(&vdev->memory_lock);
+		}
 
 		/*
 		 * If the user is writing mem/io enable (new_mem/io) and we
@@ -623,6 +625,8 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
 		*virt_cmd &= cpu_to_le16(~mask);
 		*virt_cmd |= cpu_to_le16(new_cmd & mask);
 
+		if (__vfio_pci_memory_enabled(vdev))
+			vfio_pci_dma_buf_move(vdev, false);
 		up_write(&vdev->memory_lock);
 	}
 
drivers/vfio/pci/vfio_pci_core.c +22 -6
index ba0ce0075b2f..bb97b4d94eb7 100644
@@ -700,6 +700,8 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)
 #endif
 	vfio_pci_core_disable(vdev);
 
+	vfio_pci_dma_buf_cleanup(vdev);
+
 	mutex_lock(&vdev->igate);
 	if (vdev->err_trigger) {
 		eventfd_ctx_put(vdev->err_trigger);
@@ -1244,7 +1246,10 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 	 */
 	vfio_pci_set_power_state(vdev, PCI_D0);
 
+	vfio_pci_dma_buf_move(vdev, true);
 	ret = pci_try_reset_function(vdev->pdev);
+	if (__vfio_pci_memory_enabled(vdev))
+		vfio_pci_dma_buf_move(vdev, false);
 	up_write(&vdev->memory_lock);
 
 	return ret;
@@ -1490,11 +1495,10 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_ioctl);
 
-static int vfio_pci_core_feature_token(struct vfio_device *device, u32 flags,
-				       uuid_t __user *arg, size_t argsz)
+static int vfio_pci_core_feature_token(struct vfio_pci_core_device *vdev,
+				       u32 flags, uuid_t __user *arg,
+				       size_t argsz)
 {
-	struct vfio_pci_core_device *vdev =
-		container_of(device, struct vfio_pci_core_device, vdev);
 	uuid_t uuid;
 	int ret;
 
@@ -1521,6 +1525,9 @@ static int vfio_pci_core_feature_token(struct vfio_device *device, u32 flags,
 int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
 				void __user *arg, size_t argsz)
 {
+	struct vfio_pci_core_device *vdev =
+		container_of(device, struct vfio_pci_core_device, vdev);
+
 	switch (flags & VFIO_DEVICE_FEATURE_MASK) {
 	case VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY:
 		return vfio_pci_core_pm_entry(device, flags, arg, argsz);
@@ -1530,7 +1537,9 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
 	case VFIO_DEVICE_FEATURE_LOW_POWER_EXIT:
 		return vfio_pci_core_pm_exit(device, flags, arg, argsz);
 	case VFIO_DEVICE_FEATURE_PCI_VF_TOKEN:
-		return vfio_pci_core_feature_token(device, flags, arg, argsz);
+		return vfio_pci_core_feature_token(vdev, flags, arg, argsz);
+	case VFIO_DEVICE_FEATURE_DMA_BUF:
+		return vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz);
 	default:
 		return -ENOTTY;
 	}
@@ -2083,6 +2092,7 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 	INIT_LIST_HEAD(&vdev->sriov_pfs_item);
 	init_rwsem(&vdev->memory_lock);
 	xa_init(&vdev->ctx);
+	INIT_LIST_HEAD(&vdev->dmabufs);
 
 	return 0;
 }
@@ -2463,11 +2473,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 	 * cause the PCI config space reset without restoring the original
 	 * state (saved locally in 'vdev->pm_save').
 	 */
-	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list)
+	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) {
+		vfio_pci_dma_buf_move(vdev, true);
 		vfio_pci_set_power_state(vdev, PCI_D0);
+	}
 
 	ret = pci_reset_bus(pdev);
 
+	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list)
+		if (__vfio_pci_memory_enabled(vdev))
+			vfio_pci_dma_buf_move(vdev, false);
+
 	vdev = list_last_entry(&dev_set->device_list,
 			       struct vfio_pci_core_device, vdev.dev_set_list);
 
drivers/vfio/pci/vfio_pci_priv.h +23 -0
index 5e4fa69aee16..09d3c300918c 100644
@@ -101,4 +101,27 @@ static inline bool vfio_pci_is_vga(struct pci_dev *pdev)
 	return (pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA;
 }
 
+#ifdef CONFIG_DMA_SHARED_BUFFER
+int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
+				  struct vfio_device_feature_dma_buf __user *arg,
+				  size_t argsz);
+void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev);
+void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked);
+#else
+static int
+vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
+			      struct vfio_device_feature_dma_buf __user *arg,
+			      size_t argsz)
+{
+	return -ENOTTY;
+}
+static inline void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev)
+{
+}
+static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev,
+					 bool revoked)
+{
+}
+#endif
+
 #endif
include/linux/vfio_pci_core.h +1 -0
index fbb472dd99b3..da5d8955ae56 100644
@@ -94,6 +94,7 @@ struct vfio_pci_core_device {
 	struct vfio_pci_core_device	*sriov_pf_core_dev;
 	struct notifier_block	nb;
 	struct rw_semaphore	memory_lock;
+	struct list_head	dmabufs;
 };
 
 /* Will be exported for vfio pci drivers usage */
include/uapi/linux/vfio.h +18 -0
index 2b68e6cdf190..8812b4750cc5 100644
@@ -1458,6 +1458,24 @@ struct vfio_device_feature_bus_master {
 };
 #define VFIO_DEVICE_FEATURE_BUS_MASTER 10
 
+/**
+ * Upon VFIO_DEVICE_FEATURE_GET create a dma_buf fd for the
+ * region selected.
+ *
+ * open_flags are the typical flags passed to open(2), eg O_RDWR, O_CLOEXEC,
+ * etc. offset/length specify a slice of the region to create the dmabuf from.
+ * If both are 0 then the whole region is used.
+ *
+ * Return: The fd number on success, -1 and errno is set on failure.
+ */
+struct vfio_device_feature_dma_buf {
+	__u32 region_index;
+	__u32 open_flags;
+	__u32 offset;
+	__u64 length;
+};
+#define VFIO_DEVICE_FEATURE_DMA_BUF 11
+
 /* -------- API for Type1 VFIO IOMMU -------- */
 
 /**

--
2.43.5

#4 lore raw

PATCH 4/4 vfio/pci: Allow export dmabuf without move_notify from importer

In reply to Wei Lin Guay

From: Wei Lin Guay <[email protected]>

Summary:
Support vfio to export dmabuf to importer such as RDMA NIC that does not support move_notify callback, since not all RDMA driver support on-demand-paging (ODP).

There are some use-cases such as bind accelerator that always pinned the device memory via vfio and export it to RDMA NIC such as EFA, BNXT_RE or IRDMA that does not support ODP.

---
 drivers/vfio/pci/dma_buf.c       | 32 +++++++++++++++++++++++++++-----
 drivers/vfio/pci/vfio_pci_core.c | 16 ++++++++++++++++
 drivers/vfio/pci/vfio_pci_priv.h |  7 +++++++
 3 files changed, 50 insertions(+), 5 deletions(-)
drivers/vfio/pci/dma_buf.c +27 -5
index fd772b520cd7..8017f48296cb 100644
@@ -17,6 +17,7 @@ struct vfio_pci_dma_buf {
 	unsigned int orig_nents;
 	size_t offset;
 	bool revoked;
+	bool pinned;
 };
 
 static int vfio_pci_dma_buf_attach(struct dma_buf *dmabuf,
@@ -32,17 +33,38 @@ static int vfio_pci_dma_buf_attach(struct dma_buf *dmabuf,
 	return 0;
 }
 
+bool vfio_pci_dma_buf_pinned(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_dma_buf *priv;
+	struct vfio_pci_dma_buf *tmp;
+	bool pinned = false;
+
+	down_write(&vdev->memory_lock);
+	list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm) {
+		if (!dma_buf_try_get(priv->dmabuf))
+			continue;
+		if (priv->pinned) {
+			pinned = true;
+			break;
+		}
+	}
+	up_write(&vdev->memory_lock);
+	return pinned;
+}
+
 static void vfio_pci_dma_buf_unpin(struct dma_buf_attachment *attachment)
 {
+	struct vfio_pci_dma_buf *priv = attachment->dmabuf->priv;
+
+	priv->pinned = false;
 }
 
 static int vfio_pci_dma_buf_pin(struct dma_buf_attachment *attachment)
 {
-	/*
-	 * Uses the dynamic interface but must always allow for
-	 * dma_buf_move_notify() to do revoke
-	 */
-	return -EINVAL;
+	struct vfio_pci_dma_buf *priv = attachment->dmabuf->priv;
+
+	priv->pinned = true;
+	return 0;
 }
 
 static struct sg_table *
drivers/vfio/pci/vfio_pci_core.c +16 -0
index bb97b4d94eb7..db28fa2cc9a8 100644
@@ -1246,6 +1246,13 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 	 */
 	vfio_pci_set_power_state(vdev, PCI_D0);
 
+	/*
+	 * prevent reset if dma_buf is pinned to avoid stale pinned
+	 * expose to the dmabuf exporter.
+	 */
+	if (vfio_pci_dma_buf_pinned(vdev))
+		return -EINVAL;
+
 	vfio_pci_dma_buf_move(vdev, true);
 	ret = pci_try_reset_function(vdev->pdev);
 	if (__vfio_pci_memory_enabled(vdev))
@@ -2444,6 +2451,15 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 			break;
 		}
 
+		/*
+		 * prevent reset if dma_buf is pinned to avoid stale pinned
+		 * expose to the dmabuf exporter.
+		 */
+		if (vfio_pci_dma_buf_pinned(vdev)) {
+			ret = -EINVAL;
+			break;
+		}
+
 		/*
 		 * Take the memory write lock for each device and zap BAR
 		 * mappings to prevent the user accessing the device while in
drivers/vfio/pci/vfio_pci_priv.h +7 -0
index 09d3c300918c..43c40dc4751c 100644
@@ -107,6 +107,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 				  size_t argsz);
 void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev);
 void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked);
+bool vfio_pci_dma_buf_pinned(struct vfio_pci_core_device *vdev);
 #else
 static int
 vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
@@ -115,6 +116,12 @@ vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 {
 	return -ENOTTY;
 }
+
+static inline bool vfio_pci_dma_buf_pinned(struct vfio_pci_core_device *vdev)
+{
+	return false;
+}
+
 static inline void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev)
 {
 }

--
2.43.5

#5 lore raw
In reply to Wei Lin Guay

Hi Wei Lin,

Quoted 10 lines

Subject: [PATCH 0/4] cover-letter: Allow MMIO regions to be exported through dmabuf

From: Wei Lin Guay <[email protected]>

This is another attempt to revive the patches posted by Jason Gunthorpe and Vivek Kasireddy, at
https://patchwork.kernel.org/project/linux-media/cover/0-v2-
[email protected]/
https://lwn.net/Articles/970751/

v2: https://lore.kernel.org/dri-devel/[email protected]/
addresses review comments from Alex and Jason and also includes the ability to create the dmabuf from multiple ranges. This is really needed to future-proof the feature.

Also, my understanding is that this patchset cannot proceed until Leon's series is merged:
https://lore.kernel.org/kvm/[email protected]/

Thanks,
Vivek

Quoted 68 lines

In addition to the initial proposal by Jason, another promising application is exposing memory from an AI accelerator (bound to VFIO) to an RDMA device. This would allow the RDMA device to directly access the accelerator's memory, thereby facilitating direct data transactions between the RDMA device and the accelerator.

Below is from the text/motivation from the orginal cover letter.

dma-buf has become a way to safely acquire a handle to non-struct page memory that can still have lifetime controlled by the exporter. Notably RDMA can now import dma-buf FDs and build them into MRs which allows for
PCI P2P operations. Extend this to allow vfio-pci to export MMIO memory from PCI device BARs.

This series supports a use case for SPDK where a NVMe device will be owned by SPDK through VFIO but interacting with a RDMA device. The RDMA device may directly access the NVMe CMB or directly manipulate the NVMe device's doorbell using PCI P2P.

However, as a general mechanism, it can support many other scenarios with VFIO. I imagine this dmabuf approach to be usable by iommufd as well for generic and safe P2P mappings.

This series goes after the "Break up ioctl dispatch functions to one function per ioctl" series.

v2:
- Name the new file dma_buf.c
- Restore orig_nents before freeing
- Fix reversed logic around priv->revoked
- Set priv->index
- Rebased on v2 "Break up ioctl dispatch functions"
v1: https://lore.kernel.org/r/0-v1-9e6e1739ed95+5fa-
[email protected]

Cc Oded Gabbay <[email protected]>
Cc Christian König <[email protected]>
Cc Daniel Vetter <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Maor Gottlieb <[email protected]>

Jason Gunthorpe (3):
vfio: Add vfio_device_get()
dma-buf: Add dma_buf_try_get()
vfio/pci: Allow MMIO regions to be exported through dma-buf

Wei Lin Guay (1):
vfio/pci: Allow export dmabuf without move_notify from importer

 drivers/vfio/pci/Makefile          |   1 +
 drivers/vfio/pci/dma_buf.c         | 291 +++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_config.c |   8 +-
 drivers/vfio/pci/vfio_pci_core.c   |  44 ++++-
 drivers/vfio/pci/vfio_pci_priv.h   |  30 +++
 drivers/vfio/vfio_main.c           |   1 +
 include/linux/dma-buf.h            |  13 ++
 include/linux/vfio.h               |   6 +
 include/linux/vfio_pci_core.h      |   1 +
 include/uapi/linux/vfio.h          |  18 ++
 10 files changed, 405 insertions(+), 8 deletions(-)
 create mode 100644 drivers/vfio/pci/dma_buf.c

--
2.43.5

#6 lore raw

Hi Vivek,

Quoted 15 lines · Vivek
Quoted 2 lines

Hi Wei Lin,

Quoted 10 lines

Subject: [PATCH 0/4] cover-letter: Allow MMIO regions to be exported through dmabuf

From: Wei Lin Guay <[email protected]>

This is another attempt to revive the patches posted by Jason Gunthorpe and Vivek Kasireddy, at
https://patchwork.kernel.org/project/linux-media/cover/0-v2-
[email protected]/
https://urldefense.com/v3/__https://lwn.net/Articles/970751/__;!!Bt8RZUm9aw!5uPrlSI9DhXVIeRMntADbkdWcu4YYhAzGN0OwyQ2NHxrN7bYE9f1eQBXUD8xHHPxiJorSkYhNjIRwdG4PsD2$

Quoted 4 lines

v2: https://lore.kernel.org/dri-devel/[email protected]/
addresses review comments from Alex and Jason and also includes the ability to create the dmabuf from multiple ranges. This is really needed to future-proof the feature.

The good thing is that your patch series addressed Christian’s concerns of adding dma_buf_try_get().

However, several questions regarding your v2 patch:
- The dma-buf still support redundant mmap handler? (One of review comments from v1?)
- Rather than handle different regions within a single dma-buf, would vfio-user open multiple distinct file descriptors work? For our specific use case, we don't require multiple regions and prefer Jason’s original patch.

Quoted 3 lines

Also, my understanding is that this patchset cannot proceed until Leon's series is merged:
https://lore.kernel.org/kvm/[email protected]/

Thanks for the pointer.
I will rebase my local patch series on top of that.

Thanks,
Wei Lin

Quoted 73 lines

Thanks,
Vivek

Quoted 68 lines

In addition to the initial proposal by Jason, another promising application is exposing memory from an AI accelerator (bound to VFIO) to an RDMA device. This would allow the RDMA device to directly access the accelerator's memory, thereby facilitating direct data transactions between the RDMA device and the accelerator.

Below is from the text/motivation from the orginal cover letter.

dma-buf has become a way to safely acquire a handle to non-struct page memory that can still have lifetime controlled by the exporter. Notably RDMA can now import dma-buf FDs and build them into MRs which allows for
PCI P2P operations. Extend this to allow vfio-pci to export MMIO memory from PCI device BARs.

This series supports a use case for SPDK where a NVMe device will be owned by SPDK through VFIO but interacting with a RDMA device. The RDMA device may directly access the NVMe CMB or directly manipulate the NVMe device's doorbell using PCI P2P.

However, as a general mechanism, it can support many other scenarios with VFIO. I imagine this dmabuf approach to be usable by iommufd as well for generic and safe P2P mappings.

This series goes after the "Break up ioctl dispatch functions to one function per ioctl" series.

v2:
- Name the new file dma_buf.c
- Restore orig_nents before freeing
- Fix reversed logic around priv->revoked
- Set priv->index
- Rebased on v2 "Break up ioctl dispatch functions"
v1: https://lore.kernel.org/r/0-v1-9e6e1739ed95+5fa-
[email protected]

Cc Oded Gabbay <[email protected]>
Cc Christian König <[email protected]>
Cc Daniel Vetter <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Maor Gottlieb <[email protected]>

Jason Gunthorpe (3):
vfio: Add vfio_device_get()
dma-buf: Add dma_buf_try_get()
vfio/pci: Allow MMIO regions to be exported through dma-buf

Wei Lin Guay (1):
vfio/pci: Allow export dmabuf without move_notify from importer

drivers/vfio/pci/Makefile          |   1 +
drivers/vfio/pci/dma_buf.c         | 291 +++++++++++++++++++++++++++++
drivers/vfio/pci/vfio_pci_config.c |   8 +-
drivers/vfio/pci/vfio_pci_core.c   |  44 ++++-
drivers/vfio/pci/vfio_pci_priv.h   |  30 +++
drivers/vfio/vfio_main.c           |   1 +
include/linux/dma-buf.h            |  13 ++
include/linux/vfio.h               |   6 +
include/linux/vfio_pci_core.h      |   1 +
include/uapi/linux/vfio.h          |  18 ++
10 files changed, 405 insertions(+), 8 deletions(-)

create mode 100644 drivers/vfio/pci/dma_buf.c

--
2.43.5

#7 lore raw

Hi Wei Lin,

Quoted 15 lines

Subject: Re: [PATCH 0/4] cover-letter: Allow MMIO regions to be exported through dmabuf

Quoted 8 lines
Quoted 8 lines

From: Wei Lin Guay <[email protected]>

This is another attempt to revive the patches posted by Jason Gunthorpe and Vivek Kasireddy, at
https://patchwork.kernel.org/project/linux-media/cover/0-v2-
[email protected]/

https://urldefense.com/v3/__https://lwn.net/Articles/970751/__;!!Bt8RZUm
9aw!5uPrlSI9DhXVIeRMntADbkdWcu4YYhAzGN0OwyQ2NHxrN7bYE9f1eQBX UD8xHHPxiJorSkYhNjIRwdG4PsD2$

Quoted 1 line

v2: https://lore.kernel.org/dri-devel/20240624065552.1572580-1-

[email protected]/

Quoted 8 lines
Quoted 1 line

addresses review comments from Alex and Jason and also includes the

ability

Quoted 1 line

to create the dmabuf from multiple ranges. This is really needed to future-

proof

Quoted 1 line

the feature.

The good thing is that your patch series addressed Christian’s concerns of adding dma_buf_try_get().

No, it did not address his concern. In v2, instead of adding a new function, we are now calling get_file_rcu() directly, which was suggested by Christian in the old original review.

Quoted 4 lines

However, several questions regarding your v2 patch:
- The dma-buf still support redundant mmap handler? (One of review comments from v1?)

Yeah, the mmap handler is really needed as a debugging tool given that the importer would not be able to provide access to the dmabuf's underlying memory via the CPU in any other way.

Quoted 4 lines

- Rather than handle different regions within a single dma-buf, would vfio- user open multiple distinct file descriptors work?
For our specific use case, we don't require multiple regions and prefer Jason’s original patch.

Restricting the dmabuf to a single region (or having to create multiple dmabufs to represent multiple regions/ranges associated with a single scattered buffer) would not be feasible or ideal in all cases. For instance, in my use-case, I am sharing a large framebuffer (FB) located in GPU's VRAM. And, allocating a large FB contiguously (nr_ranges = 1) in VRAM is not possible when there is memory pressure.

Furthermore, since we are adding a new UAPI with this patch/feature, we cannot go back and tweak it (to add support for nr_ranges > 1) should there be a need in the future, but you can always use nr_ranges = 1 anytime. That is why it makes sense to be flexible in terms of the number of ranges/regions.

Quoted 8 lines
Quoted 3 lines

Also, my understanding is that this patchset cannot proceed until Leon's

series is merged:

Quoted 1 line

https://lore.kernel.org/kvm/[email protected]/

Thanks for the pointer.
I will rebase my local patch series on top of that.

AFAIK, Leon's work includes new mechanism/APIs to do P2P DMA, which we should be using in this patchset. And, I think he is also planning to use the new APIs to augment dmabuf usage and not be forced to use the scatter-gather list particularly in cases where the underlying memory is not backed by struct page.

I was just waiting for all of this to happen, before posting a v3.

Thanks,
Vivek

Quoted 82 lines

Thanks,
Wei Lin

Quoted 17 lines

Thanks,
Vivek

Quoted 12 lines

In addition to the initial proposal by Jason, another promising application is exposing memory from an AI accelerator (bound to VFIO) to an RDMA device. This would allow the RDMA device to directly access the accelerator's memory, thereby facilitating direct data transactions between the RDMA device and the accelerator.

Below is from the text/motivation from the orginal cover letter.

dma-buf has become a way to safely acquire a handle to non-struct page memory that can still have lifetime controlled by the exporter. Notably RDMA can now import dma-buf FDs and build them into MRs which

allows

Quoted 5 lines
Quoted 5 lines

for
PCI P2P operations. Extend this to allow vfio-pci to export MMIO memory from PCI device BARs.

This series supports a use case for SPDK where a NVMe device will be

owned

Quoted 1 line
Quoted 1 line

by SPDK through VFIO but interacting with a RDMA device. The RDMA

device

Quoted 1 line
Quoted 1 line

may directly access the NVMe CMB or directly manipulate the NVMe

device's

Quoted 3 lines
Quoted 3 lines

doorbell using PCI P2P.

However, as a general mechanism, it can support many other scenarios

with

Quoted 46 lines
Quoted 46 lines

VFIO. I imagine this dmabuf approach to be usable by iommufd as well for generic and safe P2P mappings.

This series goes after the "Break up ioctl dispatch functions to one function per ioctl" series.

v2:
- Name the new file dma_buf.c
- Restore orig_nents before freeing
- Fix reversed logic around priv->revoked
- Set priv->index
- Rebased on v2 "Break up ioctl dispatch functions"
v1: https://lore.kernel.org/r/0-v1-9e6e1739ed95+5fa-
[email protected]

Cc Oded Gabbay <[email protected]>
Cc Christian König <[email protected]>
Cc Daniel Vetter <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Maor Gottlieb <[email protected]>

Jason Gunthorpe (3):
vfio: Add vfio_device_get()
dma-buf: Add dma_buf_try_get()
vfio/pci: Allow MMIO regions to be exported through dma-buf

Wei Lin Guay (1):
vfio/pci: Allow export dmabuf without move_notify from importer

drivers/vfio/pci/Makefile          |   1 +
drivers/vfio/pci/dma_buf.c         | 291 +++++++++++++++++++++++++++++
drivers/vfio/pci/vfio_pci_config.c |   8 +-
drivers/vfio/pci/vfio_pci_core.c   |  44 ++++-
drivers/vfio/pci/vfio_pci_priv.h   |  30 +++
drivers/vfio/vfio_main.c           |   1 +
include/linux/dma-buf.h            |  13 ++
include/linux/vfio.h               |   6 +
include/linux/vfio_pci_core.h      |   1 +
include/uapi/linux/vfio.h          |  18 ++
10 files changed, 405 insertions(+), 8 deletions(-)

create mode 100644 drivers/vfio/pci/dma_buf.c

--
2.43.5

#8 lore raw

Hi Vivek,

On 18 Dec 2024, at 08:02, Kasireddy, Vivek <[email protected]> wrote:

Hi Wei Lin,

Subject: Re: [PATCH 0/4] cover-letter: Allow MMIO regions to be exported through dmabuf

From: Wei Lin Guay <[email protected]>

This is another attempt to revive the patches posted by Jason Gunthorpe and Vivek Kasireddy, at
https://patchwork.kernel.org/project/linux-media/cover/0-v2-
[email protected]/

https://urldefense.com/v3/__https://lwn.net/Articles/970751/__;!!Bt8RZUm
9aw!5uPrlSI9DhXVIeRMntADbkdWcu4YYhAzGN0OwyQ2NHxrN7bYE9f1eQBX UD8xHHPxiJorSkYhNjIRwdG4PsD2$
v2: https://lore.kernel.org/dri-devel/20240624065552.1572580-1-
[email protected]/
addresses review comments from Alex and Jason and also includes the ability
to create the dmabuf from multiple ranges. This is really needed to future- proof
the feature.

The good thing is that your patch series addressed Christian’s concerns of adding dma_buf_try_get().
No, it did not address his concern. In v2, instead of adding a new function, we are now calling get_file_rcu() directly, which was suggested by Christian in the old original review.

However, several questions regarding your v2 patch:
- The dma-buf still support redundant mmap handler? (One of review comments from v1?)
Yeah, the mmap handler is really needed as a debugging tool given that the importer would not be able to provide access to the dmabuf's underlying memory via the CPU in any other way.

- Rather than handle different regions within a single dma-buf, would vfio- user open multiple distinct file descriptors work?
For our specific use case, we don't require multiple regions and prefer Jason’s original patch.
Restricting the dmabuf to a single region (or having to create multiple dmabufs to represent multiple regions/ranges associated with a single scattered buffer) would not be feasible or ideal in all cases. For instance, in my use-case, I am sharing a large framebuffer (FB) located in GPU's VRAM. And, allocating a large FB contiguously (nr_ranges = 1) in VRAM is not possible when there is memory pressure.

Furthermore, since we are adding a new UAPI with this patch/feature, we cannot go back and tweak it (to add support for nr_ranges > 1) should there be a need in the future, but you can always use nr_ranges = 1 anytime. That is why it makes sense to be flexible in terms of the number of ranges/regions.

Also, my understanding is that this patchset cannot proceed until Leon's series is merged:
https://lore.kernel.org/kvm/[email protected]/

Thanks for the pointer.
I will rebase my local patch series on top of that.
AFAIK, Leon's work includes new mechanism/APIs to do P2P DMA, which we should be using in this patchset. And, I think he is also planning to use the new APIs to augment dmabuf usage and not be forced to use the scatter-gather list particularly in cases where the underlying memory is not backed by struct page.

I was just waiting for all of this to happen, before posting a v3.

Is there any update or ETA for the v3? Are there any ways we can help?

Additionally, do you have any repo that we can access to begin validating our user API changes. This would greatly aid us in our software development.

Thanks,
Wei Lin

Thanks,
Vivek

Thanks,
Wei Lin

Thanks,
Vivek

In addition to the initial proposal by Jason, another promising application is exposing memory from an AI accelerator (bound to VFIO) to an RDMA device. This would allow the RDMA device to directly access the accelerator's memory, thereby facilitating direct data transactions between the RDMA device and the accelerator.

Below is from the text/motivation from the orginal cover letter.

dma-buf has become a way to safely acquire a handle to non-struct page memory that can still have lifetime controlled by the exporter. Notably RDMA can now import dma-buf FDs and build them into MRs which allows
for
PCI P2P operations. Extend this to allow vfio-pci to export MMIO memory from PCI device BARs.

This series supports a use case for SPDK where a NVMe device will be owned
by SPDK through VFIO but interacting with a RDMA device. The RDMA device
may directly access the NVMe CMB or directly manipulate the NVMe device's
doorbell using PCI P2P.

However, as a general mechanism, it can support many other scenarios with
VFIO. I imagine this dmabuf approach to be usable by iommufd as well for generic and safe P2P mappings.

This series goes after the "Break up ioctl dispatch functions to one function per ioctl" series.

v2:
- Name the new file dma_buf.c
- Restore orig_nents before freeing
- Fix reversed logic around priv->revoked
- Set priv->index
- Rebased on v2 "Break up ioctl dispatch functions"
v1: https://lore.kernel.org/r/0-v1-9e6e1739ed95+5fa-
[email protected]

Cc Oded Gabbay <[email protected]>
Cc Christian König <[email protected]>
Cc Daniel Vetter <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Maor Gottlieb <[email protected]>

Jason Gunthorpe (3):
vfio: Add vfio_device_get()
dma-buf: Add dma_buf_try_get()
vfio/pci: Allow MMIO regions to be exported through dma-buf

Wei Lin Guay (1):
vfio/pci: Allow export dmabuf without move_notify from importer

drivers/vfio/pci/Makefile          |   1 +
drivers/vfio/pci/dma_buf.c         | 291 +++++++++++++++++++++++++++++
drivers/vfio/pci/vfio_pci_config.c |   8 +-
drivers/vfio/pci/vfio_pci_core.c   |  44 ++++-
drivers/vfio/pci/vfio_pci_priv.h   |  30 +++
drivers/vfio/vfio_main.c           |   1 +
include/linux/dma-buf.h            |  13 ++
include/linux/vfio.h               |   6 +
include/linux/vfio_pci_core.h      |   1 +
include/uapi/linux/vfio.h          |  18 ++
10 files changed, 405 insertions(+), 8 deletions(-)

create mode 100644 drivers/vfio/pci/dma_buf.c

--
2.43.5

#9 lore raw

Hi Wei Lin,

[...]

Quoted 66 lines

Yeah, the mmap handler is really needed as a debugging tool given
that the
importer would not be able to provide access to the dmabuf's
underlying
memory via the CPU in any other way.

- Rather than handle different regions within a single dma-buf,
would vfio-
user open multiple distinct file descriptors work? For our specific use case, we don't require multiple regions and
prefer
Jason’s original patch.

Restricting the dmabuf to a single region (or having to create multiple
dmabufs
to represent multiple regions/ranges associated with a single scattered
buffer)
would not be feasible or ideal in all cases. For instance, in my use-case,
I am
sharing a large framebuffer (FB) located in GPU's VRAM. And,
allocating a large
FB contiguously (nr_ranges = 1) in VRAM is not possible when there is
memory
pressure.

Furthermore, since we are adding a new UAPI with this patch/feature,
we cannot
go back and tweak it (to add support for nr_ranges > 1) should there be
a need in
the future, but you can always use nr_ranges = 1 anytime. That is why
it makes
sense to be flexible in terms of the number of ranges/regions.

Also, my understanding is that this patchset cannot
proceed until Leon's

series is merged:

https://lore.kernel.org/kvm/[email protected]/

Thanks for the pointer.
I will rebase my local patch series on top of that.

AFAIK, Leon's work includes new mechanism/APIs to do P2P DMA,
which we
should be using in this patchset. And, I think he is also planning to use
the
new APIs to augment dmabuf usage and not be forced to use the
scatter-gather
list particularly in cases where the underlying memory is not backed by
struct page.

Quoted 5 lines

I was just waiting for all of this to happen, before posting a v3.

Is there any update or ETA for the v3? Are there any ways we can help?

I believe Leon's series is very close to getting merged. Once it lands, this series can be revived.

Quoted 3 lines

Additionally, do you have any repo that we can access to begin validating our user API changes. This would greatly aid us in our software development.

Sure, here is the branch associated with this series (v2):
https://gitlab.freedesktop.org/Vivek/drm-tip/-/commits/vfio_dmabuf_2

Note that, the above branch is based off of Kernel 6.10 but I think it shouldn't be too hard to forklift the patches onto 6.14. Also, here is the Qemu branch that includes patches that demonstrate and make use of this new feature:
https://gitlab.freedesktop.org/Vivek/qemu/-/commits/vfio_dmabuf_2

On a different note, if it is not too much trouble, could you please reply to emails in text (preferred format for emails on mailing lists) instead of HTML?

Thanks,
Vivek

Quoted 140 lines

Thanks,
Wei Lin

Thanks,
Vivek

Thanks,
Wei Lin

Thanks,
Vivek

In addition to the initial proposal by Jason,
another promising
application is exposing memory from an AI
accelerator (bound to VFIO)
to an RDMA device. This would allow the RDMA
device to directly access
the accelerator's memory, thereby facilitating
direct data
transactions between the RDMA device and the
accelerator.

Below is from the text/motivation from the
orginal cover letter.

dma-buf has become a way to safely acquire a
handle to non-struct page
memory that can still have lifetime controlled by
the exporter. Notably
RDMA can now import dma-buf FDs and build
them into MRs which

allows

for
PCI P2P operations. Extend this to allow vfio-pci
to export MMIO memory
from PCI device BARs.

This series supports a use case for SPDK where a
NVMe device will be

owned

by SPDK through VFIO but interacting with a
RDMA device. The RDMA

device

may directly access the NVMe CMB or directly
manipulate the NVMe

device's

doorbell using PCI P2P.

However, as a general mechanism, it can support
many other scenarios

with

VFIO. I imagine this dmabuf approach to be
usable by iommufd as well for
generic and safe P2P mappings.

This series goes after the "Break up ioctl dispatch
functions to one
function per ioctl" series.

v2:
- Name the new file dma_buf.c
- Restore orig_nents before freeing
- Fix reversed logic around priv->revoked
- Set priv->index
- Rebased on v2 "Break up ioctl dispatch
functions"

				v1: https://lore.kernel.org/r/0-v1-

9e6e1739ed95+5fa-
[email protected]
Cc: [email protected]
Cc: Oded Gabbay <[email protected]>
Cc: Christian König
<[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Leon Romanovsky <[email protected]>
Cc: Maor Gottlieb <[email protected]>
Cc: [email protected]
Signed-off-by: Jason Gunthorpe
<[email protected]>

Jason Gunthorpe (3):
vfio: Add vfio_device_get()
dma-buf: Add dma_buf_try_get() vfio/pci: Allow MMIO regions to be exported
through dma-buf

Wei Lin Guay (1):
vfio/pci: Allow export dmabuf without
move_notify from importer

drivers/vfio/pci/Makefile | 1 +
drivers/vfio/pci/dma_buf.c | 291
+++++++++++++++++++++++++++++

				drivers/vfio/pci/vfio_pci_config.c |   8 +-
				drivers/vfio/pci/vfio_pci_core.c   |  44 ++++-
				drivers/vfio/pci/vfio_pci_priv.h   |  30 +++
				drivers/vfio/vfio_main.c           |   1 +
				include/linux/dma-buf.h            |  13 ++
				include/linux/vfio.h               |   6 +
				include/linux/vfio_pci_core.h      |   1 +
				include/uapi/linux/vfio.h          |  18 ++
				10 files changed, 405 insertions(+), 8 deletions(-)
				create mode 100644 drivers/vfio/pci/dma_buf.c

				--
				2.43.5
maintainer
#10 lore raw
Quoted 4 lines · Vivek
Quoted 1 line

Is there any update or ETA for the v3? Are there any ways we can help?

I believe Leon's series is very close to getting merged. Once it lands, this series can be revived.

The recent drama has made what happens next unclear.

I would like it if interested parties could contribute reviews to Leon's v7 series to help it along.

We may want to start considering pushing ahead with this patch series and using the usual hack of abusing the scatterlist in the mean time.

Thanks,
Jason

#11 lore raw
Quoted 11 lines · Jason Gunthorpe
Quoted 4 lines · Vivek
Quoted 1 line

Is there any update or ETA for the v3? Are there any ways we can help?

I believe Leon's series is very close to getting merged. Once it lands, this series can be revived.

The recent drama has made what happens next unclear.

I would like it if interested parties could contribute reviews to Leon's v7 series to help it along.

Link to v7 https://lore.kernel.org/all/[email protected]/

Thanks

#12 lore raw
Quoted 13 lines · Leon Romanovsky
Quoted 2 lines
Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Vivek
Quoted 1 line

Is there any update or ETA for the v3? Are there any ways we can help?

I believe Leon's series is very close to getting merged. Once it lands, this series can be revived.

Quoted 6 lines
Quoted 4 lines

The recent drama has made what happens next unclear.

I would like it if interested parties could contribute reviews to Leon's v7 series to help it along.

Link to v7 https://lore.kernel.org/all/[email protected]/

Thanks, Jason/Leon for the v7. We will begin by reviewing the patch series.

Quoted 2 lines

Thanks

#13 lore raw
In reply to Jason Gunthorpe

Hi Jason,

Quoted 10 lines

Subject: Re: [PATCH 0/4] cover-letter: Allow MMIO regions to be exported through dmabuf

Quoted 4 lines · Vivek
Quoted 1 line

Is there any update or ETA for the v3? Are there any ways we can help?

I believe Leon's series is very close to getting merged. Once it lands, this series can be revived.

Quoted 7 lines

The recent drama has made what happens next unclear.

I would like it if interested parties could contribute reviews to Leon's v7 series to help it along.

We may want to start considering pushing ahead with this patch series and using the usual hack of abusing the scatterlist in the mean time.

Yeah, given that this series is gaining more adoption (in newer use-cases) and since Leon's series would likely take more time (and iterations) to get merged, I agree that it makes sense to continue using the scatterlist for now with this series.

Although, Simona has implicitly Ack'd this series, an explicit Ack would be needed from them or Christian for the dmabuf parts. I'll post a v3 soon (sometime this week) and we can take it from there.

Thanks,
Vivek

Quoted 3 lines

Thanks,
Jason

maintainer
#14 lore raw
In reply to Jason Gunthorpe

Am 26.02.25 um 14:38 schrieb Jason Gunthorpe:

Quoted 9 lines · Vivek
Quoted 4 lines
Quoted 2 lines

Is there any update or ETA for the v3? Are there any ways we can help?

I believe Leon's series is very close to getting merged. Once it lands, this series can be revived.

The recent drama has made what happens next unclear.

I would like it if interested parties could contribute reviews to Leon's v7 series to help it along.

I think demonstrating how any new interface would work with the existing importers/exporters would help.

Quoted 2 lines

We may want to start considering pushing ahead with this patch series and using the usual hack of abusing the scatterlist in the mean time.

That works for me as well.

I think when we should get away from scatterlist then that should be a separate set of patches or maybe call it a separate project to get done for all DMA-buf parties.

E.g. propose something like an iterator based interface and demonstrate that V4L and a DRM driver can live with that and that this approach works like expected.

Regards,
Christian.

Quoted 3 lines

Thanks,
Jason

maintainer
#15 lore raw
Quoted 6 lines · Christian König

Am 26.02.25 um 14:38 schrieb Jason Gunthorpe:

Quoted 5 lines · Vivek
Quoted 4 lines
Quoted 2 lines

Is there any update or ETA for the v3? Are there any ways we can help?

I believe Leon's series is very close to getting merged. Once it lands, this series can be revived.

Quoted 7 lines
Quoted 4 lines

The recent drama has made what happens next unclear.

I would like it if interested parties could contribute reviews to Leon's v7 series to help it along.

I think demonstrating how any new interface would work with the existing importers/exporters would help.

I can see that, but also it is a huge amount of work to make trial changes to all sorts of subsystems :\

Quoted 5 lines
Quoted 2 lines

We may want to start considering pushing ahead with this patch series and using the usual hack of abusing the scatterlist in the mean time.

That works for me as well.

Quoted 7 lines

I think when we should get away from scatterlist then that should be a separate set of patches or maybe call it a separate project to get done for all DMA-buf parties.

E.g. propose something like an iterator based interface and demonstrate that V4L and a DRM driver can live with that and that this approach works like expected.

Yeah, maybe so. We are some ways from that yet..

Jason

#16 lore raw
In reply to Christian König
Quoted 4 lines · Christian König

Am 26.02.25 um 14:38 schrieb Jason Gunthorpe:

Quoted 3 lines · Vivek
Quoted 2 lines
Quoted 2 lines

Is there any update or ETA for the v3? Are there any ways we can help?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I believe Leon's series is very close to getting merged. Once it lands, this series can be revived.

The recent drama has made what happens next unclear.

I would like it if interested parties could contribute reviews to Leon's v7 series to help it along.

I think demonstrating how any new interface would work with the existing importers/exporters would help.

Unfortunately, it is huge waste of time given current situation where even basic building blocks are not merged yet [1].

We do see clear path to fix dmabuf, see this roadmap [2].

[1] https://lore.kernel.org/all/20250302085717.GO53094@unreal/
[2] https://lore.kernel.org/linux-rdma/20250122071600.GC10702@unreal/T/#u

Thanks

#17 lore raw
In reply to Kasireddy, Vivek
Quoted 85 lines · Vivek
Quoted 2 lines

Hi Wei Lin,

[...]

Quoted 66 lines

Yeah, the mmap handler is really needed as a debugging tool given that the
importer would not be able to provide access to the dmabuf's underlying
memory via the CPU in any other way.

- Rather than handle different regions within a single dma-buf, would vfio-
user open multiple distinct file descriptors work?
For our specific use case, we don't require multiple regions and prefer
Jason’s original patch.

Restricting the dmabuf to a single region (or having to create multiple dmabufs
to represent multiple regions/ranges associated with a single scattered buffer)
would not be feasible or ideal in all cases. For instance, in my use-case, I am
sharing a large framebuffer (FB) located in GPU's VRAM. And, allocating a large
FB contiguously (nr_ranges = 1) in VRAM is not possible when there is memory
pressure.

Furthermore, since we are adding a new UAPI with this patch/feature, we cannot
go back and tweak it (to add support for nr_ranges > 1) should there be a need in
the future, but you can always use nr_ranges = 1 anytime. That is why it makes
sense to be flexible in terms of the number of ranges/regions.

Also, my understanding is that this patchset cannot proceed until Leon's

series is merged:

https://lore.kernel.org/kvm/[email protected]/

Thanks for the pointer.
I will rebase my local patch series on top of that.

AFAIK, Leon's work includes new mechanism/APIs to do P2P DMA, which we
should be using in this patchset. And, I think he is also planning to use the
new APIs to augment dmabuf usage and not be forced to use the scatter-gather
list particularly in cases where the underlying memory is not backed by struct page.

Quoted 5 lines

I was just waiting for all of this to happen, before posting a v3.

Is there any update or ETA for the v3? Are there any ways we can help?

I believe Leon's series is very close to getting merged. Once it lands, this series can be revived.

Quoted 3 lines

Additionally, do you have any repo that we can access to begin validating our user API changes. This would greatly aid us in our software development.

Quoted 7 lines

Sure, here is the branch associated with this series (v2):
https://urldefense.com/v3/__https://gitlab.freedesktop.org/Vivek/drm-tip/-/commits/vfio_dmabuf_2__;!!Bt8RZUm9aw!5AqOvAz6XDQHodYKrX3ALs11b3JitInzCLznNijS7PEQ2wsvhUbx0bF91aTlC-832qGyYJkZnPnV77r0u99IpQ$

Note that, the above branch is based off of Kernel 6.10 but I think it shouldn't be too hard to forklift the patches onto 6.14. Also, here is the Qemu branch that includes patches that demonstrate and make use of this new feature:
https://urldefense.com/v3/__https://gitlab.freedesktop.org/Vivek/qemu/-/commits/vfio_dmabuf_2__;!!Bt8RZUm9aw!5AqOvAz6XDQHodYKrX3ALs11b3JitInzCLznNijS7PEQ2wsvhUbx0bF91aTlC-832qGyYJkZnPnV77rIbSw9TQ$

Thanks for the pointer, Vivek.

Quoted 3 lines

On a different note, if it is not too much trouble, could you please reply to emails in text (preferred format for emails on mailing lists) instead of HTML?

Apologies. It was an issue with my email client setup again. It should be fixed now.

Thanks,
Wei Lin

Quoted 145 lines

Thanks,
Vivek

Quoted 140 lines

Thanks,
Wei Lin

Thanks,
Vivek

Thanks,
Wei Lin

Thanks,
Vivek

In addition to the initial proposal by Jason,
another promising
application is exposing memory from an AI
accelerator (bound to VFIO)
to an RDMA device. This would allow the RDMA
device to directly access
the accelerator's memory, thereby facilitating
direct data
transactions between the RDMA device and the
accelerator.

Below is from the text/motivation from the
orginal cover letter.

dma-buf has become a way to safely acquire a
handle to non-struct page
memory that can still have lifetime controlled by
the exporter. Notably
RDMA can now import dma-buf FDs and build
them into MRs which

allows

for
PCI P2P operations. Extend this to allow vfio-pci
to export MMIO memory
from PCI device BARs.

This series supports a use case for SPDK where a
NVMe device will be

owned

by SPDK through VFIO but interacting with a
RDMA device. The RDMA

device

may directly access the NVMe CMB or directly
manipulate the NVMe

device's

doorbell using PCI P2P.

However, as a general mechanism, it can support
many other scenarios

with

VFIO. I imagine this dmabuf approach to be
usable by iommufd as well for
generic and safe P2P mappings.

This series goes after the "Break up ioctl dispatch functions to one
function per ioctl" series.

v2:
- Name the new file dma_buf.c
- Restore orig_nents before freeing
- Fix reversed logic around priv->revoked
- Set priv->index
- Rebased on v2 "Break up ioctl dispatch
functions"
v1: https://lore.kernel.org/r/0-v1-
9e6e1739ed95+5fa-
[email protected]
Cc: [email protected]
Cc: Oded Gabbay <[email protected]>
Cc: Christian König
<[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Leon Romanovsky <[email protected]>
Cc: Maor Gottlieb <[email protected]>
Cc: [email protected]
Signed-off-by: Jason Gunthorpe
<[email protected]>

Jason Gunthorpe (3):
vfio: Add vfio_device_get()
dma-buf: Add dma_buf_try_get()
vfio/pci: Allow MMIO regions to be exported
through dma-buf

Wei Lin Guay (1):
vfio/pci: Allow export dmabuf without
move_notify from importer

drivers/vfio/pci/Makefile | 1 +
drivers/vfio/pci/dma_buf.c | 291
+++++++++++++++++++++++++++++

drivers/vfio/pci/vfio_pci_config.c |   8 +-
drivers/vfio/pci/vfio_pci_core.c   |  44 ++++-
drivers/vfio/pci/vfio_pci_priv.h   |  30 +++
drivers/vfio/vfio_main.c           |   1 +
include/linux/dma-buf.h            |  13 ++
include/linux/vfio.h               |   6 +
include/linux/vfio_pci_core.h      |   1 +
include/uapi/linux/vfio.h          |  18 ++
10 files changed, 405 insertions(+), 8 deletions(-)

create mode 100644 drivers/vfio/pci/dma_buf.c

--
2.43.5