Skip to contentSkip to conversation

Add ioctls to map grant refs on the external backing storage

PATCH In review Oleksii Moisieiev 2 participants 7 messages Jan 2, 2023 – Apr 21, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Add ioctls to map grant refs on the external backing storage

Hello,

Let me introduce the new ioctls, which are intended to allow gntdev to map scatter-gather table on top of the existing dmabuf, referenced by file descriptor.

When using dma-buf exporter to create dma-buf with backing storage and map it to the grant refs, provided from the domain, we've met a problem, that several HW (i.MX8 gpu in our case) do not support external buffer and requires backing storage to be created using it's native tools. That's why new ioctls were added to be able to pass existing dma-buffer fd as input parameter and use it as backing storage to export to refs.

Following calls were added:
IOCTL_GNTDEV_DMABUF_MAP_REFS_TO_BUF - map existing buffer as the backing storage and export it to the provided grant refs; IOCTL_GNTDEV_DMABUF_MAP_RELEASE - detach buffer from the grant table and set notification to unmap grant refs before releasing the external buffer. After this call the external buffer should be detroyed. IOCTL_GNTDEV_DMABUF_MAP_WAIT_RELEASED - wait for timeout until buffer is completely destroyed and gnt refs unmapped so domain could free grant pages. Should be called after buffer was destoyed.

Our setup is based on IMX8QM board. We're trying to implement zero-copy support for DomU graphics using Wayland zwp_linux_dmabuf_v1_interface implementation.

For dma-buf exporter we used i.MX8 gpu native tools to create backing storage grant-refs, received from DomU. Buffer for the backing storage was allocated using gbm_bo_create call because gpu do not support external buffer and requires backing storage to be created using it's native tools (eglCreateImageKHR returns EGL_NO_IMAGE_KHR for buffers, which were not created using gbm_bo_create).

This behaviour was also tested on Qemu setup using DRM_IOCTL_MODE_CREATE_DUMB call to create backing storage buffer.

---
Oleksii Moisieiev (3):
xen/grant-table: save page_count on map and use if during async
unmapping
dma-buf: add dma buffer release notifier callback xen/grant-table: add new ioctls to map dmabuf to existing fd

 drivers/dma-buf/dma-buf.c   |  44 ++++
 drivers/xen/gntdev-common.h |   8 +-
 drivers/xen/gntdev-dmabuf.c | 416 +++++++++++++++++++++++++++++++++++-
 drivers/xen/gntdev-dmabuf.h |   7 +
 drivers/xen/gntdev.c        | 101 ++++++++-
 drivers/xen/grant-table.c   |  73 +++++--
 include/linux/dma-buf.h     |  15 ++
 include/uapi/xen/gntdev.h   |  62 ++++++
 include/xen/grant_table.h   |   8 +
 9 files changed, 703 insertions(+), 31 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/3 xen/grant-table: save page_count on map and use if during async unmapping

Save the reference count of the page before mapping and use this value in gntdev_unmap_refs_async() call.
This is the enhancement of the
commit 3f9f1c67572f5e5e6dc84216d48d1480f3c4fcf6 ("xen/grant-table: add a mechanism to safely unmap pages that are in use").
Safe unmapping mechanism defers page that may being use (ref count > 1).

This is needed to allow to map/unmap pages, which have more than 1 reference. For example, DRM_IOCTL_MODE_CREATE_DUMB creates dma buffer with page_count = 2, which unmap call would be deferred while buffer exists because ref count will never equals 1.
This means the buffer remains mapped during DRM_IOCTL_MODE_DESTROY_DUMB call which causes an error:

Unable to handle kernel paging request at virtual address <addr>
....
Call trace:
  check_move_unevictable_folios+0xb8/0x4d0
  check_move_unevictable_pages+0x8c/0x110
  drm_gem_put_pages+0x118/0x198
  drm_gem_shmem_put_pages_locked+0x4c/0x70
  drm_gem_shmem_unpin+0x30/0x50
  virtio_gpu_cleanup_object+0x84/0x130
  virtio_gpu_cmd_unref_cb+0x18/0x2c
  virtio_gpu_dequeue_ctrl_func+0x124/0x290
  process_one_work+0x1d0/0x320
  worker_thread+0x14c/0x444
  kthread+0x10c/0x110

This enhancement allows to provide the expected page_count during map call so refs could be unmapped properly without unneeded defers.

---
 drivers/xen/grant-table.c | 16 +++++++++++++++-
 include/xen/grant_table.h |  3 +++
 2 files changed, 18 insertions(+), 1 deletion(-)
drivers/xen/grant-table.c +15 -1
index e1ec725c2819..d6576c8b6f0f 100644
@@ -1241,11 +1241,23 @@ int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
 		case GNTST_okay:
 		{
 			struct xen_page_foreign *foreign;
+			int page_cnt;
 
 			SetPageForeign(pages[i]);
 			foreign = xen_page_foreign(pages[i]);
 			foreign->domid = map_ops[i].dom;
 			foreign->gref = map_ops[i].ref;
+			page_cnt = page_count(pages[i]);
+			if (page_cnt > FOREIGN_MAX_PAGE_COUNT) {
+				/* foreign structure can't hold more than FOREIGN_MAX_PAGE_COUNT.
+				 * That's why we save page_count = 1 so safe unmap mechanism will
+				 * defer unmapping until all users stops using this page and let
+				 * caller handle page users.
+				 */
+				pr_warn_ratelimited("page have too many users. Will wait for 0 on umap\n");
+				foreign->private = 1;
+			} else
+				foreign->private = page_cnt;
 			break;
 		}
 
@@ -1308,9 +1320,11 @@ static void __gnttab_unmap_refs_async(struct gntab_unmap_queue_data* item)
 {
 	int ret;
 	int pc;
+	struct xen_page_foreign *foreign;
 
 	for (pc = 0; pc < item->count; pc++) {
-		if (page_count(item->pages[pc]) > 1) {
+		foreign = xen_page_foreign(item->pages[pc]);
+		if (page_count(item->pages[pc]) > foreign->private) {
 			unsigned long delay = GNTTAB_UNMAP_REFS_DELAY * (item->age + 1);
 			schedule_delayed_work(&item->gnttab_work,
 					      msecs_to_jiffies(delay));
include/xen/grant_table.h +3 -0
index e279be353e3f..8e220edf44ab 100644
@@ -49,6 +49,7 @@
 #include <linux/mm_types.h>
 #include <linux/page-flags.h>
 #include <linux/kernel.h>
+#include <linux/limits.h>
 
 /*
  * Technically there's no reliably invalid grant reference or grant handle,
@@ -274,9 +275,11 @@ int gnttab_unmap_refs_sync(struct gntab_unmap_queue_data *item);
 void gnttab_batch_map(struct gnttab_map_grant_ref *batch, unsigned count);
 void gnttab_batch_copy(struct gnttab_copy *batch, unsigned count);
 
+#define FOREIGN_MAX_PAGE_COUNT       U16_MAX
 
 struct xen_page_foreign {
 	domid_t domid;
+	uint16_t private;
 	grant_ref_t gref;
 };
 
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/3 dma-buf: add dma buffer release notifier callback

In reply to Oleksii Moisieiev

Add posibility to register callback on dma-buffer which is called before dma_buf->ops->release call.
This helps when external user of the dma buffer should be notified before buffer releases without changing dma-buf ops. This is needed when external dma buffer is used as backing storage for gntdev refs export and grant refs should be unmapped before dma buffer release.

---
 drivers/dma-buf/dma-buf.c | 44 +++++++++++++++++++++++++++++++++++++++
 include/linux/dma-buf.h   | 15 +++++++++++++
 2 files changed, 59 insertions(+)
drivers/dma-buf/dma-buf.c +44 -0
index efb4990b29e1..3e663ef92e1f 100644
@@ -25,6 +25,7 @@
 #include <linux/dma-resv.h>
 #include <linux/mm.h>
 #include <linux/mount.h>
+#include <linux/notifier.h>
 #include <linux/pseudo_fs.h>
 
 #include <uapi/linux/dma-buf.h>
@@ -57,6 +58,46 @@ static char *dmabuffs_dname(struct dentry *dentry, char *buffer, int buflen)
 			     dentry->d_name.name, ret > 0 ? name : "");
 }
 
+int dma_buf_register_release_notifier(struct dma_buf *dmabuf,
+			ext_release_notifier_cb ext_release_cb, void *priv)
+{
+	int ret = 0;
+
+	spin_lock(&dmabuf->ext_release_lock);
+
+	if (dmabuf->ext_release_cb) {
+		ret = -EEXIST;
+		goto unlock;
+	}
+
+	dmabuf->ext_release_cb = ext_release_cb;
+	dmabuf->ext_release_priv = priv;
+ unlock:
+	spin_unlock(&dmabuf->ext_release_lock);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(dma_buf_register_release_notifier);
+
+void dma_buf_unregister_release_notifier(struct dma_buf *dmabuf)
+{
+	spin_lock(&dmabuf->ext_release_lock);
+	dmabuf->ext_release_cb = NULL;
+	spin_unlock(&dmabuf->ext_release_lock);
+}
+EXPORT_SYMBOL_GPL(dma_buf_unregister_release_notifier);
+
+static void dma_buf_call_release_notifier(struct dma_buf *dmabuf)
+{
+	if (!dmabuf->ext_release_cb)
+		return;
+
+	spin_lock(&dmabuf->ext_release_lock);
+	dmabuf->ext_release_cb(dmabuf, dmabuf->ext_release_priv);
+	spin_unlock(&dmabuf->ext_release_lock);
+
+	dma_buf_unregister_release_notifier(dmabuf);
+}
+
 static void dma_buf_release(struct dentry *dentry)
 {
 	struct dma_buf *dmabuf;
@@ -75,6 +116,8 @@ static void dma_buf_release(struct dentry *dentry)
 	BUG_ON(dmabuf->cb_in.active || dmabuf->cb_out.active);
 
 	dma_buf_stats_teardown(dmabuf);
+	dma_buf_call_release_notifier(dmabuf);
+
 	dmabuf->ops->release(dmabuf);
 
 	if (dmabuf->resv == (struct dma_resv *)&dmabuf[1])
@@ -642,6 +685,7 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
 	init_waitqueue_head(&dmabuf->poll);
 	dmabuf->cb_in.poll = dmabuf->cb_out.poll = &dmabuf->poll;
 	dmabuf->cb_in.active = dmabuf->cb_out.active = 0;
+	spin_lock_init(&dmabuf->ext_release_lock);
 
 	if (!resv) {
 		resv = (struct dma_resv *)&dmabuf[1];
include/linux/dma-buf.h +15 -0
index 71731796c8c3..6282d56ac040 100644
@@ -287,6 +287,8 @@ struct dma_buf_ops {
 	void (*vunmap)(struct dma_buf *dmabuf, struct iosys_map *map);
 };
 
+typedef void (*ext_release_notifier_cb)(struct dma_buf *dmabuf, void *priv);
+
 /**
  * struct dma_buf - shared buffer object
  *
@@ -432,6 +434,15 @@ struct dma_buf {
 	 */
 	struct dma_resv *resv;
 
+	/** @ext_release_cb notififier callback to call on release */
+	ext_release_notifier_cb ext_release_cb;
+
+	/** @ext_release_priv private data for callback */
+	void *ext_release_priv;
+
+	/** @ext_release_lock spinlock for ext_notifier helper */
+	spinlock_t ext_release_lock;
+
 	/** @poll: for userspace poll support */
 	wait_queue_head_t poll;
 
@@ -632,4 +643,8 @@ int dma_buf_mmap(struct dma_buf *, struct vm_area_struct *,
 		 unsigned long);
 int dma_buf_vmap(struct dma_buf *dmabuf, struct iosys_map *map);
 void dma_buf_vunmap(struct dma_buf *dmabuf, struct iosys_map *map);
+
+int dma_buf_register_release_notifier(struct dma_buf *dmabuf,
+		 ext_release_notifier_cb ext_release_cb, void *priv);
+void dma_buf_unregister_release_notifier(struct dma_buf *dmabuf);
 #endif /* __DMA_BUF_H__ */
Signature · 1 line
2.25.1
#3 lore raw

PATCH 3/3 xen/grant-table: add new ioctls to map dmabuf to existing fd

In reply to Oleksii Moisieiev

Add new ioctls to allow gntdev to map scatter-gather table on top of the existing dmabuf, referenced by file descriptor.

When using dma-buf exporter to create dma-buf with backing storage and map it to the grant refs, provided from the domain, we've met a problem, that several HW (i.MX8 gpu in our case) do not support external buffer and requires backing storage to be created using it's native tools. That's why new ioctls were added to be able to pass existing dma-buffer fd as input parameter and use it as backing storage to export to refs.

Following calls were added:
IOCTL_GNTDEV_DMABUF_MAP_REFS_TO_BUF - map existing buffer as the backing storage and export it to the provided grant refs; IOCTL_GNTDEV_DMABUF_MAP_RELEASE - detach buffer from the grant table and set notification to unmap grant refs before releasing the external buffer. After this call the external buffer should be detroyed. IOCTL_GNTDEV_DMABUF_MAP_WAIT_RELEASED - wait for timeout until buffer is completely destroyed and gnt refs unmapped so domain could free grant pages.

---
 drivers/xen/gntdev-common.h |   8 +-
 drivers/xen/gntdev-dmabuf.c | 416 +++++++++++++++++++++++++++++++++++-
 drivers/xen/gntdev-dmabuf.h |   7 +
 drivers/xen/gntdev.c        | 101 ++++++++-
 drivers/xen/grant-table.c   |  57 +++--
 include/uapi/xen/gntdev.h   |  62 ++++++
 include/xen/grant_table.h   |   5 +
 7 files changed, 626 insertions(+), 30 deletions(-)
drivers/xen/gntdev-common.h +6 -2
index 9c286b2a1900..3b6980df3f9d 100644
@@ -61,6 +61,10 @@ struct gntdev_grant_map {
 	bool *being_removed;
 	struct page **pages;
 	unsigned long pages_vm_start;
+	unsigned int preserve_pages;
+
+	/* Needed to avoid allocation in gnttab_dma_free_pages(). */
+	xen_pfn_t *frames;
 
 #ifdef CONFIG_XEN_GRANT_DMA_ALLOC
 	/*
@@ -73,8 +77,6 @@ struct gntdev_grant_map {
 	int dma_flags;
 	void *dma_vaddr;
 	dma_addr_t dma_bus_addr;
-	/* Needed to avoid allocation in gnttab_dma_free_pages(). */
-	xen_pfn_t *frames;
 #endif
 
 	/* Number of live grants */
@@ -85,6 +87,8 @@ struct gntdev_grant_map {
 
 struct gntdev_grant_map *gntdev_alloc_map(struct gntdev_priv *priv, int count,
 					  int dma_flags);
+struct gntdev_grant_map *gntdev_get_alloc_from_fd(struct gntdev_priv *priv,
+					  struct sg_table *sgt, int count, int dma_flags);
 
 void gntdev_add_map(struct gntdev_priv *priv, struct gntdev_grant_map *add);
 
drivers/xen/gntdev-dmabuf.c +412 -4

518 lines not on this page · raw message

drivers/xen/gntdev-dmabuf.h +7 -0
index 3d9b9cf9d5a1..07301f12ac52 100644
@@ -21,6 +21,9 @@ void gntdev_dmabuf_fini(struct gntdev_dmabuf_priv *priv);
 long gntdev_ioctl_dmabuf_exp_from_refs(struct gntdev_priv *priv, int use_ptemod,
 				       struct ioctl_gntdev_dmabuf_exp_from_refs __user *u);
 
+long gntdev_ioctl_dmabuf_map_refs_to_buf(struct gntdev_priv *priv, int use_ptemod,
+					   struct ioctl_gntdev_dmabuf_map_refs_to_buf __user *u);
+
 long gntdev_ioctl_dmabuf_exp_wait_released(struct gntdev_priv *priv,
 					   struct ioctl_gntdev_dmabuf_exp_wait_released __user *u);
 
@@ -30,4 +33,8 @@ long gntdev_ioctl_dmabuf_imp_to_refs(struct gntdev_priv *priv,
 long gntdev_ioctl_dmabuf_imp_release(struct gntdev_priv *priv,
 				     struct ioctl_gntdev_dmabuf_imp_release __user *u);
 
+long gntdev_ioctl_dmabuf_map_release(struct gntdev_priv *priv,
+				     struct ioctl_gntdev_dmabuf_map_release __user *u);
+long gntdev_ioctl_dmabuf_map_wait_released(struct gntdev_priv *priv,
+				     struct ioctl_gntdev_dmabuf_map_wait_released __user *u);
 #endif
drivers/xen/gntdev.c +96 -5

143 lines not on this page · raw message

drivers/xen/grant-table.c +38 -19

78 lines not on this page · raw message

include/uapi/xen/gntdev.h +62 -0

66 lines not on this page · raw message

include/xen/grant_table.h +5 -0
index 8e220edf44ab..73b473474ac9 100644
@@ -250,6 +250,11 @@ int gnttab_dma_alloc_pages(struct gnttab_dma_alloc_args *args);
 int gnttab_dma_free_pages(struct gnttab_dma_alloc_args *args);
 #endif
 
+int gnttab_dma_reserve_pages(int nr_pages, struct page **pages,
+		    xen_pfn_t *frames);
+int gnttab_dma_clean_page_reservation(int nr_pages, struct page **pages,
+		    xen_pfn_t *frames);
+
 int gnttab_pages_set_private(int nr_pages, struct page **pages);
 void gnttab_pages_clear_private(int nr_pages, struct page **pages);
 
Signature · 1 line
2.25.1
#4 lore raw
In reply to Oleksii Moisieiev

[AMD Official Use Only - General]

Sorry for the messed up mail. We currently have mail problems here at AMD.

________________________________
Von: Oleksii Moisieiev <[email protected]>
Gesendet: Montag, 2. Januar 2023 14:41
An: [email protected] <[email protected]>
Cc: Oleksii Moisieiev <[email protected]>; Stefano Stabellini <[email protected]>; Oleksandr Tyshchenko <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Sumit Semwal <[email protected]>; Koenig, Christian <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Betreff: [PATCH v1 0/3] Add ioctls to map grant refs on the external backing storage

Hello,

Let me introduce the new ioctls, which are intended to allow gntdev to map scatter-gather table on top of the existing dmabuf, referenced by file descriptor.

When using dma-buf exporter to create dma-buf with backing storage and map it to the grant refs, provided from the domain, we've met a problem, that several HW (i.MX8 gpu in our case) do not support external buffer and requires backing storage to be created using it's native tools. That's why new ioctls were added to be able to pass existing dma-buffer fd as input parameter and use it as backing storage to export to refs.

This is a pretty big NAK from my side to this approach.

If you need to replace a file descriptor number local to your process then you can simply use dup2() from userspace.

If your intention here is to replace the backing store of the fd on all processes which currently have it open then please just completely forget that. This will *NEVER* ever work correctly.

Regards,
Christian.

Following calls were added:
IOCTL_GNTDEV_DMABUF_MAP_REFS_TO_BUF - map existing buffer as the backing storage and export it to the provided grant refs; IOCTL_GNTDEV_DMABUF_MAP_RELEASE - detach buffer from the grant table and set notification to unmap grant refs before releasing the external buffer. After this call the external buffer should be detroyed. IOCTL_GNTDEV_DMABUF_MAP_WAIT_RELEASED - wait for timeout until buffer is completely destroyed and gnt refs unmapped so domain could free grant pages. Should be called after buffer was destoyed.

Our setup is based on IMX8QM board. We're trying to implement zero-copy support for DomU graphics using Wayland zwp_linux_dmabuf_v1_interface implementation.

For dma-buf exporter we used i.MX8 gpu native tools to create backing storage grant-refs, received from DomU. Buffer for the backing storage was allocated using gbm_bo_create call because gpu do not support external buffer and requires backing storage to be created using it's native tools (eglCreateImageKHR returns EGL_NO_IMAGE_KHR for buffers, which were not created using gbm_bo_create).

This behaviour was also tested on Qemu setup using DRM_IOCTL_MODE_CREATE_DUMB call to create backing storage buffer.

---
Oleksii Moisieiev (3):
xen/grant-table: save page_count on map and use if during async
unmapping
dma-buf: add dma buffer release notifier callback xen/grant-table: add new ioctls to map dmabuf to existing fd

 drivers/dma-buf/dma-buf.c   |  44 ++++
 drivers/xen/gntdev-common.h |   8 +-
 drivers/xen/gntdev-dmabuf.c | 416 +++++++++++++++++++++++++++++++++++-
 drivers/xen/gntdev-dmabuf.h |   7 +
 drivers/xen/gntdev.c        | 101 ++++++++-
 drivers/xen/grant-table.c   |  73 +++++--
 include/linux/dma-buf.h     |  15 ++
 include/uapi/xen/gntdev.h   |  62 ++++++
 include/xen/grant_table.h   |   8 +
 9 files changed, 703 insertions(+), 31 deletions(-)

--
2.25.1

#5 lore raw

On 02.01.23 18:36, Koenig, Christian wrote:

[AMD Official Use Only - General]

Sorry for the messed up mail. We currently have mail problems here at AMD.

________________________________
Von: Oleksii Moisieiev <[email protected]><mailto:[email protected]>
Gesendet: Montag, 2. Januar 2023 14:41
An: [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]>
Cc: Oleksii Moisieiev <[email protected]><mailto:[email protected]>; Stefano Stabellini <[email protected]><mailto:[email protected]>; Oleksandr Tyshchenko <[email protected]><mailto:[email protected]>; [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]>; [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]>; Sumit Semwal <[email protected]><mailto:[email protected]>; Koenig, Christian <[email protected]><mailto:[email protected]>; [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]>; [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]>; [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]>
Betreff: [PATCH v1 0/3] Add ioctls to map grant refs on the external backing storage

Hello,

Let me introduce the new ioctls, which are intended to allow gntdev to map scatter-gather table on top of the existing dmabuf, referenced by file descriptor.

When using dma-buf exporter to create dma-buf with backing storage and map it to the grant refs, provided from the domain, we've met a problem, that several HW (i.MX8 gpu in our case) do not support external buffer and requires backing storage to be created using it's native tools. That's why new ioctls were added to be able to pass existing dma-buffer fd as input parameter and use it as backing storage to export to refs.

This is a pretty big NAK from my side to this approach.

If you need to replace a file descriptor number local to your process then you can simply use dup2() from userspace.

If your intention here is to replace the backing store of the fd on all processes which currently have it open then please just completely forget that. This will *NEVER* ever work correctly.

Regards,
Christian.

Hello Cristian,

Thank you for the quick response.

My goal is to provide correct buffer for the interfaces, such as zwp_linux_dmabuf_v1_interface, so zero-copy

feature will work. My suggestion is to give a possibility to use some specific buffer as backing storage where caller

takes responsibility for the provided buffer to have the correct format.

In our case we are using these calls the following way:

1) Get grefs from another Domain (We're working on the virtualized system with different Domains

working as standalone VMs that are sharing resources);

2) Create buffer using gbm_bo_create and receive fd (i.MX8 requires egl api to be called for the buffer

allocation, or eglCreateImageKHR will return EGL_NO_IMAGE_KHR during param setting for zwp_linux_dmabuf);

3) Call for IOCTL_GNTDEV_DMABUF_MAP_REFS_TO_BUF to map grefs using fd as the backing storage;

4) Call for zwp_linux_dmabuf_v1_interface and use zero-copy feature with Wayland;

5) After work finished - call for IOCTL_GNTDEV_DMABUF_MAP_RELEASE to unmap grant refs;

6) Call gbm_bo_destroy to release allocated BO object;

7) Call for IOCTL_GNTDEV_DMABUF_MAP_WAIT_RELEASED to wait until map released completely

(This includes releasing grant refs on the Domain side).

I've tested my changes on IMX8QM board using gbm_bo_create to allocate buffer and

on QEMU setup using DRM_IOCTL_MODE_CREATE_DUMB for buffer allocation.

I can provide test applications I've used for testing purposes.

Best regards,

Oleksii.

Following calls were added:
IOCTL_GNTDEV_DMABUF_MAP_REFS_TO_BUF - map existing buffer as the backing storage and export it to the provided grant refs; IOCTL_GNTDEV_DMABUF_MAP_RELEASE - detach buffer from the grant table and set notification to unmap grant refs before releasing the external buffer. After this call the external buffer should be detroyed. IOCTL_GNTDEV_DMABUF_MAP_WAIT_RELEASED - wait for timeout until buffer is completely destroyed and gnt refs unmapped so domain could free grant pages. Should be called after buffer was destoyed.

Our setup is based on IMX8QM board. We're trying to implement zero-copy support for DomU graphics using Wayland zwp_linux_dmabuf_v1_interface implementation.

For dma-buf exporter we used i.MX8 gpu native tools to create backing storage grant-refs, received from DomU. Buffer for the backing storage was allocated using gbm_bo_create call because gpu do not support external buffer and requires backing storage to be created using it's native tools (eglCreateImageKHR returns EGL_NO_IMAGE_KHR for buffers, which were not created using gbm_bo_create).

This behaviour was also tested on Qemu setup using DRM_IOCTL_MODE_CREATE_DUMB call to create backing storage buffer.

---
Oleksii Moisieiev (3):
xen/grant-table: save page_count on map and use if during async
unmapping
dma-buf: add dma buffer release notifier callback xen/grant-table: add new ioctls to map dmabuf to existing fd

 drivers/dma-buf/dma-buf.c   |  44 ++++
 drivers/xen/gntdev-common.h |   8 +-
 drivers/xen/gntdev-dmabuf.c | 416 +++++++++++++++++++++++++++++++++++-
 drivers/xen/gntdev-dmabuf.h |   7 +
 drivers/xen/gntdev.c        | 101 ++++++++-
 drivers/xen/grant-table.c   |  73 +++++--
 include/linux/dma-buf.h     |  15 ++
 include/uapi/xen/gntdev.h   |  62 ++++++
 include/xen/grant_table.h   |   8 +
 9 files changed, 703 insertions(+), 31 deletions(-)

--
2.25.1

#6 lore raw
In reply to Oleksii Moisieiev

Hello,

Just wanted to add some additional information hoping it will help to start the discussion and find a correct approach:

My goal is to provide correct buffer for the interfaces, such as zwp_linux_dmabuf_v1_interface, so zero-copy feature will work. My suggestion is to give a possibility to use some specific buffer as backing storage where caller takes responsibility for the provided buffer to have the correct format.

In our case we are using these calls the following way:
1) Get grefs from another Domain (We're working on the virtualized system with different Domains working as standalone VMs that are sharing resources);

2) Create buffer using gbm_bo_create and receive fd (i.MX8 requires egl api to be called for the buffer allocation, or eglCreateImageKHR will return EGL_NO_IMAGE_KHR during param setting for zwp_linux_dmabuf);

3) Call for IOCTL_GNTDEV_DMABUF_MAP_REFS_TO_BUF to map grefs using fd as the backing storage;

4) Call for zwp_linux_dmabuf_v1_interface and use zero-copy feature with Wayland;

5) After work finished - call for IOCTL_GNTDEV_DMABUF_MAP_RELEASE to unmap grant refs;

6) Call gbm_bo_destroy to release allocated BO object;

7) Call for IOCTL_GNTDEV_DMABUF_MAP_WAIT_RELEASED to wait until map released completely

(This includes releasing grant refs on the Domain side).

I've tested my changes on IMX8QM board using gbm_bo_create to allocate buffer and

on QEMU setup using DRM_IOCTL_MODE_CREATE_DUMB for buffer allocation.

There was a suggestion to reverse roles between exporter and imported and allocate buffer on the other side, but this approach doesn't fit to our setup.

I have a board on which starts XEN hypervisor with several virtual domains:

One domain (Dom0) has access to the hardware and is a backend side for the Graphic sharing. Another domain (DomU) is the frontend

side and runs it's Weston instance with zero-copy on the buffer, provided by the backend.

The configuration when Dom0 as Driver domain allocates dma-buf and the exports it to the DomU is not meant to be supported, see link:
https://nvd.nist.gov/vuln/detail/CVE-2021-26934

The vulnerability is that there is no way to free buffer which was exported to DomU in case if DomU crashes.

That's why I'm trying to make a solution which should work without need to switch importer and exporter.

I'm aware that my solution may be (and is) not correct, but my intend is to start the discussion in order to produce the appropriate solution.

Best regards,

Oleksii.

On 02.01.23 15:41, Oleksii Moisieiev wrote:

Hello,

Let me introduce the new ioctls, which are intended to allow gntdev to map scatter-gather table on top of the existing dmabuf, referenced by file descriptor.

When using dma-buf exporter to create dma-buf with backing storage and map it to the grant refs, provided from the domain, we've met a problem, that several HW (i.MX8 gpu in our case) do not support external buffer and requires backing storage to be created using it's native tools. That's why new ioctls were added to be able to pass existing dma-buffer fd as input parameter and use it as backing storage to export to refs.

Following calls were added:
IOCTL_GNTDEV_DMABUF_MAP_REFS_TO_BUF - map existing buffer as the backing storage and export it to the provided grant refs; IOCTL_GNTDEV_DMABUF_MAP_RELEASE - detach buffer from the grant table and set notification to unmap grant refs before releasing the external buffer. After this call the external buffer should be detroyed. IOCTL_GNTDEV_DMABUF_MAP_WAIT_RELEASED - wait for timeout until buffer is completely destroyed and gnt refs unmapped so domain could free grant pages. Should be called after buffer was destoyed.

Our setup is based on IMX8QM board. We're trying to implement zero-copy support for DomU graphics using Wayland zwp_linux_dmabuf_v1_interface implementation.

For dma-buf exporter we used i.MX8 gpu native tools to create backing storage grant-refs, received from DomU. Buffer for the backing storage was allocated using gbm_bo_create call because gpu do not support external buffer and requires backing storage to be created using it's native tools (eglCreateImageKHR returns EGL_NO_IMAGE_KHR for buffers, which were not created using gbm_bo_create).

This behaviour was also tested on Qemu setup using DRM_IOCTL_MODE_CREATE_DUMB call to create backing storage buffer.

---
Oleksii Moisieiev (3):
xen/grant-table: save page_count on map and use if during async
unmapping
dma-buf: add dma buffer release notifier callback xen/grant-table: add new ioctls to map dmabuf to existing fd

 drivers/dma-buf/dma-buf.c   |  44 ++++
 drivers/xen/gntdev-common.h |   8 +-
 drivers/xen/gntdev-dmabuf.c | 416 +++++++++++++++++++++++++++++++++++-
 drivers/xen/gntdev-dmabuf.h |   7 +
 drivers/xen/gntdev.c        | 101 ++++++++-
 drivers/xen/grant-table.c   |  73 +++++--
 include/linux/dma-buf.h     |  15 ++
 include/uapi/xen/gntdev.h   |  62 ++++++
 include/xen/grant_table.h   |   8 +
 9 files changed, 703 insertions(+), 31 deletions(-)