Add dmabuf read/write via io_uring

include/linux/dma_token.h line 28

Quoted 8 lines · Pavel Begunkov

include/linux/dma_token.h new file +8 -0

+static inline struct dma_token *
+dma_token_create(struct file *file, struct dma_token_params *params)
+{
+	struct dma_token *res;
+
+	if (!file->f_op->dma_map)
+		return ERR_PTR(-EOPNOTSUPP);
+	res = file->f_op->dma_map(file, params);

Calling the file operation ->dmap_map feels really misleading.

create_token as in the function name is already much better, but it really is not just dma, but dmabuf related, and that should really be encoded in the name.

Also why not pass the dmabuf and direction directly instead of wrapping it in the odd params struct making the whole thing hard to follow?

Add a file callback that maps a dmabuf for the given file and returns an opaque token of type struct dma_token representing the mapping.

The
implementation details are hidden from the caller, and the implementors are normally expected to extend the structure.

The callback callers will be able to pass the token with an IO request, which implemented in following patches as a new iterator type. The user should release the token once it's not needed by calling the provided release callback via appropriate helpers.

Signed-off-by Pavel Begunkov <[email protected]>

---
 include/linux/dma_token.h | 35 +++++++++++++++++++++++++++++++++++
 include/linux/fs.h        |  4 ++++
 2 files changed, 39 insertions(+)
 create mode 100644 include/linux/dma_token.h

include/linux/dma_token.h new file +35 -0

new file mode 100644
index 000000000000..9194b34282c2
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_DMA_TOKEN_H
+#define _LINUX_DMA_TOKEN_H
+
+#include <linux/dma-buf.h>
+
+struct dma_token_params {
+	struct dma_buf			*dmabuf;
+	enum dma_data_direction		dir;
+};
+
+struct dma_token {
+	void (*release)(struct dma_token *);
+};
+
+static inline void dma_token_release(struct dma_token *token)
+{
+	token->release(token);
+}
+
+static inline struct dma_token *
+dma_token_create(struct file *file, struct dma_token_params *params)
+{
+	struct dma_token *res;
+
+	if (!file->f_op->dma_map)
+		return ERR_PTR(-EOPNOTSUPP);
+	res = file->f_op->dma_map(file, params);
+
+	WARN_ON_ONCE(!IS_ERR(res) && !res->release);
+
+	return res;
+}
+
+#endif

include/linux/fs.h +4 -0

index c895146c1444..0ce9a53fabec 100644
@@ -2262,6 +2262,8 @@ struct dir_context {
 struct iov_iter;
 struct io_uring_cmd;
 struct offset_ctx;
+struct dma_token;
+struct dma_token_params;
 
 typedef unsigned int __bitwise fop_flags_t;
 

@@ -2309,6 +2311,8 @@ struct file_operations {
 	int (*uring_cmd_iopoll)(struct io_uring_cmd *, struct io_comp_batch *,
 				unsigned int poll_flags);
 	int (*mmap_prepare)(struct vm_area_desc *);
+	struct dma_token *(*dma_map)(struct file *,
+				     struct dma_token_params *);
 } __randomize_layout;
 
 /* Supports async buffered reads */

Quoted 2 lines · Pavel Begunkov

Add a file callback that maps a dmabuf for the given file and returns an opaque token of type struct dma_token representing the mapping.

I'm really scratching my head what you mean with that?

And why the heck would we need to pass a DMA-buf to a struct file?

Quoted 4 lines · Christian König

Quoted 2 lines · Pavel Begunkov

Add a file callback that maps a dmabuf for the given file and returns an opaque token of type struct dma_token representing the mapping.

Quoted 3 lines

I'm really scratching my head what you mean with that?

And why the heck would we need to pass a DMA-buf to a struct file?

I find the naming pretty confusing a well. But what this does is to tell the file system/driver that it should expect a future read_iter/write_iter operation that takes data from / puts data into the dmabuf passed to this operation.

Quoted 4 lines

I find the naming pretty confusing a well. But what this does is to tell the file system/driver that it should expect a future read_iter/write_iter operation that takes data from / puts data into the dmabuf passed to this operation.

That explanation makes much more sense.

The remaining question is why does the underlying file system / driver needs to know that it will get addresses from a DMA-buf?

Quoted 5 lines · Christian König

Quoted 4 lines

I find the naming pretty confusing a well. But what this does is to tell the file system/driver that it should expect a future read_iter/write_iter operation that takes data from / puts data into the dmabuf passed to this operation.

Quoted 4 lines

That explanation makes much more sense.

The remaining question is why does the underlying file system / driver needs to know that it will get addresses from a DMA-buf?

This eventually ends up calling dma_buf_dynamic_attach and provides a way to find the dma_buf_attachment later in the I/O path.

Quoted 6 lines · Christoph Hellwig

Quoted 5 lines · Christian König

Quoted 4 lines

I find the naming pretty confusing a well. But what this does is to tell the file system/driver that it should expect a future read_iter/write_iter operation that takes data from / puts data into the dmabuf passed to this operation.

Quoted 7 lines

Quoted 4 lines

That explanation makes much more sense.

The remaining question is why does the underlying file system / driver needs to know that it will get addresses from a DMA-buf?

This eventually ends up calling dma_buf_dynamic_attach and provides a way to find the dma_buf_attachment later in the I/O path.

Maybe it can be named as ->dma_buf_attach()? For wiring dma-buf and the importer side(nvme).

But I am wondering why not make it as one subsystem interface, such as nvme ioctl, then the whole implementation can be simplified a lot. It is reasonable because subsystem is exactly the side for consuming/importing the dma-buf.

Quoted 3 lines

But I am wondering why not make it as one subsystem interface, such as nvme ioctl, then the whole implementation can be simplified a lot. It is reasonable because subsystem is exactly the side for consuming/importing the dma-buf.

It's not an nvme specific interface, and so a file op was much more convenient.

Quoted 8 lines · Christoph Hellwig

Quoted 7 lines · Christian König

Quoted 4 lines

I find the naming pretty confusing a well. But what this does is to tell the file system/driver that it should expect a future read_iter/write_iter operation that takes data from / puts data into the dmabuf passed to this operation.

That explanation makes much more sense.

Quoted 5 lines

Quoted 2 lines

The remaining question is why does the underlying file system / driver needs to know that it will get addresses from a DMA-buf?

This eventually ends up calling dma_buf_dynamic_attach and provides a way to find the dma_buf_attachment later in the I/O path.

Maybe it can be named as ->dma_buf_attach()? For wiring dma-buf and the importer side(nvme).

But I am wondering why not make it as one subsystem interface, such as nvme ioctl, then the whole implementation can be simplified a lot. It is reasonable because subsystem is exactly the side for consuming/importing the dma-buf.

Thanks,
Ming

Quoted 3 lines

But I am wondering why not make it as one subsystem interface, such as nvme ioctl, then the whole implementation can be simplified a lot. It is reasonable because subsystem is exactly the side for consuming/importing the dma-buf.

Yeah that it might be better if it's more nvme specific came to me as well.

Quoted 5 lines · Christian König

Quoted 3 lines

But I am wondering why not make it as one subsystem interface, such as nvme ioctl, then the whole implementation can be simplified a lot. It is reasonable because subsystem is exactly the side for consuming/importing the dma-buf.

Yeah that it might be better if it's more nvme specific came to me as well.

The feature is in no way nvme specific. nvme is just the initial underlying driver. It makes total sense to support this for any high performance block device, and to pass it through file systems.

Quoted 3 lines

The feature is in no way nvme specific. nvme is just the initial underlying driver. It makes total sense to support this for any high performance block device, and to pass it through file systems.

But why does FS care the dma buffer attachment? Since high performance host controller is exactly the dma buffer attachment point.

If the callback is added in struct file_operations for wiring dma buffer and the importer(host contrller), you will see it is hard to let it cross device mapper/raid or other stackable block devices.

Quoted 6 lines · Ming Lei

Quoted 3 lines

The feature is in no way nvme specific. nvme is just the initial underlying driver. It makes total sense to support this for any high performance block device, and to pass it through file systems.

But why does FS care the dma buffer attachment? Since high performance host controller is exactly the dma buffer attachment point.

I can't parse what you're trying to say here.

Quoted 4 lines

If the callback is added in struct file_operations for wiring dma buffer and the importer(host contrller), you will see it is hard to let it cross device mapper/raid or other stackable block devices.

Why?

But even when not stacking, the registration still needs to go through the file system even for a single device, never mind multiple controlled by the file system.

Quoted 5 lines · Christoph Hellwig

Quoted 4 lines · Ming Lei

Quoted 3 lines

The feature is in no way nvme specific. nvme is just the initial underlying driver. It makes total sense to support this for any high performance block device, and to pass it through file systems.

Quoted 4 lines

Quoted 2 lines

But why does FS care the dma buffer attachment? Since high performance host controller is exactly the dma buffer attachment point.

I can't parse what you're trying to say here.

dma buffer attachment is simply none of FS's business.

Quoted 3 lines

But even when not stacking, the registration still needs to go through the file system even for a single device, never mind multiple controlled by the file system.

dma_buf can have multiple importers, so why does it have to go through FS for single device only?

If the registered buffer is attached to single device before going through FS, it can not support stacking block device, and it can't or not easily to use for multiple block device, no matter if they are behind same host controller or multiple.

include/linux/uio.h enum iter_type { · line 32

Quoted 9 lines · Pavel Begunkov

include/linux/uio.h +1 -0

index 5b127043a151..1b22594ca35b 100644
@@ -29,6 +29,7 @@ enum iter_type {
  	ITER_FOLIOQ,
  	ITER_XARRAY,
  	ITER_DISCARD,
+	ITER_DMA_TOKEN,

Please use DMABUF/dmabuf naming everywhere, this is about dmabufs and not dma in general.

lib/iov_iter.c +3 -0

+               .count = count,
+               .iov_offset = off,
+       };

We'll need bio_flagged() earlier in bio.h in the next patch, move it together with all related helpers, and mark the bio_flagged()'s bio argument as const.

Signed-off-by Pavel Begunkov <[email protected]>

Quoted 4 lines · Pavel Begunkov

We'll need bio_flagged() earlier in bio.h in the next patch, move it together with all related helpers, and mark the bio_flagged()'s bio argument as const.

Quoted 1 line

Signed-off-by Pavel Begunkov <[email protected]>

Looks good:

Reviewed-by Christoph Hellwig <[email protected]>

Maybe ask Jens to queue it up ASAP to get it out of the way?

Quoted 9 lines · Christoph Hellwig

Quoted 5 lines · Pavel Begunkov

We'll need bio_flagged() earlier in bio.h in the next patch, move it together with all related helpers, and mark the bio_flagged()'s bio argument as const.

Signed-off-by Pavel Begunkov <[email protected]>

Looks good:

Quoted 3 lines

Reviewed-by Christoph Hellwig <[email protected]>

Maybe ask Jens to queue it up ASAP to get it out of the way?

I was away, so a bit late for that. I definitely wouldn't mind if Jens pulls it in, but for a separate patch I'd need to justify it, and I don't think it brings anything meaningful in itself.

Add blk-mq infrastructure to handle dmabuf tokens. There are two main

+++ b/block/blk-mq-dma-token.c

block/blk-mq-dma-token.c new file +5 -0

+blk_status_t blk_rq_assign_dma_map(struct request *rq,
+				   struct blk_mq_dma_token *token)
+{
+	struct blk_mq_dma_map *map;
+

block/blk-mq.c +6 -0

+	if (bio_flagged(bio, BIO_DMA_TOKEN)) {
+		struct blk_mq_dma_token *token;
+		blk_status_t ret;
+
+		token = dma_token_to_blk_mq(bio->dma_token);
+		ret = blk_rq_assign_dma_map(rq, token);

On Sun, Nov 23, 2025 at 10:51:25PM +0000, Pavel Begunkov wrote:

block/bdev.c struct block_device *file_bdev(struct file *bdev_file) · line 71

Quoted 4 lines

block/bdev.c +4 -0

+	struct request_queue *q = bdev_get_queue(file_bdev(file));
+
+	if (!(file->f_flags & O_DIRECT))
+		return ERR_PTR(-EINVAL);

Shouldn't the O_DIRECT check be in the caller?

And a high-level comment explaining the fencing logic would be nice as well.

block/blk-mq-dma-token.c line 95

Quoted 2 lines

block/blk-mq-dma-token.c new file +2 -0

+static inline
+struct blk_mq_dma_map *blk_mq_get_token_map(struct blk_mq_dma_token *token)

Really odd return value / scope formatting.

block/blk-mq-dma-token.c line 104

Quoted 9 lines

block/blk-mq-dma-token.c new file +9 -0

+{
+	struct blk_mq_dma_map *map;
+
+	guard(rcu)();
+
+	map = rcu_dereference(token->map);
+	if (unlikely(!map || !percpu_ref_tryget_live_rcu(&map->refs)))
+		return NULL;
+	return map;

Please use good old rcu_read_unlock to make this readable.

Quoted 5 lines

block/blk-mq-dma-token.c new file +5 -0

+blk_status_t blk_rq_assign_dma_map(struct request *rq,
+				   struct blk_mq_dma_token *token)
+{
+	struct blk_mq_dma_map *map;
+

block/blk-mq-dma-token.c line 177

Quoted 10 lines

block/blk-mq-dma-token.c new file +10 -0

+	map = blk_mq_get_token_map(token);
+	if (map)
+		goto complete;
+
+	if (rq->cmd_flags & REQ_NOWAIT)
+		return BLK_STS_AGAIN;
+
+	map = blk_mq_create_dma_map(token);
+	if (IS_ERR(map))
+		return BLK_STS_RESOURCE;

Having a few comments, that say this is creating the map lazily would probably helper the reader. Also why not keep the !map case in the branch, as the map case should be the fast path and thus usually be straight line in the function?

block/blk-mq-dma-token.c line 186

Quoted 4 lines

block/blk-mq-dma-token.c new file +4 -0

+void blk_mq_dma_map_move_notify(struct blk_mq_dma_token *token)
+{
+	blk_mq_dma_map_remove(token);
+}

Is there a good reason for having this blk_mq_dma_map_move_notify wrapper?

Quoted 12 lines

block/blk-mq.c +11 -0

+	if (bio_flagged(bio, BIO_DMA_TOKEN)) {
+		struct blk_mq_dma_token *token;
+		blk_status_t ret;
+
+		token = dma_token_to_blk_mq(bio->dma_token);
+		ret = blk_rq_assign_dma_map(rq, token);
+		if (ret) {
+			if (ret == BLK_STS_AGAIN) {
+				bio_wouldblock_error(bio);
+			} else {
+				bio->bi_status = BLK_STS_RESOURCE;

block/blk-mq.c void blk_mq_submit_bio(struct bio *bio) · line 3235

Quoted 5 lines

block/blk-mq.c +5 -0

+				bio_endio(bio);
+			}
+			goto queue_exit;
+		}
+	}

Any reason to not just keep the dma_token_to_blk_mq? Also why is this overriding non-BLK_STS_AGAIN errors with BLK_STS_RESOURCE?

Add blk-mq infrastructure to handle dmabuf tokens. There are two main objects. The first is struct blk_mq_dma_token, which is an extension of struct dma_token and passed in an iterator. The second is struct blk_mq_dma_map, which keeps the actual mapping and unlike the token, can be ejected (e.g. by move_notify) and recreated.

The token keeps an rcu protected pointer to the mapping, so when it resolves a token into a mapping to pass it to a request, it'll do an rcu protected lookup and get a percpu reference to the mapping.

If there is no current mapping attached to a token, it'll need to be created by calling the driver (e.g. nvme) via a new callback. It requires waiting, thefore can't be done for nowait requests and couldn't happen deeper in the stack, e.g. during nvme request submission.

The structure split is needed because move_notify can request to invalidate the dma mapping at any moment, and we need a way to concurrently remove it and wait for the inflight requests using the previous mapping to complete.

Signed-off-by Pavel Begunkov <[email protected]>

---

block/Makefile                   |   1 +
block/bdev.c                     |  14 ++
block/blk-mq-dma-token.c         | 236 +++++++++++++++++++++++++++++++
block/blk-mq.c                   |  20 +++
block/fops.c                     |   1 +
include/linux/blk-mq-dma-token.h |  60 ++++++++
include/linux/blk-mq.h           |  21 +++
include/linux/blkdev.h           |   3 +
8 files changed, 356 insertions(+)

create mode 100644 block/blk-mq-dma-token.c create mode 100644 include/linux/blk-mq-dma-token.h

block/Makefile +0 -0

index c65f4da93702..0190e5aa9f00 100644
@@ -36,3 +36,4 @@ obj-$(CONFIG_BLK_INLINE_ENCRYPTION)	+= blk-crypto.o blk-crypto-profile.o \
 					   blk-crypto-sysfs.o

obj-$(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK) += blk-crypto-fallback.o
obj-$(CONFIG_BLOCK_HOLDER_DEPRECATED) += holder.o

block/Makefile +1 -0

+obj-$(CONFIG_DMA_SHARED_BUFFER) += blk-mq-dma-token.o

block/bdev.c +14 -0

index 810707cca970..da89d20f33f3 100644
@@ -28,6 +28,7 @@
 #include <linux/part_stat.h>
 #include <linux/uaccess.h>
 #include <linux/stat.h>
+#include <linux/blk-mq-dma-token.h>
 #include "../fs/internal.h"
 #include "blk.h"
 

@@ -61,6 +62,19 @@ struct block_device *file_bdev(struct file *bdev_file)
 }
 EXPORT_SYMBOL(file_bdev);
 
+struct dma_token *blkdev_dma_map(struct file *file,
+				 struct dma_token_params *params)
+{
+	struct request_queue *q = bdev_get_queue(file_bdev(file));
+
+	if (!(file->f_flags & O_DIRECT))
+		return ERR_PTR(-EINVAL);
+	if (!q->mq_ops)
+		return ERR_PTR(-EINVAL);
+
+	return blk_mq_dma_map(q, params);
+}
+
 static void bdev_write_inode(struct block_device *bdev)
 {
 	struct inode *inode = BD_INODE(bdev);

block/blk-mq-dma-token.c new file +236 -0

new file mode 100644
index 000000000000..cd62c4d09422
@@ -0,0 +1,236 @@
+#include <linux/blk-mq-dma-token.h>
+#include <linux/dma-resv.h>
+
+struct blk_mq_dma_fence {
+	struct dma_fence base;
+	spinlock_t lock;
+};
+
+static const char *blk_mq_fence_drv_name(struct dma_fence *fence)
+{
+	return "blk-mq";
+}
+
+const struct dma_fence_ops blk_mq_dma_fence_ops = {
+	.get_driver_name = blk_mq_fence_drv_name,
+	.get_timeline_name = blk_mq_fence_drv_name,
+};
+
+static void blk_mq_dma_token_free(struct blk_mq_dma_token *token)
+{
+	token->q->mq_ops->clean_dma_token(token->q, token);
+	dma_buf_put(token->dmabuf);
+	kfree(token);
+}
+
+static inline void blk_mq_dma_token_put(struct blk_mq_dma_token *token)
+{
+	if (refcount_dec_and_test(&token->refs))
+		blk_mq_dma_token_free(token);
+}
+
+static void blk_mq_dma_mapping_free(struct blk_mq_dma_map *map)
+{
+	struct blk_mq_dma_token *token = map->token;
+
+	if (map->sgt)
+		token->q->mq_ops->dma_unmap(token->q, map);
+
+	dma_fence_put(&map->fence->base);
+	percpu_ref_exit(&map->refs);
+	kfree(map);
+	blk_mq_dma_token_put(token);
+}
+
+static void blk_mq_dma_map_work_free(struct work_struct *work)
+{
+	struct blk_mq_dma_map *map = container_of(work, struct blk_mq_dma_map,
+						free_work);
+
+	dma_fence_signal(&map->fence->base);
+	blk_mq_dma_mapping_free(map);
+}
+
+static void blk_mq_dma_map_refs_free(struct percpu_ref *ref)
+{
+	struct blk_mq_dma_map *map = container_of(ref, struct blk_mq_dma_map, refs);
+
+	INIT_WORK(&map->free_work, blk_mq_dma_map_work_free);
+	queue_work(system_wq, &map->free_work);
+}
+
+static struct blk_mq_dma_map *blk_mq_alloc_dma_mapping(struct blk_mq_dma_token *token)
+{
+	struct blk_mq_dma_fence *fence = NULL;
+	struct blk_mq_dma_map *map;
+	int ret = -ENOMEM;
+
+	map = kzalloc(sizeof(*map), GFP_KERNEL);
+	if (!map)
+		return ERR_PTR(-ENOMEM);
+
+	fence = kzalloc(sizeof(*fence), GFP_KERNEL);
+	if (!fence)
+		goto err;
+
+	ret = percpu_ref_init(&map->refs, blk_mq_dma_map_refs_free, 0,
+			      GFP_KERNEL);
+	if (ret)
+		goto err;
+
+	dma_fence_init(&fence->base, &blk_mq_dma_fence_ops, &fence->lock,
+			token->fence_ctx, atomic_inc_return(&token->fence_seq));
+	spin_lock_init(&fence->lock);
+	map->fence = fence;
+	map->token = token;
+	refcount_inc(&token->refs);
+	return map;
+err:
+	kfree(map);
+	kfree(fence);
+	return ERR_PTR(ret);
+}
+
+static inline
+struct blk_mq_dma_map *blk_mq_get_token_map(struct blk_mq_dma_token *token)
+{
+	struct blk_mq_dma_map *map;
+
+	guard(rcu)();
+
+	map = rcu_dereference(token->map);
+	if (unlikely(!map || !percpu_ref_tryget_live_rcu(&map->refs)))
+		return NULL;
+	return map;
+}
+
+static struct blk_mq_dma_map *
+blk_mq_create_dma_map(struct blk_mq_dma_token *token)
+{
+	struct dma_buf *dmabuf = token->dmabuf;
+	struct blk_mq_dma_map *map;
+	long ret;
+
+	guard(mutex)(&token->mapping_lock);
+
+	map = blk_mq_get_token_map(token);
+	if (map)
+		return map;
+
+	map = blk_mq_alloc_dma_mapping(token);
+	if (IS_ERR(map))
+		return NULL;
+
+	dma_resv_lock(dmabuf->resv, NULL);
+	ret = dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_BOOKKEEP,
+				    true, MAX_SCHEDULE_TIMEOUT);
+	ret = ret ? ret : -ETIME;
+	if (ret > 0)
+		ret = token->q->mq_ops->dma_map(token->q, map);
+	dma_resv_unlock(dmabuf->resv);
+
+	if (ret)
+		return ERR_PTR(ret);
+
+	percpu_ref_get(&map->refs);
+	rcu_assign_pointer(token->map, map);
+	return map;
+}
+
+static void blk_mq_dma_map_remove(struct blk_mq_dma_token *token)
+{
+	struct dma_buf *dmabuf = token->dmabuf;
+	struct blk_mq_dma_map *map;
+	int ret;
+
+	dma_resv_assert_held(dmabuf->resv);
+
+	ret = dma_resv_reserve_fences(dmabuf->resv, 1);
+	if (WARN_ON_ONCE(ret))
+		return;
+
+	map = rcu_dereference_protected(token->map,
+					dma_resv_held(dmabuf->resv));
+	if (!map)
+		return;
+	rcu_assign_pointer(token->map, NULL);
+
+	dma_resv_add_fence(dmabuf->resv, &map->fence->base,
+			   DMA_RESV_USAGE_KERNEL);
+	percpu_ref_kill(&map->refs);
+}
+
+blk_status_t blk_rq_assign_dma_map(struct request *rq,
+				   struct blk_mq_dma_token *token)
+{
+	struct blk_mq_dma_map *map;
+
+	map = blk_mq_get_token_map(token);
+	if (map)
+		goto complete;
+
+	if (rq->cmd_flags & REQ_NOWAIT)
+		return BLK_STS_AGAIN;
+
+	map = blk_mq_create_dma_map(token);
+	if (IS_ERR(map))
+		return BLK_STS_RESOURCE;
+complete:
+	rq->dma_map = map;
+	return BLK_STS_OK;
+}
+
+void blk_mq_dma_map_move_notify(struct blk_mq_dma_token *token)
+{
+	blk_mq_dma_map_remove(token);
+}
+
+static void blk_mq_release_dma_mapping(struct dma_token *base_token)
+{
+	struct blk_mq_dma_token *token = dma_token_to_blk_mq(base_token);
+	struct dma_buf *dmabuf = token->dmabuf;
+
+	dma_resv_lock(dmabuf->resv, NULL);
+	blk_mq_dma_map_remove(token);
+	dma_resv_unlock(dmabuf->resv);
+
+	blk_mq_dma_token_put(token);
+}
+
+struct dma_token *blk_mq_dma_map(struct request_queue *q,
+				  struct dma_token_params *params)
+{
+	struct dma_buf *dmabuf = params->dmabuf;
+	struct blk_mq_dma_token *token;
+	int ret;
+
+	if (!q->mq_ops->dma_map || !q->mq_ops->dma_unmap ||
+	    !q->mq_ops->init_dma_token || !q->mq_ops->clean_dma_token)
+		return ERR_PTR(-EINVAL);
+
+	token = kzalloc(sizeof(*token), GFP_KERNEL);
+	if (!token)
+		return ERR_PTR(-ENOMEM);
+
+	get_dma_buf(dmabuf);
+	token->fence_ctx = dma_fence_context_alloc(1);
+	token->dmabuf = dmabuf;
+	token->dir = params->dir;
+	token->base.release = blk_mq_release_dma_mapping;
+	token->q = q;
+	refcount_set(&token->refs, 1);
+	mutex_init(&token->mapping_lock);
+
+	if (!blk_get_queue(q)) {
+		kfree(token);
+		return ERR_PTR(-EFAULT);
+	}
+
+	ret = token->q->mq_ops->init_dma_token(token->q, token);
+	if (ret) {
+		kfree(token);
+		blk_put_queue(q);
+		return ERR_PTR(ret);
+	}
+	return &token->base;
+}

block/blk-mq.c +2 -0

index f2650c97a75e..1ff3a7e3191b 100644
@@ -29,6 +29,7 @@
 #include <linux/blk-crypto.h>
 #include <linux/part_stat.h>
 #include <linux/sched/isolation.h>
+#include <linux/blk-mq-dma-token.h>
 
 #include <trace/events/block.h>
 

@@ -439,6 +440,7 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data,
 	rq->nr_integrity_segments = 0;
 	rq->end_io = NULL;
 	rq->end_io_data = NULL;
+	rq->dma_map = NULL;
 
 	blk_crypto_rq_set_defaults(rq);
 	INIT_LIST_HEAD(&rq->queuelist);

Quoted 36 lines · Pavel Begunkov

Add blk-mq infrastructure to handle dmabuf tokens. There are two main objects. The first is struct blk_mq_dma_token, which is an extension of struct dma_token and passed in an iterator. The second is struct blk_mq_dma_map, which keeps the actual mapping and unlike the token, can be ejected (e.g. by move_notify) and recreated.

The token keeps an rcu protected pointer to the mapping, so when it resolves a token into a mapping to pass it to a request, it'll do an rcu protected lookup and get a percpu reference to the mapping.

If there is no current mapping attached to a token, it'll need to be created by calling the driver (e.g. nvme) via a new callback. It requires waiting, thefore can't be done for nowait requests and couldn't happen deeper in the stack, e.g. during nvme request submission.

The structure split is needed because move_notify can request to invalidate the dma mapping at any moment, and we need a way to concurrently remove it and wait for the inflight requests using the previous mapping to complete.

Signed-off-by Pavel Begunkov <[email protected]>

---
block/Makefile                   |   1 +
block/bdev.c                     |  14 ++ block/blk-mq-dma-token.c         | 236 +++++++++++++++++++++++++++++++ block/blk-mq.c                   |  20 +++
block/fops.c                     |   1 + include/linux/blk-mq-dma-token.h |  60 ++++++++ include/linux/blk-mq.h           |  21 +++ include/linux/blkdev.h           |   3 +

8 files changed, 356 insertions(+)

create mode 100644 block/blk-mq-dma-token.c create mode 100644 include/linux/blk-mq-dma-token.h

block/Makefile +0 -0

index c65f4da93702..0190e5aa9f00 100644

Quoted 21 lines

block/blk-mq.c +2 -0

index f2650c97a75e..1ff3a7e3191b 100644
@@ -29,6 +29,7 @@
 #include <linux/blk-crypto.h>
 #include <linux/part_stat.h>
 #include <linux/sched/isolation.h>
+#include <linux/blk-mq-dma-token.h>
 
 #include <trace/events/block.h>
 

@@ -439,6 +440,7 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data,
     rq->nr_integrity_segments = 0;
     rq->end_io = NULL;
     rq->end_io_data = NULL;
+    rq->dma_map = NULL;
 
     blk_crypto_rq_set_defaults(rq);
     INIT_LIST_HEAD(&rq->queuelist);

block/blk-mq.c static void __blk_mq_free_request(struct request *rq) · line 799

Quoted 4 lines

block/blk-mq.c +1 -0

@@ -794,6 +796,7 @@ static void __blk_mq_free_request(struct request *rq)
     blk_pm_mark_last_busy(rq);
     rq->mq_hctx = NULL;
 
+    blk_rq_drop_dma_map(rq);

blk_rq_drop_dma_map(rq), needs to be added in blk_mq_end_request_batch as well[1], otherwise I am seeing we leave with increased reference count in dma-buf exporter side.

Thanks,
Nitesh

[1]

block/blk-mq-dma-token.c line 83

Quoted 4 lines

block/blk-mq-dma-token.c new file +4 -0

+
+       dma_fence_init(&fence->base, &blk_mq_dma_fence_ops, &fence->lock,
+                       token->fence_ctx, atomic_inc_return(&token->fence_seq));
+       spin_lock_init(&fence->lock);

nit lock should be initialized before handing its address to dma_fence_init()

Why do you care about alignment to the controller page size?

drivers/nvme/host/pci.c static void nvme_release_descriptor_pools(struct nvme_dev *dev) · line 555

Quoted 10 lines

drivers/nvme/host/pci.c +10 -0

+	for_each_sgtable_dma_sg(sgt, sg, tmp) {
+		dma_addr_t dma = sg_dma_address(sg);
+		unsigned long sg_len = sg_dma_len(sg);
+
+		while (sg_len) {
+			dma_list[i++] = dma;
+			dma += NVME_CTRL_PAGE_SIZE;
+			sg_len -= NVME_CTRL_PAGE_SIZE;
+		}
+	}

Why does this build controller pages sized chunks?

drivers/nvme/host/pci.c +16 -0

+static void nvme_sync_dma(struct nvme_dev *nvme_dev, struct request *req,
+			  enum dma_data_direction dir)
+{
+	struct blk_mq_dma_map *map = req->dma_map;
+	int length = blk_rq_payload_bytes(req);
+	bool for_cpu = dir == DMA_FROM_DEVICE;
+	struct device *dev = nvme_dev->dev;
+	dma_addr_t *dma_list = map->private;
+	struct bio *bio = req->bio;
+	int offset, map_idx;
+
+	offset = bio->bi_iter.bi_bvec_done;
+	map_idx = offset / NVME_CTRL_PAGE_SIZE;
+	length += offset & (NVME_CTRL_PAGE_SIZE - 1);
+
+	while (length > 0) {

Add an ability to register a dmabuf backed io_uring buffer. It also needs know which device to use for attachment, for that it takes target_fd and extracts the device through the new file op. Unlike normal buffers, it also retains the target file so that any imports from ineligible requests can be rejected in next patches.

Suggested-by Vishal Verma <[email protected]>

Suggested-by David Wei <[email protected]>

Signed-off-by Pavel Begunkov <[email protected]>

---
 io_uring/rsrc.c | 106 +++++++++++++++++++++++++++++++++++++++++++++++-
 io_uring/rsrc.h |   1 +
 2 files changed, 106 insertions(+), 1 deletion(-)

io_uring/rsrc.c +63 -0

index 691f9645d04c..7dfebf459dd0 100644
@@ -10,6 +10,8 @@
 #include <linux/compat.h>
 #include <linux/io_uring.h>
 #include <linux/io_uring/cmd.h>
+#include <linux/dma-buf.h>
+#include <linux/dma_token.h>
 
 #include <uapi/linux/io_uring.h>
 

@@ -802,6 +804,106 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages,
 	return true;
 }
 
+struct io_regbuf_dma {
+	struct dma_token		*token;
+	struct file			*target_file;
+	struct dma_buf			*dmabuf;
+};
+
+static void io_release_reg_dmabuf(void *priv)
+{
+	struct io_regbuf_dma *db = priv;
+
+	dma_token_release(db->token);
+	dma_buf_put(db->dmabuf);
+	fput(db->target_file);
+	kfree(db);
+}
+
+static struct io_rsrc_node *io_register_dmabuf(struct io_ring_ctx *ctx,
+						struct io_uring_reg_buffer *rb,
+						struct iovec *iov)
+{
+	struct dma_token_params params = {};
+	struct io_rsrc_node *node = NULL;
+	struct io_mapped_ubuf *imu = NULL;
+	struct io_regbuf_dma *regbuf = NULL;
+	struct file *target_file = NULL;
+	struct dma_buf *dmabuf = NULL;
+	struct dma_token *token;
+	int ret;
+
+	if (iov->iov_base || iov->iov_len)
+		return ERR_PTR(-EFAULT);
+
+	node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);
+	if (!node) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
+	imu = io_alloc_imu(ctx, 0);
+	if (!imu) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
+	regbuf = kzalloc(sizeof(*regbuf), GFP_KERNEL);
+	if (!regbuf) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
+	target_file = fget(rb->target_fd);
+	if (!target_file) {
+		ret = -EBADF;
+		goto err;
+	}
+
+	dmabuf = dma_buf_get(rb->dmabuf_fd);
+	if (IS_ERR(dmabuf)) {
+		ret = PTR_ERR(dmabuf);
+		dmabuf = NULL;
+		goto err;

Quoted 10 lines · Pavel Begunkov

Add an ability to register a dmabuf backed io_uring buffer. It also needs know which device to use for attachment, for that it takes target_fd and extracts the device through the new file op. Unlike normal buffers, it also retains the target file so that any imports from ineligible requests can be rejected in next patches.

Suggested-by Vishal Verma <[email protected]>

Suggested-by David Wei <[email protected]>

Signed-off-by Pavel Begunkov <[email protected]>

---

Quoted 11 lines

io_uring/rsrc.c +11 -0

+	dmabuf = dma_buf_get(rb->dmabuf_fd);
+	if (IS_ERR(dmabuf)) {
+		ret = PTR_ERR(dmabuf);
+		dmabuf = NULL;
+		goto err;
+	}
+
+	params.dmabuf = dmabuf;
+	params.dir = DMA_BIDIRECTIONAL;
+	token = dma_token_create(target_file, &params);
+	if (IS_ERR(token)) {

io_uring/rsrc.c bool io_check_coalesce_buffer(struct page **page_array, int nr_pages, · line 877

Quoted 4 lines

io_uring/rsrc.c +4 -0

+		ret = PTR_ERR(token);
+		goto err;
+	}
+

This way looks less flexible, for example, the same dma-buf may be used on IOs to multiple disks, then it needs to be registered for each target file.

Picking up the work on supporting dmabuf in the read/write path.

There
are two main changes. First, it doesn't pass a dma addresss directly by rather wraps it into an opaque structure, which is extended and understood by the target driver.

The second big change is support for dynamic attachments, which added a good part of complexity (see Patch 5). I kept the main machinery in nvme at first, but move_notify can ask to kill the dma mapping asynchronously, and any new IO would need to wait during submission, thus it was moved to blk-mq. That also introduced an extra callback layer b/w driver and blk-mq.

There are some rough corners, and I'm not perfectly happy about the complexity and layering. For v3 I'll try to move the waiting up in the stack to io_uring wrapped into library helpers.

For now, I'm interested what is the best way to test move_notify? And how dma_resv_reserve_fences() errors should be handled in move_notify?

The uapi didn't change, after registration it looks like a normal io_uring registered buffer and can be used as such. Only non-vectored fixed reads/writes are allowed. Pseudo code:

// registration
reg_buf_idx = 0;
io_uring_update_buffer(ring, reg_buf_idx, { dma_buf_fd, file_fd });

// request creation io_uring_prep_read_fixed(sqe, file_fd, buffer_offset,

                         buffer_size, file_offset, reg_buf_idx);

And as previously, a good bunch of code was taken from Keith's series [1].

liburing based example:

git: https://github.com/isilence/liburing.git dmabuf-rw

link https://github.com/isilence/liburing/tree/dmabuf-rw

[1] https://lore.kernel.org/io-uring/[email protected]/

Pavel Begunkov (11):
file: add callback for pre-mapping dmabuf
iov_iter: introduce iter type for pre-registered dma
block: move around bio flagging helpers
block: introduce dma token backed bio type
block: add infra to handle dmabuf tokens
nvme-pci: add support for dmabuf reggistration
nvme-pci: implement dma_token backed requests io_uring/rsrc: add imu flags
io_uring/rsrc: extended reg buffer registration io_uring/rsrc: add dmabuf-backed buffer registeration io_uring/rsrc: implement dmabuf regbuf import

 block/Makefile                   |   1 +
 block/bdev.c                     |  14 ++
 block/bio.c                      |  21 +++
 block/blk-merge.c                |  23 +++
 block/blk-mq-dma-token.c         | 236 +++++++++++++++++++++++++++++++
 block/blk-mq.c                   |  20 +++
 block/blk.h                      |   3 +-
 block/fops.c                     |   3 +
 drivers/nvme/host/pci.c          | 217 ++++++++++++++++++++++++++++
 include/linux/bio.h              |  49 ++++---
 include/linux/blk-mq-dma-token.h |  60 ++++++++
 include/linux/blk-mq.h           |  21 +++
 include/linux/blk_types.h        |   8 +-
 include/linux/blkdev.h           |   3 +
 include/linux/dma_token.h        |  35 +++++
 include/linux/fs.h               |   4 +
 include/linux/uio.h              |  10 ++
 include/uapi/linux/io_uring.h    |  13 +-
 io_uring/rsrc.c                  | 201 +++++++++++++++++++++++---
 io_uring/rsrc.h                  |  23 ++-
 io_uring/rw.c                    |   7 +-
 lib/iov_iter.c                   |  30 +++-
 22 files changed, 948 insertions(+), 54 deletions(-)
 create mode 100644 block/blk-mq-dma-token.c
 create mode 100644 include/linux/blk-mq-dma-token.h
 create mode 100644 include/linux/dma_token.h

Quoted 1 line · Pavel Begunkov

Picking up the work on supporting dmabuf in the read/write path.

IIRC that work was completely stopped because it violated core dma_fence and DMA-buf rules and after some private discussion was considered not doable in general.

Or am I mixing something up here?

Since I don't see any dma_fence implementation at all that might actually be the case.

On the other hand we have direct I/O from DMA-buf working for quite a while, just not upstream and without io_uring support.

Quoted 6 lines · Christian König

Quoted 1 line · Pavel Begunkov

Picking up the work on supporting dmabuf in the read/write path.

IIRC that work was completely stopped because it violated core dma_fence and DMA-buf rules and after some private discussion was considered not doable in general.

Or am I mixing something up here?

The time gap is purely due to me being busy. I wasn't CC'ed to those private discussions you mentioned, but the v1 feedback was to use dynamic attachments and avoid passing dma address arrays directly.

https://lore.kernel.org/all/[email protected]/

I'm lost on what part is not doable. Can you elaborate on the core dma-fence dma-buf rules?

Quoted 1 line

Since I don't see any dma_fence implementation at all that might actually be the case.

See Patch 5, struct blk_mq_dma_fence. It's used in the move_notify callback and is signaled when all inflight IO using the current mapping are complete. All new IO requests will try to recreate the mapping, and hence potentially wait with dma_resv_wait_timeout().

Quoted 1 line

On the other hand we have direct I/O from DMA-buf working for quite a while, just not upstream and without io_uring support.

Have any reference?

Quoted 12 lines · Pavel Begunkov

Quoted 6 lines · Christian König

Quoted 1 line · Pavel Begunkov

Picking up the work on supporting dmabuf in the read/write path.

IIRC that work was completely stopped because it violated core dma_fence and DMA-buf rules and after some private discussion was considered not doable in general.

Or am I mixing something up here?

The time gap is purely due to me being busy. I wasn't CC'ed to those private discussions you mentioned, but the v1 feedback was to use dynamic attachments and avoid passing dma address arrays directly.

Quoted 4 lines

https://lore.kernel.org/all/[email protected]/

I'm lost on what part is not doable. Can you elaborate on the core dma-fence dma-buf rules?

I most likely mixed that up, in other words that was a different discussion.

When you use dma_fences to indicate async completion of events you need to be super duper careful that you only do this for in flight events, have the fence creation in the right order etc...

For example once the fence is created you can't make any memory allocations any more, that's why we have this dance of reserving fence slots, creating the fence and then adding it.

Quoted 6 lines

Quoted 1 line

Since I don't see any dma_fence implementation at all that might actually be the case.

See Patch 5, struct blk_mq_dma_fence. It's used in the move_notify callback and is signaled when all inflight IO using the current mapping are complete. All new IO requests will try to recreate the mapping, and hence potentially wait with dma_resv_wait_timeout().

Without looking at the code that approach sounds more or less correct to me.

Quoted 3 lines

Quoted 1 line

On the other hand we have direct I/O from DMA-buf working for quite a while, just not upstream and without io_uring support.

Have any reference?

There is a WIP feature in AMDs GPU driver package for ROCm.

But that can't be used as general purpose DMA-buf approach, because it makes use of internal knowledge about how the GPU driver is using the backing store.

BTW when you use DMA addresses from DMA-buf always keep in mind that this memory can be written by others at the same time, e.g. you can't do things like compute a CRC first, then write to backing store and finally compare CRC.

Quoted 18 lines · Christian König

Quoted 16 lines · Pavel Begunkov

Quoted 6 lines · Christian König

Quoted 1 line · Pavel Begunkov

Picking up the work on supporting dmabuf in the read/write path.

IIRC that work was completely stopped because it violated core dma_fence and DMA-buf rules and after some private discussion was considered not doable in general.

Or am I mixing something up here?

The time gap is purely due to me being busy. I wasn't CC'ed to those private discussions you mentioned, but the v1 feedback was to use dynamic attachments and avoid passing dma address arrays directly.

https://lore.kernel.org/all/[email protected]/

I'm lost on what part is not doable. Can you elaborate on the core dma-fence dma-buf rules?

Quoted 3 lines

I most likely mixed that up, in other words that was a different discussion.

When you use dma_fences to indicate async completion of events you need to be super duper careful that you only do this for in flight events, have the fence creation in the right order etc...

I'm curious, what can happen if there is new IO using a move_notify()ed mapping, but let's say it's guaranteed to complete strictly before dma_buf_unmap_attachment() and the fence is signaled? Is there some loss of data or corruption that can happen?

sg_table = map_attach()         | move_notify()                   |
  -> add_fence(fence)           |
                                | issue_IO(sg_table) | // IO completed unmap_attachment(sg_table)      | signal_fence(fence)             |

Quoted 1 line

For example once the fence is created you can't make any memory allocations any more, that's why we have this dance of reserving fence slots, creating the fence and then adding it.

Looks I have some terminology gap here. By "memory allocations" you don't mean kmalloc, right? I assume it's about new users of the mapping.

Quoted 9 lines

Quoted 6 lines

Quoted 1 line

Since I don't see any dma_fence implementation at all that might actually be the case.

See Patch 5, struct blk_mq_dma_fence. It's used in the move_notify callback and is signaled when all inflight IO using the current mapping are complete. All new IO requests will try to recreate the mapping, and hence potentially wait with dma_resv_wait_timeout().

Without looking at the code that approach sounds more or less correct to me.

Quoted 7 lines

Quoted 3 lines

Quoted 1 line

On the other hand we have direct I/O from DMA-buf working for quite a while, just not upstream and without io_uring support.

Have any reference?

There is a WIP feature in AMDs GPU driver package for ROCm.

But that can't be used as general purpose DMA-buf approach, because it makes use of internal knowledge about how the GPU driver is using the backing store.

Got it

Quoted 1 line

BTW when you use DMA addresses from DMA-buf always keep in mind that this memory can be written by others at the same time, e.g. you can't do things like compute a CRC first, then write to backing store and finally compare CRC.

Right. The direct IO path also works with user pages, so the constraints are similar in this regard.

Quoted 23 lines · Pavel Begunkov

Quoted 21 lines · Christian König

Quoted 8 lines · Pavel Begunkov

Quoted 6 lines · Christian König

Quoted 1 line · Pavel Begunkov

Picking up the work on supporting dmabuf in the read/write path.

IIRC that work was completely stopped because it violated core dma_fence and DMA-buf rules and after some private discussion was considered not doable in general.

Or am I mixing something up here?

Quoted 8 lines

The time gap is purely due to me being busy. I wasn't CC'ed to those private discussions you mentioned, but the v1 feedback was to use dynamic attachments and avoid passing dma address arrays directly.

https://lore.kernel.org/all/[email protected]/

I'm lost on what part is not doable. Can you elaborate on the core dma-fence dma-buf rules?

I most likely mixed that up, in other words that was a different discussion.

When you use dma_fences to indicate async completion of events you need to be super duper careful that you only do this for in flight events, have the fence creation in the right order etc...

Quoted 4 lines

I'm curious, what can happen if there is new IO using a move_notify()ed mapping, but let's say it's guaranteed to complete strictly before dma_buf_unmap_attachment() and the fence is signaled? Is there some loss of data or corruption that can happen?

The problem is that you can't guarantee that because you run into deadlocks.

As soon as a dma_fence() is created and published by calling add_fence it can be memory management loops back and depends on that fence.

So you actually can't issue any new IO which might block the unmap operation.

Quoted 11 lines

sg_table = map_attach()         | move_notify()                   |
  -> add_fence(fence)           |
                                | issue_IO(sg_table) | // IO completed
unmap_attachment(sg_table)      | signal_fence(fence)             |

Quoted 1 line

For example once the fence is created you can't make any memory allocations any more, that's why we have this dance of reserving fence slots, creating the fence and then adding it.

Quoted 3 lines

Looks I have some terminology gap here. By "memory allocations" you don't mean kmalloc, right? I assume it's about new users of the mapping.

kmalloc() as well as get_free_page() is exactly what is meant here.

You can't make any memory allocation any more after creating/publishing a dma_fence.

The usually flow is the following:

1. Lock dma_resv object
2. Prepare I/O operation, make all memory allocations etc...
3. Allocate dma_fence object
4. Push I/O operation to the HW, making sure that you don't allocate memory any more.
5. Call dma_resv_add_fence(with fence allocate in #3).
6. Unlock dma_resv object

If you stride from that you most likely end up in a deadlock sooner or later.

This series significantly reduces the IOMMU/DMA overhead for I/O, particularly when the IOMMU is configured in STRICT or LAZY mode. I modified t/io_uring in fio to exercise this path and tested with an Intel Optane device. On my setup, I see the following improvement:

- STRICT: before = 570 KIOPS, after = 5.01 MIOPS
- LAZY: before = 1.93 MIOPS, after = 5.01 MIOPS
- PASSTHROUGH: before = 5.01 MIOPS, after = 5.01 MIOPS

The STRICT/LAZY numbers clearly show the benefit of avoiding per-I/O dma_map/dma_unmap and reusing the pre-mapped DMA addresses.

Picking up the work on supporting dmabuf in the read/write path. There are two main changes. First, it doesn't pass a dma addresss directly by rather wraps it into an opaque structure, which is extended and understood by the target driver.

The second big change is support for dynamic attachments, which added a good part of complexity (see Patch 5). I kept the main machinery in nvme at first, but move_notify can ask to kill the dma mapping asynchronously, and any new IO would need to wait during submission, thus it was moved to blk-mq. That also introduced an extra callback layer b/w driver and blk-mq.

[...]