Add dmabuf read/write via io_uring
Subsystems (16)
v5 cover Add dmabuf read/write via io_uring
The patch set allows to register a dmabuf to an io_uring instance for a specified file and use it with io_uring read / write requests. The infrastructure is not tied to io_uring and there could be more users in the future. A similar idea was attempted some years ago by Keith [1] (from where I borrowed a good number of changes). Later it was brough back up to life by Tushar and Vishal.
It's an opt-in feature for files, and they need to implement a new file operation to use it. Only NVMe block devices are supported in this series. The user API is built on top of io_uring's "registered buffers", where a dmabuf is registered in a special way, but after it can be used as any other "registered buffer" with IORING_OP_{READ,WRITE}_FIXED requests. It's created via a new file operation and the resulted map is then passed through the I/O stack in a new iterator type. There is some additional infrastructure to glue it together, count requests, manage lifetime and implement invalidation.
Tushar, William, Phil as well as Kanchan, Anuj and Nitesh did a lot of testing and experimentation on various devices with previous versions of the patch set (separate thanks for trying it out and investigating problems). Earlier benchmarks by Anuj for IOMMU optimisations with udmabuf showed:
STRICT: before = 570 KIOPS, after = 5.01 MIOPS
LAZY: before = 1.93 MIOPS, after = 5.01 MIOPS
PASSTHROUGH: before = 5.01 MIOPS, after = 5.01 MIOPS
# Patch set structure:
- Patches 1-2 introduce internal API and infrastructure mediating io_uring and target subsystem / devices
- Patches 3-8: block layer support + prep patches
- Patches 9-11 implement NVMe support
- Patches 12-16 add io_uring support and uapi. Patch 14 specifically enables dma-buf support, the rest are preparation patches.
There are some liburing tests that can serve as an example:
git: https://github.com/isilence/liburing.git rw-dmabuf-tests-v5
url: https://github.com/isilence/liburing/tree/rw-dmabuf-tests-v5
The patches are based on Jens' for-next. Also available as a branch:
git: https://github.com/isilence/linux.git rw-dmabuf-v5
url: https://github.com/isilence/linux/tree/rw-dmabuf-v5
[1] https://lore.kernel.org/io-uring/[email protected]/
v5: - Reject dma-buf with buffered IO for raw bdev
- Add lim->max_segments bio splitting
- Add bio_iov_iter_set() helper
- Fix io_uring uapi validation
- Other minor changes
NVMe:
- Rename nvme_pci_sgl_set_data() to nvme_pci_dma_iter_set_sgl(), split into its own prep patch.
- Convert segment walk loops to do-while.
- Drop adjacent segment coalescing logic.
- Drop first_dma/first_len from nvme_pci_dmabuf_sgl_nents()
- Remove entries > NVME_MAX_SEGS bailout; moving to the block layer.
- Factor SGL vs PRP decision making into a helper.
v4: - https://lore.kernel.org/all/[email protected]/
- Add sgl support from Anuj
- Move it under drivers/dma-buf/ and rename
- Fix mis-sized allocations
- Fix io_uring re-import mishandling
- Drop map before io_uring "task work"
- Move blk-mq callback to block_device_operations
- Convert bio flag to REQ_OP*
- Other small changes
v3: https://lore.kernel.org/io-uring/[email protected]/
- Rework io_uring registration
- Move token/map infrastructure code out of blk-mq
- Simplify callbacks: remove a separate blk-mq table, which was mostly just forwarding calls (to nvme).
- Don't skip dma sync depending on request direction
- Fix a couple of hangs
- Rename s/dma/dmabuf/
- Other small changes
v2: - Don't pass raw dma addresses, wrap it into a driver specific object
- Split into two objects: token and map
- Implement move_notify
Anuj Gupta (2):
nvme-pci: rename nvme_pci_sgl_set_data to nvme_pci_dma_iter_set_sgl
nvme-pci: add SGL support for the dmabuf path
Pavel Begunkov (14):
dma-buf: introduce initial file I/O infrastructure
iov_iter: add iterator type for dmabuf maps
block: rename bi_bvec_done
block: always adjust bi_offset on bio_advance_iter
block: move bvec init into __bio_clone
block: introduce bio_iov_iter_set()
block: introduce dma map backed bio type
block: add dma-buf support for raw bdev
nvme-pci: implement dma-buf backed requests io_uring/rsrc: introduce buf registration structure io_uring/rsrc: extend buffer update io_uring/rsrc: add uncloneable regbuf flag io_uring/rsrc: add regbuf import flags io_uring/rsrc: add dmabuf backed registered buffers
Documentation/block/biovecs.rst | 8 +- block/bio.c | 33 ++- block/blk-map.c | 2 +- block/blk-merge.c | 45 +++- block/blk-mq-dma.c | 2 +- block/blk.h | 2 +- block/fops.c | 35 ++- drivers/block/loop.c | 2 +- drivers/block/zloop.c | 2 +- drivers/dma-buf/Makefile | 2 +- drivers/dma-buf/dma-buf-io.c | 278 +++++++++++++++++++++ drivers/md/dm-io-rewind.c | 16 +- drivers/md/dm-pcache/segment.c | 4 +- drivers/nvdimm/btt.c | 2 +- drivers/nvme/host/core.c | 12 + drivers/nvme/host/nvme.h | 2 + drivers/nvme/host/pci.c | 424 +++++++++++++++++++++++++++++++- drivers/nvme/host/tcp.c | 2 +- fs/btrfs/misc.h | 2 +- include/linux/bio.h | 23 +- include/linux/blk-mq.h | 7 + include/linux/blk_types.h | 14 +- include/linux/blkdev.h | 2 + include/linux/bvec.h | 17 +- include/linux/dma-buf-io.h | 92 +++++++ include/linux/fs.h | 2 + include/linux/io_uring_types.h | 5 + include/linux/uio.h | 11 + include/uapi/linux/io_uring.h | 31 ++- io_uring/io_uring.c | 3 +- io_uring/net.c | 4 +- io_uring/rsrc.c | 280 ++++++++++++++++++--- io_uring/rsrc.h | 45 +++- io_uring/rw.c | 6 +- lib/iov_iter.c | 31 ++- net/ceph/messenger.c | 4 +- 36 files changed, 1321 insertions(+), 131 deletions(-) create mode 100644 drivers/dma-buf/dma-buf-io.c create mode 100644 include/linux/dma-buf-io.h
Signature · 1 line
2.54.0
PATCH v5 1/16 dma-buf: introduce initial file I/O infrastructure
The goal is to be able to natively use dma-buf in the read-write / IO path. This patch adds basic building blocks serving as a glue and API between drivers and upper layer subsystems providing the uAPI. Later patches implement it for NVMe raw block devices and expose it to the user space via io_uring.
There are two main objects. struct dma_buf_io_ctx and struct dma_buf_io_map. The ctx is used during initial registration and serves as an interface between the upper layer user like io_uring and to the importer subsystem / driver. The map represents the actual dma map established for the target device[s] with dma_buf_map_attachment() and stored in a device specific format. The context is created via a new file operation ->init_dma_buf_io_ctx.
The ctx-map separation exists to support map invalidation (see dma_buf_io_invalidate_mappings()). A ctx can create multiple maps during its lifetime, but there can only be no more than one (active) map attached to it. Invalidation drops the active map if present, and the next map will only be attempted to be created once there is a new request that wants to use the dma-buf IO ctx.
The primary task of the dma_buf_io_map object is to count requests using it and to wait for their completion when we want to destroy the DMA map. Invalidation is delayed by inserting a dma fence, which is signaled when all flight requests are completed.
[un]mapping and any work with dma addresses is delegated to the importer driver via an ops table stored in the ctx, see struct dma_buf_io_ops. Only the target driver / subsystem knows about devices it wants to use the dma-buf with, especially in case of multi-device filesystems or stacking in the future.
--- drivers/dma-buf/Makefile | 2 +- drivers/dma-buf/dma-buf-io.c | 278 +++++++++++++++++++++++++++++++++++ include/linux/dma-buf-io.h | 92 ++++++++++++ include/linux/fs.h | 2 + 4 files changed, 373 insertions(+), 1 deletion(-) create mode 100644 drivers/dma-buf/dma-buf-io.c create mode 100644 include/linux/dma-buf-io.h
drivers/dma-buf/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \ - dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o + dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o dma-buf-io.o obj-$(CONFIG_DMABUF_HEAPS) += dma-heap.o obj-$(CONFIG_DMABUF_HEAPS) += heaps/ obj-$(CONFIG_SYNC_FILE) += sync_file.o
drivers/dma-buf/dma-buf-io.c new file +278 -0
278 lines not on this page · raw message
include/linux/dma-buf-io.h new file +92 -0
92 lines not on this page · raw message
include/linux/fs.h +2 -0
@@ -1914,6 +1914,7 @@ struct dir_context {#define COPY_FILE_SPLICE (1 << 0)struct io_uring_cmd ; +struct dma_buf_io_ctx ;struct offset_ctx ;typedef unsigned int __bitwise fop_flags_t ;
@@ -1962,6 +1963,7 @@ struct file_operations {int (*uring_cmd_iopoll)(struct io_uring_cmd *,struct io_comp_batch *,unsigned int poll_flags);int (*mmap_prepare)(struct vm_area_desc *); +int (*init_dma_buf_io_ctx)(struct file *,struct dma_buf_io_ctx *); } __randomize_layout;/* Supports async buffered reads */
Signature · 1 line
2.54.0
Christoph Hellwig
maintainerRe: [PATCH v5 01/16] dma-buf: introduce initial file I/O infrastructure
Quoted 372 lines · Pavel Begunkov
The goal is to be able to natively use dma-buf in the read-write / IO path. This patch adds basic building blocks serving as a glue and API between drivers and upper layer subsystems providing the uAPI. Later patches implement it for NVMe raw block devices and expose it to the user space via
io_uring.There are two main objects. struct
dma_buf_io_ctxand structdma_buf_io_map. The ctx is used during initial registration and serves as an interface between the upper layer user likeio_uringand to the importer subsystem / driver. The map represents the actual dma map established for the target device[s] withdma_buf_map_attachment()and stored in a device specific format. The context is created via a new file operation ->init_dma_buf_io_ctx.The ctx-map separation exists to support map invalidation (see
dma_buf_io_invalidate_mappings()). A ctx can create multiple maps during its lifetime, but there can only be no more than one (active) map attached to it. Invalidation drops the active map if present, and the next map will only be attempted to be created once there is a new request that wants to use the dma-buf IO ctx.The primary task of the
dma_buf_io_mapobject is to count requests using it and to wait for their completion when we want to destroy the DMA map. Invalidation is delayed by inserting a dma fence, which is signaled when all flight requests are completed.[un]mapping and any work with dma addresses is delegated to the importer driver via an ops table stored in the ctx, see struct
dma_buf_io_ops. Only the target driver / subsystem knows about devices it wants to use the dma-buf with, especially in case of multi-device filesystems or stacking in the future.Signed-off-by Pavel Begunkov <[email protected]>--- drivers/dma-buf/Makefile | 2 +- drivers/dma-buf/dma-buf-io.c | 278 +++++++++++++++++++++++++++++++++++ include/linux/dma-buf-io.h | 92 ++++++++++++ include/linux/fs.h | 2 + 4 files changed, 373 insertions(+), 1 deletion(-) create mode 100644 drivers/dma-buf/dma-buf-io.c create mode 100644 include/linux/dma-buf-io.hdrivers/dma-buf/Makefile +1
-1@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \ - dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o + dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o dma-buf-io.o obj-$(CONFIG_DMABUF_HEAPS) += dma-heap.o obj-$(CONFIG_DMABUF_HEAPS) += heaps/ obj-$(CONFIG_SYNC_FILE) += sync_file.odrivers/dma-buf/dma-buf-io.c new file +278
-0@@ -0,0 +1,278 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Common infrastructure for supporing dma-buf in the I/O path. +* +* Copyright (C) 2026 Pavel Begunkov <[email protected]> +*/ +#include <linux/dma-buf-io.h> +#include <linux/dma-resv.h> + +struct dma_buf_io_fence { +struct dma_fence base; +spinlock_t lock; +}; + +static const char *dma_buf_io_fence_drv_name (struct dma_fence *fence) +{ +/* default fence release kfree's the base pointer */ +BUILD_BUG_ON (offsetof(struct dma_buf_io_fence , base)); + +return "dma-buf-io-ctx" ; +} + +static const char *dma_buf_io_fence_timeline_name (struct dma_fence *fence) +{ +return "dma-buf-io-ctx" ; +} + +const struct dma_fence_ops dma_buf_io_fence_ops = { + .get_driver_name = dma_buf_io_fence_drv_name, + .get_timeline_name = dma_buf_io_fence_timeline_name, +}; + +static void dma_buf_io_ctx_destroy_work (struct work_struct *work) +{ + struct dma_buf_io_ctx *ctx = container_of(work, struct dma_buf_io_ctx, + release_work); + +if (WARN_ON_ONCE (refcount_read (&ctx->refs))) +return ; + + ctx->dev_ops->release (ctx); +dma_buf_put (ctx->dmabuf); +kfree (ctx); +} + +static void dma_buf_io_map_release_work (struct work_struct *work) +{ + struct dma_buf_io_map *map = container_of(work, struct dma_buf_io_map, + release_work); +struct dma_buf_io_fence *fence = map->fence; +struct dma_buf_io_ctx *ctx = map->ctx; +struct dma_buf *dmabuf = ctx->dmabuf; + +/* the release path must wait for fences */ +if (WARN_ON_ONCE (refcount_read (&ctx->refs) ==0 )) +return ; + +/* Prevent from destoying the ctx while unmapping */ +refcount_inc (&ctx->refs); + +/* +* There are no more requests using the map, we can signal the fence. +* It should be done before taking the resv lock as someone could be +* waiting for the fence while holding the lock. +*/ +dma_fence_signal (&fence->base); + +dma_resv_lock (dmabuf->resv,NULL ); + ctx->dev_ops->unmap (ctx, map); +dma_resv_unlock (dmabuf->resv); + +dma_fence_put (&fence->base); +percpu_ref_exit (&map->refs); +kfree (map); + +if (refcount_dec_and_test (&ctx->refs)) { +/* +* Destruction needs to wait for I/O and dma fences. Defer it to +* simplify locking. +*/ +INIT_WORK (&ctx->release_work, dma_buf_io_ctx_destroy_work); +queue_work (system_wq, &ctx->release_work); + } +} + +static void dma_buf_io_map_refs_release (struct percpu_ref *ref) +{ + struct dma_buf_io_map *map = container_of(ref, struct dma_buf_io_map, refs); + +/* might sleep, use a worker */ +INIT_WORK (&map->release_work, dma_buf_io_map_release_work); +queue_work (system_wq, &map->release_work); +} + +int dma_buf_io_init_map (struct dma_buf_io_ctx *ctx,struct dma_buf_io_map *map) +{ +struct dma_buf_io_fence *fence =NULL ; +int ret; + + fence =kzalloc (sizeof (*fence),GFP_KERNEL ); +if (!fence) +return -ENOMEM ; + + ret =percpu_ref_init (&map->refs, dma_buf_io_map_refs_release,0 ,GFP_KERNEL ); +if (ret) { +kfree (fence); +return ret; + } + +spin_lock_init (&fence->lock); +dma_fence_init (&fence->base, &dma_buf_io_fence_ops, &fence->lock, + ctx->fence_ctx,atomic_inc_return (&ctx->fence_seq)); + map->fence = fence; + map->ctx = ctx; +return 0 ; +} +EXPORT_SYMBOL_NS_GPL (dma_buf_io_init_map,"DMA_BUF" ); + +struct dma_buf_io_map *dma_buf_io_create_map (struct dma_buf_io_ctx *ctx) +{ +struct dma_buf *dmabuf = ctx->dmabuf; +struct dma_buf_io_map *map; +long ret; + +retry: +/* +* ->dmabuf_map() will be calling dma_buf_map_attachment(), for which +* we'll need to wait for fences. Do a bit nicer and try to wait +* without the resv lock first. +*/ + ret =dma_resv_wait_timeout (dmabuf->resv,DMA_RESV_USAGE_KERNEL , + true,MAX_SCHEDULE_TIMEOUT ); +if (!ret) + ret = -EAGAIN ; +if (ret <0 ) +return ERR_PTR (ret); + + ret =dma_resv_lock_interruptible (dmabuf->resv,NULL ); +if (ret) +return ERR_PTR (ret); + + map =dma_buf_io_get_map (ctx); +if (map) { + ret =0 ; + goto out; + } + +if (dma_resv_wait_timeout (dmabuf->resv,DMA_RESV_USAGE_KERNEL , + true,0 ) <0 ) { +dma_resv_unlock (dmabuf->resv); + goto retry; + } + + map = ctx->dev_ops->map (ctx); +if (IS_ERR (map)) { + ret =PTR_ERR (map); + goto out; + } + +if (WARN_ON_ONCE (!map->seg_shift)) +return ERR_PTR (-EFAULT ); + +percpu_ref_get (&map->refs); +rcu_assign_pointer (ctx->map, map); +out: +dma_resv_unlock (dmabuf->resv); +if (ret <0 ) +return ERR_PTR (ret); +return map; +} + +static void dma_buf_io_drop_map (struct dma_buf_io_ctx *ctx) +{ +struct dma_buf *dmabuf = ctx->dmabuf; +struct dma_buf_io_map *map; +int ret; + +dma_resv_assert_held (dmabuf->resv); + + map =rcu_dereference_protected (ctx->map, +dma_resv_held (dmabuf->resv)); +if (!map) +return ; +rcu_assign_pointer (ctx->map,NULL ); + + ret =dma_resv_reserve_fences (dmabuf->resv,1 ); +if (WARN_ON_ONCE (ret)) { +struct dma_fence *fence = &map->fence->base; + +dma_fence_get (fence); +percpu_ref_kill (&map->refs); +dma_fence_wait (fence, false); +dma_fence_put (fence); +return ; + } + +dma_resv_add_fence (dmabuf->resv, &map->fence->base, +DMA_RESV_USAGE_KERNEL ); +/* +* Delay destruction until all inflight requests using the map are +* gone. It'll also signal the fence then. +*/ +percpu_ref_kill (&map->refs); +} + +void dma_buf_io_invalidate_mappings (struct dma_buf_io_ctx *ctx) +{ +dma_buf_io_drop_map (ctx); +} +EXPORT_SYMBOL_NS_GPL (dma_buf_io_invalidate_mappings,"DMA_BUF" ); + +static void dma_buf_io_ctx_release_work (struct work_struct *work) +{ + struct dma_buf_io_ctx *ctx = container_of(work, struct dma_buf_io_ctx, + release_work); +struct dma_buf *dmabuf = ctx->dmabuf; +long ret; + +dma_resv_lock (dmabuf->resv,NULL ); +/* Remove the last map, there should be no new ones going forward. */ +dma_buf_io_drop_map (ctx); +dma_resv_unlock (dmabuf->resv); + +/* Wait until all maps are destroyed. */ + ret =dma_resv_wait_timeout (dmabuf->resv,DMA_RESV_USAGE_KERNEL , + false,MAX_SCHEDULE_TIMEOUT ); + +if (WARN_ON_ONCE (ret <=0 )) +return ; +if (WARN_ON_ONCE (rcu_dereference_protected (ctx->map, true))) +return ; + +if (refcount_dec_and_test (&ctx->refs)) +dma_buf_io_ctx_destroy_work (&ctx->release_work); +} + +void dma_buf_io_ctx_release (struct dma_buf_io_ctx *ctx) +{ +/* +* Destruction needs to wait for I/O and dma fences. Defer it to +* simplify locking. +*/ +INIT_WORK (&ctx->release_work, dma_buf_io_ctx_release_work); +queue_work (system_wq, &ctx->release_work); +} + +int dma_buf_io_ctx_create (struct file *file, +struct dma_buf_io_ctx *ctx, +struct dma_buf *dmabuf, +enum dma_data_direction dir) +{ +int ret; + +if (!file->f_op->init_dma_buf_io_ctx) +return -EOPNOTSUPP ; + +memset (ctx,0 ,sizeof (*ctx)); + ctx->fence_ctx =dma_fence_context_alloc (1 ); + ctx->dir = dir; + ctx->dmabuf = dmabuf; +refcount_set (&ctx->refs,1 ); +get_dma_buf (dmabuf); + + ret = file->f_op->init_dma_buf_io_ctx (file, ctx); +if (ret) { +memset (ctx,0 ,sizeof (*ctx)); +dma_buf_put (dmabuf); +return ret; + } + +if (WARN_ON_ONCE (!ctx->dev_ops || + !ctx->dev_ops->map || + !ctx->dev_ops->unmap || + !ctx->dev_ops->release)) +return -EINVAL ; + +return ret; +}include/linux/dma-buf-io.h new file +28
-0@@ -0,0 +1,92 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __DMA_BUF_IO_H__ +#define __DMA_BUF_IO_H__ + +#include <linux/dma-buf.h> + +struct dma_buf_io_fence; +struct dma_buf_io_ctx; +struct dma_buf_io_map; + +struct dma_buf_io_ops { + /* + * Create a new map for the given ctx. Called with the reservation + * lock held. + */ + struct dma_buf_io_map *(*map)(struct dma_buf_io_ctx *ctx); + + /* + * Clean up device specific parts of the @map. Called with the + * reservation lock held. + */ + void (*unmap)(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map *map); + + /* + * The user tries to destroy the ctx. Release all device specific + * parts of the token. + */ + void (*release)(struct dma_buf_io_ctx *);
include/linux/dma-buf-io.h line 37
Quoted 10 lines
include/linux/dma-buf-io.h new file +9
-0+}; + +struct dma_buf_io_map { +/* +* Counts attached requests and other users. Device specific unmapping +* is deferred until all refs are dropped. +*/ +struct percpu_ref refs; +unsigned seg_shift;
Can you add a big fat comment explaining how seg_shift works here? I'm looking at the later patch using it for splitting and am very confused about it.
Re: [PATCH v5 01/16] dma-buf: introduce initial file I/O infrastructure
In reply to Pavel Begunkovdrivers/dma-buf/dma-buf-io.c line 152
Quoted 5 lines · Pavel Begunkov
drivers/dma-buf/dma-buf-io.c new file +5
-0+if (dma_resv_wait_timeout (dmabuf->resv,DMA_RESV_USAGE_KERNEL , + true,0 ) <0 ) { +dma_resv_unlock (dmabuf->resv); + goto retry; + }
Shouldn't this be <= 0? With a zero timeout dma_resv_wait_timeout() returns 0 for a still pending fence; only -ERESTARTSYS is negative.
Thanks,
Sidong
Re: [PATCH v5 01/16] dma-buf: introduce initial file I/O infrastructure
Quoted 9 lines · Sidong Yang
drivers/dma-buf/dma-buf-io.c line 152
Quoted 5 lines · Pavel Begunkov
drivers/dma-buf/dma-buf-io.c new file +5
-0+if (dma_resv_wait_timeout (dmabuf->resv,DMA_RESV_USAGE_KERNEL , + true,0 ) <0 ) { +dma_resv_unlock (dmabuf->resv); + goto retry; + }Shouldn't this be <= 0? With a zero timeout
dma_resv_wait_timeout()returns 0 for a still pending fence; only-ERESTARTSYSis negative.
You're right, and this chunk is gone in v6
Signature · 1 line
Pavel Begunkov
PATCH v5 2/16 iov_iter: add iterator type for dmabuf maps
In reply to Pavel BegunkovIntroduce a new iterator type for dmabuf maps. The map in an opaque object with internals and format specific to the subsystem / driver, and only it can use that subsystem / driver for issuing IO. The task of the middle layers is to pass the map / iterator further down, maybe doing basic splitting and length checking. The iterator can only be used by operations of the file the associated map was created for.
--- include/linux/uio.h | 11 +++++++++++ lib/iov_iter.c | 29 +++++++++++++++++++++++------ 2 files changed, 34 insertions(+), 6 deletions(-)
include/linux/uio.h +11 -0
@@ -12,6 +12,7 @@struct page ;struct folio_queue ; +struct dma_buf_io_map ;typedef unsigned int __bitwise iov_iter_extraction_t ;
@@ -29,6 +30,7 @@ enum iter_type {ITER_FOLIOQ ,ITER_XARRAY ,ITER_DISCARD , + ITER_DMABUF_MAP, };#define ITER_SOURCE 1 // == WRITE
@@ -71,6 +73,7 @@ struct iov_iter {const struct folio_queue *folioq;struct xarray *xarray;void __user *ubuf; +struct dma_buf_io_map *dmabuf_map; };size_t count; };
@@ -155,6 +158,11 @@ static inline bool iov_iter_is_xarray(const struct iov_iter *i)return iov_iter_type (i) ==ITER_XARRAY ; } +static inline bool iov_iter_is_dmabuf_map (const struct iov_iter *i) +{ +return iov_iter_type (i) ==ITER_DMABUF_MAP ; +} +static inline unsigned char iov_iter_rw (const struct iov_iter *i) {return i->data_source ?WRITE :READ ;
@@ -300,6 +308,9 @@ void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction, unsigned int first_slot, unsigned int offset, size_t count);void iov_iter_xarray (struct iov_iter *i,unsigned int direction,struct xarray *xarray,loff_t start,size_t count); +void iov_iter_dmabuf_map (struct iov_iter *i,unsigned int direction, +struct dma_buf_io_map *map, +loff_t off,size_t count);ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start); ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages,
lib/iov_iter.c +23 -6
@@ -575,7 +575,8 @@ void iov_iter_advance(struct iov_iter *i, size_t size) {if (unlikely (i->count < size)) size = i->count; -if (likely (iter_is_ubuf (i)) ||unlikely (iov_iter_is_xarray (i))) { +if (likely (iter_is_ubuf (i)) ||unlikely (iov_iter_is_xarray (i)) || +unlikely (iov_iter_is_dmabuf_map (i))) { i->iov_offset += size; i->count -= size; }else if (likely (iter_is_iovec (i) ||iov_iter_is_kvec (i))) {
@@ -631,7 +632,8 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll)return ; } unroll -= i->iov_offset; -if (iov_iter_is_xarray (i) ||iter_is_ubuf (i)) { +if (iov_iter_is_xarray (i) ||iter_is_ubuf (i) || +iov_iter_is_dmabuf_map (i)) {BUG ();/* We should never go beyond the start of the specified * range since we might then be straying into pages that * aren't pinned.
@@ -775,6 +777,20 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, }EXPORT_SYMBOL (iov_iter_xarray); +void iov_iter_dmabuf_map (struct iov_iter *i,unsigned int direction, +struct dma_buf_io_map *map, +loff_t off,size_t count) +{ +WARN_ON (direction & ~(READ |WRITE )); + *i = (struct iov_iter ){ + .iter_type =ITER_DMABUF_MAP , + .data_source = direction, + .dmabuf_map = map, + .count = count, + .iov_offset = off, + }; +} +/** * iov_iter_discard - Initialise an I/O iterator that discards data * @i: The iterator to initialise.
@@ -841,7 +857,7 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)unsigned long iov_iter_alignment (const struct iov_iter *i) { -if (likely (iter_is_ubuf (i))) { +if (likely (iter_is_ubuf (i)) ||iov_iter_is_dmabuf_map (i)) {size_t size = i->count;if (size)return ((unsigned long )i->ubuf + i->iov_offset) | size;
@@ -872,7 +888,7 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i)size_t size = i->count;unsigned k; -if (iter_is_ubuf (i)) +if (iter_is_ubuf (i) ||iov_iter_is_dmabuf_map (i))return 0 ;if (WARN_ON (!iter_is_iovec (i)))
@@ -1469,11 +1485,12 @@ EXPORT_SYMBOL_GPL(import_ubuf);void iov_iter_restore (struct iov_iter *i,struct iov_iter_state *state) {if (WARN_ON_ONCE (!iov_iter_is_bvec (i) && !iter_is_iovec (i) && - !iter_is_ubuf (i)) && !iov_iter_is_kvec (i)) + !iter_is_ubuf (i) && !iov_iter_is_kvec (i) && + !iov_iter_is_dmabuf_map (i)))return ; i->iov_offset = state->iov_offset; i->count = state->count; -if (iter_is_ubuf (i)) +if (iter_is_ubuf (i) ||iov_iter_is_dmabuf_map (i))return ;/* * For the *vec iters, nr_segs + iov is constant - if we increment
Signature · 1 line
2.54.0
Re: [PATCH v5 02/16] iov_iter: add iterator type for dmabuf maps
lib/iov_iter.c static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i) · line 863
Quoted 9 lines · Pavel Begunkov
lib/iov_iter.c +1
-1@@ -841,7 +857,7 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)unsigned long iov_iter_alignment (const struct iov_iter *i) { -if (likely (iter_is_ubuf (i))) { +if (likely (iter_is_ubuf (i)) ||iov_iter_is_dmabuf_map (i)) {size_t size = i->count;if (size)return ((unsigned long )i->ubuf + i->iov_offset) | size;
dmabuf_map shares the same union slot as ubuf, so this reads the map pointer as a user address. gap_alignment() below already returns 0 correctly for dmabuf - this should too. Checked the rest of this patch (advance/revert/restore) - none of them dereference the union pointer for dmabuf, so it's isolated to this one spot. Doesn't affect the current series, nothing reaches this with a dmabuf iter yet.
Quoted 7 lines
lib/iov_iter.c +1
-1@@ -872,7 +888,7 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i)size_t size = i->count;unsigned k; -if (iter_is_ubuf (i)) +if (iter_is_ubuf (i) ||iov_iter_is_dmabuf_map (i))return 0 ;
Re: [PATCH v5 02/16] iov_iter: add iterator type for dmabuf maps
Quoted 17 lines · Anuj gupta
lib/iov_iter.c static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i) · line 863
Quoted 9 lines · Pavel Begunkov
lib/iov_iter.c +1
-1@@ -841,7 +857,7 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)unsigned long iov_iter_alignment (const struct iov_iter *i) { -if (likely (iter_is_ubuf (i))) { +if (likely (iter_is_ubuf (i)) ||iov_iter_is_dmabuf_map (i)) {size_t size = i->count;if (size)return ((unsigned long )i->ubuf + i->iov_offset) | size;
dmabuf_mapshares the same union slot as ubuf, so this reads the map pointer as a user address.gap_alignment()below already returns 0 correctly for dmabuf - this should too. Checked the rest of this patch (advance/revert/restore) - none of them dereference the union pointer for dmabuf, so it's isolated to this one spot. Doesn't affect the current series, nothing reaches this with a dmabuf iter yet.
Ah yes, that's nonsense, it should've checked iov_offset instead. Thanks
Signature · 1 line
Pavel Begunkov
PATCH v5 3/16 block: rename bi_bvec_done
In reply to Pavel Begunkovstruct bvec_iter::bi_bvec_done is used an offset in the current bvec, let's rename it accordingly for better clarity. We're also going to use it for non-bvec based iteration, so drop the "bvec" part.
--- Documentation/block/biovecs.rst | 8 ++++---- block/bio.c | 4 ++-- block/blk-merge.c | 8 ++++---- block/blk-mq-dma.c | 2 +- block/blk.h | 2 +- drivers/block/loop.c | 2 +- drivers/block/zloop.c | 2 +- drivers/md/dm-io-rewind.c | 10 +++++----- drivers/md/dm-pcache/segment.c | 4 ++-- drivers/nvdimm/btt.c | 2 +- drivers/nvme/host/tcp.c | 2 +- fs/btrfs/misc.h | 2 +- include/linux/bvec.h | 14 +++++++------- io_uring/net.c | 4 ++-- lib/iov_iter.c | 2 +- net/ceph/messenger.c | 4 ++-- 16 files changed, 36 insertions(+), 36 deletions(-)
Documentation/block/biovecs.rst +4 -4
@@ -16,16 +16,16 @@ bv_len by the number of bytes completed in that biovec. In the new scheme of things, everything that must be mutated in order to partially complete a bio is segregated into struct bvec_iter: bi_sector, bi_size and bi_idx have been moved there; and instead of modifying bv_offset -and bv_len, struct bvec_iter has bi_bvec_done, which represents the number of +and bv_len, struct bvec_iter has bi_offset, which represents the number of bytes completed in the current bvec. There are a bunch of new helper macros for hiding the gory details - in particular, presenting the illusion of partially completed biovecs so that -normal code doesn't have to deal with bi_bvec_done. +normal code doesn't have to deal with bi_offset. * Driver code should no longer refer to biovecs directly; we now have bio_iovec() and bio_iter_iovec() macros that return literal struct biovecs, - constructed from the raw biovecs but taking into account bi_bvec_done and + constructed from the raw biovecs but taking into account bi_offset and bi_size. bio_for_each_segment() has been updated to take a bvec_iter argument
@@ -101,7 +101,7 @@ Other implications: I.e. instead of using bio_iovec_idx() (or bio->bi_iovec[bio->bi_idx]), you now use bio_iter_iovec(), which takes a bvec_iter and returns a literal struct bio_vec - constructed on the fly from the raw biovec but - taking into account bi_bvec_done (and bi_size). + taking into account bi_offset (and bi_size). * bi_vcnt can't be trusted or relied upon by driver code - i.e. anything that doesn't actually own the bio. The reason is twofold: firstly, it's not
block/bio.c +2 -2
@@ -229,7 +229,7 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table, bio->bi_iter.bi_sector =0 ; bio->bi_iter.bi_size =0 ; bio->bi_iter.bi_idx =0 ; - bio->bi_iter.bi_bvec_done =0 ; + bio->bi_iter.bi_offset =0 ; bio->bi_end_io =NULL ; bio->bi_private =NULL ;#ifdef CONFIG_BLK_CGROUP
@@ -1188,7 +1188,7 @@ void bio_iov_bvec_set(struct bio *bio, const struct iov_iter *iter) bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_idx =0 ; - bio->bi_iter.bi_bvec_done = iter->iov_offset; + bio->bi_iter.bi_offset = iter->iov_offset; bio->bi_iter.bi_size =iov_iter_count (iter);bio_set_flag (bio,BIO_CLONED ); }
block/blk-merge.c +4 -4
@@ -33,7 +33,7 @@ static inline void bio_get_last_bvec(struct bio *bio, struct bio_vec *bv)bio_advance_iter (bio, &iter, iter.bi_size); -if (!iter.bi_bvec_done) +if (!iter.bi_offset) idx = iter.bi_idx -1 ;else /* in the middle of bvec */ idx = iter.bi_idx;
@@ -41,11 +41,11 @@ static inline void bio_get_last_bvec(struct bio *bio, struct bio_vec *bv) *bv = bio->bi_io_vec[idx];/* -* iter.bi_bvec_done records actual length of the last bvec +* iter.bi_offset records actual length of the last bvec * if this bio ends in the middle of one io vector */ -if (iter.bi_bvec_done) - bv->bv_len = iter.bi_bvec_done; +if (iter.bi_offset) + bv->bv_len = iter.bi_offset; } static inline bool bio_will_gap(struct request_queue *q,
block/blk-mq-dma.c +1 -1
@@ -44,7 +44,7 @@ static bool blk_map_iter_next(struct request *req, struct blk_map_iter *iter,* one could be merged into it. This typically happens when moving to * the next bio, but some callers also don't pack bvecs tight. */ -while (!iter->iter.bi_size || !iter->iter.bi_bvec_done) { +while (!iter->iter.bi_size || !iter->iter.bi_offset) {struct bio_vec next;if (!__blk_map_iter_next (iter))
block/blk.h +1 -1
@@ -406,7 +406,7 @@ static inline bool bio_may_need_split(struct bio *bio,return true; bv =__bvec_iter_bvec (bio->bi_io_vec, bio->bi_iter); -if (bio->bi_iter.bi_size > bv->bv_len - bio->bi_iter.bi_bvec_done) +if (bio->bi_iter.bi_size > bv->bv_len - bio->bi_iter.bi_offset)return true;if ((bv->bv_offset | bv->bv_len) & lim->dma_alignment)return true;
drivers/block/loop.c +1 -1
@@ -379,7 +379,7 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd,iov_iter_bvec (&iter, rw,__bvec_iter_bvec (rq->bio->bi_io_vec, rq->bio->bi_iter), nr_bvec,blk_rq_bytes (rq)); - iter.iov_offset = rq->bio->bi_iter.bi_bvec_done; + iter.iov_offset = rq->bio->bi_iter.bi_offset; }atomic_set (&cmd->ref,2 );
drivers/block/zloop.c +1 -1
@@ -554,7 +554,7 @@ static int zloop_do_rw(struct zloop_cmd *cmd)iov_iter_bvec (&iter, rw,__bvec_iter_bvec (rq->bio->bi_io_vec, rq->bio->bi_iter), nr_bvec,blk_rq_bytes (rq)); - iter.iov_offset = rq->bio->bi_iter.bi_bvec_done; + iter.iov_offset = rq->bio->bi_iter.bi_offset; } cmd->iocb.ki_pos = (cmd->sector - zone->start) <<SECTOR_SHIFT ;
drivers/md/dm-io-rewind.c +5 -5
@@ -16,12 +16,12 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec *bv,int idx; iter->bi_size += bytes; -if (bytes <= iter->bi_bvec_done) { - iter->bi_bvec_done -= bytes; +if (bytes <= iter->bi_offset) { + iter->bi_offset -= bytes;return true; } - bytes -= iter->bi_bvec_done; + bytes -= iter->bi_offset; idx = iter->bi_idx -1 ;while (idx >=0 && bytes && bytes > bv[idx].bv_len) {
@@ -32,13 +32,13 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec *bv,if (WARN_ONCE (idx <0 && bytes,"Attempted to rewind iter beyond bvec's boundaries\n" )) { iter->bi_size -= bytes; - iter->bi_bvec_done =0 ; + iter->bi_offset =0 ; iter->bi_idx =0 ;return false; } iter->bi_idx = idx; - iter->bi_bvec_done = bv[idx].bv_len - bytes; + iter->bi_offset = bv[idx].bv_len - bytes;return true; }
drivers/md/dm-pcache/segment.c +2 -2
@@ -14,7 +14,7 @@ int segment_copy_to_bio(struct pcache_segment *segment,iov_iter_bvec (&iter,ITER_DEST , &bio->bi_io_vec[bio->bi_iter.bi_idx],bio_segments (bio), bio->bi_iter.bi_size); - iter.iov_offset = bio->bi_iter.bi_bvec_done; + iter.iov_offset = bio->bi_iter.bi_offset;if (bio_off)iov_iter_advance (&iter, bio_off);
@@ -35,7 +35,7 @@ int segment_copy_from_bio(struct pcache_segment *segment,iov_iter_bvec (&iter,ITER_SOURCE , &bio->bi_io_vec[bio->bi_iter.bi_idx],bio_segments (bio), bio->bi_iter.bi_size); - iter.iov_offset = bio->bi_iter.bi_bvec_done; + iter.iov_offset = bio->bi_iter.bi_offset;if (bio_off)iov_iter_advance (&iter, bio_off);
drivers/nvdimm/btt.c +1 -1
@@ -1155,7 +1155,7 @@ static int btt_rw_integrity(struct btt *btt, struct bio_integrity_payload *bip, bv =bvec_iter_bvec (bip->bip_vec, bip->bip_iter);/* * The 'bv' obtained from bvec_iter_bvec has its .bv_len and -* .bv_offset already adjusted for iter->bi_bvec_done, and we +* .bv_offset already adjusted for iter->bi_offset, and we * can use those directly */
drivers/nvme/host/tcp.c +1 -1
@@ -357,7 +357,7 @@ static void nvme_tcp_init_iter(struct nvme_tcp_request *req,iov_iter_bvec (&req->iter, dir,__bvec_iter_bvec (bio->bi_io_vec, bio->bi_iter), nr_bvec, bio->bi_iter.bi_size); - req->iter.iov_offset = bio->bi_iter.bi_bvec_done; + req->iter.iov_offset = bio->bi_iter.bi_offset; } }
fs/btrfs/misc.h +1 -1
@@ -74,7 +74,7 @@ static inline struct bvec_iter init_bvec_iter_for_bio(struct bio *bio) .bi_sector = 0, .bi_size = bio_size, .bi_idx = 0, - .bi_bvec_done = 0, + .bi_offset = 0, }; }
include/linux/bvec.h +7 -7
@@ -110,7 +110,7 @@ struct bvec_iter {/* * Current offset in the bvec entry pointed to by `bi_idx`. */ -unsigned int bi_bvec_done; +unsigned int bi_offset; } __packed__aligned(4) ;struct bvec_iter_all {
@@ -135,14 +135,14 @@ mp_bvec_iter_page(const struct bio_vec *bvecs, const struct bvec_iter iter) static __always_inline unsigned intmp_bvec_iter_len (const struct bio_vec *bvecs,const struct bvec_iter iter) { -return min (__bvec_iter_bvec (bvecs, iter)->bv_len - iter.bi_bvec_done, +return min (__bvec_iter_bvec (bvecs, iter)->bv_len - iter.bi_offset, iter.bi_size); } static __always_inline unsigned intmp_bvec_iter_offset (const struct bio_vec *bvecs,const struct bvec_iter iter) { -return __bvec_iter_bvec (bvecs, iter)->bv_offset + iter.bi_bvec_done; +return __bvec_iter_bvec (bvecs, iter)->bv_offset + iter.bi_offset; }static __always_inline unsigned int
@@ -204,7 +204,7 @@ static inline bool bvec_iter_advance(const struct bio_vec *bv, } iter->bi_size -= bytes; - bytes += iter->bi_bvec_done; + bytes += iter->bi_offset;while (bytes && bytes >= bv[idx].bv_len) { bytes -= bv[idx].bv_len;
@@ -212,7 +212,7 @@ static inline bool bvec_iter_advance(const struct bio_vec *bv, } iter->bi_idx = idx; - iter->bi_bvec_done = bytes; + iter->bi_offset = bytes;return true; }
@@ -223,13 +223,13 @@ static inline bool bvec_iter_advance(const struct bio_vec *bv,static inline void bvec_iter_advance_single (const struct bio_vec *bv,struct bvec_iter *iter,unsigned int bytes) { -unsigned int done = iter->bi_bvec_done + bytes; +unsigned int done = iter->bi_offset + bytes;if (done == bv[iter->bi_idx].bv_len) { done =0 ; iter->bi_idx++; } - iter->bi_bvec_done = done; + iter->bi_offset = done; iter->bi_size -= bytes; }
io_uring/net.c +2 -2
@@ -1471,7 +1471,7 @@ static int io_sg_from_iter(struct sk_buff *skb,return zerocopy_fill_skb_from_iter (skb, from, length); bi.bi_size =min (from->count, length); - bi.bi_bvec_done = from->iov_offset; + bi.bi_offset = from->iov_offset; bi.bi_idx =0 ;while (bi.bi_size && frag <MAX_SKB_FRAGS ) {
@@ -1490,7 +1490,7 @@ static int io_sg_from_iter(struct sk_buff *skb, from->bvec += bi.bi_idx; from->nr_segs -= bi.bi_idx; from->count -= copied; - from->iov_offset = bi.bi_bvec_done; + from->iov_offset = bi.bi_offset; skb->data_len += copied; skb->len += copied;
lib/iov_iter.c +1 -1
@@ -1651,7 +1651,7 @@ static ssize_t iov_iter_extract_bvec_pages(struct iov_iter *i, } bi.bi_idx =0 ; bi.bi_size = maxsize; - bi.bi_bvec_done = skip; + bi.bi_offset = skip; maxpages =want_pages_array (pages, maxsize, skip, maxpages);if (!maxpages)
net/ceph/messenger.c +2 -2
@@ -762,7 +762,7 @@ static bool ceph_msg_data_bio_advance(struct ceph_msg_data_cursor *cursor,if (!cursor->resid)return false;/* no more data */ -if (!bytes || (it->iter.bi_size && it->iter.bi_bvec_done && +if (!bytes || (it->iter.bi_size && it->iter.bi_offset && page ==bio_iter_page (it->bio, it->iter)))return false;/* more bytes to process in this segment */
@@ -817,7 +817,7 @@ static bool ceph_msg_data_bvecs_advance(struct ceph_msg_data_cursor *cursor,if (!cursor->resid)return false;/* no more data */ -if (!bytes || (cursor->bvec_iter.bi_bvec_done && +if (!bytes || (cursor->bvec_iter.bi_offset && page ==bvec_iter_page (bvecs, cursor->bvec_iter)))return false;/* more bytes to process in this segment */
Signature · 1 line
2.54.0
PATCH v5 4/16 block: always adjust bi_offset on bio_advance_iter
In reply to Pavel BegunkovExtend bio_no_advance_iter() iteration to also increment the offset. It's not currently needed because the currently listed request types don't have a buffer, but will be useful once we add bios backed by dma-buf, which don't have a bvec but work with a single range.
--- drivers/md/dm-io-rewind.c | 6 ++++-- include/linux/bio.h | 12 ++++++++---- 2 files changed, 12 insertions(+), 6 deletions(-)
drivers/md/dm-io-rewind.c +4 -2
@@ -113,10 +113,12 @@ static inline void dm_bio_rewind_iter(const struct bio *bio, iter->bi_sector -= bytes >>9 ;/* No advance means no rewind */ -if (bio_no_advance_iter (bio)) +if (bio_no_advance_iter (bio)) { iter->bi_size += bytes; -else + iter->bi_offset -= bytes; + }else {dm_bvec_iter_rewind (bio->bi_io_vec, iter, bytes); + } }/**
include/linux/bio.h +8 -4
@@ -113,11 +113,13 @@ static inline void bio_advance_iter(const struct bio *bio, { iter->bi_sector += bytes >>9 ; -if (bio_no_advance_iter (bio)) +if (bio_no_advance_iter (bio)) { iter->bi_size -= bytes; -else + iter->bi_offset += bytes; + }else {bvec_iter_advance (bio->bi_io_vec, iter, bytes);/* TODO: It is reasonable to complete bio with error here. */ + } }/* @bytes should be less or equal to bvec[i->bi_idx].bv_len */
@@ -127,10 +129,12 @@ static inline void bio_advance_iter_single(const struct bio *bio, { iter->bi_sector += bytes >>9 ; -if (bio_no_advance_iter (bio)) +if (bio_no_advance_iter (bio)) { iter->bi_size -= bytes; -else + iter->bi_offset += bytes; + }else {bvec_iter_advance_single (bio->bi_io_vec, iter, bytes); + } }void __bio_advance (struct bio *,unsigned bytes);
Signature · 1 line
2.54.0
PATCH v5 5/16 block: move bvec init into __bio_clone
In reply to Pavel BegunkovConsolidate bi_io_vec assignment for cloning in __bio_clone to keep any further changes in one place.
--- block/bio.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
block/bio.c +2 -3
@@ -860,6 +860,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp) bio->bi_write_hint = bio_src->bi_write_hint; bio->bi_write_stream = bio_src->bi_write_stream; bio->bi_iter = bio_src->bi_iter; + bio->bi_io_vec = bio_src->bi_io_vec;if (bio->bi_bdev) { if (bio->bi_bdev == bio_src->bi_bdev &&
@@ -902,8 +903,6 @@ struct bio *bio_alloc_clone(struct block_device *bdev, struct bio *bio_src,bio_put (bio);return NULL ; } - bio->bi_io_vec = bio_src->bi_io_vec; -return bio; }EXPORT_SYMBOL (bio_alloc_clone);
@@ -923,7 +922,7 @@ int bio_init_clone(struct block_device *bdev, struct bio *bio, {int ret; -bio_init (bio, bdev, bio_src->bi_io_vec,0 , bio_src->bi_opf); +bio_init (bio, bdev,NULL ,0 , bio_src->bi_opf); ret =__bio_clone (bio, bio_src, gfp);if (ret)bio_uninit (bio);
Signature · 1 line
2.54.0
PATCH v5 6/16 block: introduce bio_iov_iter_set()
In reply to Pavel BegunkovIn preparation to supporting dma-buf backed iterators and bios, introduce bio_iov_iter_set() which attempts to set up the bio directly from the given iterator. For now, it only supports bvec and expects users to check the result and fall back to other means if fails, but later we'll add more types.
--- block/bio.c | 13 ++++++++----- block/blk-map.c | 2 +- block/fops.c | 16 +++++++--------- include/linux/bio.h | 2 +- 4 files changed, 17 insertions(+), 16 deletions(-)
block/bio.c +8 -5
@@ -1181,8 +1181,11 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty) }EXPORT_SYMBOL_GPL (__bio_release_pages); -void bio_iov_bvec_set (struct bio *bio,const struct iov_iter *iter) +bool bio_iov_iter_set (struct bio *bio,const struct iov_iter *iter) { +if (!iov_iter_is_bvec (iter)) +return false; +WARN_ON_ONCE (bio->bi_max_vecs); bio->bi_io_vec = (struct bio_vec *)iter->bvec;
@@ -1190,6 +1193,7 @@ void bio_iov_bvec_set(struct bio *bio, const struct iov_iter *iter) bio->bi_iter.bi_offset = iter->iov_offset; bio->bi_iter.bi_size =iov_iter_count (iter);bio_set_flag (bio,BIO_CLONED ); +return true; }/*
@@ -1284,10 +1288,9 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,if (WARN_ON_ONCE (bio_flagged (bio,BIO_CLONED )))return -EIO ; -if (iov_iter_is_bvec (iter)) { -bio_iov_bvec_set (bio, iter); - -if (!bio_iov_bvec_aligned (bio, mem_align_mask)) +if (bio_iov_iter_set (bio, iter)) { +if (iov_iter_is_bvec (iter) && + !bio_iov_bvec_aligned (bio, mem_align_mask))return -EINVAL ;iov_iter_advance (iter, bio->bi_iter.bi_size);
block/blk-map.c +1 -1
@@ -473,7 +473,7 @@ static int blk_rq_map_user_bvec(struct request *rq, const struct iov_iter *iter) bio =blk_rq_map_bio_alloc (rq,0 ,GFP_KERNEL );if (!bio)return -ENOMEM ; -bio_iov_bvec_set (bio, iter); +bio_iov_iter_set (bio, iter); ret =blk_rq_append_bio (rq, bio);if (ret)
block/fops.c +7 -9
@@ -342,15 +342,13 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, bio->bi_end_io = blkdev_bio_end_io_async; bio->bi_ioprio = iocb->ki_ioprio; -if (iov_iter_is_bvec (iter)) { -/* -* Users don't rely on the iterator being in any particular -* state for async I/O returning -EIOCBQUEUED, hence we can -* avoid expensive iov_iter_advance(). Bypass -* bio_iov_iter_get_pages() and set the bvec directly. -*/ -bio_iov_bvec_set (bio, iter); - }else { +/* +* Users don't rely on the iterator being in any particular +* state for async I/O returning -EIOCBQUEUED, hence we can +* avoid expensive iov_iter_advance(). Bypass +* bio_iov_iter_get_pages() and set the bvec directly. +*/ +if (!bio_iov_iter_set (bio, iter)) { ret =blkdev_iov_iter_get_pages (bio, iter, bdev);if (unlikely (ret)) goto out_bio_put;
include/linux/bio.h +1 -1
@@ -522,7 +522,7 @@ int bdev_rw_virt(struct block_device *bdev, sector_t sector, void *data,int bio_iov_iter_get_pages (struct bio *bio,struct iov_iter *iter,unsigned mem_align_mask,unsigned len_align_mask); -void bio_iov_bvec_set (struct bio *bio,const struct iov_iter *iter); +bool bio_iov_iter_set (struct bio *bio,const struct iov_iter *iter);void __bio_release_pages (struct bio *bio,bool mark_dirty);extern void bio_set_pages_dirty (struct bio *bio);extern void bio_check_pages_dirty (struct bio *bio);
Signature · 1 line
2.54.0
Re: [PATCH v5 06/16] block: introduce bio_iov_iter_set()
Looks good:
PATCH v5 7/16 block: introduce dma map backed bio type
In reply to Pavel BegunkovPremapped buffers don't require a generic bio_vec since these have already been dma mapped. Repurpose the bi_io_vec space to strore dmabuf maps as they are mutually exclusive.
--- block/bio.c | 15 +++++++++++++-- block/blk-merge.c | 37 +++++++++++++++++++++++++++++++++++++ block/fops.c | 2 +- include/linux/bio.h | 9 +++++---- include/linux/blk-mq.h | 7 +++++++ include/linux/blk_types.h | 14 +++++++++++++- include/linux/bvec.h | 3 ++- 7 files changed, 78 insertions(+), 9 deletions(-)
block/bio.c +13 -2
@@ -860,7 +860,11 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp) bio->bi_write_hint = bio_src->bi_write_hint; bio->bi_write_stream = bio_src->bi_write_stream; bio->bi_iter = bio_src->bi_iter; - bio->bi_io_vec = bio_src->bi_io_vec; + +if (op_is_dmabuf (bio->bi_opf)) + bio->bi_dmabuf_map = bio_src->bi_dmabuf_map; +else + bio->bi_io_vec = bio_src->bi_io_vec;if (bio->bi_bdev) { if (bio->bi_bdev == bio_src->bi_bdev &&
@@ -1183,16 +1187,23 @@ EXPORT_SYMBOL_GPL(__bio_release_pages);bool bio_iov_iter_set (struct bio *bio,const struct iov_iter *iter) { -if (!iov_iter_is_bvec (iter)) +if (!iov_iter_is_bvec (iter) && !iov_iter_is_dmabuf_map (iter))return false;WARN_ON_ONCE (bio->bi_max_vecs); +static_assert (offsetof(struct bio , bi_io_vec) == + offsetof(struct bio , bi_dmabuf_map)); +static_assert (offsetof(struct iov_iter , bvec) == + offsetof(struct iov_iter , dmabuf_map)); + bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_idx =0 ; bio->bi_iter.bi_offset = iter->iov_offset; bio->bi_iter.bi_size =iov_iter_count (iter);bio_set_flag (bio,BIO_CLONED ); +if (iov_iter_is_dmabuf_map (iter)) + bio->bi_opf |=REQ_NOMERGE |REQ_DMABUF ;return true; }
block/blk-merge.c +37 -0
@@ -9,6 +9,7 @@#include <linux/blk-integrity.h> #include <linux/part_stat.h> #include <linux/blk-cgroup.h> +#include <linux/dma-buf-io.h> #include <trace/events/block.h>
@@ -319,6 +320,28 @@ static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv,return bv->bv_offset | (bvprv->bv_offset + bvprv->bv_len); } +static inline int bio_split_io_at_dmabuf (struct bio *bio, +const struct queue_limits *lim,unsigned *segs, +unsigned max_bytes,unsigned len_align_mask, +unsigned start_align_mask) +{ +unsigned bytes =min (bio->bi_iter.bi_size, max_bytes); +unsigned seg_shift = bio->bi_dmabuf_map->seg_shift; +unsigned offset = bio->bi_iter.bi_offset & ((1U << seg_shift) -1 ); + +if ((bio->bi_iter.bi_offset & start_align_mask) || + (bio->bi_iter.bi_size & len_align_mask)) +return -EINVAL ; + +/* single contiguous range into the dma-buf */ + *segs =1 ; + + bytes =min (bytes, ((unsigned )lim->max_segments << seg_shift) - offset); +if (bytes != bio->bi_iter.bi_size) +return bytes; +return 0 ; +} +/** * bio_split_io_at - check if and where to split a bio * @bio: [in] bio to be split
@@ -346,6 +369,19 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim, len_align_mask |= (bc->bc_key->crypto_cfg.data_unit_size -1 ); } +if (op_is_dmabuf (bio->bi_opf)) { +int ret; + + ret =bio_split_io_at_dmabuf (bio, lim, &nsegs, max_bytes, + len_align_mask, start_align_mask); +if (ret <0 ) +return ret; +if (!ret) + goto out; + bytes = ret; + goto split; + } +bio_for_each_bvec (bv, bio, iter) {if (bv.bv_offset & start_align_mask || bv.bv_len & len_align_mask)
@@ -376,6 +412,7 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim, bvprvp = &bvprv; } +out: *segs = nsegs; bio->bi_bvec_gap_bit =ffs (gaps);return 0 ;
block/fops.c +1 -1
@@ -346,7 +346,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,* Users don't rely on the iterator being in any particular * state for async I/O returning -EIOCBQUEUED, hence we can * avoid expensive iov_iter_advance(). Bypass -* bio_iov_iter_get_pages() and set the bvec directly. +* bio_iov_iter_get_pages() and set the bvec/dmabuf directly. */ if (!bio_iov_iter_set (bio, iter)) { ret =blkdev_iov_iter_get_pages (bio, iter, bdev);
include/linux/bio.h +5 -4
@@ -80,7 +80,8 @@ static inline bool bio_no_advance_iter(const struct bio *bio) {return bio_op (bio) ==REQ_OP_DISCARD ||bio_op (bio) ==REQ_OP_SECURE_ERASE || -bio_op (bio) ==REQ_OP_WRITE_ZEROES ; +bio_op (bio) ==REQ_OP_WRITE_ZEROES || +op_is_dmabuf (bio->bi_opf); }static inline void *bio_data (struct bio *bio)
@@ -438,12 +439,12 @@ static inline void bio_wouldblock_error(struct bio *bio)/* * Calculate number of bvec segments that should be allocated to fit data -* pointed by @iter. If @iter is backed by bvec it's going to be reused -* instead of allocating a new one. +* pointed by @iter. If @iter is backed by a bvec or a dmabuf, the bvec array / +* the dma map are going to be reused, and so no extra allocation is required. */ static inline int bio_iov_vecs_to_alloc (struct iov_iter *iter,int max_segs) { -if (iov_iter_is_bvec (iter)) +if (iov_iter_is_bvec (iter) ||iov_iter_is_dmabuf_map (iter))return 0 ;return iov_iter_npages (iter, max_segs); }
include/linux/blk-mq.h +7 -0
@@ -1017,6 +1017,13 @@ static inline void *blk_mq_rq_to_pdu(struct request *rq)return rq +1 ; } +static inline bool blk_mq_rq_is_dmabuf (struct request *rq) +{ +if (!IS_ENABLED (CONFIG_DMA_SHARED_BUFFER )) +return false; +return rq->bio &&op_is_dmabuf (rq->bio->bi_opf); +} +static inline struct blk_mq_hw_ctx *queue_hctx (struct request_queue *q,int id) {struct blk_mq_hw_ctx *hctx;
include/linux/blk_types.h +13 -1
@@ -233,7 +233,12 @@ struct bio {atomic_t __bi_remaining;/* The actual vec list, preserved by bio_reset() */ -struct bio_vec *bi_io_vec; +union { +struct bio_vec *bi_io_vec; +/* Driver specific dma map, valid IFF REQ_DMABUF is set */ +struct dma_buf_io_map *bi_dmabuf_map; + }; +struct bvec_iter bi_iter;union {
@@ -402,6 +407,7 @@ enum req_flag_bits { __REQ_DRV,/* for driver use */ __REQ_FS_PRIVATE,/* for file system (submitter) use */ __REQ_ATOMIC,/* for atomic write operations */ + __REQ_DMABUF, /* Using premmaped dma buffers *//* * Command specific flags, keep last: */
@@ -434,6 +440,7 @@ enum req_flag_bits {#define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV)#define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE)#define REQ_ATOMIC (__force blk_opf_t)(1ULL << __REQ_ATOMIC) +#define REQ_DMABUF (__force blk_opf_t)(1ULL << __REQ_DMABUF)#define REQ_NOUNMAP (__force blk_opf_t)(1ULL << __REQ_NOUNMAP)
@@ -487,6 +494,11 @@ static inline bool op_is_discard(blk_opf_t op)return (op &REQ_OP_MASK ) ==REQ_OP_DISCARD ; } +static inline bool op_is_dmabuf (blk_opf_t op) +{ +return op &REQ_DMABUF ; +} +/* * Check if a bio or request operation is a zone management operation. */
include/linux/bvec.h +2 -1
@@ -108,7 +108,8 @@ struct bvec_iter {unsigned int bi_idx;/* -* Current offset in the bvec entry pointed to by `bi_idx`. +* Current offset in the bvec entry pointed to by `bi_idx` or into +* a dma-buf map. */ unsigned int bi_offset; } __packed__aligned(4) ;
Signature · 1 line
2.54.0
Christoph Hellwig
maintainerRe: [PATCH v5 07/16] block: introduce dma map backed bio type
Quoted 81 lines · Pavel Begunkov
Premapped buffers don't require a generic
bio_vecsince these have already been dma mapped. Repurpose thebi_io_vecspace to strore dmabuf maps as they are mutually exclusive.Suggested-by Keith Busch <[email protected]>Signed-off-by Pavel Begunkov <[email protected]>--- block/bio.c | 15 +++++++++++++-- block/blk-merge.c | 37 +++++++++++++++++++++++++++++++++++++ block/fops.c | 2 +- include/linux/bio.h | 9 +++++---- include/linux/blk-mq.h | 7 +++++++ include/linux/blk_types.h | 14 +++++++++++++- include/linux/bvec.h | 3 ++- 7 files changed, 78 insertions(+), 9 deletions(-)block/bio.c +13
-2@@ -860,7 +860,11 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp) bio->bi_write_hint = bio_src->bi_write_hint; bio->bi_write_stream = bio_src->bi_write_stream; bio->bi_iter = bio_src->bi_iter; - bio->bi_io_vec = bio_src->bi_io_vec; + +if (op_is_dmabuf (bio->bi_opf)) + bio->bi_dmabuf_map = bio_src->bi_dmabuf_map; +else + bio->bi_io_vec = bio_src->bi_io_vec;if (bio->bi_bdev) { if (bio->bi_bdev == bio_src->bi_bdev &&@@ -1183,16 +1187,23 @@ EXPORT_SYMBOL_GPL(__bio_release_pages);bool bio_iov_iter_set (struct bio *bio,const struct iov_iter *iter) { -if (!iov_iter_is_bvec (iter)) +if (!iov_iter_is_bvec (iter) && !iov_iter_is_dmabuf_map (iter))return false;WARN_ON_ONCE (bio->bi_max_vecs); +static_assert (offsetof(struct bio , bi_io_vec) == + offsetof(struct bio , bi_dmabuf_map)); +static_assert (offsetof(struct iov_iter , bvec) == + offsetof(struct iov_iter , dmabuf_map)); + bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_idx =0 ; bio->bi_iter.bi_offset = iter->iov_offset; bio->bi_iter.bi_size =iov_iter_count (iter);bio_set_flag (bio,BIO_CLONED ); +if (iov_iter_is_dmabuf_map (iter)) + bio->bi_opf |=REQ_NOMERGE |REQ_DMABUF ;return true; }block/blk-merge.c +8
-0@@ -9,6 +9,7 @@#include <linux/blk-integrity.h> #include <linux/part_stat.h> #include <linux/blk-cgroup.h> +#include <linux/dma-buf-io.h> #include <trace/events/block.h> @@ -319,6 +320,28 @@ static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv,return bv->bv_offset | (bvprv->bv_offset + bvprv->bv_len); } +static inline int bio_split_io_at_dmabuf (struct bio *bio, +const struct queue_limits *lim,unsigned *segs, +unsigned max_bytes,unsigned len_align_mask, +unsigned start_align_mask) +{ +unsigned bytes =min (bio->bi_iter.bi_size, max_bytes); +unsigned seg_shift = bio->bi_dmabuf_map->seg_shift;
block/blk-merge.c static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv, · line 339
Quoted 10 lines
block/blk-merge.c +10
-0+unsigned offset = bio->bi_iter.bi_offset & ((1U << seg_shift) -1 ); + +if ((bio->bi_iter.bi_offset & start_align_mask) || + (bio->bi_iter.bi_size & len_align_mask)) +return -EINVAL ; + +/* single contiguous range into the dma-buf */ + *segs =1 ; + + bytes =min (bytes, ((unsigned )lim->max_segments << seg_shift) - offset);
I guess ->seg_shift is some sort of encoding of a max segment size?
Comments please.
Re: [PATCH v5 07/16] block: introduce dma map backed bio type
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Pavel Begunkov
Premapped buffers don't require a generic
bio_vecsince these have already been dma mapped. Repurpose thebi_io_vecspace to strore dmabuf maps as they are mutually exclusive.Suggested-by Keith Busch <[email protected]>Signed-off-by Pavel Begunkov <[email protected]>---
...>>
Quoted 12 lines
Quoted 12 lines
block/blk-merge.c +12
-0+static inline int bio_split_io_at_dmabuf (struct bio *bio, +const struct queue_limits *lim,unsigned *segs, +unsigned max_bytes,unsigned len_align_mask, +unsigned start_align_mask) +{ +unsigned bytes =min (bio->bi_iter.bi_size, max_bytes); +unsigned seg_shift = bio->bi_dmabuf_map->seg_shift; +unsigned offset = bio->bi_iter.bi_offset & ((1U << seg_shift) -1 ); + +if ((bio->bi_iter.bi_offset & start_align_mask) || + (bio->bi_iter.bi_size & len_align_mask)) +return -EINVAL ;
Quoted 8 lines
block/blk-merge.c static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv, · line 339
Quoted 5 lines
block/blk-merge.c +5
-0+ +/* single contiguous range into the dma-buf */ + *segs =1 ; + + bytes =min (bytes, ((unsigned )lim->max_segments << seg_shift) - offset);I guess ->seg_shift is some sort of encoding of a max segment size?
I'll add a comment.
We rather need the minimum segment size, you can always split large ones. I calculated it even stricter as the least common multiple pow2 to avoid divs here.
If the device supports N segments and we know that each segment is at least M bytes, then we should be able to issue IO of size N*M of full segments. The line above truncates the bio size using that + offset adjustments.
I guess it might be more straightforward to calculate the worst case number of segments and then adjust the splitting size, but since we don't return the number of segments it's more computations. E.g.
seg_size = 1U << seg_shift;
nsegs = (bio->bi_iter.bi_size + offset + seg_size - 1) / seg_size;
if (nsegs > lim->max_segments) {
nsegs = lim->max_segments; // unused after the block
bytes = seg_size * nsegs - offset;
}Signature · 1 line
Pavel Begunkov
PATCH v5 8/16 block: add dma-buf support for raw bdev
In reply to Pavel BegunkovAdd a simple proxy implementation of init_dma_buf_io_ctx() forwarding the call to a new struct block_device_operations operation. Also reject dma-buf backed iterators for buffered IO.
--- block/fops.c | 19 +++++++++++++++++++ include/linux/blkdev.h | 2 ++ 2 files changed, 21 insertions(+)
block/fops.c +19 -0
@@ -767,6 +767,9 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from) ret =direct_write_fallback (iocb, from, ret,blkdev_buffered_write (iocb, from)); }else { +if (unlikely (iov_iter_is_dmabuf_map (from))) +return -EOPNOTSUPP ; +/* * Take i_rwsem and invalidate_lock to avoid racing with * set_blocksize changing i_blkbits/folio order and punching
@@ -821,6 +824,8 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to)if (ret <0 || !count) goto reexpand; } +if (unlikely (iov_iter_is_dmabuf_map (to))) +return -EOPNOTSUPP ;/* * Take i_rwsem and invalidate_lock to avoid racing with set_blocksize
@@ -924,6 +929,19 @@ static int blkdev_mmap_prepare(struct vm_area_desc *desc)return generic_file_mmap_prepare (desc); } +static int blkdev_init_dma_buf_io_ctx (struct file *file, +struct dma_buf_io_ctx *ctx) +{ +struct block_device *bdev =file_bdev (file); +struct gendisk *disk = bdev->bd_disk; + +if (!(file->f_flags &O_DIRECT )) +return -EINVAL ; +if (!disk->fops->init_dma_buf_io_ctx) +return -EINVAL ; +return disk->fops->init_dma_buf_io_ctx (bdev, ctx); +} +const struct file_operations def_blk_fops = { .open = blkdev_open, .release = blkdev_release,
@@ -942,6 +960,7 @@ const struct file_operations def_blk_fops = { .fallocate = blkdev_fallocate, .uring_cmd = blkdev_uring_cmd, .fop_flags = FOP_BUFFER_RASYNC | FOP_DONTCACHE, + .init_dma_buf_io_ctx = blkdev_init_dma_buf_io_ctx, };static __init int blkdev_init (void )
include/linux/blkdev.h +2 -0
@@ -1682,6 +1682,8 @@ struct block_device_operations {/* returns the length of the identifier or a negative errno: */ int (*get_unique_id)(struct gendisk *disk,u8 id[16 ],enum blk_unique_id id_type); +int (*init_dma_buf_io_ctx)(struct block_device *, +struct dma_buf_io_ctx *);struct module *owner;const struct pr_ops *pr_ops;
Signature · 1 line
2.54.0
PATCH v5 9/16 nvme-pci: implement dma-buf backed requests
In reply to Pavel BegunkovEnable BIO_DMABUF_MAP backed requests. On registration we map the dma-buf and store it as a prp list, which is then used to initialise requests.
--- drivers/nvme/host/core.c | 12 ++ drivers/nvme/host/nvme.h | 2 + drivers/nvme/host/pci.c | 265 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 279 insertions(+)
drivers/nvme/host/core.c +12 -0
@@ -2676,6 +2676,17 @@ static int nvme_report_zones(struct gendisk *disk, sector_t sector,#define nvme_report_zones NULL#endif /* CONFIG_BLK_DEV_ZONED */ +static int nvme_init_dma_buf_io_ctx (struct block_device *bdev, +struct dma_buf_io_ctx *ctx) +{ +struct nvme_ns *ns = bdev->bd_disk->private_data; +struct nvme_ctrl *ctrl = ns->ctrl; + +if (!ctrl->ops->init_dma_buf_io_ctx) +return -EINVAL ; +return ctrl->ops->init_dma_buf_io_ctx (ctrl, ctx); +} +const struct block_device_operations nvme_bdev_ops = { .owner =THIS_MODULE , .ioctl = nvme_ioctl,
@@ -2686,6 +2697,7 @@ const struct block_device_operations nvme_bdev_ops = { .get_unique_id = nvme_get_unique_id, .report_zones = nvme_report_zones, .pr_ops = &nvme_pr_ops, + .init_dma_buf_io_ctx = nvme_init_dma_buf_io_ctx, }; static int nvme_wait_ready(struct nvme_ctrl *ctrl, u32 mask, u32 val,
drivers/nvme/host/nvme.h +2 -0
@@ -652,6 +652,8 @@ struct nvme_ctrl_ops {int (*get_address)(struct nvme_ctrl *ctrl,char *buf,int size);void (*print_device_info)(struct nvme_ctrl *ctrl);bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl); +int (*init_dma_buf_io_ctx)(struct nvme_ctrl *ctrl, +struct dma_buf_io_ctx *ctx);unsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl,bool is_admin); };
drivers/nvme/host/pci.c +265 -0
312 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [PATCH v5 09/16] nvme-pci: implement dma-buf backed requests
Quoted 20 lines · Pavel Begunkov
drivers/nvme/host/pci.c +20
-0+ map =kmalloc_flex (*map, dma_list, nr_entries); +if (!map) +return ERR_PTR (-ENOMEM ); + + sgt =dma_buf_map_attachment (attach, ctx->dir); +if (IS_ERR (sgt)) { + ret =PTR_ERR (sgt); + sgt =NULL ; + goto err; + } + +for_each_sgtable_dma_sg (sgt, sg, tmp) { +dma_addr_t dma_addr =sg_dma_address (sg); +unsigned long sg_len =sg_dma_len (sg); + +if (sg_len %NVME_CTRL_PAGE_SIZE ) { + ret = -EINVAL ; + goto err; + } + seg_shift =min (seg_shift,__ffs (sg_len));
drivers/nvme/host/pci.c static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled) · line 2476
Quoted 10 lines
drivers/nvme/host/pci.c +10
-0+ +while (sg_len) { + map->dma_list[i++] = dma_addr; + dma_addr +=NVME_CTRL_PAGE_SIZE ; + sg_len -=NVME_CTRL_PAGE_SIZE ; + } + } + +if (WARN_ON_ONCE (seg_shift <NVME_CTRL_PAGE_SHIFT )) +return ERR_PTR (-EFAULT );
This should be 'goto err;' instead of a direct return. If this ever triggers, it leaks the map and already established sgt mapping.
Quoted 14 lines
drivers/nvme/host/pci.c +14
-0+ + ret =dma_buf_io_init_map (ctx, &map->base); +if (ret) + goto err; + map->base.seg_shift = seg_shift; + map->nr_entries = nr_entries; + map->sgt = sgt; +return &map->base; +err: +if (sgt) +dma_buf_unmap_attachment (attach, sgt, ctx->dir); +kfree (map); +return ERR_PTR (ret); +}
Re: [PATCH v5 09/16] nvme-pci: implement dma-buf backed requests
Quoted 1 line · Anuj Gupta/Anuj Gupta
On 8/1/2026 9:16 PM, Pavel Begunkov wrote:
...>> + if (WARN_ON_ONCE(seg_shift < NVME_CTRL_PAGE_SHIFT))
Quoted 4 lines
drivers/nvme/host/pci.c static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled) · line 2476
Quoted 1 line
drivers/nvme/host/pci.c +1
-0+return ERR_PTR (-EFAULT );This should be 'goto err;' instead of a direct return. If this ever triggers, it leaks the map and already established sgt mapping.
I can change it as I'm going to respin the series anyway, but it doesn't really matter here as it's a warning and shouldn't happen.
Signature · 1 line
Pavel Begunkov
Christoph Hellwig
maintainerRe: [PATCH v5 09/16] nvme-pci: implement dma-buf backed requests
In reply to Pavel Begunkovdrivers/nvme/host/pci.c static void nvme_free_descriptors(struct request *req) · line 933
Quoted 2 lines
drivers/nvme/host/pci.c +2
-0+static blk_status_t nvme_rq_setup_dmabuf_map (struct request *req, +struct nvme_queue *nvmeq)
Two tab indents for prototype continuations everywhere please.
PATCH v5 10/16 nvme-pci: rename nvme_pci_sgl_set_data to nvme_pci_dma_iter_set_sgl
In reply to Pavel BegunkovFrom: Anuj Gupta <[email protected]>
Rename the blk_dma_iter based nvme_pci_sgl_set_data() to nvme_pci_dma_iter_set_sgl().
--- drivers/nvme/host/pci.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/nvme/host/pci.c +5 -5
@@ -1287,7 +1287,7 @@ static blk_status_t nvme_pci_setup_data_prp(struct request *req,return BLK_STS_IOERR ; } -static void nvme_pci_sgl_set_data (struct nvme_sgl_desc *sge, +static void nvme_pci_dma_iter_set_sgl (struct nvme_sgl_desc *sge,struct blk_dma_iter *iter) { sge->addr =cpu_to_le64 (iter->addr);
@@ -1317,7 +1317,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request *req, iod->cmd.common.flags =NVME_CMD_SGL_METABUF ;if (entries ==1 ||blk_rq_dma_map_coalesce (&iod->dma_state)) { -nvme_pci_sgl_set_data (&iod->cmd.common.dptr.sgl, iter); +nvme_pci_dma_iter_set_sgl (&iod->cmd.common.dptr.sgl, iter); iod->total_len += iter->len;return BLK_STS_OK ; }
@@ -1339,7 +1339,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request *req, iter->status =BLK_STS_IOERR ;break ; } -nvme_pci_sgl_set_data (&sg_list[mapped++], iter); +nvme_pci_dma_iter_set_sgl (&sg_list[mapped++], iter); iod->total_len += iter->len; }while (blk_rq_dma_map_iter_next (req, nvmeq->dev->dev, iter));
@@ -1502,13 +1502,13 @@ static blk_status_t nvme_pci_setup_meta_iter(struct request *req) iod->cmd.common.metadata =cpu_to_le64 (sgl_dma);if (entries ==1 ) { iod->meta_total_len = iter.len; -nvme_pci_sgl_set_data (sg_list, &iter); +nvme_pci_dma_iter_set_sgl (sg_list, &iter);return BLK_STS_OK ; } sgl_dma +=sizeof (*sg_list);do { -nvme_pci_sgl_set_data (&sg_list[++i], &iter); +nvme_pci_dma_iter_set_sgl (&sg_list[++i], &iter); iod->meta_total_len += iter.len; }while (blk_rq_integrity_dma_map_iter_next (req, dev->dev, &iter));
Signature · 1 line
2.54.0
Re: [PATCH v5 10/16] nvme-pci: rename nvme_pci_sgl_set_data to nvme_pci_dma_iter_set_sgl
Looks good:
PATCH v5 11/16 nvme-pci: add SGL support for the dmabuf path
In reply to Pavel BegunkovFrom: Anuj Gupta <[email protected]>
Add SGL support in addition to PRP for dmabuf-backed requests, building the descriptor list from the mapping's sg_table.
--- drivers/nvme/host/pci.c | 153 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 149 insertions(+), 4 deletions(-)
drivers/nvme/host/pci.c +149 -4
@@ -1287,12 +1287,18 @@ static blk_status_t nvme_pci_setup_data_prp(struct request *req,return BLK_STS_IOERR ; } +static void nvme_pci_sgl_set_data (struct nvme_sgl_desc *sge, +dma_addr_t addr,u32 len) +{ + sge->addr =cpu_to_le64 (addr); + sge->length =cpu_to_le32 (len); + sge->type =NVME_SGL_FMT_DATA_DESC <<4 ; +} +static void nvme_pci_dma_iter_set_sgl (struct nvme_sgl_desc *sge,struct blk_dma_iter *iter) { - sge->addr =cpu_to_le64 (iter->addr); - sge->length =cpu_to_le32 (iter->len); - sge->type =NVME_SGL_FMT_DATA_DESC <<4 ; +nvme_pci_sgl_set_data (sge, iter->addr, iter->len); } static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
@@ -1303,6 +1309,145 @@ static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge, sge->type =NVME_SGL_FMT_LAST_SEG_DESC <<4 ; } +static unsigned int nvme_pci_dmabuf_sgl_nents (struct request *req) +{ +struct bio *bio = req->bio; +struct nvme_dmabuf_map *map =to_nvme_dmabuf_map (bio->bi_dmabuf_map); +struct scatterlist *sg; +unsigned long tmp; +size_t offset = bio->bi_iter.bi_offset; +size_t remaining =blk_rq_payload_bytes (req); +unsigned int nents =0 ; + +for_each_sgtable_dma_sg (map->sgt, sg, tmp) { +size_t sg_len =sg_dma_len (sg); + +if (!remaining) +break ; +if (offset >= sg_len) { + offset -= sg_len; +continue ; + } + + sg_len -= offset; + offset =0 ; + +do { +size_t chunk =min (remaining, sg_len); + + nents++; + sg_len -= chunk; + remaining -= chunk; + }while (sg_len && remaining); + } + +if (unlikely (remaining)) +return 0 ; + +return nents; +} + +static blk_status_t nvme_rq_setup_dmabuf_sgl (struct request *req, +struct nvme_queue *nvmeq,unsigned int entries) +{ +struct nvme_iod *iod =blk_mq_rq_to_pdu (req); +struct bio *bio = req->bio; +struct nvme_dmabuf_map *map =to_nvme_dmabuf_map (bio->bi_dmabuf_map); +size_t length =blk_rq_payload_bytes (req); +struct nvme_sgl_desc *sg_list = &iod->cmd.common.dptr.sgl; +bool pooled = entries >1 ; +dma_addr_t sgl_dma =0 ; +unsigned int mapped =0 ; +unsigned long tmp; +struct scatterlist *sg; +size_t offset = bio->bi_iter.bi_offset; +size_t remaining = length; + +if (!entries) +return BLK_STS_IOERR ; + + iod->cmd.common.flags =NVME_CMD_SGL_METABUF ; + iod->total_len = length; + +nvme_dmabuf_map_sync_for_device (nvmeq->dev, req); + +/* entries == 1 fits in the inline descriptor; more needs a pool. */ +if (pooled) { +if (entries <=NVME_SMALL_POOL_SIZE /sizeof (*sg_list)) + iod->flags |=IOD_SMALL_DESCRIPTOR ; + + sg_list =dma_pool_alloc (nvme_dma_pool (nvmeq, iod),GFP_ATOMIC , + &sgl_dma); +if (!sg_list) +return BLK_STS_RESOURCE ; + iod->descriptors[iod->nr_descriptors++] = sg_list; + } + +for_each_sgtable_dma_sg (map->sgt, sg, tmp) { +size_t sg_len =sg_dma_len (sg); +dma_addr_t addr =sg_dma_address (sg); + +if (!remaining) +break ; +if (offset >= sg_len) { + offset -= sg_len; +continue ; + } + + addr += offset; + sg_len -= offset; + offset =0 ; + +do { +u32 chunk =min_t (size_t, remaining, sg_len); + +if (WARN_ON_ONCE (mapped == entries)) + goto err_free; +nvme_pci_sgl_set_data (&sg_list[mapped++], addr, chunk); + + addr += chunk; + sg_len -= chunk; + remaining -= chunk; + }while (sg_len && remaining); + } + +if (unlikely (remaining)) + goto err_free; + +if (pooled) +nvme_pci_sgl_set_seg (&iod->cmd.common.dptr.sgl, sgl_dma, + mapped); +return BLK_STS_OK ; +err_free: +if (pooled) { + iod->nr_descriptors--; +dma_pool_free (nvme_dma_pool (nvmeq, iod), sg_list, sgl_dma); + } +return BLK_STS_IOERR ; +} + +static blk_status_t nvme_rq_setup_dmabuf (struct request *req, +struct nvme_queue *nvmeq,enum nvme_use_sgl use_sgl) +{ +unsigned int entries; +size_t avg_seg; + +if (use_sgl ==SGL_UNSUPPORTED ) +return nvme_rq_setup_dmabuf_map (req, nvmeq); + + entries =nvme_pci_dmabuf_sgl_nents (req); + +if (use_sgl ==SGL_FORCED ) +return nvme_rq_setup_dmabuf_sgl (req, nvmeq, entries); + + avg_seg = entries ? +DIV_ROUND_UP (blk_rq_payload_bytes (req), entries) :0 ; +if (sgl_threshold && avg_seg >= sgl_threshold) +return nvme_rq_setup_dmabuf_sgl (req, nvmeq, entries); + +return nvme_rq_setup_dmabuf_map (req, nvmeq); +} +static blk_status_t nvme_pci_setup_data_sgl (struct request *req,struct blk_dma_iter *iter) {
@@ -1398,7 +1543,7 @@ static blk_status_t nvme_map_data(struct request *req)blk_status_t ret;if (blk_mq_rq_is_dmabuf (req)) -return nvme_rq_setup_dmabuf_map (req, nvmeq); +return nvme_rq_setup_dmabuf (req, nvmeq, use_sgl);/* * Try to skip the DMA iterator for single segment requests, as that
Signature · 1 line
2.54.0
Re: [PATCH v5 11/16] nvme-pci: add SGL support for the dmabuf path
Looks good:
PATCH v5 12/16 io_uring/rsrc: introduce buf registration structure
In reply to Pavel BegunkovIn preparation to following changes, instead of passing an iovec for buffer registration introduce a new structure. It'll be moved to uapi later, but for now it's initialised early from a user provided iovec.
--- io_uring/rsrc.c | 47 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 15 deletions(-)
io_uring/rsrc.c +32 -15
@@ -27,8 +27,13 @@ struct io_rsrc_update {u32 offset; }; +struct io_uring_regbuf_desc { +__u64 uaddr; +__u64 size; +}; +static struct io_rsrc_node *io_sqe_buffer_register (struct io_ring_ctx *ctx, -struct iovec *iov); +struct io_uring_regbuf_desc *desc);static int hpage_acct_ref (struct io_ring_ctx *ctx,struct page *hpage,bool *acct_new)
@@ -81,6 +86,15 @@ static bool hpage_acct_unref(struct io_ring_ctx *ctx, struct page *hpage)#define IO_CACHED_BVECS_SEGS 32 +static void io_iov_to_regbuf_desc (const struct iovec *iov, +struct io_uring_regbuf_desc *desc) +{ + *desc = (struct io_uring_regbuf_desc ) { + .uaddr = (u64 )(uintptr_t )iov->iov_base, + .size = iov->iov_len, + }; +} +int __io_account_mem (struct user_struct *user,unsigned long nr_pages) {unsigned long page_limit, cur_pages, new_pages;
@@ -381,6 +395,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,return -EINVAL ;for (done =0 ; done < nr_args; done++) { +struct io_uring_regbuf_desc desc;struct io_rsrc_node *node;u64 tag =0 ;
@@ -394,7 +409,9 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, err = -EFAULT ;break ; } - node =io_sqe_buffer_register (ctx, iov); + +io_iov_to_regbuf_desc (iov, &desc); + node =io_sqe_buffer_register (ctx, &desc);if (IS_ERR (node)) { err =PTR_ERR (node);break ;
@@ -853,26 +870,26 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages, }static struct io_rsrc_node *io_sqe_buffer_register (struct io_ring_ctx *ctx, -struct iovec *iov) +struct io_uring_regbuf_desc *desc) { +unsigned long uaddr = (unsigned long )desc->uaddr; +size_t size = desc->size;struct io_mapped_ubuf *imu =NULL ;struct page **pages =NULL ;struct io_rsrc_node *node;unsigned long off; -size_t size;int ret, nr_pages, i;struct io_imu_folio_data data;bool coalesced = false; -if (!iov->iov_base) { -if (iov->iov_len) +if (!uaddr) { +if (size)return ERR_PTR (-EFAULT );/* remove the buffer without installing a new one */ return NULL ; } - ret =io_validate_user_buf_range ((unsigned long )iov->iov_base, - iov->iov_len); + ret =io_validate_user_buf_range (uaddr, size);if (ret)return ERR_PTR (ret);
@@ -881,8 +898,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,return ERR_PTR (-ENOMEM ); ret = -ENOMEM ; - pages =io_pin_pages ((unsigned long ) iov->iov_base, iov->iov_len, - &nr_pages); + pages =io_pin_pages (uaddr, size, &nr_pages);if (IS_ERR (pages)) { ret =PTR_ERR (pages); pages =NULL ;
@@ -904,10 +920,9 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,if (ret) goto done; - size = iov->iov_len;/* store original address for later verification */ - imu->ubuf = (unsigned long ) iov->iov_base; - imu->len = iov->iov_len; + imu->ubuf = uaddr; + imu->len = size; imu->folio_shift =PAGE_SHIFT ; imu->release = io_release_ubuf; imu->priv = imu;
@@ -917,7 +932,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, imu->folio_shift = data.folio_shift;refcount_set (&imu->refs,1 ); - off = (unsigned long )iov->iov_base & ~PAGE_MASK ; + off = uaddr & ~PAGE_MASK ;if (coalesced) off += data.first_folio_page_idx <<PAGE_SHIFT ;
@@ -969,6 +984,7 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,memset (iov,0 ,sizeof (*iov));for (i =0 ; i < nr_args; i++) { +struct io_uring_regbuf_desc desc;struct io_rsrc_node *node;u64 tag =0 ;
@@ -992,7 +1008,8 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, } } - node =io_sqe_buffer_register (ctx, iov); +io_iov_to_regbuf_desc (iov, &desc); + node =io_sqe_buffer_register (ctx, &desc);if (IS_ERR (node)) { ret =PTR_ERR (node);break ;
Signature · 1 line
2.54.0
PATCH v5 13/16 io_uring/rsrc: extend buffer update
In reply to Pavel BegunkovWe need to pass more information to buffer registration than we can fit into a single struct iovec. This patch allows users to optionally pass struct io_uring_regbuf_desc. Apart from having more space for future use cases, it also introduces registration types.
Currently, the type can be either of IO_REGBUF_TYPE_UADDR, which mirrors the iovec path, or IO_REGBUF_TYPE_EMPTY for leaving a buffer table slot empty. The next patch introduces a dmabuf backed type, and can be useful for other extensions like splicing a list of user addresses (i.e. iovec[]), interoperability with zcrx, kernel allocated memory like was brough up by Cristoph. Note, the type only represents a registration option, which is distinct from how io_uring internally stores it.
The flags field is not used yet but always useful to have, e.g. we can encode read-only / write-only restrictions using it.
--- include/uapi/linux/io_uring.h | 27 +++++++++++++- io_uring/rsrc.c | 69 ++++++++++++++++++++++------------- 2 files changed, 69 insertions(+), 27 deletions(-)
include/uapi/linux/io_uring.h +26 -1
@@ -790,13 +790,38 @@ struct io_uring_rsrc_update {struct io_uring_rsrc_update2 {__u32 offset; -__u32 resv; +__u32 flags;__aligned_u64 data;__aligned_u64 tags;__u32 nr;__u32 resv2; }; +/* struct io_uring_rsrc_update2::flags */ +enum io_uring_rsrc_reg_flags { +/* +* Use the extended descriptor format for buffer updates, +* see struct io_uring_regbuf_desc +*/ +IORING_RSRC_UPDATE_EXTENDED =1U <<1 , +}; + +/* Buffer registration type, passed in struct io_uring_regbuf_desc::type */ +enum io_uring_regbuf_type { +IO_REGBUF_TYPE_EMPTY , +IO_REGBUF_TYPE_UADDR , + + __IO_REGBUF_TYPE_MAX, +}; + +struct io_uring_regbuf_desc { +__u32 type;/* enum io_uring_regbuf_type */ +__u32 flags; +__u64 size; +__u64 uaddr; +__u64 __resv[7 ]; +}; +/* Skip updating fd indexes set to this value in the fd table */ #define IORING_REGISTER_FILES_SKIP (-2)
io_uring/rsrc.c +43 -26
151 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH v5 14/16 io_uring/rsrc: add uncloneable regbuf flag
In reply to Pavel BegunkovIt's hard to implement cloning if the internal structure needs to be mutable and/or relies on other ring resources. In preparation to such buffer types, add a flag indicating that the buffer can't be cloned. It might be possible to add cloning in the future for them, but that would likely need reallocating the structure and reacquiring resources in case by case manner.
--- io_uring/rsrc.c | 5 +++++ io_uring/rsrc.h | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-)
io_uring/rsrc.c +5 -0
@@ -1381,6 +1381,11 @@ static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctxif (!src_node) { dst_node =NULL ; }else { +if (src_node->buf->flags &IO_REGBUF_F_UNCLONEABLE ) { +io_rsrc_data_free (ctx, &data); +return -ENOMEM ; + } + dst_node =io_rsrc_node_alloc (ctx,IORING_RSRC_BUFFER );if (!dst_node) {io_rsrc_data_free (ctx, &data);
io_uring/rsrc.h +2 -1
@@ -29,7 +29,8 @@ enum { };enum { -IO_REGBUF_F_KBUF =1 , +IO_REGBUF_F_KBUF =1 <<0 , +IO_REGBUF_F_UNCLONEABLE =1 <<1 , };struct io_mapped_ubuf {
Signature · 1 line
2.54.0
PATCH v5 15/16 io_uring/rsrc: add regbuf import flags
In reply to Pavel BegunkovWe'll have special registered buffer types that can't be used with all opcodes and need special handling. Add separate flags to control registered buffer import, which will be used to specify what kind of buffers the caller can handle.
--- io_uring/rsrc.c | 8 ++++---- io_uring/rsrc.h | 24 ++++++++++++++++++++---- 2 files changed, 24 insertions(+), 8 deletions(-)
io_uring/rsrc.c +4 -4
@@ -1245,9 +1245,9 @@ inline struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,return NULL ; } -int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter, +int __io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter,u64 buf_addr,size_t len,int ddir, -unsigned issue_flags) +unsigned issue_flags,unsigned import_flags) {struct io_rsrc_node *node;
@@ -1656,9 +1656,9 @@ static int io_kern_bvec_size(struct iovec *iov, unsigned nr_iovs,return 0 ; } -int io_import_reg_vec (int ddir,struct iov_iter *iter, +int __io_import_reg_vec (int ddir,struct iov_iter *iter,struct io_kiocb *req,struct iou_vec *vec, -unsigned nr_iovs,unsigned issue_flags) +unsigned nr_iovs,unsigned issue_flags,unsigned import_flags) {struct io_rsrc_node *node;struct io_mapped_ubuf *imu;
io_uring/rsrc.h +20 -4
@@ -65,12 +65,28 @@ int io_rsrc_data_alloc(struct io_rsrc_data *data, unsigned nr);struct io_rsrc_node *io_find_buf_node (struct io_kiocb *req,unsigned issue_flags); -int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter, +int __io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter,u64 buf_addr,size_t len,int ddir, -unsigned issue_flags); -int io_import_reg_vec (int ddir,struct iov_iter *iter, +unsigned issue_flags,unsigned import_flags); +int __io_import_reg_vec (int ddir,struct iov_iter *iter,struct io_kiocb *req,struct iou_vec *vec, -unsigned nr_iovs,unsigned issue_flags); +unsigned nr_iovs,unsigned issue_flags, +unsigned import_flags); + +static inline int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter, +u64 buf_addr,size_t len,int ddir, +unsigned issue_flags) +{ +return __io_import_reg_buf (req, iter, buf_addr, len, ddir, issue_flags,0 ); +} + +static inline int io_import_reg_vec (int ddir,struct iov_iter *iter, +struct io_kiocb *req,struct iou_vec *vec, +unsigned nr_iovs,unsigned issue_flags) +{ +return __io_import_reg_vec (ddir, iter, req, vec, nr_iovs, issue_flags,0 ); +} +int io_prep_reg_iovec (struct io_kiocb *req,struct iou_vec *iv,const struct iovec __user *uvec,size_t uvec_segs);
Signature · 1 line
2.54.0
PATCH v5 16/16 io_uring/rsrc: add dmabuf backed registered buffers
In reply to Pavel BegunkovImplement dmabuf backed registered buffers. To register them, the user should specify IO_REGBUF_TYPE_DMABUF for the regitration and pass the desired dmabuf fd and a file for which it should be registered.
From there, it can be used with io_uring read/write requests IORING_OP_{READ,WRITE}_FIXED) as normal. The requests should be issued against the file specified during registration, and otherwise they'll be failed. The user should also be prepared to handle spurious -EAGAIN by reissuing the request.
Internally, dmabuf registered buffers is an optin feature for io_uring request opcodes and they should pass a special flag on import to use it.
--- include/linux/io_uring_types.h | 5 + include/uapi/linux/io_uring.h | 6 +- io_uring/io_uring.c | 3 +- io_uring/rsrc.c | 167 ++++++++++++++++++++++++++++++++- io_uring/rsrc.h | 18 ++++ io_uring/rw.c | 6 +- 6 files changed, 197 insertions(+), 8 deletions(-)
include/linux/io_uring_types.h +5 -0
@@ -10,6 +10,7 @@struct iou_loop_params ;struct io_uring_bpf_ops ; +struct dma_buf_io_map ;enum {/*
@@ -594,6 +595,7 @@ enum {REQ_F_IMPORT_BUFFER_BIT ,REQ_F_SQE_COPIED_BIT ,REQ_F_IOPOLL_BIT , +REQ_F_DROP_DMABUF_BIT ,/* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT,
@@ -689,6 +691,8 @@ enum {REQ_F_SQE_COPIED =IO_REQ_FLAG (REQ_F_SQE_COPIED_BIT ),/* request must be iopolled to completion (set in ->issue()) */ REQ_F_IOPOLL =IO_REQ_FLAG (REQ_F_IOPOLL_BIT ), +/* there is a dma map attached to request that needs to be dropped */ + REQ_F_DROP_DMABUF = IO_REQ_FLAG(REQ_F_DROP_DMABUF_BIT), }; struct io_tw_req {
@@ -811,6 +815,7 @@ struct io_kiocb {/* custom credentials, valid IFF REQ_F_CREDS is set */ const struct cred *creds;struct io_wq_work work; +struct dma_buf_io_map *dmabuf_map;struct io_big_cqe {u64 extra1;
include/uapi/linux/io_uring.h +5 -1
@@ -810,6 +810,7 @@ enum io_uring_rsrc_reg_flags {enum io_uring_regbuf_type {IO_REGBUF_TYPE_EMPTY ,IO_REGBUF_TYPE_UADDR , +IO_REGBUF_TYPE_DMABUF , __IO_REGBUF_TYPE_MAX, };
@@ -819,7 +820,10 @@ struct io_uring_regbuf_desc {__u32 flags;__u64 size;__u64 uaddr; -__u64 __resv[7 ]; + +__s32 dmabuf_fd; +__s32 target_fd; +__u64 __resv[6 ]; };/* Skip updating fd indexes set to this value in the fd table */
io_uring/io_uring.c +2 -1
@@ -109,7 +109,7 @@#define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT | IO_REQ_LINK_FLAGS | \ REQ_F_REISSUE | REQ_F_POLLED | \ - IO_REQ_CLEAN_FLAGS) + IO_REQ_CLEAN_FLAGS | REQ_F_DROP_DMABUF)#define IO_TCTX_REFS_CACHE_NR (1U << 10)
@@ -1114,6 +1114,7 @@ static void io_free_batch_list(struct io_ring_ctx *ctx,io_queue_next (req);if (unlikely (req->flags &IO_REQ_CLEAN_FLAGS ))io_clean_op (req); +io_req_drop_dmabuf (req); }io_put_file (req);io_req_put_rsrc_nodes (req);
io_uring/rsrc.c +163 -4
210 lines not on this page · raw message
io_uring/rsrc.h +18 -0
@@ -31,6 +31,11 @@ enum {enum {IO_REGBUF_F_KBUF =1 <<0 ,IO_REGBUF_F_UNCLONEABLE =1 <<1 , +IO_REGBUF_F_DMABUF =1 <<3 , +}; + +enum { +IO_REGBUF_IMPORT_ALLOW_DMABUF =1 <<1 , };struct io_mapped_ubuf {
@@ -173,4 +178,17 @@ static inline void io_alloc_cache_vec_kasan(struct iou_vec *iv) io_vec_free(iv); } +void io_drop_dmabuf_node (struct io_kiocb *req); + +static inline void io_req_drop_dmabuf (struct io_kiocb *req) +{ +if (!IS_ENABLED (CONFIG_DMA_SHARED_BUFFER )) +return ; +if (!(req->flags &REQ_F_DROP_DMABUF )) +return ; +if (WARN_ON_ONCE (!(req->flags &REQ_F_BUF_NODE ))) +return ; +io_drop_dmabuf_node (req); +} +#endif
io_uring/rw.c +4 -2
@@ -380,8 +380,8 @@ static int io_init_rw_fixed(struct io_kiocb *req, unsigned int issue_flags,if (io->bytes_done)return 0 ; - ret =io_import_reg_buf (req, &io->iter, rw->addr, rw->len, ddir, - issue_flags); + ret =__io_import_reg_buf (req, &io->iter, rw->addr, rw->len, ddir, + issue_flags,IO_REGBUF_IMPORT_ALLOW_DMABUF );iov_iter_save_state (&io->iter, &io->iter_state);return ret; }
@@ -591,6 +591,8 @@ static void io_complete_rw(struct kiocb *kiocb, long res) struct io_rw *rw = container_of(kiocb, struct io_rw, kiocb);struct io_kiocb *req =cmd_to_io_kiocb (rw); +io_req_drop_dmabuf (req); +__io_complete_rw_common (req, res);io_req_set_res (req,io_fixup_rw_res (req, res),0 ); req->io_task_work.func = io_req_rw_complete;
Signature · 1 line
2.54.0
Re: (subset) [PATCH v5 00/16] Add dmabuf read/write via io_uring
In reply to Pavel BegunkovQuoted 8 lines · Pavel Begunkov
The patch set allows to register a dmabuf to an
io_uringinstance for a specified file and use it withio_uringread / write requests. The infrastructure is not tied toio_uringand there could be more users in the future. A similar idea was attempted some years ago by Keith [1] (from where I borrowed a good number of changes). Later it was brough back up to life by Tushar and Vishal.[...]
Applied, thanks!
[05/16] block: move bvec init into __bio_clone
commit: e228404b05c8293f6a1286856e49ba3e55da1933
[06/16] block: introduce bio_iov_iter_set()
commit: 8b8755e00847da1027fe6624277e43d289e2300eBest regards,
Signature · 1 line
Jens Axboe
Quoted 8 lines · Pavel Begunkov
The patch set allows to register a dmabuf to an
io_uringinstance for a specified file and use it withio_uringread / write requests. The infrastructure is not tied toio_uringand there could be more users in the future. A similar idea was attempted some years ago by Keith [1] (from where I borrowed a good number of changes). Later it was brough back up to life by Tushar and Vishal.[...]
Applied, thanks!
[01/16] dma-buf: introduce initial file I/O infrastructure
(no commit info)
[02/16] iov_iter: add iterator type for dmabuf maps
(no commit info)
[03/16] block: rename bi_bvec_done
(no commit info)
[04/16] block: always adjust bi_offset on bio_advance_iter
(no commit info)
[05/16] block: move bvec init into __bio_clone
(no commit info)
[06/16] block: introduce bio_iov_iter_set()
(no commit info)
[07/16] block: introduce dma map backed bio type
(no commit info)
[08/16] block: add dma-buf support for raw bdev
(no commit info)
[09/16] nvme-pci: implement dma-buf backed requests
(no commit info)
[10/16] nvme-pci: rename nvme_pci_sgl_set_data to nvme_pci_dma_iter_set_sgl
(no commit info)
[11/16] nvme-pci: add SGL support for the dmabuf path
(no commit info)
[12/16] io_uring/rsrc: introduce buf registration structure
(no commit info)
[13/16] io_uring/rsrc: extend buffer update
(no commit info)
[14/16] io_uring/rsrc: add uncloneable regbuf flag
(no commit info)
[15/16] io_uring/rsrc: add regbuf import flags
(no commit info)
[16/16] io_uring/rsrc: add dmabuf backed registered buffers
(no commit info)
Best regards,
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 9 lines · Jens Axboe
Quoted 7 lines · Pavel Begunkov
The patch set allows to register a dmabuf to an
io_uringinstance for a specified file and use it withio_uringread / write requests. The infrastructure is not tied toio_uringand there could be more users in the future. A similar idea was attempted some years ago by Keith [1] (from where I borrowed a good number of changes). Later it was brough back up to life by Tushar and Vishal.
Quoted 3 lines
Quoted 1 line
[...]
Applied, thanks!
Eh no it is not applied, was just looking at the series locally and I guess it got added to the b4 ty pile.
I am, however, wondering if we'll see a v6 of this soon? I had some comments on the nvme dma mapping, that can be improved, but I'd rather defer that to post getting this included.
If the series is dormant, just let us know, and we can pick up the bits and pieces.
Signature · 1 line
Jens Axboe
Quoted 12 lines · Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 8 lines · Pavel Begunkov
The patch set allows to register a dmabuf to an
io_uringinstance for a specified file and use it withio_uringread / write requests. The infrastructure is not tied toio_uringand there could be more users in the future. A similar idea was attempted some years ago by Keith [1] (from where I borrowed a good number of changes). Later it was brough back up to life by Tushar and Vishal.[...]
Quoted 8 lines
Quoted 1 line
Applied, thanks!
Eh no it is not applied, was just looking at the series locally and I guess it got added to the b4 ty pile.
I am, however, wondering if we'll see a v6 of this soon? I had some comments on the nvme dma mapping, that can be improved, but I'd rather defer that to post getting this included.
Working on it
Signature · 1 line
Pavel Begunkov