zero-copy RX for io_uring
Subsystems (3)
cover zero-copy RX for io_uring
This series introduces network RX zerocopy for io_uring.
This is an evolution of the earlier zctap work, re-targeted to use io_uring as the userspace API. The code is intends to provide a ZC RX path for upper-level networking protocols (aka TCP and UDP), with a focus on focuses on host-provided memory (not GPU memory).
This patch contains the upper-level core code required for operation, but does not not contain the network driver side changes required for true zero-copy operation. The io_uring RECV_ZC opcode will work without hardware support, albeit in copy mode.
The intent is to use a network driver which provides header/data splitting, so the frame header which is processed by the networking stack is not placed in user memory.
The code is successfully receiving a zero-copy TCP stream from a remote sender.
There is a liburing fork providing the needed wrappers:
https://github.com/jlemon/liburing/tree/zctap
Which contains an examples/io_uring-net test application exercising these features. A sample run:
# ./io_uring-net -i eth1 -q 20 -p 9999 -r 3000
copy bytes: 1938872
ZC bytes: 996683008
Total bytes: 998621880, nsec:1025219375
Rate: 7.79 Gb/sIf no queue is specified, then non-zc mode is used:
# ./io_uring-net -p 9999
copy bytes: 998621880
ZC bytes: 0
Total bytes: 998621880, nsec:1051515726
Rate: 7.60 Gb/sThere is also an iperf3 fork as well:
https://github.com/jlemon/iperf/tree/io_uring
This allows running single tests with either:
* select (normal iperf3)
* io_uring READ
* io_uring RECV_ZC copy mode
* io_uring RECV_ZC hardware mode
Current testing shows similar BW between RECV_ZC and READ modes (running at 22Gbit/sec), but a reduction of ~50% of MemBW.
High level description:
The application allocates a frame backing store, and provides this to the kernel for use. An interface queue is requested from the networking device, and incoming frames are deposited into the provided memory region. The NIC should provide a header splitting feature, so only the frame payload is placed in the user space area.
Responsibility for correctly steering incoming frames to the queue is outside the scope of this work - it is assumed that the user has set steering rules up separately.
Incoming frames are sent up the stack as skb's and eventually land in the application's socket receive queue. This differs from AF_XDP, which receives raw frames directly to userspace, without protocol processing.
The RECV_ZC opcode then returns an iov[] style vector which points to the data in userspace memory. When the application has completed processing of the data, the buffers are returned back to the kernel through a fill ring for reuse.
Jonathan Lemon (15):
io_uring: add zctap ifq definition
netdevice: add SETUP_ZCTAP to the netdev_bpf structure
io_uring: add register ifq opcode
io_uring: create a zctap region for a mapped buffer
io_uring: mark pages in ifq region with zctap information.
io_uring: Provide driver API for zctap packet buffers.
io_uring: Allocate zctap device buffers and dma map them.
io_uring: Add zctap buffer get/put functions and refcounting.
skbuff: Introduce SKBFL_FIXED_FRAG and skb_fixed()
io_uring: Allocate a uarg for use by the ifq RX
io_uring: Define the zctap iov[] returned to the user.
io_uring: add OP_RECV_ZC command.
io_uring: Make remove_ifq_region a delayed work call
io_uring: Add a buffer caching mechanism for zctap.
io_uring: Notify the application as the fillq is drained.
include/linux/io_uring.h | 47 ++ include/linux/io_uring_types.h | 12 + include/linux/netdevice.h | 6 + include/linux/skbuff.h | 10 +- include/uapi/linux/io_uring.h | 24 + io_uring/Makefile | 3 +- io_uring/io_uring.c | 8 + io_uring/kbuf.c | 13 + io_uring/kbuf.h | 2 + io_uring/net.c | 123 ++++ io_uring/opdef.c | 15 + io_uring/zctap.c | 1001 ++++++++++++++++++++++++++++++++ io_uring/zctap.h | 31 + 13 files changed, 1293 insertions(+), 2 deletions(-) create mode 100644 io_uring/zctap.c create mode 100644 io_uring/zctap.h
Signature · 1 line
2.30.2
PATCH 1/15 io_uring: add zctap ifq definition
Add structure definition for io_zctap_ifq for use by lower level networking hooks.
--- include/linux/io_uring_types.h | 11 +++++++++++ 1 file changed, 11 insertions(+)
include/linux/io_uring_types.h +11 -0
@@ -322,6 +322,7 @@ struct io_ring_ctx {struct io_mapped_ubuf *dummy_ubuf;struct io_rsrc_data *file_data;struct io_rsrc_data *buf_data; +struct io_zctap_ifq *zctap_ifq;struct delayed_work rsrc_put_work;struct llist_head rsrc_put_llist;
@@ -577,4 +578,14 @@ struct io_overflow_cqe { struct io_uring_cqe cqe; }; +struct io_zctap_ifq { +struct net_device *dev; +struct io_ring_ctx *ctx; +void *region; +struct ubuf_info *uarg; +u16 queue_id; +u16 id; +u16 fill_bgid; +}; +#endif
Signature · 1 line
2.30.2
PATCH 2/15 netdevice: add SETUP_ZCTAP to the netdev_bpf structure
In reply to Jonathan LemonThis command requests the networking device setup or teardown a new interface queue, backed by a region of user supplied memory.
The queue will be managed by io-uring.
--- include/linux/netdevice.h | 6 ++++++ 1 file changed, 6 insertions(+)
include/linux/netdevice.h +6 -0
@@ -980,6 +980,7 @@ enum bpf_netdev_command {BPF_OFFLOAD_MAP_ALLOC ,BPF_OFFLOAD_MAP_FREE ,XDP_SETUP_XSK_POOL , + XDP_SETUP_ZCTAP, };struct bpf_prog_offload_ops ;
@@ -1018,6 +1019,11 @@ struct netdev_bpf {struct xsk_buff_pool *pool;u16 queue_id; } xsk; +/* XDP_SETUP_ZCTAP */ +struct { +struct io_zctap_ifq *ifq; +u16 queue_id; + } zct; }; };
Signature · 1 line
2.30.2
PATCH 3/15 io_uring: add register ifq opcode
In reply to Jonathan LemonAdd initial support for support for hooking in zero-copy interface queues to io_uring. This command requests a user-managed queue from the specified network device.
This only includes the register opcode, unregistration is currently done implicitly when the ring is removed.
--- include/uapi/linux/io_uring.h | 15 ++++ io_uring/Makefile | 3 +- io_uring/io_uring.c | 8 ++ io_uring/zctap.c | 134 ++++++++++++++++++++++++++++++++++ io_uring/zctap.h | 9 +++ 5 files changed, 168 insertions(+), 1 deletion(-) create mode 100644 io_uring/zctap.c create mode 100644 io_uring/zctap.h
include/uapi/linux/io_uring.h +15 -0
@@ -490,6 +490,9 @@ enum {/* register a range of fixed file slots for automatic slot allocation */ IORING_REGISTER_FILE_ALLOC_RANGE =25 , +/* register a network ifq for zerocopy RX */ +IORING_REGISTER_IFQ =26 , +/* this goes last */ IORING_REGISTER_LAST };
@@ -666,6 +669,18 @@ struct io_uring_recvmsg_out {__u32 flags; }; +/* +* Argument for IORING_REGISTER_IFQ +*/ +struct io_uring_ifq_req { +__u32 ifindex; +__u16 queue_id; +__u16 ifq_id; +__u16 fill_bgid; +__u16 region_id; +__u16 resv[2 ]; +}; + #ifdef __cplusplus }#endif
io_uring/Makefile +2 -1
@@ -7,5 +7,6 @@ obj-$(CONFIG_IO_URING) += io_uring.o xattr.o nop.o fs.o splice.o \ openclose.o uring_cmd.o epoll.o \ statx.o net.o msg_ring.o timeout.o \ sqpoll.o fdinfo.o tctx.o poll.o \ - cancel.o kbuf.o rsrc.o rw.o opdef.o notif.o + cancel.o kbuf.o rsrc.o rw.o opdef.o \ + notif.o zctap.o obj-$(CONFIG_IO_WQ) += io-wq.o
io_uring/io_uring.c +8 -0
@@ -91,6 +91,7 @@#include "cancel.h" #include "net.h" #include "notif.h" +#include "zctap.h" #include "timeout.h" #include "poll.h"
@@ -2799,6 +2800,7 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)__io_cqring_overflow_flush (ctx, true);xa_for_each (&ctx->personalities, index, creds)io_unregister_personality (ctx, index); +io_unregister_zctap_all (ctx);if (ctx->rings)io_poll_remove_all (ctx,NULL , true);mutex_unlock (&ctx->uring_lock);
@@ -4031,6 +4033,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_register_file_alloc_range (ctx, arg);break ; +case IORING_REGISTER_IFQ : + ret = -EINVAL ; +if (!arg || nr_args !=1 ) +break ; + ret =io_register_ifq (ctx, arg); +break ;default : ret = -EINVAL ;break ;
io_uring/zctap.c new file +134 -0
134 lines not on this page · raw message
io_uring/zctap.h new file +9 -0
@@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 +#ifndef IOU_ZCTAP_H +#define IOU_ZCTAP_H + +int io_register_ifq (struct io_ring_ctx *ctx, +struct io_uring_ifq_req __user *arg); +void io_unregister_zctap_all (struct io_ring_ctx *ctx); + +#endif
Signature · 1 line
2.30.2
PATCH 4/15 io_uring: create a zctap region for a mapped buffer
In reply to Jonathan LemonThis function takes all of a memory region that was previously registered with io_uring, and assigns it as the backing store for the specified ifq, binding the pages to a specific device.
The entire region is registered instead of providing individual buffers, as this allows the hardware to select the optimal buffer size for incoming packets.
The region is registered as part of the register_ifq opcode, instead of separately, since the ifq ring requires memory when it is created.
--- io_uring/zctap.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++- io_uring/zctap.h | 2 ++ 2 files changed, 64 insertions(+), 1 deletion(-)
io_uring/zctap.c +62 -1
@@ -6,16 +6,73 @@#include <linux/mm.h> #include <linux/io_uring.h> #include <linux/netdevice.h> +#include <linux/nospec.h> #include <uapi/linux/io_uring.h> #include "io_uring.h" #include "zctap.h" +#include "rsrc.h" +#include "kbuf.h" #define NR_ZCTAP_IFQS 1 +struct ifq_region { +struct io_mapped_ubuf *imu; +int free_count; +int nr_pages; +u16 id; +struct page *freelist[]; +}; +typedef int (*bpf_op_t )(struct net_device *dev,struct netdev_bpf *bpf); +static void io_remove_ifq_region (struct ifq_region *ifr) +{ +kvfree (ifr); +} + +int io_provide_ifq_region (struct io_zctap_ifq *ifq,u16 id) +{ +struct io_ring_ctx *ctx = ifq->ctx; +struct io_mapped_ubuf *imu; +struct ifq_region *ifr; +int i, nr_pages; +struct page *page; + +/* XXX for now, only allow one region per ifq. */ +if (ifq->region) +return -EFAULT ; + +if (unlikely (id >= ctx->nr_user_bufs)) +return -EFAULT ; + id =array_index_nospec (id, ctx->nr_user_bufs); + imu = ctx->user_bufs[id]; + +/* XXX check region is page aligned */ +if (imu->ubuf & ~PAGE_MASK || imu->ubuf_end & ~PAGE_MASK ) +return -EFAULT ; + + nr_pages = imu->nr_bvecs; + ifr =kvmalloc (struct_size (ifr, freelist, nr_pages),GFP_KERNEL ); +if (!ifr) +return -ENOMEM ; + + ifr->nr_pages = nr_pages; + ifr->imu = imu; + ifr->free_count = nr_pages; + ifr->id = id; + +for (i =0 ; i < nr_pages; i++) { + page = imu->bvec[i].bv_page; + ifr->freelist[i] = page; + } + + ifq->region = ifr; + +return 0 ; +} +static int __io_queue_mgmt (struct net_device *dev,struct io_zctap_ifq *ifq,u16 queue_id) {
@@ -60,6 +117,8 @@ static void io_zctap_ifq_free(struct io_zctap_ifq *ifq) {if (ifq->queue_id !=-1 )io_close_zctap_ifq (ifq, ifq->queue_id); +if (ifq->region) +io_remove_ifq_region (ifq->region);if (ifq->dev)dev_put (ifq->dev);kfree (ifq);
@@ -95,7 +154,9 @@ int io_register_ifq(struct io_ring_ctx *ctx,if (!ifq->dev) goto out; -/* region attachment TBD */ + err =io_provide_ifq_region (ifq, req.region_id); +if (err) + goto out; err =io_open_zctap_ifq (ifq, req.queue_id);if (err)
io_uring/zctap.h +2 -0
@@ -6,4 +6,6 @@ int io_register_ifq(struct io_ring_ctx *ctx, struct io_uring_ifq_req __user *arg);void io_unregister_zctap_all (struct io_ring_ctx *ctx); +int io_provide_ifq_region (struct io_zctap_ifq *ifq,u16 id); +#endif
Signature · 1 line
2.30.2
PATCH 5/15 io_uring: mark pages in ifq region with zctap information.
In reply to Jonathan LemonThe network stack passes up pages, which must be mapped to zctap device buffers in order to get the reference count and other items. Mark the page as private, and use the page_private field to record the lookup and ownership information.
--- io_uring/zctap.c | 61 ++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 56 insertions(+), 5 deletions(-)
io_uring/zctap.c +56 -5
@@ -27,18 +27,68 @@ struct ifq_region {typedef int (*bpf_op_t )(struct net_device *dev,struct netdev_bpf *bpf); +static void zctap_set_page_info (struct page *page,u64 info) +{ +set_page_private (page, info); +} + +static u64 zctap_mk_page_info (u16 region_id,u16 pgid) +{ +return (u64 )0xface <<48 | (u64 )region_id <<16 | (u64 )pgid; +} +static void io_remove_ifq_region (struct ifq_region *ifr) { +struct io_mapped_ubuf *imu; +struct page *page; +int i; + + imu = ifr->imu; +for (i =0 ; i < ifr->nr_pages; i++) { + page = imu->bvec[i].bv_page; + +ClearPagePrivate (page); +set_page_private (page,0 ); + } +kvfree (ifr); } +static int io_zctap_map_region (struct ifq_region *ifr) +{ +struct io_mapped_ubuf *imu; +struct page *page; +u64 info; +int i; + + imu = ifr->imu; +for (i =0 ; i < ifr->nr_pages; i++) { + page = imu->bvec[i].bv_page; +if (PagePrivate (page)) + goto out; +SetPagePrivate (page); + info =zctap_mk_page_info (ifr->id, i); +zctap_set_page_info (page, info); + ifr->freelist[i] = page; + } +return 0 ; + +out: +while (i--) { + page = imu->bvec[i].bv_page; +ClearPagePrivate (page); +set_page_private (page,0 ); + } +return -EEXIST ; +} +int io_provide_ifq_region (struct io_zctap_ifq *ifq,u16 id) {struct io_ring_ctx *ctx = ifq->ctx;struct io_mapped_ubuf *imu;struct ifq_region *ifr; -int i, nr_pages; -struct page *page; +int nr_pages; +int err;/* XXX for now, only allow one region per ifq. */ if (ifq->region)
@@ -63,9 +113,10 @@ int io_provide_ifq_region(struct io_zctap_ifq *ifq, u16 id) ifr->free_count = nr_pages; ifr->id = id; -for (i =0 ; i < nr_pages; i++) { - page = imu->bvec[i].bv_page; - ifr->freelist[i] = page; + err =io_zctap_map_region (ifr); +if (err) { +kvfree (ifr); +return err; } ifq->region = ifr;
Signature · 1 line
2.30.2
Re: [PATCH v1 05/15] io_uring: mark pages in ifq region with zctap information.
Quoted 4 lines · Jonathan Lemon
The network stack passes up pages, which must be mapped to zctap device buffers in order to get the reference count and other items. Mark the page as private, and use the
page_privatefield to record the lookup and ownership information.
Who coordinate ownership of page_private here? What other parts of the kernel could touch these pages?
Re: [PATCH v1 05/15] io_uring: mark pages in ifq region with zctap information.
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Jonathan Lemon
The network stack passes up pages, which must be mapped to zctap device buffers in order to get the reference count and other items. Mark the page as private, and use the
page_privatefield to record the lookup and ownership information.Who coordinate ownership of
page_privatehere? What other parts of the kernel could touch these pages?
This may be an issue. The driver, network stack, and application all utilize the pages, which have been pinned by io_uring. If the pages are passed to another subsystem which wants to use the private area for its own purposes, there will be a conflict.
There doesn’t seem to be a good way to maintain long term information in the current page structure. A resolution for this would be using an external lookup indexed by the page, or changing the information passed up in the skb.
—
Jonathan
PATCH 6/15 io_uring: Provide driver API for zctap packet buffers.
In reply to Jonathan LemonIntroduce 'struct io_zctap_buf', representing a buffer used by the network drivers, and the get/put functions which are used by the network driver to obtain the buffers.
The code for these will be fleshed out in a following patch.
--- include/linux/io_uring.h | 47 ++++++++++++++++++++++++++++++++++++++++ io_uring/zctap.c | 23 ++++++++++++++++++++ 2 files changed, 70 insertions(+)
include/linux/io_uring.h +47 -0
@@ -32,6 +32,13 @@ struct io_uring_cmd {u8 pdu[32 ];/* available inline for free use */ }; +struct io_zctap_buf { +dma_addr_t dma; +struct page *page; +atomic_t refcount; +u8 _pad[4 ]; +}; +#if defined(CONFIG_IO_URING )int io_uring_cmd_import_fixed (u64 ubuf,unsigned long len,int rw,struct iov_iter *iter,void *ioucmd);
@@ -44,6 +51,21 @@ void __io_uring_free(struct task_struct *tsk);void io_uring_unreg_ringfd (void );const char *io_uring_get_opcode (u8 opcode); +struct io_zctap_ifq ; +struct io_zctap_buf *io_zctap_get_buf (struct io_zctap_ifq *ifq,int refc); +void io_zctap_put_buf (struct io_zctap_ifq *ifq,struct io_zctap_buf *buf); +void io_zctap_put_buf_refs (struct io_zctap_ifq *ifq,struct io_zctap_buf *buf, +unsigned count); +bool io_zctap_put_page (struct io_zctap_ifq *ifq,struct page *page); + +static inline dma_addr_t io_zctap_buf_dma (struct io_zctap_buf *buf) +{ +return buf->dma; +} +static inline struct page *io_zctap_buf_page (struct io_zctap_buf *buf) +{ +return buf->page; +}static inline void io_uring_files_cancel (void ) {if (current->io_uring) {
@@ -92,6 +114,31 @@ static inline const char *io_uring_get_opcode(u8 opcode) {return "" ; } +static inline dma_addr_t io_zctap_buf_dma (struct io_zctap_buf *buf) +{ +return 0 ; +} +static inline struct page *io_zctap_buf_page (struct io_zctap_buf *buf) +{ +return NULL ; +} +static inline struct io_zctap_buf *io_zctap_get_buf (struct io_zctap_ifq *ifq, +int refc) +{ +return NULL ; +} +void io_zctap_put_buf (struct io_zctap_ifq *ifq,struct io_zctap_buf *buf) +{ +} +void io_zctap_put_buf_refs (struct io_zctap_ifq *ifq,struct io_zctap_buf *buf, +unsigned count) +{ +} +bool io_zctap_put_page (struct io_zctap_ifq *ifq,struct page *page) +{ +return false; +} +#endif #endif
io_uring/zctap.c +23 -0
@@ -37,6 +37,29 @@ static u64 zctap_mk_page_info(u16 region_id, u16 pgid)return (u64 )0xface <<48 | (u64 )region_id <<16 | (u64 )pgid; } +struct io_zctap_buf *io_zctap_get_buf (struct io_zctap_ifq *ifq,int refc) +{ +return NULL ; +} +EXPORT_SYMBOL (io_zctap_get_buf); + +void io_zctap_put_buf (struct io_zctap_ifq *ifq,struct io_zctap_buf *buf) +{ +} +EXPORT_SYMBOL (io_zctap_put_buf); + +void io_zctap_put_buf_refs (struct io_zctap_ifq *ifq,struct io_zctap_buf *buf, +unsigned count) +{ +} +EXPORT_SYMBOL (io_zctap_put_buf_refs); + +bool io_zctap_put_page (struct io_zctap_ifq *ifq,struct page *page) +{ +return false; +} +EXPORT_SYMBOL (io_zctap_put_page); +static void io_remove_ifq_region (struct ifq_region *ifr) {struct io_mapped_ubuf *imu;
Signature · 1 line
2.30.2
Re: [PATCH v1 06/15] io_uring: Provide driver API for zctap packet buffers.
io_uring/zctap.c static u64 zctap_mk_page_info(u16 region_id, u16 pgid) · line 49
Quoted 10 lines · Jonathan Lemon
io_uring/zctap.c +10
-0+struct io_zctap_buf *io_zctap_get_buf (struct io_zctap_ifq *ifq,int refc) +{ +return NULL ; +} +EXPORT_SYMBOL (io_zctap_get_buf); + +void io_zctap_put_buf (struct io_zctap_ifq *ifq,struct io_zctap_buf *buf) +{ +} +EXPORT_SYMBOL (io_zctap_put_buf);
Adding stubs without anything in them is rather pointless.
Also why are these exported? I can't find any modular users anywhere. Even if so, any low-level uring zero copy functionality should be EXPORT_SYMBOL_GPL, and only added once the modular users show up and are discussed.
Re: [PATCH v1 06/15] io_uring: Provide driver API for zctap packet buffers.
Quoted 13 lines · Christoph Hellwig
io_uring/zctap.c static u64 zctap_mk_page_info(u16 region_id, u16 pgid) · line 49
Quoted 10 lines · Jonathan Lemon
io_uring/zctap.c +10
-0+struct io_zctap_buf *io_zctap_get_buf (struct io_zctap_ifq *ifq,int refc) +{ +return NULL ; +} +EXPORT_SYMBOL (io_zctap_get_buf); + +void io_zctap_put_buf (struct io_zctap_ifq *ifq,struct io_zctap_buf *buf) +{ +} +EXPORT_SYMBOL (io_zctap_put_buf);Adding stubs without anything in them is rather pointless.
This patch just introduces the API, for easier reviewing (and handling some patch dependencies). Patch 8 adds the initial implementation.
Quoted 4 lines
Also why are these exported? I can't find any modular users anywhere. Even if so, any low-level uring zero copy functionality should be
EXPORT_SYMBOL_GPL, and only added once the modular users show up and are discussed.
I just copied EXPORT_SYMBOL from io_uring.c, and have no preference as to which export variant is used.
I was testing against several drivers, which were loadable modules for ease of development. For example, Broadcom’s in-tree bnxt driver does not (currently) have steering support, so testing was against their external driver package while waiting for the features to land upstream.
Signature · 1 line
Jonathan
PATCH 7/15 io_uring: Allocate zctap device buffers and dma map them.
In reply to Jonathan LemonThe goal is to register a memory region with the device, and later specify the desired packet buffer size. The code currently assumes a page size.
Create the desired number of zctap buffers and DMA map them to the target device, recording the dma address for later use.
Hold a page reference while the page is dma mapped.
Change the freelist from an array of page pointers to an index into the device buffer list.
--- io_uring/zctap.c | 78 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 63 insertions(+), 15 deletions(-)
io_uring/zctap.c +63 -15
@@ -18,11 +18,14 @@#define NR_ZCTAP_IFQS 1struct ifq_region { +struct io_zctap_ifq *ifq;/* only for delayed_work */ struct io_mapped_ubuf *imu;int free_count;int nr_pages;u16 id; -struct page *freelist[]; + +struct io_zctap_buf *buf; +u16 freelist[]; };typedef int (*bpf_op_t )(struct net_device *dev,struct netdev_bpf *bpf);
@@ -60,49 +63,85 @@ bool io_zctap_put_page(struct io_zctap_ifq *ifq, struct page *page) }EXPORT_SYMBOL (io_zctap_put_page); +static inline struct device * +netdev2device (struct net_device *dev) +{ +return dev->dev.parent;/* from SET_NETDEV_DEV() */ +} +static void io_remove_ifq_region (struct ifq_region *ifr) { -struct io_mapped_ubuf *imu; -struct page *page; +struct device *device =netdev2device (ifr->ifq->dev); +struct io_zctap_buf *buf;int i; - imu = ifr->imu;for (i =0 ; i < ifr->nr_pages; i++) { - page = imu->bvec[i].bv_page; - -ClearPagePrivate (page); -set_page_private (page,0 ); + buf = &ifr->buf[i]; +set_page_private (buf->page,0 ); +ClearPagePrivate (buf->page); +dma_unmap_page_attrs (device, buf->dma,PAGE_SIZE , +DMA_BIDIRECTIONAL , +DMA_ATTR_SKIP_CPU_SYNC ); +put_page (buf->page); } +kvfree (ifr->buf);kvfree (ifr); } -static int io_zctap_map_region (struct ifq_region *ifr) +static int io_zctap_map_region (struct ifq_region *ifr,struct device *device) {struct io_mapped_ubuf *imu; +struct io_zctap_buf *buf;struct page *page; +dma_addr_t addr; +int i, err;u64 info; -int i; imu = ifr->imu;for (i =0 ; i < ifr->nr_pages; i++) { page = imu->bvec[i].bv_page; -if (PagePrivate (page)) + +if (PagePrivate (page)) { + err = -EEXIST ; goto out; + } +SetPagePrivate (page); info =zctap_mk_page_info (ifr->id, i);zctap_set_page_info (page, info); - ifr->freelist[i] = page; + + buf = &ifr->buf[i]; + addr =dma_map_page_attrs (device, page,0 ,PAGE_SIZE , +DMA_BIDIRECTIONAL , +DMA_ATTR_SKIP_CPU_SYNC ); +if (dma_mapping_error (device, addr)) { +set_page_private (page,0 ); +ClearPagePrivate (page); + err = -ENOMEM ; + goto out; + } + buf->dma = addr; + buf->page = page; +atomic_set (&buf->refcount,0 ); +get_page (page); + + ifr->freelist[i] = i; }return 0 ; out:while (i--) { page = imu->bvec[i].bv_page; -ClearPagePrivate (page);set_page_private (page,0 ); +ClearPagePrivate (page); + buf = &ifr->buf[i]; +dma_unmap_page_attrs (device, buf->dma,PAGE_SIZE , +DMA_BIDIRECTIONAL , +DMA_ATTR_SKIP_CPU_SYNC ); +put_page (page); } -return -EEXIST ; +return err; }int io_provide_ifq_region (struct io_zctap_ifq *ifq,u16 id)
@@ -131,13 +170,22 @@ int io_provide_ifq_region(struct io_zctap_ifq *ifq, u16 id)if (!ifr)return -ENOMEM ; + ifr->buf =kvmalloc_array (nr_pages,sizeof (*ifr->buf),GFP_KERNEL ); +if (!ifr->buf) { +kvfree (ifr); +return -ENOMEM ; + } + ifr->nr_pages = nr_pages; ifr->imu = imu; ifr->free_count = nr_pages; ifr->id = id; - err =io_zctap_map_region (ifr); + ifr->ifq = ifq;/* XXX */ + + err =io_zctap_map_region (ifr,netdev2device (ifq->dev));if (err) { +kvfree (ifr->buf);kvfree (ifr);return err; }
Signature · 1 line
2.30.2
Re: [PATCH v1 07/15] io_uring: Allocate zctap device buffers and dma map them.
io_uring/zctap.c bool io_zctap_put_page(struct io_zctap_ifq *ifq, struct page *page) · line 117
Quoted 7 lines
io_uring/zctap.c +6
-0+dma_unmap_page_attrs (device, buf->dma,PAGE_SIZE , +DMA_BIDIRECTIONAL , +DMA_ATTR_SKIP_CPU_SYNC ); + addr =dma_map_page_attrs (device, page,0 ,PAGE_SIZE , +DMA_BIDIRECTIONAL , +DMA_ATTR_SKIP_CPU_SYNC );
You can't just magically skip cpu syncs. The flag is only valid for mappings of already mapped memory when following a very careful protocol. I can't see any indications of that beeing true here, but maybe I'm just missing something.
Re: [PATCH v1 07/15] io_uring: Allocate zctap device buffers and dma map them.
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines
io_uring/zctap.c +3
-0+dma_unmap_page_attrs (device, buf->dma,PAGE_SIZE , +DMA_BIDIRECTIONAL , +DMA_ATTR_SKIP_CPU_SYNC );
Quoted 8 lines
io_uring/zctap.c bool io_zctap_put_page(struct io_zctap_ifq *ifq, struct page *page) · line 117
Quoted 3 lines
io_uring/zctap.c +3
-0+ addr =dma_map_page_attrs (device, page,0 ,PAGE_SIZE , +DMA_BIDIRECTIONAL , +DMA_ATTR_SKIP_CPU_SYNC );You can't just magically skip cpu syncs. The flag is only valid for mappings of already mapped memory when following a very careful protocol. I can't see any indications of that beeing true here, but maybe I'm just missing something.
This was copied from page_pool_dma_map(). The same logic applies - io_uring pins the memory, zctap creates a pool that is dma mapped, and the driver performs dma_sync_single_*() when receiving the packet. —
Jonathan
PATCH 8/15 io_uring: Add zctap buffer get/put functions and refcounting.
In reply to Jonathan LemonFlesh out the driver API functions introduced earlier.
The driver gets a buffer with the specified reference count.
If the driver specifies a large refcount (bias), it decrements this as skb fragments go up the stack, and the driver releases the references when finished with the buffer.
When ownership of the fragment is transferred to the user, a user refcount is incremented, and correspondingly decremented when returned. When all refcounts are released, the buffer is safe to reuse. The user/kernel split is needed to differentiate between "safe to reuse the buffer" and "still in use by the kernel".
The locking here can likely be improved.
--- io_uring/kbuf.c | 13 +++++ io_uring/kbuf.h | 2 + io_uring/zctap.c | 131 ++++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 145 insertions(+), 1 deletion(-)
io_uring/kbuf.c +13 -0
@@ -188,6 +188,19 @@ void __user *io_buffer_select(struct io_kiocb *req, size_t *len,return ret; } +/* Called from the network driver, in napi context. */ +u64 io_zctap_buffer (struct io_kiocb *req,size_t *len) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_buffer_list *bl; +void __user *ret =NULL ; + + bl =io_buffer_get_list (ctx, req->buf_index); +if (likely (bl)) + ret =io_ring_buffer_select (req, len, bl,IO_URING_F_UNLOCKED ); +return (u64 )ret; +} +static __cold int io_init_bl_list (struct io_ring_ctx *ctx) {int i;
io_uring/kbuf.h +2 -0
@@ -50,6 +50,8 @@ unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags);void io_kbuf_recycle_legacy (struct io_kiocb *req,unsigned issue_flags); +u64 io_zctap_buffer (struct io_kiocb *req,size_t *len); +static inline void io_kbuf_recycle_ring (struct io_kiocb *req) {/*
io_uring/zctap.c +130 -1
162 lines not on this page · raw message
Signature · 1 line
2.30.2
PATCH 9/15 skbuff: Introduce SKBFL_FIXED_FRAG and skb_fixed()
In reply to Jonathan LemonWhen a skb marked as zerocopy goes up the network stack, during RX, it calls skb_orphan_frags_rx. This is designed to catch TX zerocopy data being redirected back up the stack, not new zerocopy fragments coming up from the driver.
Currently, since the skb is marked as zerocopy, skb_copy_ubufs() is called, defeating the point of zerocopy-RX.
Have the driver mark the fragments as fixed, so they are not copied.
--- include/linux/skbuff.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
include/linux/skbuff.h +9 -1
@@ -516,6 +516,9 @@ enum {* use frags only up until ubuf_info is released */ SKBFL_MANAGED_FRAG_REFS =BIT (4 ), + +/* don't move or copy the fragment */ + SKBFL_FIXED_FRAG = BIT(5), };#define SKBFL_ZEROCOPY_FRAG (SKBFL_ZEROCOPY_ENABLE | SKBFL_SHARED_FRAG)
@@ -1653,6 +1656,11 @@ static inline bool skb_zcopy_managed(const struct sk_buff *skb)return skb_shinfo (skb)->flags &SKBFL_MANAGED_FRAG_REFS ; } +static inline bool skb_fixed (const struct sk_buff *skb) +{ +return skb_shinfo (skb)->flags &SKBFL_FIXED_FRAG ; +} +static inline bool skb_pure_zcopy_same (const struct sk_buff *skb1,const struct sk_buff *skb2) {
@@ -3089,7 +3097,7 @@ static inline int skb_orphan_frags(struct sk_buff *skb, gfp_t gfp_mask)/* Frags must be orphaned, even if refcounted, if skb might loop to rx path */ static inline int skb_orphan_frags_rx (struct sk_buff *skb,gfp_t gfp_mask) { -if (likely (!skb_zcopy (skb))) +if (likely (!skb_zcopy (skb) ||skb_fixed (skb)))return 0 ;return skb_copy_ubufs (skb, gfp_mask); }
Signature · 1 line
2.30.2
PATCH 10/15 io_uring: Allocate a uarg for use by the ifq RX
In reply to Jonathan LemonCreate a static uarg which is attached to zerocopy RX buffers, and add a callback to handle freeing the skb.
As the skb is marked as zerocopy, it bypasses the default network skb fragment destructor and uses our callback. This handles our buffer refcounts, and releases the ZC buffer back to the freelist.
Add the put_page() implementations, which release the fragments. This may also be called by drivers during cleanup.
--- io_uring/zctap.c | 64 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 56 insertions(+), 8 deletions(-)
io_uring/zctap.c +56 -8
@@ -30,6 +30,12 @@ struct ifq_region {u16 freelist[]; }; +/* XXX get around not having "struct ubuf_info" defined in io_uring_types.h */ +struct io_zctap_ifq_priv { +struct io_zctap_ifq ifq; +struct ubuf_info uarg; +}; +typedef int (*bpf_op_t )(struct net_device *dev,struct netdev_bpf *bpf);static void zctap_set_page_info (struct page *page,u64 info)
@@ -52,6 +58,16 @@ static u16 zctap_page_id(const struct page *page)return zctap_page_info (page) &0xffff ; } +static bool zctap_page_magic (const struct page *page) +{ +return (zctap_page_info (page) >>48 ) ==0xface ; +} + +static bool zctap_page_ours (struct page *page) +{ +return PagePrivate (page) &&zctap_page_magic (page); +} +/* driver bias cannot be larger than this */ #define IO_ZCTAP_UREF 0x10000#define IO_ZCTAP_KREF_MASK (IO_ZCTAP_UREF - 1)
@@ -70,7 +86,9 @@ static bool io_zctap_put_buf_uref(struct io_zctap_buf *buf)return atomic_sub_and_test (IO_ZCTAP_UREF , &buf->refcount); } -/* gets a user-supplied buffer from the fill queue */ +/* gets a user-supplied buffer from the fill queue +* note: may drain N entries, but still have no usable buffers +*/ static struct io_zctap_buf *io_zctap_get_buffer (struct io_zctap_ifq *ifq,u16 *buf_pgid) {
@@ -185,9 +203,19 @@ void io_zctap_put_buf_refs(struct io_zctap_ifq *ifq, struct io_zctap_buf *buf, }EXPORT_SYMBOL (io_zctap_put_buf_refs); +/* could be called by the stack as it drops/recycles the skbs */ bool io_zctap_put_page (struct io_zctap_ifq *ifq,struct page *page) { -return false; +struct ifq_region *ifr; +u16 pgid; + +if (!zctap_page_ours (page)) +return false; + + ifr = ifq->region;/* only one */ + pgid =zctap_page_id (page); +io_zctap_put_buf (ifq, &ifr->buf[pgid]); +return true; }EXPORT_SYMBOL (io_zctap_put_page);
@@ -351,17 +379,35 @@ static int io_close_zctap_ifq(struct io_zctap_ifq *ifq, u16 queue_id)return __io_queue_mgmt (ifq->dev,NULL , queue_id); } +static void io_zctap_ifq_callback (struct sk_buff *skb,struct ubuf_info *uarg, +bool success) +{ +struct skb_shared_info *shinfo =skb_shinfo (skb); +struct io_zctap_ifq_priv *priv; +struct page *page; +int i; + + priv = container_of(uarg, struct io_zctap_ifq_priv, uarg); + +for (i =0 ; i < shinfo->nr_frags; i++) { + page =skb_frag_page (&shinfo->frags[i]); +if (!io_zctap_put_page (&priv->ifq, page)) +__skb_frag_unref (&shinfo->frags[i], skb->pp_recycle); + } +} +static struct io_zctap_ifq *io_zctap_ifq_alloc (struct io_ring_ctx *ctx) { -struct io_zctap_ifq *ifq; +struct io_zctap_ifq_priv *priv; - ifq =kzalloc (sizeof (*ifq),GFP_KERNEL ); -if (!ifq) + priv =kzalloc (sizeof (*priv),GFP_KERNEL ); +if (!priv)return NULL ; - ifq->ctx = ctx; - ifq->queue_id =-1 ; -return ifq; + priv->ifq.ctx = ctx; + priv->ifq.queue_id =-1 ; + priv->ifq.uarg = &priv->uarg; +return &priv->ifq; }static void io_zctap_ifq_free (struct io_zctap_ifq *ifq)
@@ -399,6 +445,8 @@ int io_register_ifq(struct io_ring_ctx *ctx,return -ENOMEM ; ifq->fill_bgid = req.fill_bgid; + ifq->uarg->callback = io_zctap_ifq_callback; + ifq->uarg->flags =SKBFL_ALL_ZEROCOPY |SKBFL_FIXED_FRAG ; err = -ENODEV ; ifq->dev =dev_get_by_index (&init_net, req.ifindex);
Signature · 1 line
2.30.2
PATCH 11/15 io_uring: Define the zctap iov[] returned to the user.
In reply to Jonathan LemonWhen performing a ZC receive, instead of returning data directly to the user, an iov[] structure is returned referencing the data in user space.
The application locates the base address of the data by performing address computations on bgid:bid. The off/len applies to the base address, resulting in the data segment.
The bgid:bid identifying the buffer should later be placed in the ifq's fill ring, which returns the buffer back to the kernel.
--- include/uapi/linux/io_uring.h | 8 ++++++++ 1 file changed, 8 insertions(+)
include/uapi/linux/io_uring.h +8 -0
@@ -681,6 +681,14 @@ struct io_uring_ifq_req {__u16 resv[2 ]; }; +struct io_uring_zctap_iov { +__u32 off; +__u32 len; +__u16 bgid; +__u16 bid; +__u16 resv[2 ]; +}; + #ifdef __cplusplus }#endif
Signature · 1 line
2.30.2
PATCH 12/15 io_uring: add OP_RECV_ZC command.
In reply to Jonathan LemonThe recvzc opcode uses a metadata buffer either supplied directly with buf/len, or indirectly from the buffer group. The expectation is that this buffer is then filled with an array of io_uring_zctap_iov structures, which point to the data in user-memory.
The opcode uses addr3 with an encoded format:
addr3 = (readlen << 32) | (copy_bgid << 16);
Readlen specifies the maximum amount of data which should be read from the socket. The amount of returned data is also limited by the number of iovs which the metadata area can hold.
As a fallback, if the desired skb data is not already present in user memory, then a seprate buffer is obtained from the copy_bgid and the data is copied into user memory, which is returned as an iov[] structure.
This may happen possibly due to system misconfiguration, imprecise header splitting, running the RECV_ZC opcode without hardware zero-copy support, or other network stack follies.
--- include/uapi/linux/io_uring.h | 1 + io_uring/net.c | 123 ++++++++++++ io_uring/opdef.c | 15 ++ io_uring/zctap.c | 340 ++++++++++++++++++++++++++++++++++ io_uring/zctap.h | 20 ++ 5 files changed, 499 insertions(+)
include/uapi/linux/io_uring.h +1 -0
@@ -215,6 +215,7 @@ enum io_uring_op {IORING_OP_URING_CMD ,IORING_OP_SEND_ZC ,IORING_OP_SENDMSG_ZC , +IORING_OP_RECV_ZC ,/* this goes last, obviously */ IORING_OP_LAST,
io_uring/net.c +123 -0
141 lines not on this page · raw message
io_uring/opdef.c +15 -0
@@ -33,6 +33,7 @@#include "poll.h" #include "cancel.h" #include "rw.h" +#include "zctap.h" static int io_no_issue (struct io_kiocb *req,unsigned int issue_flags) {
@@ -521,6 +522,20 @@ const struct io_op_def io_op_defs[] = { .fail = io_sendrecv_fail, #else .prep = io_eopnotsupp_prep, +#endif + }, + [IORING_OP_RECV_ZC ] = { + .name ="RECV_ZC" , + .needs_file =1 , + .unbound_nonreg_file =1 , + .pollin =1 , + .buffer_select =1 , + .ioprio =1 , +#if defined(CONFIG_NET) + .prep = io_recvzc_prep, + .issue = io_recvzc, +#else + .prep = io_eopnotsupp_prep, #endif }, };
io_uring/zctap.c +340 -0
367 lines not on this page · raw message
io_uring/zctap.h +20 -0
@@ -2,10 +2,30 @@#ifndef IOU_ZCTAP_H #define IOU_ZCTAP_H +struct zctap_read_desc { +struct io_ring_ctx *ctx; +struct iov_iter *iter; +u32 iov_space; +u32 iov_limit; +u32 recv_limit; + +struct io_kiocb req; +u8 *buf; +size_t offset; +size_t buflen; + +u16 copy_bgid;/* XXX move to register ifq? */ +}; +int io_register_ifq (struct io_ring_ctx *ctx,struct io_uring_ifq_req __user *arg);void io_unregister_zctap_all (struct io_ring_ctx *ctx);int io_provide_ifq_region (struct io_zctap_ifq *ifq,u16 id); +int io_recvzc (struct io_kiocb *req,unsigned int issue_flags); +int io_recvzc_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe); +int io_zctap_recv (struct socket *sock,struct zctap_read_desc *zrd, +struct msghdr *msg,unsigned int flags); +#endif
Signature · 1 line
2.30.2
PATCH 13/15 io_uring: Make remove_ifq_region a delayed work call
In reply to Jonathan LemonThe page backing store should not be removed until all ouststanding packets are returned. The packets may be inflight, owned by the driver or sitting in a socket buffer.
The region holds a reference to the ifq, and when the ifq is closed, a delayed work item is scheduled which checks that all pages have been returned. When complete, then the region releases the ifq reference so it can be freed.
Currently, the work item will exit and leak pages after a timeout expires. This should not happen in normal operation.
--- include/linux/io_uring_types.h | 1 + io_uring/zctap.c | 77 +++++++++++++++++++++++++--------- 2 files changed, 59 insertions(+), 19 deletions(-)
include/linux/io_uring_types.h +1 -0
@@ -583,6 +583,7 @@ struct io_zctap_ifq {struct io_ring_ctx *ctx;void *region;struct ubuf_info *uarg; +refcount_t refcount;u16 queue_id;u16 id;u16 fill_bgid;
io_uring/zctap.c +58 -19
172 lines not on this page · raw message
Signature · 1 line
2.30.2
PATCH 14/15 io_uring: Add a buffer caching mechanism for zctap.
In reply to Jonathan LemonThis is based on the same concept as the page pool.
Here, there are 4 separate buffer sources:
cache - small (128) cache the driver can use locklessly. ptr_ring - buffers freed through skb_release_data() fillq - entries returned from the application freelist - spinlock protected pool of free entries.
The driver first tries the lockless cache, before attempting to refill it from the ptr ring. If there are still no buffers, then the fill ring is examined, before going to the freelist.
If the ptr_ring is full when buffers are released as the skb is dropped (or the driver returns the buffers), then they are placed back on the freelist.
--- io_uring/zctap.c | 128 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 99 insertions(+), 29 deletions(-)
io_uring/zctap.c +99 -29
@@ -18,8 +18,12 @@#define NR_ZCTAP_IFQS 1 +#define REGION_CACHE_COUNT 128 +#define REGION_REFILL_COUNT 64 +struct ifq_region {struct io_zctap_ifq *ifq; +int cache_count;int free_count;int nr_pages;u16 id;
@@ -28,6 +32,10 @@ struct ifq_region {struct delayed_work release_work;unsigned long delay_end; +u16 cache[REGION_CACHE_COUNT ]; + +struct ptr_ring ring; +struct io_zctap_buf *buf;u16 freelist[]; };
@@ -103,8 +111,29 @@ static bool io_zctap_put_buf_uref(struct io_zctap_buf *buf)return atomic_sub_and_test (IO_ZCTAP_UREF , &buf->refcount); } +/* if on exit/teardown path, can skip this work */ +static void io_zctap_recycle_buf (struct ifq_region *ifr, +struct io_zctap_buf *buf) +{ +int rc; + +if (in_serving_softirq ()) + rc =ptr_ring_produce (&ifr->ring, buf); +else + rc =ptr_ring_produce_bh (&ifr->ring, buf); + +if (rc) { +spin_lock (&ifr->freelist_lock); + + ifr->freelist[ifr->free_count++] = buf - ifr->buf; + +spin_unlock (&ifr->freelist_lock); + } +} +/* gets a user-supplied buffer from the fill queue * note: may drain N entries, but still have no usable buffers +* XXX add retry limit? */ static struct io_zctap_buf *io_zctap_get_buffer (struct io_zctap_ifq *ifq,u16 *buf_pgid)
@@ -150,40 +179,71 @@ static struct io_zctap_buf *io_zctap_get_buffer(struct io_zctap_ifq *ifq,return buf; } -/* if on exit/teardown path, can skip this work */ -static void io_zctap_recycle_buf (struct ifq_region *ifr, -struct io_zctap_buf *buf) +static int io_zctap_get_buffers (struct io_zctap_ifq *ifq,u16 *cache,int n) { -spin_lock (&ifr->freelist_lock); +struct io_zctap_buf *buf; +int i; - ifr->freelist[ifr->free_count++] = buf - ifr->buf; - -spin_unlock (&ifr->freelist_lock); +for (i =0 ; i < n; i++) { + buf =io_zctap_get_buffer (ifq, &cache[i]); +if (!buf) +break ; + } +return i; }struct io_zctap_buf *io_zctap_get_buf (struct io_zctap_ifq *ifq,int refc) { -struct ifq_region *ifr = ifq->region;struct io_zctap_buf *buf; +struct ifq_region *ifr; +int count;u16 pgid; + ifr = ifq->region; +if (ifr->cache_count) + goto out; + +if (!__ptr_ring_empty (&ifr->ring)) { +do { + buf =__ptr_ring_consume (&ifr->ring); +if (!buf) +break ; + ifr->cache[ifr->cache_count++] = buf - ifr->buf; + }while (ifr->cache_count <REGION_REFILL_COUNT ); + +if (ifr->cache_count) + goto out; + } + + count =io_zctap_get_buffers (ifq, ifr->cache,REGION_REFILL_COUNT ); + ifr->cache_count += count; + +if (ifr->cache_count) + goto out; +spin_lock (&ifr->freelist_lock); - buf =NULL ; -if (ifr->free_count) { - pgid = ifr->freelist[--ifr->free_count]; - buf = &ifr->buf[pgid]; - } + count =min_t (int, ifr->free_count,REGION_CACHE_COUNT ); + ifr->free_count -= count; + ifr->cache_count += count; +memcpy (ifr->cache, &ifr->freelist[ifr->free_count], + count *sizeof (u16));spin_unlock (&ifr->freelist_lock); -if (!buf) { - buf =io_zctap_get_buffer (ifq, &pgid); -if (!buf) -return NULL ; - } +if (ifr->cache_count) + goto out; -WARN_ON (atomic_read (&buf->refcount)); +return NULL ; + +out: + pgid = ifr->cache[--ifr->cache_count]; + buf = &ifr->buf[pgid]; + +WARN_RATELIMIT (atomic_read (&buf->refcount), +"pgid:%d refc:%d cache_count:%d\n" , + pgid,atomic_read (&buf->refcount), + ifr->cache_count);atomic_set (&buf->refcount, refc &IO_ZCTAP_KREF_MASK );return buf;
@@ -278,6 +338,7 @@ static void io_remove_ifq_region_work(struct work_struct *work) }io_zctap_ifq_put (ifr->ifq); +ptr_ring_cleanup (&ifr->ring,NULL );kvfree (ifr->buf);kvfree (ifr); }
@@ -365,16 +426,18 @@ int io_provide_ifq_region(struct io_zctap_ifq *ifq, u16 id)if (imu->ubuf & ~PAGE_MASK || imu->ubuf_end & ~PAGE_MASK )return -EFAULT ; + err = -ENOMEM ; nr_pages = imu->nr_bvecs; ifr =kvmalloc (struct_size (ifr, freelist, nr_pages),GFP_KERNEL );if (!ifr) -return -ENOMEM ; + goto fail; ifr->buf =kvmalloc_array (nr_pages,sizeof (*ifr->buf),GFP_KERNEL ); -if (!ifr->buf) { -kvfree (ifr); -return -ENOMEM ; - } +if (!ifr->buf) + goto fail_buf; + +if (ptr_ring_init (&ifr->ring,1024 ,GFP_KERNEL )) + goto fail_ring;spin_lock_init (&ifr->freelist_lock); ifr->nr_pages = nr_pages;
@@ -382,18 +445,25 @@ int io_provide_ifq_region(struct io_zctap_ifq *ifq, u16 id) ifr->id = id; ifr->ifq = ifq; ifr->delay_end =0 ; + ifr->cache_count =0 ; err =io_zctap_map_region (ifr, imu); -if (err) { -kvfree (ifr->buf); -kvfree (ifr); -return err; - } +if (err) + goto fail_map; ifq->region = ifr;refcount_inc (&ifq->refcount);return 0 ; + +fail_map: +ptr_ring_cleanup (&ifr->ring,NULL ); +fail_ring: +kvfree (ifr->buf); +fail_buf: +kvfree (ifr); +fail: +return err; } static int __io_queue_mgmt(struct net_device *dev, struct io_zctap_ifq *ifq,
Signature · 1 line
2.30.2
PATCH 15/15 io_uring: Notify the application as the fillq is drained.
In reply to Jonathan LemonUserspace maintains a free count of space available in the fillq, and only returns entries based on the available space. As the kernel removes these entries, it needs to notify the application so more buffers can be queued.
Only one outstanding notifier per queue is used, and it provides the most recent count of entries removed from the queue.
Also post a notifier when the NIC is unable to obtain any buffers. When this happens, the NIC may just drop packets or stall.
--- io_uring/zctap.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)
io_uring/zctap.c +58 -0
@@ -15,6 +15,7 @@#include "zctap.h" #include "rsrc.h" #include "kbuf.h" +#include "refs.h" #define NR_ZCTAP_IFQS 1
@@ -26,7 +27,9 @@ struct ifq_region {int cache_count;int free_count;int nr_pages; +int taken;u16 id; +bool empty;spinlock_t freelist_lock;struct delayed_work release_work;
@@ -44,8 +47,14 @@ struct ifq_region {struct io_zctap_ifq_priv {struct io_zctap_ifq ifq;struct ubuf_info uarg; +struct io_kiocb req; }; +static struct io_kiocb *io_zctap_ifq_notifier (struct io_zctap_ifq *ifq) +{ +return &((struct io_zctap_ifq_priv *)ifq)->req; +} +static void io_zctap_ifq_put (struct io_zctap_ifq *ifq);typedef int (*bpf_op_t )(struct net_device *dev,struct netdev_bpf *bpf);
@@ -131,6 +140,34 @@ static void io_zctap_recycle_buf(struct ifq_region *ifr, } } +struct io_zctap_notif { +struct file *file; +u64 udata; +int res; +int cflags; +}; + +static void io_zctap_post_notify (struct io_kiocb *req,bool *locked) +{ + struct io_zctap_notif *n = io_kiocb_to_cmd(req, struct io_zctap_notif); + +io_post_aux_cqe (req->ctx, n->udata, n->res, n->cflags, true); +io_req_task_complete (req, locked); +} + +static void io_zctap_notify (struct io_kiocb *req,int bgid,int count) +{ + struct io_zctap_notif *n = io_kiocb_to_cmd(req, struct io_zctap_notif); + + n->udata =0xface0000 ;/* XXX */ + n->res = (bgid <<16 ) | count; + n->cflags =IORING_CQE_F_BUFFER |IORING_CQE_F_NOTIF ; + +req_ref_get (req); + req->io_task_work.func = io_zctap_post_notify; +io_req_task_work_add (req); +} +/* gets a user-supplied buffer from the fill queue * note: may drain N entries, but still have no usable buffers * XXX add retry limit?
@@ -159,6 +196,7 @@ static struct io_zctap_buf *io_zctap_get_buffer(struct io_zctap_ifq *ifq,if (!addr)return NULL ; + ifr->taken++; pgid = addr &0xffff ; region_id = (addr >>16 ) &0xffff ;if (region_id) {
@@ -196,6 +234,7 @@ struct io_zctap_buf *io_zctap_get_buf(struct io_zctap_ifq *ifq, int refc) {struct io_zctap_buf *buf;struct ifq_region *ifr; +struct io_kiocb *req;int count;u16 pgid;
@@ -218,6 +257,12 @@ struct io_zctap_buf *io_zctap_get_buf(struct io_zctap_ifq *ifq, int refc) count =io_zctap_get_buffers (ifq, ifr->cache,REGION_REFILL_COUNT ); ifr->cache_count += count; + req =io_zctap_ifq_notifier (ifq); +if (ifr->taken &&atomic_read (&req->refs) ==1 ) { +io_zctap_notify (req, ifq->fill_bgid, ifr->taken); + ifr->taken =0 ; + } +if (ifr->cache_count) goto out;
@@ -234,11 +279,17 @@ struct io_zctap_buf *io_zctap_get_buf(struct io_zctap_ifq *ifq, int refc)if (ifr->cache_count) goto out; +if (!ifr->empty &&atomic_read (&req->refs) ==1 ) { +io_zctap_notify (req, ifq->fill_bgid,0 ); + ifr->empty = true; + } +return NULL ; out: pgid = ifr->cache[--ifr->cache_count]; buf = &ifr->buf[pgid]; + ifr->empty = false; WARN_RATELIMIT(atomic_read(&buf->refcount), "pgid:%d refc:%d cache_count:%d\n",
@@ -445,6 +496,8 @@ int io_provide_ifq_region(struct io_zctap_ifq *ifq, u16 id) ifr->id = id; ifr->ifq = ifq; ifr->delay_end =0 ; + ifr->taken =0 ; + ifr->empty = false; ifr->cache_count =0 ; err =io_zctap_map_region (ifr, imu);
@@ -533,6 +586,11 @@ static struct io_zctap_ifq *io_zctap_ifq_alloc(struct io_ring_ctx *ctx) priv->ifq.ctx = ctx; priv->ifq.queue_id =-1 ; priv->ifq.uarg = &priv->uarg; + + priv->req.ctx = ctx; + priv->req.task = current; +io_req_set_refcount (&priv->req); +return &priv->ifq; }
Signature · 1 line
2.30.2
Quoted 32 lines · Jonathan Lemon
This series introduces network RX zerocopy for
io_uring.This is an evolution of the earlier zctap work, re-targeted to use
io_uringas the userspace API. The code is intends to provide a ZC RX path for upper-level networking protocols (aka TCP and UDP), with a focus on focuses on host-provided memory (not GPU memory).This patch contains the upper-level core code required for operation, but does not not contain the network driver side changes required for true zero-copy operation. The
io_uringRECV_ZCopcode will work without hardware support, albeit in copy mode.The intent is to use a network driver which provides header/data splitting, so the frame header which is processed by the networking stack is not placed in user memory.
The code is successfully receiving a zero-copy TCP stream from a remote sender.
There is a liburing fork providing the needed wrappers:
https://github.com/jlemon/liburing/tree/zctap
Which contains an examples/io_uring-net test application exercising these features. A sample run:
# ./io_uring-net -i eth1 -q 20 -p 9999 -r 3000 copy bytes: 1938872 ZC bytes: 996683008 Total bytes: 998621880, nsec:1025219375 Rate: 7.79 Gb/s
Quoted 7 lines
If no queue is specified, then non-zc mode is used:
# ./io_uring-net -p 9999 copy bytes: 998621880 ZC bytes: 0 Total bytes: 998621880, nsec:1051515726 Rate: 7.60 Gb/s
Haven't dive into your test case yet, but the performance data looks disappointing
I don't know why we need zerocopy if we can't get a big performance gain.
Have you tested large messages with jumbo or LRO enabled ?
Thanks
Quoted 72 lines
There is also an iperf3 fork as well:
https://github.com/jlemon/iperf/tree/io_uring
This allows running single tests with either:
* select (normal iperf3)
*io_uringREAD
*io_uringRECV_ZCcopy mode
*io_uringRECV_ZChardware modeCurrent testing shows similar BW between
RECV_ZCand READ modes (running at 22Gbit/sec), but a reduction of ~50% of MemBW.High level description:
The application allocates a frame backing store, and provides this to the kernel for use. An interface queue is requested from the networking device, and incoming frames are deposited into the provided memory region. The NIC should provide a header splitting feature, so only the frame payload is placed in the user space area.
Responsibility for correctly steering incoming frames to the queue is outside the scope of this work - it is assumed that the user has set steering rules up separately.
Incoming frames are sent up the stack as skb's and eventually land in the application's socket receive queue. This differs from
AF_XDP, which receives raw frames directly to userspace, without protocol processing.The
RECV_ZCopcode then returns an iov[] style vector which points to the data in userspace memory. When the application has completed processing of the data, the buffers are returned back to the kernel through a fill ring for reuse.Jonathan Lemon (15):
io_uring: add zctap ifq definition
netdevice: addSETUP_ZCTAPto thenetdev_bpfstructure
io_uring: add register ifq opcode
io_uring: create a zctap region for a mapped buffer
io_uring: mark pages in ifq region with zctap information.
io_uring: Provide driver API for zctap packet buffers.
io_uring: Allocate zctap device buffers and dma map them.
io_uring: Add zctap buffer get/put functions and refcounting.
skbuff: IntroduceSKBFL_FIXED_FRAGandskb_fixed()
io_uring: Allocate a uarg for use by the ifq RX
io_uring: Define the zctap iov[] returned to the user.
io_uring: addOP_RECV_ZCcommand.
io_uring: Makeremove_ifq_regiona delayed work call
io_uring: Add a buffer caching mechanism for zctap.
io_uring: Notify the application as the fillq is drained.include/linux/io_uring.h | 47 ++ include/linux/io_uring_types.h | 12 + include/linux/netdevice.h | 6 + include/linux/skbuff.h | 10 +- include/uapi/linux/io_uring.h | 24 + io_uring/Makefile | 3 +- io_uring/io_uring.c | 8 + io_uring/kbuf.c | 13 + io_uring/kbuf.h | 2 + io_uring/net.c | 123 ++++ io_uring/opdef.c | 15 + io_uring/zctap.c | 1001 ++++++++++++++++++++++++++++++++ io_uring/zctap.h | 31 + 13 files changed, 1293 insertions(+), 2 deletions(-)create mode 100644
io_uring/zctap.c
create mode 100644io_uring/zctap.hSignature · 1 line
2.30.2
Quoted 6 lines · Dust Li
Haven't dive into your test case yet, but the performance data looks disappointing
I don't know why we need zerocopy if we can't get a big performance gain.
The cited numbers are intended to show that there is no network bandwidth performance drop from using RECV_ZC. For the systems under test (25Gb, 50Gb), we are getting linerate already, so there wouldn't be any BW improvement.
The zerocopy gains come from a reduction in memory bandwith, removing the user memcpy overhead, reducing the CPU usage. This is why I put up the changes for iperf3, so cpu and membw usage metrics can be collected while under network load.
Quoted 1 line
Have you tested large messages with jumbo or LRO enabled ?
Haven't tested jumbo frames, but these are with GRO support.
Signature · 1 line
Jonathan