zero-copy RX for io_uring
Subsystems (4)
cover zero-copy RX for io_uring
This series is a RFC for io_uring/zctap. This is an evolution of the earlier zctap work, re-targeted to use io_uring as the userspace API. The current code is intended to provide a zero-copy RX path for upper-level networking protocols (aka TCP and UDP). The current draft focuses on host-provided memory (not GPU memory).
This RFC contains the upper-level core code required for operation, with the intent of soliciting feedback on the general API. This does not contain the network driver side changes required for complete operation. Also please note that as an RFC, there are some things which are incomplete or in need of rework.
The intent is to use a network driver which provides header/data splitting, so the frame header (which is processed by the networking stack) does not reside in user memory.
The code is roughly working (in that it has successfully received a TCP stream from a remote sender), but as an RFC, the intent is to solicit feedback on the API and overall design. The current code will also work with system pages, copying the data out to the application - this is intended as a fallback/testing path.
High level description:
The application allocates a frame backing store, and provides this to the kernel for use. An interface queue is requested from the networking device, and incoming frames are deposited into the provided memory region.
Responsibility for correctly steering incoming frames to the queue is outside the scope of this work - it is assumed that the user has set steering rules up separately.
Incoming frames are sent up the stack as skb's and eventually land in the application's socket receive queue. This differs from AF_XDP, which receives raw frames directly to userspace, without protocol processing.
The RECV_ZC opcode then returns an iov[] style vector which points to the data in userspace memory. When the application has completed processing of the data, the buffer is returned back to the kernel through a fill ring for reuse.
Jonathan Lemon (9):
io_uring: add zctap ifq definition
netdevice: add SETUP_ZCTAP to the netdev_bpf structure
io_uring: add register ifq opcode
io_uring: add provide_ifq_region opcode
io_uring: Add io_uring zctap iov structure and helpers
io_uring: introduce reference tracking for user pages.
page_pool: add page allocation and free hooks.
io_uring: provide functions for the page_pool.
io_uring: add OP_RECV_ZC command.
include/linux/io_uring.h | 24 ++ include/linux/io_uring_types.h | 10 + include/linux/netdevice.h | 6 + include/net/page_pool.h | 6 + include/uapi/linux/io_uring.h | 26 ++ io_uring/Makefile | 3 +- io_uring/io_uring.c | 10 + io_uring/kbuf.c | 13 + io_uring/kbuf.h | 2 + io_uring/net.c | 123 ++++++ io_uring/opdef.c | 23 + io_uring/zctap.c | 749 +++++++++++++++++++++++++++++++++ io_uring/zctap.h | 20 + net/core/page_pool.c | 41 +- 14 files changed, 1048 insertions(+), 8 deletions(-) create mode 100644 io_uring/zctap.c create mode 100644 io_uring/zctap.h
Signature · 1 line
2.30.2
RFC 1/9 io_uring: add zctap ifq definition
Add structure definition for io_zctap_ifq for use by lower level networking hooks.
--- include/linux/io_uring_types.h | 9 +++++++++ 1 file changed, 9 insertions(+)
include/linux/io_uring_types.h +9 -0
@@ -323,6 +323,7 @@ struct io_ring_ctx {struct io_mapped_ubuf *dummy_ubuf;struct io_rsrc_data *file_data;struct io_rsrc_data *buf_data; +struct xarray zctap_ifq_xa;struct delayed_work rsrc_put_work;struct llist_head rsrc_put_llist;
@@ -578,4 +579,12 @@ struct io_overflow_cqe { struct io_uring_cqe cqe; }; +struct io_zctap_ifq { +struct net_device *dev; +struct io_ring_ctx *ctx; +u16 queue_id; +u16 id; +u16 fill_bgid; +}; +#endif
Signature · 1 line
2.30.2
RFC 2/9 netdevice: add SETUP_ZCTAP to the netdev_bpf structure
In reply to Jonathan LemonThis command requests the networking device setup or teardown a new interface queue, backed by a region of user supplied memory.
The queue will be managed by io-uring.
--- include/linux/netdevice.h | 6 ++++++ 1 file changed, 6 insertions(+)
include/linux/netdevice.h +6 -0
@@ -979,6 +979,7 @@ enum bpf_netdev_command {BPF_OFFLOAD_MAP_ALLOC ,BPF_OFFLOAD_MAP_FREE ,XDP_SETUP_XSK_POOL , + XDP_SETUP_ZCTAP, };struct bpf_prog_offload_ops ;
@@ -1017,6 +1018,11 @@ struct netdev_bpf {struct xsk_buff_pool *pool;u16 queue_id; } xsk; +/* XDP_SETUP_ZCTAP */ +struct { +struct io_zctap_ifq *ifq; +u16 queue_id; + } zct; }; };
Signature · 1 line
2.30.2
RFC 3/9 io_uring: add register ifq opcode
In reply to Jonathan LemonAdd initial support for support for hooking in zero-copy interface queues to io_uring. This command requests a user-managed queue from the specified network device.
This only includes the register opcode, unregistration is currently done implicitly when the ring is removed.
--- include/uapi/linux/io_uring.h | 14 ++++ io_uring/Makefile | 3 +- io_uring/io_uring.c | 10 +++ io_uring/zctap.c | 146 ++++++++++++++++++++++++++++++++++ io_uring/zctap.h | 11 +++ 5 files changed, 183 insertions(+), 1 deletion(-) create mode 100644 io_uring/zctap.c create mode 100644 io_uring/zctap.h
include/uapi/linux/io_uring.h +14 -0
@@ -473,6 +473,9 @@ enum {/* register a range of fixed file slots for automatic slot allocation */ IORING_REGISTER_FILE_ALLOC_RANGE =25 , +/* register a network ifq for zerocopy RX */ +IORING_REGISTER_IFQ =26 , +/* this goes last */ IORING_REGISTER_LAST };
@@ -649,6 +652,17 @@ struct io_uring_recvmsg_out {__u32 flags; }; +/* +* Argument for IORING_REGISTER_IFQ +*/ +struct io_uring_ifq_req { +__u32 ifindex; +__u16 queue_id; +__u16 ifq_id; +__u16 fill_bgid; +__u16 __pad[3 ]; +}; + #ifdef __cplusplus }#endif
io_uring/Makefile +2 -1
@@ -7,5 +7,6 @@ obj-$(CONFIG_IO_URING) += io_uring.o xattr.o nop.o fs.o splice.o \ openclose.o uring_cmd.o epoll.o \ statx.o net.o msg_ring.o timeout.o \ sqpoll.o fdinfo.o tctx.o poll.o \ - cancel.o kbuf.o rsrc.o rw.o opdef.o notif.o + cancel.o kbuf.o rsrc.o rw.o opdef.o \ + notif.o zctap.o obj-$(CONFIG_IO_WQ) += io-wq.o
io_uring/io_uring.c +10 -0
@@ -91,6 +91,7 @@#include "cancel.h" #include "net.h" #include "notif.h" +#include "zctap.h" #include "timeout.h" #include "poll.h"
@@ -321,6 +322,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)INIT_WQ_LIST (&ctx->locked_free_list);INIT_DELAYED_WORK (&ctx->fallback_work, io_fallback_req_func);INIT_WQ_LIST (&ctx->submit_state.compl_reqs); +xa_init (&ctx->zctap_ifq_xa);return ctx; err:kfree (ctx->dummy_ubuf);
@@ -2639,6 +2641,8 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)__io_cqring_overflow_flush (ctx, true);xa_for_each (&ctx->personalities, index, creds)io_unregister_personality (ctx, index); +xa_for_each (&ctx->zctap_ifq_xa, index, creds) +io_unregister_zctap_ifq (ctx, index);if (ctx->rings)io_poll_remove_all (ctx,NULL , true);mutex_unlock (&ctx->uring_lock);
@@ -3839,6 +3843,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_register_file_alloc_range (ctx, arg);break ; +case IORING_REGISTER_IFQ : + ret = -EINVAL ; +if (!arg || nr_args !=1 ) +break ; + ret =io_register_ifq (ctx, arg); +break ;default : ret = -EINVAL ;break ;
io_uring/zctap.c new file +146 -0
146 lines not on this page · raw message
io_uring/zctap.h new file +11 -0
@@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-2.0 +#ifndef IOU_ZCTAP_H +#define IOU_ZCTAP_H + +int io_register_ifq (struct io_ring_ctx *ctx, +struct io_uring_ifq_req __user *arg); +int io_unregister_ifq (struct io_ring_ctx *ctx, +struct io_uring_ifq_req __user *arg); +int io_unregister_zctap_ifq (struct io_ring_ctx *ctx,unsigned long index); + +#endif
Signature · 1 line
2.30.2
RFC 4/9 io_uring: add provide_ifq_region opcode
In reply to Jonathan LemonThis opcode takes part or all of a memory region that was previously registered with io_uring, and assigns it as the backing store for the specified ifq.
The entire region is registered instead of providing individual bufferrs, as this allows the hardware to select the optimal buffer size for incoming packets.
--- include/linux/io_uring_types.h | 1 + include/uapi/linux/io_uring.h | 1 + io_uring/opdef.c | 9 ++++ io_uring/zctap.c | 96 ++++++++++++++++++++++++++++++++++ io_uring/zctap.h | 4 ++ 5 files changed, 111 insertions(+)
include/linux/io_uring_types.h +1 -0
@@ -582,6 +582,7 @@ struct io_overflow_cqe {struct io_zctap_ifq {struct net_device *dev;struct io_ring_ctx *ctx; +void *region;/* XXX relocate? */ u16 queue_id;u16 id;u16 fill_bgid;
include/uapi/linux/io_uring.h +1 -0
@@ -206,6 +206,7 @@ enum io_uring_op {IORING_OP_SOCKET ,IORING_OP_URING_CMD ,IORING_OP_SEND_ZC , +IORING_OP_PROVIDE_IFQ_REGION ,/* this goes last, obviously */ IORING_OP_LAST,
io_uring/opdef.c +9 -0
@@ -33,6 +33,7 @@#include "poll.h" #include "cancel.h" #include "rw.h" +#include "zctap.h" static int io_no_issue (struct io_kiocb *req,unsigned int issue_flags) {
@@ -488,6 +489,14 @@ const struct io_op_def io_op_defs[] = { .prep = io_eopnotsupp_prep, #endif }, + [IORING_OP_PROVIDE_IFQ_REGION ] = { + .audit_skip =1 , + .iopoll =1 , + .buffer_select =1 , + .name ="PROVIDE_IFQ_REGION" , + .prep = io_provide_ifq_region_prep, + .issue = io_provide_ifq_region, + }, };const char *io_uring_get_opcode (u8 opcode)
io_uring/zctap.c +96 -0
110 lines not on this page · raw message
io_uring/zctap.h +4 -0
@@ -8,4 +8,8 @@ int io_unregister_ifq(struct io_ring_ctx *ctx, struct io_uring_ifq_req __user *arg);int io_unregister_zctap_ifq (struct io_ring_ctx *ctx,unsigned long index); +int io_provide_ifq_region_prep (struct io_kiocb *req, +const struct io_uring_sqe *sqe); +int io_provide_ifq_region (struct io_kiocb *req,unsigned int issue_flags); +#endif
Signature · 1 line
2.30.2
RFC 5/9 io_uring: Add io_uring zctap iov structure and helpers
In reply to Jonathan LemonWith networking zero-copy receive, the incoming data is placed directly into user-supplied buffers. Instead of returning the buffer address, return the buffer group id and buffer id, and let the application figure out the base address.
Add helpers for storing and retrieving the encoding, which is stored in the page_private field. This will be used in the zerocopy RX routine, when handling pages from skb fragments.
--- include/uapi/linux/io_uring.h | 10 +++++++++ io_uring/zctap.c | 39 ++++++++++++++++++++++++++++++++++- 2 files changed, 48 insertions(+), 1 deletion(-)
include/uapi/linux/io_uring.h +10 -0
@@ -664,6 +664,16 @@ struct io_uring_ifq_req {__u16 __pad[3 ]; }; +struct io_uring_zctap_iov { +__u32 off; +__u32 len; +__u16 bgid; +__u16 bid; +__u16 ifq_id; +__u16 resv; +}; + + #ifdef __cplusplus }#endif
io_uring/zctap.c +38 -1
@@ -19,6 +19,26 @@ static DEFINE_XARRAY_ALLOC1(io_zctap_ifq_xa);typedef int (*bpf_op_t )(struct net_device *dev,struct netdev_bpf *bpf); +static u64 zctap_page_info (u16 region_id,u16 pgid,u16 ifq_id) +{ +return (u64 )region_id <<32 | (u64 )pgid <<16 | ifq_id; +} + +static u16 zctap_page_region_id (const struct page *page) +{ +return (page_private (page) >>32 ) &0xffff ; +} + +static u16 zctap_page_id (const struct page *page) +{ +return (page_private (page) >>16 ) &0xffff ; +} + +static u16 zctap_page_ifq_id (const struct page *page) +{ +return page_private (page) &0xffff ; +} +static int __io_queue_mgmt (struct net_device *dev,struct io_zctap_ifq *ifq,u16 *queue_id) {
@@ -213,8 +233,9 @@ int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags) { struct io_ifq_region *r = io_kiocb_to_cmd(req, struct io_ifq_region);struct ifq_region *ifr; -int i, idx, nr_pages; +int i, id, idx, nr_pages;struct page *page; +u64 info; nr_pages = r->len >>PAGE_SHIFT ; idx = (r->addr - req->imu->ubuf) >>PAGE_SHIFT ;
@@ -231,12 +252,28 @@ int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags) ifr->start = r->addr; ifr->end = r->addr + r->len; + id = r->ifq->id;for (i =0 ; i < nr_pages; i++, idx++) { page = req->imu->bvec[idx].bv_page; +if (PagePrivate (page)) + goto out; +SetPagePrivate (page); + info =zctap_page_info (r->bgid, idx + i, id); +set_page_private (page, info); ifr->page[i] = page; }WRITE_ONCE (r->ifq->region, ifr);return 0 ; +out: +while (i--) { + page = req->imu->bvec[idx + i].bv_page; +ClearPagePrivate (page); +set_page_private (page,0 ); + } + +kvfree (ifr); + +return -EEXIST ; }
Signature · 1 line
2.30.2
RFC 6/9 io_uring: introduce reference tracking for user pages.
In reply to Jonathan LemonThis is currently a WIP.
If only part of a page is used by a skb fragment, and then provided to the user, the page should not be reused by the kernel until all sub-page fragments are not in use.
If only full pages are used (and not sub-page fragments), then this code shouldn't be needed.
--- io_uring/zctap.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
io_uring/zctap.c +34 -0
@@ -183,9 +183,36 @@ struct ifq_region {int count;int imu_idx;int nr_pages; +u8 *page_uref;struct page *page[]; }; +static void io_add_page_uref (struct ifq_region *ifr,u16 pgid) +{ +if (WARN_ON (!ifr)) +return ; + +if (WARN_ON (pgid < ifr->imu_idx)) +return ; + + ifr->page_uref[pgid - ifr->imu_idx]++; +} + +static bool io_put_page_last_uref (struct ifq_region *ifr,u64 addr) +{ +int idx; + +if (WARN_ON (addr < ifr->start || addr > ifr->end)) +return false; + + idx = (addr - ifr->start) >>PAGE_SHIFT ; + +if (WARN_ON (!ifr->page_uref[idx])) +return false; + +return --ifr->page_uref[idx] ==0 ; +} +int io_provide_ifq_region_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
@@ -244,6 +271,11 @@ int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags)if (!ifr)return -ENOMEM ; + ifr->page_uref =kvmalloc_array (nr_pages,sizeof (u8),GFP_KERNEL ); +if (!ifr->page_uref) { +kvfree (ifr); +return -ENOMEM ; + } ifr->nr_pages = nr_pages; ifr->imu_idx = idx;
@@ -261,6 +293,7 @@ int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags) info =zctap_page_info (r->bgid, idx + i, id);set_page_private (page, info); ifr->page[i] = page; + ifr->page_uref[i] =0 ; }WRITE_ONCE (r->ifq->region, ifr);
@@ -273,6 +306,7 @@ int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags)set_page_private (page,0 ); } +kvfree (ifr->page_uref);kvfree (ifr);return -EEXIST ;
Signature · 1 line
2.30.2
RFC 7/9 page_pool: add page allocation and free hooks.
In reply to Jonathan LemonIn order to allow for user-allocated page backing, add hooks to the page pool so pages can be obtained and released from a user-supplied provider instead of the system page allocator.
skbs are marked with skb_mark_for_recycle() if they contain pages belonging to a page pool, and page_put() will deliver the pages back to the pool instead of freeing them to the system page allocator.
--- include/net/page_pool.h | 6 ++++++ net/core/page_pool.c | 41 ++++++++++++++++++++++++++++++++++------- 2 files changed, 40 insertions(+), 7 deletions(-)
include/net/page_pool.h +6 -0
@@ -82,6 +82,12 @@ struct page_pool_params {unsigned int offset;/* DMA addr offset */ void (*init_callback)(struct page *page,void *arg);void *init_arg; +struct page *(*alloc_pages)(void *arg,int nid,gfp_t gfp, +unsigned int order); +unsigned long (*alloc_bulk)(void *arg,gfp_t gfp,int nid, +unsigned long nr_pages, +struct page **page_array); +void (*put_page)(void *arg,struct page *page); };#ifdef CONFIG_PAGE_POOL_STATS
net/core/page_pool.c +34 -7
82 lines not on this page · raw message
Signature · 1 line
2.30.2
RFC 8/9 io_uring: provide functions for the page_pool.
In reply to Jonathan LemonThese functions are called by the page_pool, in order to refill the pool with user-supplied pages, or returning excess pages back from the pool.
If no pages are present in the region cache, then an attempt is made to obtain more pages from the interface fill queue.
--- include/linux/io_uring.h | 24 ++++++++++++ io_uring/kbuf.c | 13 +++++++ io_uring/kbuf.h | 2 + io_uring/zctap.c | 82 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 121 insertions(+)
include/linux/io_uring.h +24 -0
@@ -37,6 +37,14 @@ void __io_uring_free(struct task_struct *tsk);void io_uring_unreg_ringfd (void );const char *io_uring_get_opcode (u8 opcode); +struct io_zctap_ifq ; +struct page *io_zctap_ifq_get_page (struct io_zctap_ifq *ifq, +unsigned int order); +unsigned long io_zctap_ifq_get_bulk (struct io_zctap_ifq *ifq, +unsigned long nr_pages, +struct page **page_array); +bool io_zctap_ifq_put_page (struct io_zctap_ifq *ifq,struct page *page); +static inline void io_uring_files_cancel (void ) {if (current->io_uring) {
@@ -80,6 +88,22 @@ static inline const char *io_uring_get_opcode(u8 opcode) {return "" ; } +static inline struct page *io_zctap_ifq_get_page (struct io_zctap_ifq *ifq, +unsigned int order) +{ +return NULL ; +} +sttaic unsigned long io_zctap_ifq_get_bulk (struct io_zctap_ifq *ifq, +unsigned long nr_pages, +struct page **page_array) +{ +return 0 ; +} +bool io_zctap_ifq_put_page (struct io_zctap_ifq *ifq,struct page *page) +{ +return false; +} +#endif #endif
io_uring/kbuf.c +13 -0
@@ -188,6 +188,19 @@ void __user *io_buffer_select(struct io_kiocb *req, size_t *len,return ret; } +/* XXX May called from the driver, in napi context. */ +u64 io_zctap_buffer (struct io_kiocb *req,size_t *len) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_buffer_list *bl; +void __user *ret =NULL ; + + bl =io_buffer_get_list (ctx, req->buf_index); +if (likely (bl)) + ret =io_ring_buffer_select (req, len, bl,IO_URING_F_UNLOCKED ); +return (u64 )ret; +} +static __cold int io_init_bl_list (struct io_ring_ctx *ctx) {int i;
io_uring/kbuf.h +2 -0
@@ -50,6 +50,8 @@ unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags);void io_kbuf_recycle_legacy (struct io_kiocb *req,unsigned issue_flags); +u64 io_zctap_buffer (struct io_kiocb *req,size_t *len); +static inline void io_kbuf_recycle_ring (struct io_kiocb *req) {/*
io_uring/zctap.c +82 -0
85 lines not on this page · raw message
Signature · 1 line
2.30.2
RFC 9/9 io_uring: add OP_RECV_ZC command.
In reply to Jonathan LemonThis is still a WIP. The current code (temporarily) uses addr3 as a hack in order to leverage code in io_recvmsg_prep.
The recvzc opcode uses a metadata buffer either supplied directly with buf/len, or indirectly from the buffer group. The expectation is that this buffer is then filled with an array of io_uring_zctap_iov structures, which point to the data in user-memory.
addr3 = (readlen << 32) | (copy_bgid << 16) | ctx->ifq_id;
The amount of returned data is limited by the number of iovs that the metadata area can hold, and also the readlen parameter.
As a fallback (and for testing purposes), if the skb data is not present in user memory (perhaps due to system misconfiguration), then a seprate buffer is obtained from the copy_bgid and the data is copied into user-memory.
--- include/uapi/linux/io_uring.h | 1 + io_uring/net.c | 123 ++++++++++++ io_uring/opdef.c | 14 ++ io_uring/zctap.c | 354 ++++++++++++++++++++++++++++++++++ io_uring/zctap.h | 5 + 5 files changed, 497 insertions(+)
include/uapi/linux/io_uring.h +1 -0
@@ -207,6 +207,7 @@ enum io_uring_op {IORING_OP_URING_CMD ,IORING_OP_SEND_ZC ,IORING_OP_PROVIDE_IFQ_REGION , +IORING_OP_RECV_ZC ,/* this goes last, obviously */ IORING_OP_LAST,
io_uring/net.c +123 -0
141 lines not on this page · raw message
io_uring/opdef.c +14 -0
@@ -497,6 +497,20 @@ const struct io_op_def io_op_defs[] = { .prep = io_provide_ifq_region_prep, .issue = io_provide_ifq_region, }, + [IORING_OP_RECV_ZC] = { + .name = "RECV_ZC", + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollin = 1, + .buffer_select = 1, + .ioprio = 1, +#if defined(CONFIG_NET ) + .prep = io_recvzc_prep, + .issue = io_recvzc, +#else + .prep = io_eopnotsupp_prep, +#endif + }, };const char *io_uring_get_opcode (u8 opcode)
io_uring/zctap.c +354 -0
363 lines not on this page · raw message
io_uring/zctap.h +5 -0
@@ -12,4 +12,9 @@ int io_provide_ifq_region_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);int io_provide_ifq_region (struct io_kiocb *req,unsigned int issue_flags); +int io_recvzc (struct io_kiocb *req,unsigned int issue_flags); +int io_recvzc_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe); +int io_zctap_recv (struct io_zctap_ifq *ifq,struct socket *sock, +struct msghdr *msg,int flags,u32 datalen,u16 copy_bgid); +#endif
Signature · 1 line
2.30.2
Quoted 38 lines · Jonathan Lemon
This series is a RFC for
io_uring/zctap. This is an evolution of the earlier zctap work, re-targeted to useio_uringas the userspace API. The current code is intended to provide a zero-copy RX path for upper-level networking protocols (aka TCP and UDP). The current draft focuses on host-provided memory (not GPU memory).This RFC contains the upper-level core code required for operation, with the intent of soliciting feedback on the general API. This does not contain the network driver side changes required for complete operation. Also please note that as an RFC, there are some things which are incomplete or in need of rework.
The intent is to use a network driver which provides header/data splitting, so the frame header (which is processed by the networking stack) does not reside in user memory.
The code is roughly working (in that it has successfully received a TCP stream from a remote sender), but as an RFC, the intent is to solicit feedback on the API and overall design. The current code will also work with system pages, copying the data out to the application - this is intended as a fallback/testing path.
High level description:
The application allocates a frame backing store, and provides this to the kernel for use. An interface queue is requested from the networking device, and incoming frames are deposited into the provided memory region.
Responsibility for correctly steering incoming frames to the queue is outside the scope of this work - it is assumed that the user has set steering rules up separately.
Incoming frames are sent up the stack as skb's and eventually land in the application's socket receive queue. This differs from
AF_XDP, which receives raw frames directly to userspace, without protocol processing.
Quoted 4 lines
The
RECV_ZCopcode then returns an iov[] style vector which points to the data in userspace memory. When the application has completed processing of the data, the buffer is returned back to the kernel through a fill ring for reuse.
Interesting work ! Any userspace demo and performance data ?
Quoted 32 lines
Jonathan Lemon (9):
io_uring: add zctap ifq definition
netdevice: addSETUP_ZCTAPto thenetdev_bpfstructure
io_uring: add register ifq opcode
io_uring: addprovide_ifq_regionopcode
io_uring: Addio_uringzctap iov structure and helpers
io_uring: introduce reference tracking for user pages.
page_pool: add page allocation and free hooks.
io_uring: provide functions for thepage_pool.
io_uring: addOP_RECV_ZCcommand.include/linux/io_uring.h | 24 ++ include/linux/io_uring_types.h | 10 + include/linux/netdevice.h | 6 + include/net/page_pool.h | 6 + include/uapi/linux/io_uring.h | 26 ++ io_uring/Makefile | 3 +- io_uring/io_uring.c | 10 + io_uring/kbuf.c | 13 + io_uring/kbuf.h | 2 + io_uring/net.c | 123 ++++++ io_uring/opdef.c | 23 + io_uring/zctap.c | 749 +++++++++++++++++++++++++++++++++ io_uring/zctap.h | 20 + net/core/page_pool.c | 41 +- 14 files changed, 1048 insertions(+), 8 deletions(-)create mode 100644
io_uring/zctap.c
create mode 100644io_uring/zctap.hSignature · 1 line
2.30.2
Quoted 39 lines · dust.li
Quoted 38 lines · Jonathan Lemon
This series is a RFC for
io_uring/zctap. This is an evolution of the earlier zctap work, re-targeted to useio_uringas the userspace API. The current code is intended to provide a zero-copy RX path for upper-level networking protocols (aka TCP and UDP). The current draft focuses on host-provided memory (not GPU memory).This RFC contains the upper-level core code required for operation, with the intent of soliciting feedback on the general API. This does not contain the network driver side changes required for complete operation. Also please note that as an RFC, there are some things which are incomplete or in need of rework.
The intent is to use a network driver which provides header/data splitting, so the frame header (which is processed by the networking stack) does not reside in user memory.
The code is roughly working (in that it has successfully received a TCP stream from a remote sender), but as an RFC, the intent is to solicit feedback on the API and overall design. The current code will also work with system pages, copying the data out to the application - this is intended as a fallback/testing path.
High level description:
The application allocates a frame backing store, and provides this to the kernel for use. An interface queue is requested from the networking device, and incoming frames are deposited into the provided memory region.
Responsibility for correctly steering incoming frames to the queue is outside the scope of this work - it is assumed that the user has set steering rules up separately.
Incoming frames are sent up the stack as skb's and eventually land in the application's socket receive queue. This differs from
AF_XDP, which receives raw frames directly to userspace, without protocol processing.
Quoted 6 lines
Quoted 4 lines
The
RECV_ZCopcode then returns an iov[] style vector which points to the data in userspace memory. When the application has completed processing of the data, the buffer is returned back to the kernel through a fill ring for reuse.Interesting work ! Any userspace demo and performance data ?
Coming soon! I'm hoping to get feedback on the overall API though, did you have any thoughts here?
Signature · 1 line
Jonathan