Skip to contentSkip to conversation

zero-copy RX for io_uring

RFC Superseded Jonathan Lemon 3 participants 12 messages Oct 7, 2022 – Oct 10, 2022

Subsystems (4)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover zero-copy RX for io_uring

This series is a RFC for io_uring/zctap. This is an evolution of the earlier zctap work, re-targeted to use io_uring as the userspace API. The current code is intended to provide a zero-copy RX path for upper-level networking protocols (aka TCP and UDP). The current draft focuses on host-provided memory (not GPU memory).

This RFC contains the upper-level core code required for operation, with the intent of soliciting feedback on the general API. This does not contain the network driver side changes required for complete operation. Also please note that as an RFC, there are some things which are incomplete or in need of rework.

The intent is to use a network driver which provides header/data splitting, so the frame header (which is processed by the networking stack) does not reside in user memory.

The code is roughly working (in that it has successfully received a TCP stream from a remote sender), but as an RFC, the intent is to solicit feedback on the API and overall design. The current code will also work with system pages, copying the data out to the application - this is intended as a fallback/testing path.

High level description:

The application allocates a frame backing store, and provides this to the kernel for use. An interface queue is requested from the networking device, and incoming frames are deposited into the provided memory region.

Responsibility for correctly steering incoming frames to the queue is outside the scope of this work - it is assumed that the user has set steering rules up separately.

Incoming frames are sent up the stack as skb's and eventually land in the application's socket receive queue. This differs from AF_XDP, which receives raw frames directly to userspace, without protocol processing.

The RECV_ZC opcode then returns an iov[] style vector which points to the data in userspace memory. When the application has completed processing of the data, the buffer is returned back to the kernel through a fill ring for reuse.

Jonathan Lemon (9):
io_uring: add zctap ifq definition
netdevice: add SETUP_ZCTAP to the netdev_bpf structure
io_uring: add register ifq opcode
io_uring: add provide_ifq_region opcode
io_uring: Add io_uring zctap iov structure and helpers
io_uring: introduce reference tracking for user pages.
page_pool: add page allocation and free hooks.
io_uring: provide functions for the page_pool.
io_uring: add OP_RECV_ZC command.

 include/linux/io_uring.h       |  24 ++
 include/linux/io_uring_types.h |  10 +
 include/linux/netdevice.h      |   6 +
 include/net/page_pool.h        |   6 +
 include/uapi/linux/io_uring.h  |  26 ++
 io_uring/Makefile              |   3 +-
 io_uring/io_uring.c            |  10 +
 io_uring/kbuf.c                |  13 +
 io_uring/kbuf.h                |   2 +
 io_uring/net.c                 | 123 ++++++
 io_uring/opdef.c               |  23 +
 io_uring/zctap.c               | 749 +++++++++++++++++++++++++++++++++
 io_uring/zctap.h               |  20 +
 net/core/page_pool.c           |  41 +-
 14 files changed, 1048 insertions(+), 8 deletions(-)
 create mode 100644 io_uring/zctap.c
 create mode 100644 io_uring/zctap.h
Signature · 1 line
2.30.2
#1 lore raw

RFC 1/9 io_uring: add zctap ifq definition

Add structure definition for io_zctap_ifq for use by lower level networking hooks.

---
 include/linux/io_uring_types.h | 9 +++++++++
 1 file changed, 9 insertions(+)
include/linux/io_uring_types.h +9 -0
index 677a25d44d7f..680fbf1f34e7 100644
@@ -323,6 +323,7 @@ struct io_ring_ctx {
 	struct io_mapped_ubuf		*dummy_ubuf;
 	struct io_rsrc_data		*file_data;
 	struct io_rsrc_data		*buf_data;
+	struct xarray			zctap_ifq_xa;
 
 	struct delayed_work		rsrc_put_work;
 	struct llist_head		rsrc_put_llist;
@@ -578,4 +579,12 @@ struct io_overflow_cqe {
 	struct io_uring_cqe cqe;
 };
 
+struct io_zctap_ifq {
+	struct net_device	*dev;
+	struct io_ring_ctx	*ctx;
+	u16			queue_id;
+	u16			id;
+	u16			fill_bgid;
+};
+
 #endif
Signature · 1 line
2.30.2
#2 lore raw

RFC 2/9 netdevice: add SETUP_ZCTAP to the netdev_bpf structure

In reply to Jonathan Lemon

This command requests the networking device setup or teardown a new interface queue, backed by a region of user supplied memory.

The queue will be managed by io-uring.

---
 include/linux/netdevice.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/linux/netdevice.h +6 -0
index 9f42fc871c3b..49ecfc276411 100644
@@ -979,6 +979,7 @@ enum bpf_netdev_command {
 	BPF_OFFLOAD_MAP_ALLOC,
 	BPF_OFFLOAD_MAP_FREE,
 	XDP_SETUP_XSK_POOL,
+	XDP_SETUP_ZCTAP,
 };
 
 struct bpf_prog_offload_ops;
@@ -1017,6 +1018,11 @@ struct netdev_bpf {
 			struct xsk_buff_pool *pool;
 			u16 queue_id;
 		} xsk;
+		/* XDP_SETUP_ZCTAP */
+		struct {
+			struct io_zctap_ifq *ifq;
+			u16 queue_id;
+		} zct;
 	};
 };
 
Signature · 1 line
2.30.2
#3 lore raw

RFC 3/9 io_uring: add register ifq opcode

In reply to Jonathan Lemon

Add initial support for support for hooking in zero-copy interface queues to io_uring. This command requests a user-managed queue from the specified network device.

This only includes the register opcode, unregistration is currently done implicitly when the ring is removed.

---
 include/uapi/linux/io_uring.h |  14 ++++
 io_uring/Makefile             |   3 +-
 io_uring/io_uring.c           |  10 +++
 io_uring/zctap.c              | 146 ++++++++++++++++++++++++++++++++++
 io_uring/zctap.h              |  11 +++
 5 files changed, 183 insertions(+), 1 deletion(-)
 create mode 100644 io_uring/zctap.c
 create mode 100644 io_uring/zctap.h
include/uapi/linux/io_uring.h +14 -0
index 6b83177fd41d..bc5108d65c0a 100644
@@ -473,6 +473,9 @@ enum {
 	/* register a range of fixed file slots for automatic slot allocation */
 	IORING_REGISTER_FILE_ALLOC_RANGE	= 25,
 
+	/* register a network ifq for zerocopy RX */
+	IORING_REGISTER_IFQ			= 26,
+
 	/* this goes last */
 	IORING_REGISTER_LAST
 };
@@ -649,6 +652,17 @@ struct io_uring_recvmsg_out {
 	__u32 flags;
 };
 
+/*
+ * Argument for IORING_REGISTER_IFQ
+ */
+struct io_uring_ifq_req {
+	__u32	ifindex;
+	__u16	queue_id;
+	__u16	ifq_id;
+	__u16	fill_bgid;
+	__u16	__pad[3];
+};
+
 #ifdef __cplusplus
 }
 #endif
io_uring/Makefile +2 -1
index 8cc8e5387a75..9d87e2e45ef9 100644
@@ -7,5 +7,6 @@ obj-$(CONFIG_IO_URING)		+= io_uring.o xattr.o nop.o fs.o splice.o \
 					openclose.o uring_cmd.o epoll.o \
 					statx.o net.o msg_ring.o timeout.o \
 					sqpoll.o fdinfo.o tctx.o poll.o \
-					cancel.o kbuf.o rsrc.o rw.o opdef.o notif.o
+					cancel.o kbuf.o rsrc.o rw.o opdef.o \
+					notif.o zctap.o
 obj-$(CONFIG_IO_WQ)		+= io-wq.o
io_uring/io_uring.c +10 -0
index b9640ad5069f..8dd988b33af0 100644
@@ -91,6 +91,7 @@
 #include "cancel.h"
 #include "net.h"
 #include "notif.h"
+#include "zctap.h"
 
 #include "timeout.h"
 #include "poll.h"
@@ -321,6 +322,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)
 	INIT_WQ_LIST(&ctx->locked_free_list);
 	INIT_DELAYED_WORK(&ctx->fallback_work, io_fallback_req_func);
 	INIT_WQ_LIST(&ctx->submit_state.compl_reqs);
+	xa_init(&ctx->zctap_ifq_xa);
 	return ctx;
 err:
 	kfree(ctx->dummy_ubuf);
@@ -2639,6 +2641,8 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)
 		__io_cqring_overflow_flush(ctx, true);
 	xa_for_each(&ctx->personalities, index, creds)
 		io_unregister_personality(ctx, index);
+	xa_for_each(&ctx->zctap_ifq_xa, index, creds)
+		io_unregister_zctap_ifq(ctx, index);
 	if (ctx->rings)
 		io_poll_remove_all(ctx, NULL, true);
 	mutex_unlock(&ctx->uring_lock);
@@ -3839,6 +3843,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,
 			break;
 		ret = io_register_file_alloc_range(ctx, arg);
 		break;
+	case IORING_REGISTER_IFQ:
+		ret = -EINVAL;
+		if (!arg || nr_args != 1)
+			break;
+		ret = io_register_ifq(ctx, arg);
+		break;
 	default:
 		ret = -EINVAL;
 		break;
io_uring/zctap.c new file +146 -0

146 lines not on this page · raw message

io_uring/zctap.h new file +11 -0
new file mode 100644
index 000000000000..bda15d218fe3
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0
+#ifndef IOU_ZCTAP_H
+#define IOU_ZCTAP_H
+
+int io_register_ifq(struct io_ring_ctx *ctx,
+		    struct io_uring_ifq_req __user *arg);
+int io_unregister_ifq(struct io_ring_ctx *ctx,
+		      struct io_uring_ifq_req __user *arg);
+int io_unregister_zctap_ifq(struct io_ring_ctx *ctx, unsigned long index);
+
+#endif
Signature · 1 line
2.30.2
#4 lore raw

RFC 4/9 io_uring: add provide_ifq_region opcode

In reply to Jonathan Lemon

This opcode takes part or all of a memory region that was previously registered with io_uring, and assigns it as the backing store for the specified ifq.

The entire region is registered instead of providing individual bufferrs, as this allows the hardware to select the optimal buffer size for incoming packets.

---
 include/linux/io_uring_types.h |  1 +
 include/uapi/linux/io_uring.h  |  1 +
 io_uring/opdef.c               |  9 ++++
 io_uring/zctap.c               | 96 ++++++++++++++++++++++++++++++++++
 io_uring/zctap.h               |  4 ++
 5 files changed, 111 insertions(+)
include/linux/io_uring_types.h +1 -0
index 680fbf1f34e7..56257e8afd0a 100644
@@ -582,6 +582,7 @@ struct io_overflow_cqe {
 struct io_zctap_ifq {
 	struct net_device	*dev;
 	struct io_ring_ctx	*ctx;
+	void			*region;	/* XXX relocate? */
 	u16			queue_id;
 	u16			id;
 	u16			fill_bgid;
include/uapi/linux/io_uring.h +1 -0
index bc5108d65c0a..3b392f8270dc 100644
@@ -206,6 +206,7 @@ enum io_uring_op {
 	IORING_OP_SOCKET,
 	IORING_OP_URING_CMD,
 	IORING_OP_SEND_ZC,
+	IORING_OP_PROVIDE_IFQ_REGION,
 
 	/* this goes last, obviously */
 	IORING_OP_LAST,
io_uring/opdef.c +9 -0
index c4dddd0fd709..bf28c43117c3 100644
@@ -33,6 +33,7 @@
 #include "poll.h"
 #include "cancel.h"
 #include "rw.h"
+#include "zctap.h"
 
 static int io_no_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
@@ -488,6 +489,14 @@ const struct io_op_def io_op_defs[] = {
 		.prep			= io_eopnotsupp_prep,
 #endif
 	},
+	[IORING_OP_PROVIDE_IFQ_REGION] = {
+		.audit_skip		= 1,
+		.iopoll			= 1,
+		.buffer_select		= 1,
+		.name			= "PROVIDE_IFQ_REGION",
+		.prep			= io_provide_ifq_region_prep,
+		.issue			= io_provide_ifq_region,
+	},
 };
 
 const char *io_uring_get_opcode(u8 opcode)
io_uring/zctap.c +96 -0

110 lines not on this page · raw message

io_uring/zctap.h +4 -0
index bda15d218fe3..709c803220f4 100644
@@ -8,4 +8,8 @@ int io_unregister_ifq(struct io_ring_ctx *ctx,
 		      struct io_uring_ifq_req __user *arg);
 int io_unregister_zctap_ifq(struct io_ring_ctx *ctx, unsigned long index);
 
+int io_provide_ifq_region_prep(struct io_kiocb *req,
+                               const struct io_uring_sqe *sqe);
+int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags);
+
 #endif
Signature · 1 line
2.30.2
#5 lore raw

RFC 5/9 io_uring: Add io_uring zctap iov structure and helpers

In reply to Jonathan Lemon

With networking zero-copy receive, the incoming data is placed directly into user-supplied buffers. Instead of returning the buffer address, return the buffer group id and buffer id, and let the application figure out the base address.

Add helpers for storing and retrieving the encoding, which is stored in the page_private field. This will be used in the zerocopy RX routine, when handling pages from skb fragments.

---
 include/uapi/linux/io_uring.h | 10 +++++++++
 io_uring/zctap.c              | 39 ++++++++++++++++++++++++++++++++++-
 2 files changed, 48 insertions(+), 1 deletion(-)
include/uapi/linux/io_uring.h +10 -0
index 3b392f8270dc..145d55280919 100644
@@ -664,6 +664,16 @@ struct io_uring_ifq_req {
 	__u16	__pad[3];
 };
 
+struct io_uring_zctap_iov {
+	__u32	off;
+	__u32	len;
+	__u16	bgid;
+	__u16	bid;
+	__u16	ifq_id;
+	__u16	resv;
+};
+
+
 #ifdef __cplusplus
 }
 #endif
io_uring/zctap.c +38 -1
index 728f7c938b7b..58b4c5417650 100644
@@ -19,6 +19,26 @@ static DEFINE_XARRAY_ALLOC1(io_zctap_ifq_xa);
 
 typedef int (*bpf_op_t)(struct net_device *dev, struct netdev_bpf *bpf);
 
+static u64 zctap_page_info(u16 region_id, u16 pgid, u16 ifq_id)
+{
+	return (u64)region_id << 32 | (u64)pgid << 16 | ifq_id;
+}
+
+static u16 zctap_page_region_id(const struct page *page)
+{
+	return (page_private(page) >> 32) & 0xffff;
+}
+
+static u16 zctap_page_id(const struct page *page)
+{
+	return (page_private(page) >> 16) & 0xffff;
+}
+
+static u16 zctap_page_ifq_id(const struct page *page)
+{
+	return page_private(page) & 0xffff;
+}
+
 static int __io_queue_mgmt(struct net_device *dev, struct io_zctap_ifq *ifq,
 			   u16 *queue_id)
 {
@@ -213,8 +233,9 @@ int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags)
 {
 	struct io_ifq_region *r = io_kiocb_to_cmd(req, struct io_ifq_region);
 	struct ifq_region *ifr;
-	int i, idx, nr_pages;
+	int i, id, idx, nr_pages;
 	struct page *page;
+	u64 info;
 
 	nr_pages = r->len >> PAGE_SHIFT;
 	idx = (r->addr - req->imu->ubuf) >> PAGE_SHIFT;
@@ -231,12 +252,28 @@ int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags)
 	ifr->start = r->addr;
 	ifr->end = r->addr + r->len;
 
+	id = r->ifq->id;
 	for (i = 0; i < nr_pages; i++, idx++) {
 		page = req->imu->bvec[idx].bv_page;
+		if (PagePrivate(page))
+			goto out;
+		SetPagePrivate(page);
+		info = zctap_page_info(r->bgid, idx + i, id);
+		set_page_private(page, info);
 		ifr->page[i] = page;
 	}
 
 	WRITE_ONCE(r->ifq->region,  ifr);
 
 	return 0;
+out:
+	while (i--) {
+		page = req->imu->bvec[idx + i].bv_page;
+		ClearPagePrivate(page);
+		set_page_private(page, 0);
+	}
+
+	kvfree(ifr);
+
+	return -EEXIST;
 }
Signature · 1 line
2.30.2
#6 lore raw

RFC 6/9 io_uring: introduce reference tracking for user pages.

In reply to Jonathan Lemon

This is currently a WIP.

If only part of a page is used by a skb fragment, and then provided to the user, the page should not be reused by the kernel until all sub-page fragments are not in use.

If only full pages are used (and not sub-page fragments), then this code shouldn't be needed.

---
 io_uring/zctap.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
io_uring/zctap.c +34 -0
index 58b4c5417650..9db3421fb9fa 100644
@@ -183,9 +183,36 @@ struct ifq_region {
 	int			count;
 	int			imu_idx;
 	int			nr_pages;
+	u8			*page_uref;
 	struct page		*page[];
 };
 
+static void io_add_page_uref(struct ifq_region *ifr, u16 pgid)
+{
+	if (WARN_ON(!ifr))
+		return;
+
+	if (WARN_ON(pgid < ifr->imu_idx))
+		return;
+
+	ifr->page_uref[pgid - ifr->imu_idx]++;
+}
+
+static bool io_put_page_last_uref(struct ifq_region *ifr, u64 addr)
+{
+	int idx;
+
+	if (WARN_ON(addr < ifr->start || addr > ifr->end))
+		return false;
+
+	idx = (addr - ifr->start) >> PAGE_SHIFT;
+
+	if (WARN_ON(!ifr->page_uref[idx]))
+		return false;
+
+	return --ifr->page_uref[idx] == 0;
+}
+
 int io_provide_ifq_region_prep(struct io_kiocb *req,
 			       const struct io_uring_sqe *sqe)
 {
@@ -244,6 +271,11 @@ int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags)
 	if (!ifr)
 		return -ENOMEM;
 
+	ifr->page_uref = kvmalloc_array(nr_pages, sizeof(u8), GFP_KERNEL);
+	if (!ifr->page_uref) {
+		kvfree(ifr);
+		return -ENOMEM;
+	}
 
 	ifr->nr_pages = nr_pages;
 	ifr->imu_idx = idx;
@@ -261,6 +293,7 @@ int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags)
 		info = zctap_page_info(r->bgid, idx + i, id);
 		set_page_private(page, info);
 		ifr->page[i] = page;
+		ifr->page_uref[i] = 0;
 	}
 
 	WRITE_ONCE(r->ifq->region,  ifr);
@@ -273,6 +306,7 @@ int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags)
 		set_page_private(page, 0);
 	}
 
+	kvfree(ifr->page_uref);
 	kvfree(ifr);
 
 	return -EEXIST;
Signature · 1 line
2.30.2
#7 lore raw

RFC 7/9 page_pool: add page allocation and free hooks.

In reply to Jonathan Lemon

In order to allow for user-allocated page backing, add hooks to the page pool so pages can be obtained and released from a user-supplied provider instead of the system page allocator.

skbs are marked with skb_mark_for_recycle() if they contain pages belonging to a page pool, and page_put() will deliver the pages back to the pool instead of freeing them to the system page allocator.

---
 include/net/page_pool.h |  6 ++++++
 net/core/page_pool.c    | 41 ++++++++++++++++++++++++++++++++++-------
 2 files changed, 40 insertions(+), 7 deletions(-)
include/net/page_pool.h +6 -0
index 813c93499f20..85c8423f9a7e 100644
@@ -82,6 +82,12 @@ struct page_pool_params {
 	unsigned int	offset;  /* DMA addr offset */
 	void (*init_callback)(struct page *page, void *arg);
 	void *init_arg;
+	struct page *(*alloc_pages)(void *arg, int nid, gfp_t gfp,
+				    unsigned int order);
+	unsigned long (*alloc_bulk)(void *arg, gfp_t gfp, int nid,
+				    unsigned long nr_pages,
+				    struct page **page_array);
+	void (*put_page)(void *arg, struct page *page);
 };
 
 #ifdef CONFIG_PAGE_POOL_STATS
net/core/page_pool.c +34 -7

82 lines not on this page · raw message

Signature · 1 line
2.30.2
#8 lore raw

RFC 8/9 io_uring: provide functions for the page_pool.

In reply to Jonathan Lemon

These functions are called by the page_pool, in order to refill the pool with user-supplied pages, or returning excess pages back from the pool.

If no pages are present in the region cache, then an attempt is made to obtain more pages from the interface fill queue.

---
 include/linux/io_uring.h | 24 ++++++++++++
 io_uring/kbuf.c          | 13 +++++++
 io_uring/kbuf.h          |  2 +
 io_uring/zctap.c         | 82 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 121 insertions(+)
include/linux/io_uring.h +24 -0
index 4a2f6cc5a492..b92e65e0a469 100644
@@ -37,6 +37,14 @@ void __io_uring_free(struct task_struct *tsk);
 void io_uring_unreg_ringfd(void);
 const char *io_uring_get_opcode(u8 opcode);
 
+struct io_zctap_ifq;
+struct page *io_zctap_ifq_get_page(struct io_zctap_ifq *ifq,
+                                   unsigned int order);
+unsigned long io_zctap_ifq_get_bulk(struct io_zctap_ifq *ifq,
+                                    unsigned long nr_pages,
+                                    struct page **page_array);
+bool io_zctap_ifq_put_page(struct io_zctap_ifq *ifq, struct page *page);
+
 static inline void io_uring_files_cancel(void)
 {
 	if (current->io_uring) {
@@ -80,6 +88,22 @@ static inline const char *io_uring_get_opcode(u8 opcode)
 {
 	return "";
 }
+static inline struct page *io_zctap_ifq_get_page(struct io_zctap_ifq *ifq,
+                                   unsigned int order)
+{
+	return NULL;
+}
+sttaic unsigned long io_zctap_ifq_get_bulk(struct io_zctap_ifq *ifq,
+                                    unsigned long nr_pages,
+                                    struct page **page_array)
+{
+	return 0;
+}
+bool io_zctap_ifq_put_page(struct io_zctap_ifq *ifq, struct page *page)
+{
+	return false;
+}
+
 #endif
 
 #endif
io_uring/kbuf.c +13 -0
index 25cd724ade18..caae2755e3d5 100644
@@ -188,6 +188,19 @@ void __user *io_buffer_select(struct io_kiocb *req, size_t *len,
 	return ret;
 }
 
+/* XXX May called from the driver, in napi context. */
+u64 io_zctap_buffer(struct io_kiocb *req, size_t *len)
+{
+	struct io_ring_ctx *ctx = req->ctx;
+	struct io_buffer_list *bl;
+	void __user *ret = NULL;
+
+	bl = io_buffer_get_list(ctx, req->buf_index);
+	if (likely(bl))
+		ret = io_ring_buffer_select(req, len, bl, IO_URING_F_UNLOCKED);
+	return (u64)ret;
+}
+
 static __cold int io_init_bl_list(struct io_ring_ctx *ctx)
 {
 	int i;
io_uring/kbuf.h +2 -0
index 746fbf31a703..1379e0e9f870 100644
@@ -50,6 +50,8 @@ unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags);
 
 void io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags);
 
+u64 io_zctap_buffer(struct io_kiocb *req, size_t *len);
+
 static inline void io_kbuf_recycle_ring(struct io_kiocb *req)
 {
 	/*
io_uring/zctap.c +82 -0

85 lines not on this page · raw message

Signature · 1 line
2.30.2
#9 lore raw

RFC 9/9 io_uring: add OP_RECV_ZC command.

In reply to Jonathan Lemon

This is still a WIP. The current code (temporarily) uses addr3 as a hack in order to leverage code in io_recvmsg_prep.

The recvzc opcode uses a metadata buffer either supplied directly with buf/len, or indirectly from the buffer group. The expectation is that this buffer is then filled with an array of io_uring_zctap_iov structures, which point to the data in user-memory.

    addr3 = (readlen << 32) | (copy_bgid << 16) | ctx->ifq_id;

The amount of returned data is limited by the number of iovs that the metadata area can hold, and also the readlen parameter.

As a fallback (and for testing purposes), if the skb data is not present in user memory (perhaps due to system misconfiguration), then a seprate buffer is obtained from the copy_bgid and the data is copied into user-memory.

---
 include/uapi/linux/io_uring.h |   1 +
 io_uring/net.c                | 123 ++++++++++++
 io_uring/opdef.c              |  14 ++
 io_uring/zctap.c              | 354 ++++++++++++++++++++++++++++++++++
 io_uring/zctap.h              |   5 +
 5 files changed, 497 insertions(+)
include/uapi/linux/io_uring.h +1 -0
index 145d55280919..3c31a966687e 100644
@@ -207,6 +207,7 @@ enum io_uring_op {
 	IORING_OP_URING_CMD,
 	IORING_OP_SEND_ZC,
 	IORING_OP_PROVIDE_IFQ_REGION,
+	IORING_OP_RECV_ZC,
 
 	/* this goes last, obviously */
 	IORING_OP_LAST,
io_uring/net.c +123 -0

141 lines not on this page · raw message

io_uring/opdef.c +14 -0
index bf28c43117c3..f3782e7b707b 100644
@@ -497,6 +497,20 @@ const struct io_op_def io_op_defs[] = {
 		.prep			= io_provide_ifq_region_prep,
 		.issue			= io_provide_ifq_region,
 	},
+	[IORING_OP_RECV_ZC] = {
+		.name			= "RECV_ZC",
+		.needs_file		= 1,
+		.unbound_nonreg_file	= 1,
+		.pollin			= 1,
+		.buffer_select		= 1,
+		.ioprio			= 1,
+#if defined(CONFIG_NET)
+		.prep			= io_recvzc_prep,
+		.issue			= io_recvzc,
+#else
+		.prep			= io_eopnotsupp_prep,
+#endif
+	},
 };
 
 const char *io_uring_get_opcode(u8 opcode)
io_uring/zctap.c +354 -0

363 lines not on this page · raw message

io_uring/zctap.h +5 -0
index 709c803220f4..2c3e23a6a07a 100644
@@ -12,4 +12,9 @@ int io_provide_ifq_region_prep(struct io_kiocb *req,
                                const struct io_uring_sqe *sqe);
 int io_provide_ifq_region(struct io_kiocb *req, unsigned int issue_flags);
 
+int io_recvzc(struct io_kiocb *req, unsigned int issue_flags);
+int io_recvzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
+int io_zctap_recv(struct io_zctap_ifq *ifq, struct socket *sock,
+		  struct msghdr *msg, int flags, u32 datalen, u16 copy_bgid);
+
 #endif
Signature · 1 line
2.30.2
#10 lore raw
In reply to Jonathan Lemon
Quoted 38 lines · Jonathan Lemon

This series is a RFC for io_uring/zctap. This is an evolution of the earlier zctap work, re-targeted to use io_uring as the userspace API. The current code is intended to provide a zero-copy RX path for upper-level networking protocols (aka TCP and UDP). The current draft focuses on host-provided memory (not GPU memory).

This RFC contains the upper-level core code required for operation, with the intent of soliciting feedback on the general API. This does not contain the network driver side changes required for complete operation. Also please note that as an RFC, there are some things which are incomplete or in need of rework.

The intent is to use a network driver which provides header/data splitting, so the frame header (which is processed by the networking stack) does not reside in user memory.

The code is roughly working (in that it has successfully received a TCP stream from a remote sender), but as an RFC, the intent is to solicit feedback on the API and overall design. The current code will also work with system pages, copying the data out to the application - this is intended as a fallback/testing path.

High level description:

The application allocates a frame backing store, and provides this to the kernel for use. An interface queue is requested from the networking device, and incoming frames are deposited into the provided memory region.

Responsibility for correctly steering incoming frames to the queue is outside the scope of this work - it is assumed that the user has set steering rules up separately.

Incoming frames are sent up the stack as skb's and eventually land in the application's socket receive queue. This differs from AF_XDP, which receives raw frames directly to userspace, without protocol processing.

Quoted 4 lines

The RECV_ZC opcode then returns an iov[] style vector which points to the data in userspace memory. When the application has completed processing of the data, the buffer is returned back to the kernel through a fill ring for reuse.

Interesting work ! Any userspace demo and performance data ?

Quoted 32 lines

Jonathan Lemon (9):
io_uring: add zctap ifq definition
netdevice: add SETUP_ZCTAP to the netdev_bpf structure
io_uring: add register ifq opcode
io_uring: add provide_ifq_region opcode
io_uring: Add io_uring zctap iov structure and helpers
io_uring: introduce reference tracking for user pages.
page_pool: add page allocation and free hooks.
io_uring: provide functions for the page_pool.
io_uring: add OP_RECV_ZC command.

include/linux/io_uring.h       |  24 ++
include/linux/io_uring_types.h |  10 +
include/linux/netdevice.h      |   6 +
include/net/page_pool.h        |   6 +
include/uapi/linux/io_uring.h  |  26 ++
io_uring/Makefile              |   3 +-
io_uring/io_uring.c            |  10 +
io_uring/kbuf.c                |  13 +
io_uring/kbuf.h                |   2 +
io_uring/net.c                 | 123 ++++++
io_uring/opdef.c               |  23 +
io_uring/zctap.c               | 749 +++++++++++++++++++++++++++++++++
io_uring/zctap.h               |  20 +
net/core/page_pool.c           |  41 +-
14 files changed, 1048 insertions(+), 8 deletions(-)

create mode 100644 io_uring/zctap.c
create mode 100644 io_uring/zctap.h

Signature · 1 line
2.30.2
#11 lore raw
Quoted 39 lines · dust.li
Quoted 38 lines · Jonathan Lemon

This series is a RFC for io_uring/zctap. This is an evolution of the earlier zctap work, re-targeted to use io_uring as the userspace API. The current code is intended to provide a zero-copy RX path for upper-level networking protocols (aka TCP and UDP). The current draft focuses on host-provided memory (not GPU memory).

This RFC contains the upper-level core code required for operation, with the intent of soliciting feedback on the general API. This does not contain the network driver side changes required for complete operation. Also please note that as an RFC, there are some things which are incomplete or in need of rework.

The intent is to use a network driver which provides header/data splitting, so the frame header (which is processed by the networking stack) does not reside in user memory.

The code is roughly working (in that it has successfully received a TCP stream from a remote sender), but as an RFC, the intent is to solicit feedback on the API and overall design. The current code will also work with system pages, copying the data out to the application - this is intended as a fallback/testing path.

High level description:

The application allocates a frame backing store, and provides this to the kernel for use. An interface queue is requested from the networking device, and incoming frames are deposited into the provided memory region.

Responsibility for correctly steering incoming frames to the queue is outside the scope of this work - it is assumed that the user has set steering rules up separately.

Incoming frames are sent up the stack as skb's and eventually land in the application's socket receive queue. This differs from AF_XDP, which receives raw frames directly to userspace, without protocol processing.

Quoted 6 lines
Quoted 4 lines

The RECV_ZC opcode then returns an iov[] style vector which points to the data in userspace memory. When the application has completed processing of the data, the buffer is returned back to the kernel through a fill ring for reuse.

Interesting work ! Any userspace demo and performance data ?

Coming soon! I'm hoping to get feedback on the overall API though, did you have any thoughts here?

Signature · 1 line
Jonathan