Skip to contentSkip to conversation

Add support for incremental buffer consumption

RFC PATCH Superseded Jens Axboe 1 participant 4 messages Aug 12, 2024 io_uring

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

cover Add support for incremental buffer consumption

Hi,

The recommended way to use io_uring for networking workloads is to use ring provided buffers. The application sets up a ring (or several) for buffers, and puts buffers for receiving data into them. When a recv completes, the completion contains information on which buffer data was received into. You can even use bundles with receive, and receive data into multiple buffers at the same time.

This all works fine, but has some limitations in that a buffer is always fully consumed. This patchset adds support for partial consumption of a buffer. This, in turn, allows an application to supply fewer buffers for receives, but of a much larger size. For example, rather than add a ton of 1500b buffers for receiving data, the application can just add one large buffer. Whenever data is received, only the current head part of the buffer is consumed and used. This leads to less iteration of buffers, and also eliminates any potential wasteage of memory if some of the receives only partially fill a provided buffer.

Patchset is lightly tested, passes current tests and also the new test cases I wrote for it. The liburing 'pbuf-ring-inc' branch has extra tests and support for this.

Using incrementally consumed buffers from an application point of view is fairly trivial. Just pass the flag IOU_PBUF_RING_INC to io_uring_setup_buf_ring(), and this marks this buffer group ID as being incrementally consumed. Outside of that, the application just needs to keep track of where the current read/recv point is at. See patch 3 for details.

Patch 1+2 are just basic prep patches, patch 3 is the meat of it. But still pretty darn simple. Note that this feature ONLY works with ring provide buffers, not with legacy/classic provided buffers. Code can also be found here, along with some other patches on top which aren't strictly related:

https://git.kernel.dk/cgit/linux/log/?h=io_uring-net-coalesce

and it's based on 6.11-rc3 with the pending 6.12 io_uring patches pulled in first.

Comments/reviews welcome! I'll add support for this to examples/proxy in the liburing repo, and can provide some performance results post that.

 include/uapi/linux/io_uring.h |  8 ++++++
 io_uring/io_uring.c           |  2 +-
 io_uring/kbuf.c               | 28 +++++++++---------
 io_uring/kbuf.h               | 54 ++++++++++++++++++++++++++---------
 io_uring/net.c                |  8 +++---
 io_uring/rw.c                 |  8 +++---
 6 files changed, 71 insertions(+), 37 deletions(-)
Signature · 1 line
Jens Axboe
author maintainer
#1 lore raw

PATCH 1/3 io_uring/kbuf: add io_kbuf_commit() helper

Committing the selected ring buffer is currently done in two different spots, combine it into a helper and just call that.

---
 io_uring/kbuf.c |  3 +--
 io_uring/kbuf.h | 14 ++++++++++----
 2 files changed, 11 insertions(+), 6 deletions(-)
io_uring/kbuf.c +1 -2
index 277b8e66a8cb..5e27d8b936c2 100644
@@ -171,9 +171,8 @@ static void __user *io_ring_buffer_select(struct io_kiocb *req, size_t *len,
 		 * the transfer completes (or if we get -EAGAIN and must poll of
 		 * retry).
 		 */
-		req->flags &= ~REQ_F_BUFFERS_COMMIT;
+		io_kbuf_commit(req, bl, 1);
 		req->buf_list = NULL;
-		bl->head++;
 	}
 	return u64_to_user_ptr(buf->addr);
 }
io_uring/kbuf.h +10 -4
index 2ed141d7662e..c9798663cd9f 100644
@@ -121,15 +121,21 @@ static inline bool io_kbuf_recycle(struct io_kiocb *req, unsigned issue_flags)
 	return false;
 }
 
+static inline void io_kbuf_commit(struct io_kiocb *req,
+				  struct io_buffer_list *bl, int nr)
+{
+	if (unlikely(!(req->flags & REQ_F_BUFFERS_COMMIT)))
+		return;
+	bl->head += nr;
+	req->flags &= ~REQ_F_BUFFERS_COMMIT;
+}
+
 static inline void __io_put_kbuf_ring(struct io_kiocb *req, int nr)
 {
 	struct io_buffer_list *bl = req->buf_list;
 
 	if (bl) {
-		if (req->flags & REQ_F_BUFFERS_COMMIT) {
-			bl->head += nr;
-			req->flags &= ~REQ_F_BUFFERS_COMMIT;
-		}
+		io_kbuf_commit(req, bl, nr);
 		req->buf_index = bl->bgid;
 	}
 	req->flags &= ~REQ_F_BUFFER_RING;
Signature · 1 line
2.43.0
author maintainer
#2 lore raw

PATCH 2/3 io_uring/kbuf: move io_ring_head_to_buf() to kbuf.h

In reply to Jens Axboe

In preparation for using this helper in kbuf.h as well, move it there and turn it into a macro.

---
 io_uring/kbuf.c | 6 ------
 io_uring/kbuf.h | 3 +++
 2 files changed, 3 insertions(+), 6 deletions(-)
io_uring/kbuf.c +0 -6
index 5e27d8b936c2..64f5bb91a28b 100644
@@ -132,12 +132,6 @@ static int io_provided_buffers_select(struct io_kiocb *req, size_t *len,
 	return 0;
 }
 
-static struct io_uring_buf *io_ring_head_to_buf(struct io_uring_buf_ring *br,
-						__u16 head, __u16 mask)
-{
-	return &br->bufs[head & mask];
-}
-
 static void __user *io_ring_buffer_select(struct io_kiocb *req, size_t *len,
 					  struct io_buffer_list *bl,
 					  unsigned int issue_flags)
io_uring/kbuf.h +3 -0
index c9798663cd9f..b7da3ce880bf 100644
@@ -121,6 +121,9 @@ static inline bool io_kbuf_recycle(struct io_kiocb *req, unsigned issue_flags)
 	return false;
 }
 
+/* Mapped buffer ring, return io_uring_buf from head */
+#define io_ring_head_to_buf(br, head, mask)	&(br)->bufs[(head) & (mask)]
+
 static inline void io_kbuf_commit(struct io_kiocb *req,
 				  struct io_buffer_list *bl, int nr)
 {
Signature · 1 line
2.43.0
author maintainer
#3 lore raw

PATCH 3/3 io_uring/kbuf: add support for incremental buffer consumption

In reply to Jens Axboe

By default, any recv/read operation that uses provided buffers will consume at least 1 buffer fully (and maybe more, in case of bundles). This adds support for incremental consumption, meaning that an application may add large buffers, and each read/recv will just consume the part of the buffer that it needs.

For example, let's say an application registers 1MB buffers in a provided buffer ring, for streaming receives. If it gets a short recv, then the full 1MB buffer will be consumed and passed back to the application. With incremental consumption, only the part that was actually used is consumed, and the buffer remains the current one.

This means that both the application and the kernel needs to keep track of what the current receive point is. Each recv will still pass back a buffer ID and the size consumed, the only difference is that before the next receive would always be the next buffer in the ring. Now the same buffer ID may return multiple receives, each at an offset into that buffer from where the previous receive left off. Example:

Application registers a provided buffer ring, and adds two 32K buffers to the ring.

Buffer1 address: 0x1000000 (buffer ID 0)
Buffer2 address: 0x2000000 (buffer ID 1)

A recv completion is received with the following values:

cqe->res 0x1000 (4k bytes received)
cqe->flags 0x1 (IORING_CQE_F_BUFFER set, buffer ID 0)

and the application now knows that 4096b of data is available at 0x1000000, the start of that buffer. Now the next receive comes in:

cqe->res 0x2000 (8k bytes received)
cqe->flags 0x1 (IORING_CQE_F_BUFFER set, buffer ID 0)

which tells the application that 8k is available where the last completion left off, at 0x1001000. Next completion is:

cqe->res 0x5000 (20k bytes received)
cqe->flags 0x1 (IORING_CQE_F_BUFFER set, buffer ID 0)

and the application now knows that 20k of data is available at 0x1003000, which is where the previous receive ended. The next completion is then:

cqe->res 0x1000 (4k bytes received)
cqe->flags 0x10001 (IORING_CQE_F_BUFFER set, buffer ID 1)

which tells the application that buffer ID 1 is now the current one, hence there's 4k of valid data at 0x2000000. 0x2001000 will be the next receive point for this buffer ID.

Once a buffer has been fully consumed, the buffer ring head is incremented and the next receive will indicate the next buffer ID in the CQE cflags.

An application can request incremental consumption by setting IOU_PBUF_RING_INC in the provided buffer ring registration. Outside of that, any provided buffer ring setup and buffer additions is done like before, no changes there. The only change is in how an application may see multiple completions for the same buffer ID, hence needing to know where the next receive will happen.

Light testing done, works with recv, send (w/bundle), read and read mshot. Don't particularly like the extra argument for the kbuf put helpers.

Note that like existing provided buffer rings, this should not be used with IOSQE_ASYNC, as both really require the ring to remain locked over the duration of the buffer selection and the operation completion. It will consume a buffer otherwise regardless of the size of the IO done.

---
 include/uapi/linux/io_uring.h |  8 +++++++
 io_uring/io_uring.c           |  2 +-
 io_uring/kbuf.c               | 21 +++++++++++------
 io_uring/kbuf.h               | 43 ++++++++++++++++++++++++-----------
 io_uring/net.c                |  8 +++----
 io_uring/rw.c                 |  8 +++----
 6 files changed, 61 insertions(+), 29 deletions(-)
include/uapi/linux/io_uring.h +8 -0
index 2aaf7ee256ac..aae91d10e412 100644
@@ -707,9 +707,17 @@ struct io_uring_buf_ring {
  *			mmap(2) with the offset set as:
  *			IORING_OFF_PBUF_RING | (bgid << IORING_OFF_PBUF_SHIFT)
  *			to get a virtual mapping for the ring.
+ * IOU_PBUF_RING_INC:	If set, buffers consumed from this buffer ring can be
+ *			consumed incrementally. Normally one (or more) buffers
+ *			are fully consumed. With incremental consumptions, it's
+ *			feasible to register big ranges of buffers, and each
+ *			use of it will consume only as much as it needs. This
+ *			requires that both the kernel and application keep
+ *			track of where the current read/recv index is at.
  */
 enum io_uring_register_pbuf_ring_flags {
 	IOU_PBUF_RING_MMAP	= 1,
+	IOU_PBUF_RING_INC	= 2,
 };
 
 /* argument for IORING_(UN)REGISTER_PBUF_RING */
io_uring/io_uring.c +1 -1
index 3942db160f18..390d44cbfce8 100644
@@ -904,7 +904,7 @@ void io_req_defer_failed(struct io_kiocb *req, s32 res)
 	lockdep_assert_held(&req->ctx->uring_lock);
 
 	req_set_fail(req);
-	io_req_set_res(req, res, io_put_kbuf(req, IO_URING_F_UNLOCKED));
+	io_req_set_res(req, res, io_put_kbuf(req, res, IO_URING_F_UNLOCKED));
 	if (def->fail)
 		def->fail(req);
 	io_req_complete_defer(req);
io_uring/kbuf.c +14 -7
index 64f5bb91a28b..793b2454acca 100644
@@ -165,7 +165,7 @@ static void __user *io_ring_buffer_select(struct io_kiocb *req, size_t *len,
 		 * the transfer completes (or if we get -EAGAIN and must poll of
 		 * retry).
 		 */
-		io_kbuf_commit(req, bl, 1);
+		io_kbuf_commit(req, bl, *len, 1);
 		req->buf_list = NULL;
 	}
 	return u64_to_user_ptr(buf->addr);
@@ -241,16 +241,21 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg,
 
 	req->buf_index = buf->bid;
 	do {
+		int len = buf->len;
+
 		/* truncate end piece, if needed */
-		if (buf->len > arg->max_len)
-			buf->len = arg->max_len;
+		if (len > arg->max_len) {
+			len = arg->max_len;
+			if (!(bl->flags & IOBL_INC))
+				buf->len = len;
+		}
 
 		iov->iov_base = u64_to_user_ptr(buf->addr);
-		iov->iov_len = buf->len;
+		iov->iov_len = len;
 		iov++;
 
-		arg->out_len += buf->len;
-		arg->max_len -= buf->len;
+		arg->out_len += len;
+		arg->max_len -= len;
 		if (!arg->max_len)
 			break;
 
@@ -672,7 +677,7 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
 
 	if (reg.resv[0] || reg.resv[1] || reg.resv[2])
 		return -EINVAL;
-	if (reg.flags & ~IOU_PBUF_RING_MMAP)
+	if (reg.flags & ~(IOU_PBUF_RING_MMAP | IOU_PBUF_RING_INC))
 		return -EINVAL;
 	if (!(reg.flags & IOU_PBUF_RING_MMAP)) {
 		if (!reg.ring_addr)
@@ -710,6 +715,8 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
 	if (!ret) {
 		bl->nr_entries = reg.ring_entries;
 		bl->mask = reg.ring_entries - 1;
+		if (reg.flags & IOU_PBUF_RING_INC)
+			bl->flags |= IOBL_INC;
 
 		io_buffer_add_list(ctx, bl, reg.bgid);
 		return 0;
io_uring/kbuf.h +30 -13

92 lines not on this page · raw message

io_uring/net.c +4 -4
index dc83a35b8af4..e312fc1ed7de 100644
@@ -499,11 +499,11 @@ static inline bool io_send_finish(struct io_kiocb *req, int *ret,
 	unsigned int cflags;
 
 	if (!(sr->flags & IORING_RECVSEND_BUNDLE)) {
-		cflags = io_put_kbuf(req, issue_flags);
+		cflags = io_put_kbuf(req, *ret, issue_flags);
 		goto finish;
 	}
 
-	cflags = io_put_kbufs(req, io_bundle_nbufs(kmsg, *ret), issue_flags);
+	cflags = io_put_kbufs(req, *ret, io_bundle_nbufs(kmsg, *ret), issue_flags);
 
 	if (bundle_finished || req->flags & REQ_F_BL_EMPTY)
 		goto finish;
@@ -844,13 +844,13 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret,
 		cflags |= IORING_CQE_F_SOCK_NONEMPTY;
 
 	if (sr->flags & IORING_RECVSEND_BUNDLE) {
-		cflags |= io_put_kbufs(req, io_bundle_nbufs(kmsg, *ret),
+		cflags |= io_put_kbufs(req, *ret, io_bundle_nbufs(kmsg, *ret),
 				      issue_flags);
 		/* bundle with no more immediate buffers, we're done */
 		if (req->flags & REQ_F_BL_EMPTY)
 			goto finish;
 	} else {
-		cflags |= io_put_kbuf(req, issue_flags);
+		cflags |= io_put_kbuf(req, *ret, issue_flags);
 	}
 
 	/*
io_uring/rw.c +4 -4
index c004d21e2f12..f5e0694538b9 100644
@@ -511,7 +511,7 @@ void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts)
 	io_req_io_end(req);
 
 	if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))
-		req->cqe.flags |= io_put_kbuf(req, 0);
+		req->cqe.flags |= io_put_kbuf(req, req->cqe.res, 0);
 
 	io_req_rw_cleanup(req, 0);
 	io_req_task_complete(req, ts);
@@ -593,7 +593,7 @@ static int kiocb_done(struct io_kiocb *req, ssize_t ret,
 			 */
 			io_req_io_end(req);
 			io_req_set_res(req, final_ret,
-				       io_put_kbuf(req, issue_flags));
+				       io_put_kbuf(req, ret, issue_flags));
 			io_req_rw_cleanup(req, issue_flags);
 			return IOU_OK;
 		}
@@ -975,7 +975,7 @@ int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags)
 		 * Put our buffer and post a CQE. If we fail to post a CQE, then
 		 * jump to the termination path. This request is then done.
 		 */
-		cflags = io_put_kbuf(req, issue_flags);
+		cflags = io_put_kbuf(req, ret, issue_flags);
 		rw->len = 0; /* similarly to above, reset len to 0 */
 
 		if (io_req_post_cqe(req, ret, cflags | IORING_CQE_F_MORE)) {
@@ -1167,7 +1167,7 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin)
 		if (!smp_load_acquire(&req->iopoll_completed))
 			break;
 		nr_events++;
-		req->cqe.flags = io_put_kbuf(req, 0);
+		req->cqe.flags = io_put_kbuf(req, req->cqe.res, 0);
 		if (req->opcode != IORING_OP_URING_CMD)
 			io_req_rw_cleanup(req, 0);
 	}
Signature · 1 line
2.43.0