Skip to contentSkip to conversation

io_uring/net: support registered buffer for plain send and recv

PATCH Applied Ming Lei 2 participants 4 messages Jun 8 io_uring

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

v2 cover io_uring/net: support registered buffer for plain send and recv

From: Ming Lei <[email protected]>

Hi,

This series wires IORING_RECVSEND_FIXED_BUF into the plain IORING_OP_SEND and IORING_OP_RECV paths; so far the flag has only been honoured on the SEND_ZC path.

Motivation: targets such as ublk's NBD backend want to push/pull I/O data directly to/from an io_uring registered buffer over a plain send/recv on a TCP socket, avoiding the per-I/O import and page pinning while keeping single-CQE completion. The SEND_ZC path is left untouched.

Patch 1 is the kernel change. Patch 2 adds a liburing test and is meant for the liburing tree.

Changes since v1:
- Reject IORING_SEND_VECTORIZED on the plain IORING_OP_SEND fixed-buffer
path: the plain io_send() issue path imports a single registered buffer and has no vectorized-regbuf import step, so the combination silently did nothing before. (Jens Axboe)
- Add the matching send-vectorized negative case to the liburing test.
- Clarify in the commit log that SEND_ZC is unaffected.

v1: https://lore.kernel.org/io-uring/[email protected]/

Ming Lei (2):
io_uring/net: support registered buffer for plain send and recv
test: add fixed-buf-send-recv for registered buffer send/recv

 io_uring/net.c             |  47 ++++++-
 test/Makefile              |   1 +
 test/fixed-buf-send-recv.c | 311 +++++++++++++++++++++++++++++++++++++
 3 files changed, 357 insertions(+), 2 deletions(-)

--
2.54.0

author
#1 lore raw

PATCH v2 1/2 io_uring/net: support registered buffer for plain send and recv

From: Ming Lei <[email protected]>

So far IORING_RECVSEND_FIXED_BUF is only honoured on the SEND_ZC path, even though the import wiring is already present for plain send and completely absent for recv. Targets such as ublk's NBD backend want to push/pull I/O data directly to/from an io_uring registered buffer over a plain send/recv on a TCP socket.

Wire IORING_RECVSEND_FIXED_BUF into the plain IORING_OP_SEND and IORING_OP_RECV paths:

- Accept the flag in SENDMSG_FLAGS / RECVMSG_FLAGS and, at prep time, restrict it to the non-vectorized IORING_OP_SEND / IORING_OP_RECV opcodes. It is mutually exclusive with buffer select, bundles and (for recv) multishot, and records sqe->buf_index.

- For recv, set REQ_F_IMPORT_BUFFER in setup so the registered buffer is imported lazily at issue time, mirroring the send path.

- In io_send()/io_recv(), import the registered buffer via io_import_reg_buf() (ITER_SOURCE for send, ITER_DEST for recv) and clear REQ_F_IMPORT_BUFFER. The resulting bvec iter persists in async_data, so MSG_WAITALL partial send/recv retries resume at the right offset.

---
 io_uring/net.c | 47 +++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 45 insertions(+), 2 deletions(-)
io_uring/net.c +45 -2
index f01f1d25e930..c2bbf9dd2790 100644
@@ -418,7 +418,8 @@ static int io_sendmsg_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe
 	return io_net_import_vec(req, kmsg, msg.msg_iov, msg.msg_iovlen, ITER_SOURCE);
 }
 
-#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE | IORING_SEND_VECTORIZED)
+#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE | \
+			IORING_SEND_VECTORIZED | IORING_RECVSEND_FIXED_BUF)
 
 int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 {
@@ -431,6 +432,15 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 	sr->flags = READ_ONCE(sqe->ioprio);
 	if (sr->flags & ~SENDMSG_FLAGS)
 		return -EINVAL;
+	if (sr->flags & IORING_RECVSEND_FIXED_BUF) {
+		/* registered buffer send only supported for plain IORING_OP_SEND */
+		if (req->opcode != IORING_OP_SEND ||
+		    (sr->flags & IORING_RECVSEND_BUNDLE) ||
+		    (sr->flags & IORING_SEND_VECTORIZED) ||
+		    (req->flags & REQ_F_BUFFER_SELECT))
+			return -EINVAL;
+		req->buf_index = READ_ONCE(sqe->buf_index);
+	}
 	sr->msg_flags = READ_ONCE(sqe->msg_flags) | MSG_NOSIGNAL;
 	if (sr->msg_flags & MSG_DONTWAIT)
 		req->flags |= REQ_F_NOWAIT;
@@ -662,6 +672,15 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags)
 	    (sr->flags & IORING_RECVSEND_POLL_FIRST))
 		return -EAGAIN;
 
+	if (req->flags & REQ_F_IMPORT_BUFFER) {
+		ret = io_import_reg_buf(req, &kmsg->msg.msg_iter,
+					(u64)(uintptr_t)sr->buf, sr->len,
+					ITER_SOURCE, issue_flags);
+		if (unlikely(ret))
+			return ret;
+		req->flags &= ~REQ_F_IMPORT_BUFFER;
+	}
+
 	flags = sr->msg_flags;
 	if (issue_flags & IO_URING_F_NONBLOCK)
 		flags |= MSG_DONTWAIT;
@@ -777,6 +796,10 @@ static int io_recvmsg_prep_setup(struct io_kiocb *req)
 
 		if (req->flags & REQ_F_BUFFER_SELECT)
 			return 0;
+		if (sr->flags & IORING_RECVSEND_FIXED_BUF) {
+			req->flags |= REQ_F_IMPORT_BUFFER;
+			return 0;
+		}
 		return import_ubuf(ITER_DEST, sr->buf, sr->len,
 				   &kmsg->msg.msg_iter);
 	}
@@ -785,7 +808,7 @@ static int io_recvmsg_prep_setup(struct io_kiocb *req)
 }
 
 #define RECVMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECV_MULTISHOT | \
-			IORING_RECVSEND_BUNDLE)
+			IORING_RECVSEND_BUNDLE | IORING_RECVSEND_FIXED_BUF)
 
 int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 {
@@ -803,6 +826,14 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 	sr->flags = READ_ONCE(sqe->ioprio);
 	if (sr->flags & ~RECVMSG_FLAGS)
 		return -EINVAL;
+	if (sr->flags & IORING_RECVSEND_FIXED_BUF) {
+		/* registered buffer recv only for plain IORING_OP_RECV */
+		if (req->opcode != IORING_OP_RECV ||
+		    (sr->flags & (IORING_RECV_MULTISHOT | IORING_RECVSEND_BUNDLE)) ||
+		    (req->flags & REQ_F_BUFFER_SELECT))
+			return -EINVAL;
+		req->buf_index = READ_ONCE(sqe->buf_index);
+	}
 	sr->msg_flags = READ_ONCE(sqe->msg_flags);
 	if (sr->msg_flags & MSG_DONTWAIT)
 		req->flags |= REQ_F_NOWAIT;
@@ -1199,6 +1230,18 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)
 	if (force_nonblock)
 		flags |= MSG_DONTWAIT;
 
+	if (req->flags & REQ_F_IMPORT_BUFFER) {
+		ret = io_import_reg_buf(req, &kmsg->msg.msg_iter,
+					(u64)(uintptr_t)sr->buf, sr->len,
+					ITER_DEST, issue_flags);
+		if (unlikely(ret)) {
+			kmsg->msg.msg_inq = -1;
+			sel.buf_list = NULL;
+			goto out_free;
+		}
+		req->flags &= ~REQ_F_IMPORT_BUFFER;
+	}
+
 retry_multishot:
 	sel.buf_list = NULL;
 	if (io_do_buffer_select(req)) {
Signature · 1 line
2.54.0
author
#2 lore raw

PATCH v2 2/2 test: add fixed-buf-send-recv for registered buffer send/recv

In reply to Ming Lei

From: Ming Lei <[email protected]>

Exercise IORING_RECVSEND_FIXED_BUF on plain IORING_OP_SEND and
IORING_OP_RECV: send-fixed, recv-fixed and both-fixed roundtrips with non-zero offsets into distinct registered buffers, a large MSG_WAITALL transfer to cover the persisted bvec iter across partial retries, plus negative cases (sendmsg/bundle/recv-multishot rejected with -EINVAL and a bad buf_index returning -EFAULT).

---
 test/Makefile              |   1 +
 test/fixed-buf-send-recv.c | 311 +++++++++++++++++++++++++++++++++++++
 2 files changed, 312 insertions(+)
 create mode 100644 test/fixed-buf-send-recv.c
test/Makefile +1 -0
index d1cd2470..effb3bae 100644
@@ -132,6 +132,7 @@ test_srcs := \
 	file-verify.c \
 	fixed-buf-iter.c \
 	fixed-buf-merge.c \
+	fixed-buf-send-recv.c \
 	fixed-hugepage.c \
 	fixed-link.c \
 	fixed-reuse.c \
test/fixed-buf-send-recv.c new file +311 -0

311 lines not on this page · raw message

Signature · 1 line
2.54.0
maintainer Applied
#3 lore raw

Re: (subset) [PATCH v2 0/2] io_uring/net: support registered buffer for plain send and recv

In reply to Ming Lei
Quoted 10 lines · Ming Lei

This series wires IORING_RECVSEND_FIXED_BUF into the plain IORING_OP_SEND and IORING_OP_RECV paths; so far the flag has only been honoured on the SEND_ZC path.

Motivation: targets such as ublk's NBD backend want to push/pull I/O data directly to/from an io_uring registered buffer over a plain send/recv on a TCP socket, avoiding the per-I/O import and page pinning while keeping single-CQE completion. The SEND_ZC path is left untouched.

[...]

Applied, thanks!

[1/2] io_uring/net: support registered buffer for plain send and recv

      commit: 57ed21fad4022d595c6654d3b4d2b2083a79ee25

Best regards,

Signature · 1 line
Jens Axboe