io_uring/net: support registered buffer for plain send and recv
v2 cover io_uring/net: support registered buffer for plain send and recv
From: Ming Lei <[email protected]>
Hi,
This series wires IORING_RECVSEND_FIXED_BUF into the plain IORING_OP_SEND and IORING_OP_RECV paths; so far the flag has only been honoured on the SEND_ZC path.
Motivation: targets such as ublk's NBD backend want to push/pull I/O data directly to/from an io_uring registered buffer over a plain send/recv on a TCP socket, avoiding the per-I/O import and page pinning while keeping single-CQE completion. The SEND_ZC path is left untouched.
Patch 1 is the kernel change. Patch 2 adds a liburing test and is meant for the liburing tree.
Changes since v1:
- Reject IORING_SEND_VECTORIZED on the plain IORING_OP_SEND fixed-buffer
path: the plain io_send() issue path imports a single registered buffer and has no vectorized-regbuf import step, so the combination silently did nothing before. (Jens Axboe)
- Add the matching send-vectorized negative case to the liburing test.
- Clarify in the commit log that SEND_ZC is unaffected.
v1: https://lore.kernel.org/io-uring/[email protected]/
Ming Lei (2):
io_uring/net: support registered buffer for plain send and recv
test: add fixed-buf-send-recv for registered buffer send/recv
io_uring/net.c | 47 ++++++- test/Makefile | 1 + test/fixed-buf-send-recv.c | 311 +++++++++++++++++++++++++++++++++++++ 3 files changed, 357 insertions(+), 2 deletions(-)
--
2.54.0
PATCH v2 1/2 io_uring/net: support registered buffer for plain send and recv
From: Ming Lei <[email protected]>
So far IORING_RECVSEND_FIXED_BUF is only honoured on the SEND_ZC path, even though the import wiring is already present for plain send and completely absent for recv. Targets such as ublk's NBD backend want to push/pull I/O data directly to/from an io_uring registered buffer over a plain send/recv on a TCP socket.
Wire IORING_RECVSEND_FIXED_BUF into the plain IORING_OP_SEND and IORING_OP_RECV paths:
- Accept the flag in SENDMSG_FLAGS / RECVMSG_FLAGS and, at prep time, restrict it to the non-vectorized IORING_OP_SEND / IORING_OP_RECV opcodes. It is mutually exclusive with buffer select, bundles and (for recv) multishot, and records sqe->buf_index.
- For recv, set REQ_F_IMPORT_BUFFER in setup so the registered buffer is imported lazily at issue time, mirroring the send path.
- In io_send()/io_recv(), import the registered buffer via io_import_reg_buf() (ITER_SOURCE for send, ITER_DEST for recv) and clear REQ_F_IMPORT_BUFFER. The resulting bvec iter persists in async_data, so MSG_WAITALL partial send/recv retries resume at the right offset.
--- io_uring/net.c | 47 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 45 insertions(+), 2 deletions(-)
io_uring/net.c +45 -2
@@ -418,7 +418,8 @@ static int io_sendmsg_setup(struct io_kiocb *req, const struct io_uring_sqe *sqereturn io_net_import_vec (req, kmsg, msg.msg_iov, msg.msg_iovlen,ITER_SOURCE ); } -#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE | IORING_SEND_VECTORIZED) +#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE | \ + IORING_SEND_VECTORIZED | IORING_RECVSEND_FIXED_BUF)int io_sendmsg_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
@@ -431,6 +432,15 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->flags =READ_ONCE (sqe->ioprio);if (sr->flags & ~SENDMSG_FLAGS )return -EINVAL ; +if (sr->flags &IORING_RECVSEND_FIXED_BUF ) { +/* registered buffer send only supported for plain IORING_OP_SEND */ +if (req->opcode !=IORING_OP_SEND || + (sr->flags &IORING_RECVSEND_BUNDLE ) || + (sr->flags &IORING_SEND_VECTORIZED ) || + (req->flags &REQ_F_BUFFER_SELECT )) +return -EINVAL ; + req->buf_index =READ_ONCE (sqe->buf_index); + } sr->msg_flags =READ_ONCE (sqe->msg_flags) |MSG_NOSIGNAL ;if (sr->msg_flags &MSG_DONTWAIT ) req->flags |=REQ_F_NOWAIT ;
@@ -662,6 +672,15 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST))return -EAGAIN ; +if (req->flags &REQ_F_IMPORT_BUFFER ) { + ret =io_import_reg_buf (req, &kmsg->msg.msg_iter, + (u64 )(uintptr_t )sr->buf, sr->len, +ITER_SOURCE , issue_flags); +if (unlikely (ret)) +return ret; + req->flags &= ~REQ_F_IMPORT_BUFFER ; + } + flags = sr->msg_flags;if (issue_flags &IO_URING_F_NONBLOCK ) flags |=MSG_DONTWAIT ;
@@ -777,6 +796,10 @@ static int io_recvmsg_prep_setup(struct io_kiocb *req)if (req->flags &REQ_F_BUFFER_SELECT )return 0 ; +if (sr->flags &IORING_RECVSEND_FIXED_BUF ) { + req->flags |=REQ_F_IMPORT_BUFFER ; +return 0 ; + }return import_ubuf (ITER_DEST , sr->buf, sr->len, &kmsg->msg.msg_iter); }
@@ -785,7 +808,7 @@ static int io_recvmsg_prep_setup(struct io_kiocb *req) }#define RECVMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECV_MULTISHOT | \ - IORING_RECVSEND_BUNDLE) + IORING_RECVSEND_BUNDLE | IORING_RECVSEND_FIXED_BUF)int io_recvmsg_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
@@ -803,6 +826,14 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->flags =READ_ONCE (sqe->ioprio);if (sr->flags & ~RECVMSG_FLAGS )return -EINVAL ; +if (sr->flags &IORING_RECVSEND_FIXED_BUF ) { +/* registered buffer recv only for plain IORING_OP_RECV */ +if (req->opcode !=IORING_OP_RECV || + (sr->flags & (IORING_RECV_MULTISHOT |IORING_RECVSEND_BUNDLE )) || + (req->flags &REQ_F_BUFFER_SELECT )) +return -EINVAL ; + req->buf_index =READ_ONCE (sqe->buf_index); + } sr->msg_flags =READ_ONCE (sqe->msg_flags);if (sr->msg_flags &MSG_DONTWAIT ) req->flags |=REQ_F_NOWAIT ;
@@ -1199,6 +1230,18 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)if (force_nonblock) flags |=MSG_DONTWAIT ; +if (req->flags &REQ_F_IMPORT_BUFFER ) { + ret =io_import_reg_buf (req, &kmsg->msg.msg_iter, + (u64 )(uintptr_t )sr->buf, sr->len, +ITER_DEST , issue_flags); +if (unlikely (ret)) { + kmsg->msg.msg_inq =-1 ; + sel.buf_list =NULL ; + goto out_free; + } + req->flags &= ~REQ_F_IMPORT_BUFFER ; + } + retry_multishot: sel.buf_list =NULL ;if (io_do_buffer_select (req)) {
Signature · 1 line
2.54.0
PATCH v2 2/2 test: add fixed-buf-send-recv for registered buffer send/recv
In reply to Ming LeiFrom: Ming Lei <[email protected]>
Exercise IORING_RECVSEND_FIXED_BUF on plain IORING_OP_SEND andIORING_OP_RECV: send-fixed, recv-fixed and both-fixed roundtrips with non-zero offsets into distinct registered buffers, a large MSG_WAITALL transfer to cover the persisted bvec iter across partial retries, plus negative cases (sendmsg/bundle/recv-multishot rejected with -EINVAL and a bad buf_index returning -EFAULT).
--- test/Makefile | 1 + test/fixed-buf-send-recv.c | 311 +++++++++++++++++++++++++++++++++++++ 2 files changed, 312 insertions(+) create mode 100644 test/fixed-buf-send-recv.c
test/Makefile +1 -0
@@ -132,6 +132,7 @@ test_srcs := \ file-verify.c \ fixed-buf-iter.c \ fixed-buf-merge.c \ + fixed-buf-send-recv.c \ fixed-hugepage.c \ fixed-link.c \ fixed-reuse.c \
test/fixed-buf-send-recv.c new file +311 -0
311 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: (subset) [PATCH v2 0/2] io_uring/net: support registered buffer for plain send and recv
In reply to Ming LeiQuoted 10 lines · Ming Lei
This series wires
IORING_RECVSEND_FIXED_BUFinto the plainIORING_OP_SENDandIORING_OP_RECVpaths; so far the flag has only been honoured on theSEND_ZCpath.Motivation: targets such as ublk's NBD backend want to push/pull I/O data directly to/from an
io_uringregistered buffer over a plain send/recv on a TCP socket, avoiding the per-I/O import and page pinning while keeping single-CQE completion. TheSEND_ZCpath is left untouched.[...]
Applied, thanks!
[1/2] io_uring/net: support registered buffer for plain send and recv
commit: 57ed21fad4022d595c6654d3b4d2b2083a79ee25
Best regards,
Signature · 1 line
Jens Axboe