io_uring/net: support registered buffer for plain send and recv
cover io_uring/net: support registered buffer for plain send and recv
From: Ming Lei <[email protected]>
Hi,
This series wires IORING_RECVSEND_FIXED_BUF into the plain IORING_OP_SEND and IORING_OP_RECV paths. So far the flag has only been honoured on the SEND_ZC path, even though the import wiring is already present for plain send and completely absent for recv.
Motivation: targets such as ublk's NBD backend want to push/pull I/O data directly to/from an io_uring registered buffer over a plain send/recv on a TCP socket, without the SEND_ZC notification machinery.
The flag is accepted at prep time for the non-vectorized IORING_OP_SEND / IORING_OP_RECV opcodes only, and is mutually exclusive with buffer select, bundles and (for recv) multishot. The registered buffer is imported lazily at issue time via io_import_reg_buf() (mirroring the existing send path), and the resulting bvec iter persists in async_data so MSG_WAITALL partial send/recv retries resume at the right offset.
Patch 1 is the kernel change.
Patch 2 adds a liburing test and is meant for the liburing tree. It covers send-fixed/recv-fixed/both-fixed roundtrips (with non-zero offsets into distinct registered buffers), a large MSG_WAITALL transfer that exercises the persisted bvec iter across partial retries, and the negative validation cases (sendmsg/bundle/recv-multishot rejected with -EINVAL, bad buf_index returning -EFAULT).
Ming Lei (2):
io_uring/net: support registered buffer for plain send and recv
test: add fixed-buf-send-recv for registered buffer send/recv
io_uring/net.c | 46 ++++++- test/Makefile | 1 + test/fixed-buf-send-recv.c | 300 +++++++++++++++++++++++++++++++++++++ 3 files changed, 345 insertions(+), 2 deletions(-)
--
2.54.0
PATCH 1/2 io_uring/net: support registered buffer for plain send and recv
From: Ming Lei <[email protected]>
So far IORING_RECVSEND_FIXED_BUF is only honoured on the SEND_ZC path, even though the import wiring is already present for plain send and completely absent for recv. Targets such as ublk's NBD backend want to push/pull I/O data directly to/from an io_uring registered buffer over a plain send/recv on a TCP socket, without the SEND_ZC notification machinery.
Wire IORING_RECVSEND_FIXED_BUF into the plain IORING_OP_SEND and IORING_OP_RECV paths:
- Accept the flag in SENDMSG_FLAGS / RECVMSG_FLAGS and, at prep time, restrict it to the non-vectorized IORING_OP_SEND / IORING_OP_RECV opcodes. It is mutually exclusive with buffer select, bundles and (for recv) multishot, and records sqe->buf_index.
- For recv, set REQ_F_IMPORT_BUFFER in setup so the registered buffer is imported lazily at issue time, mirroring the send path.
- In io_send()/io_recv(), import the registered buffer via io_import_reg_buf() (ITER_SOURCE for send, ITER_DEST for recv) and clear REQ_F_IMPORT_BUFFER. The resulting bvec iter persists in async_data, so MSG_WAITALL partial send/recv retries resume at the right offset.
--- io_uring/net.c | 46 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 44 insertions(+), 2 deletions(-)
io_uring/net.c +44 -2
@@ -418,7 +418,8 @@ static int io_sendmsg_setup(struct io_kiocb *req, const struct io_uring_sqe *sqereturn io_net_import_vec (req, kmsg, msg.msg_iov, msg.msg_iovlen,ITER_SOURCE ); } -#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE | IORING_SEND_VECTORIZED) +#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE | \ + IORING_SEND_VECTORIZED | IORING_RECVSEND_FIXED_BUF)int io_sendmsg_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
@@ -431,6 +432,14 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->flags =READ_ONCE (sqe->ioprio);if (sr->flags & ~SENDMSG_FLAGS )return -EINVAL ; +if (sr->flags &IORING_RECVSEND_FIXED_BUF ) { +/* registered buffer send only supported for plain IORING_OP_SEND */ +if (req->opcode !=IORING_OP_SEND || + (sr->flags &IORING_RECVSEND_BUNDLE ) || + (req->flags &REQ_F_BUFFER_SELECT )) +return -EINVAL ; + req->buf_index =READ_ONCE (sqe->buf_index); + } sr->msg_flags =READ_ONCE (sqe->msg_flags) |MSG_NOSIGNAL ;if (sr->msg_flags &MSG_DONTWAIT ) req->flags |=REQ_F_NOWAIT ;
@@ -662,6 +671,15 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST))return -EAGAIN ; +if (req->flags &REQ_F_IMPORT_BUFFER ) { + ret =io_import_reg_buf (req, &kmsg->msg.msg_iter, + (u64 )(uintptr_t )sr->buf, sr->len, +ITER_SOURCE , issue_flags); +if (unlikely (ret)) +return ret; + req->flags &= ~REQ_F_IMPORT_BUFFER ; + } + flags = sr->msg_flags;if (issue_flags &IO_URING_F_NONBLOCK ) flags |=MSG_DONTWAIT ;
@@ -777,6 +795,10 @@ static int io_recvmsg_prep_setup(struct io_kiocb *req)if (req->flags &REQ_F_BUFFER_SELECT )return 0 ; +if (sr->flags &IORING_RECVSEND_FIXED_BUF ) { + req->flags |=REQ_F_IMPORT_BUFFER ; +return 0 ; + }return import_ubuf (ITER_DEST , sr->buf, sr->len, &kmsg->msg.msg_iter); }
@@ -785,7 +807,7 @@ static int io_recvmsg_prep_setup(struct io_kiocb *req) }#define RECVMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECV_MULTISHOT | \ - IORING_RECVSEND_BUNDLE) + IORING_RECVSEND_BUNDLE | IORING_RECVSEND_FIXED_BUF)int io_recvmsg_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
@@ -803,6 +825,14 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->flags =READ_ONCE (sqe->ioprio);if (sr->flags & ~RECVMSG_FLAGS )return -EINVAL ; +if (sr->flags &IORING_RECVSEND_FIXED_BUF ) { +/* registered buffer recv only for plain IORING_OP_RECV */ +if (req->opcode !=IORING_OP_RECV || + (sr->flags & (IORING_RECV_MULTISHOT |IORING_RECVSEND_BUNDLE )) || + (req->flags &REQ_F_BUFFER_SELECT )) +return -EINVAL ; + req->buf_index =READ_ONCE (sqe->buf_index); + } sr->msg_flags =READ_ONCE (sqe->msg_flags);if (sr->msg_flags &MSG_DONTWAIT ) req->flags |=REQ_F_NOWAIT ;
@@ -1199,6 +1229,18 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)if (force_nonblock) flags |=MSG_DONTWAIT ; +if (req->flags &REQ_F_IMPORT_BUFFER ) { + ret =io_import_reg_buf (req, &kmsg->msg.msg_iter, + (u64 )(uintptr_t )sr->buf, sr->len, +ITER_DEST , issue_flags); +if (unlikely (ret)) { + kmsg->msg.msg_inq =-1 ; + sel.buf_list =NULL ; + goto out_free; + } + req->flags &= ~REQ_F_IMPORT_BUFFER ; + } + retry_multishot: sel.buf_list =NULL ;if (io_do_buffer_select (req)) {
Signature · 1 line
2.54.0
Jens Axboe
maintainerQuoted 6 lines · Ming Lei
io_uring/net.c +0
-0@@ -431,6 +432,14 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->flags =READ_ONCE (sqe->ioprio);
io_uring/net.c int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) · line 442
Quoted 10 lines
io_uring/net.c +8
-0if (sr->flags & ~SENDMSG_FLAGS )return -EINVAL ; +if (sr->flags &IORING_RECVSEND_FIXED_BUF ) { +/* registered buffer send only supported for plain IORING_OP_SEND */ +if (req->opcode !=IORING_OP_SEND || + (sr->flags &IORING_RECVSEND_BUNDLE ) || + (req->flags &REQ_F_BUFFER_SELECT )) +return -EINVAL ; + req->buf_index =READ_ONCE (sqe->buf_index); + }
I think this should either reject IORING_SEND_VECTORIZED, or if there's a use case for it, ensure that it actually works.
Outside of that, change seems straight forward to me.
Signature · 1 line
Jens Axboe
PATCH 2/2 liburing test: add fixed-buf-send-recv for registered buffer send/recv
In reply to Ming LeiFrom: Ming Lei <[email protected]>
Exercise IORING_RECVSEND_FIXED_BUF on plain IORING_OP_SEND andIORING_OP_RECV: send-fixed, recv-fixed and both-fixed roundtrips with non-zero offsets into distinct registered buffers, a large MSG_WAITALL transfer to cover the persisted bvec iter across partial retries, plus negative cases (sendmsg/bundle/recv-multishot rejected with -EINVAL and a bad buf_index returning -EFAULT).
--- test/Makefile | 1 + test/fixed-buf-send-recv.c | 300 +++++++++++++++++++++++++++++++++++++ 2 files changed, 301 insertions(+) create mode 100644 test/fixed-buf-send-recv.c
test/Makefile +1 -0
@@ -132,6 +132,7 @@ test_srcs := \ file-verify.c \ fixed-buf-iter.c \ fixed-buf-merge.c \ + fixed-buf-send-recv.c \ fixed-hugepage.c \ fixed-link.c \ fixed-reuse.c \
test/fixed-buf-send-recv.c new file +300 -0
300 lines not on this page · raw message
Signature · 1 line
2.54.0
Hello Jens,
Ping...
Thanks,
Ming