io_uring/rw: transform single vector readv/writev into ubuf
Jens Axboe
maintainerPATCH 1/1 io_uring/rw: transform single vector readv/writev into ubuf
It's very common to have applications that use vectored reads or writes, even if they only pass in a single segment. Obviously they should be using read/write at that point, but...
Vectored IO comes with the downside of needing to retain iovec state, and hence they require and allocation and state copy if they end up getting deferred. Additionally, they also require extra cleanup when completed as the memory as the allocated state memory has to be freed.
Automatically transform single segment IORING_OP_{READV,WRITEV} into IORING_OP_{READ,WRITE}, and hence into an ITER_UBUF. Outside of being more efficient if needing deferral, ITER_UBUF is also more efficient for normal processing compared to ITER_IOVEC, as they don't require iteration. The latter is apparent when running peak testing, where using IORING_OP_READV to randomly read 24 drives previously scored:
IOPS=72.54M, BW=35.42GiB/s, IOS/call=32/31
IOPS=73.35M, BW=35.81GiB/s, IOS/call=32/31
IOPS=72.71M, BW=35.50GiB/s, IOS/call=32/31
IOPS=73.29M, BW=35.78GiB/s, IOS/call=32/32
IOPS=73.45M, BW=35.86GiB/s, IOS/call=32/32
IOPS=73.19M, BW=35.74GiB/s, IOS/call=31/32
IOPS=72.89M, BW=35.59GiB/s, IOS/call=32/31
IOPS=73.07M, BW=35.68GiB/s, IOS/call=32/32
and after the change we get:
IOPS=77.31M, BW=37.75GiB/s, IOS/call=32/31
IOPS=77.32M, BW=37.75GiB/s, IOS/call=32/32
IOPS=77.45M, BW=37.81GiB/s, IOS/call=31/31
IOPS=77.47M, BW=37.83GiB/s, IOS/call=32/32
IOPS=77.14M, BW=37.67GiB/s, IOS/call=32/32
IOPS=77.14M, BW=37.66GiB/s, IOS/call=31/31
IOPS=77.37M, BW=37.78GiB/s, IOS/call=32/32
IOPS=77.25M, BW=37.72GiB/s, IOS/call=32/32
which is a nice win as well.
---
io_uring/rw.c +16 -1
@@ -402,7 +402,22 @@ static struct iovec *__io_import_iovec(int ddir, struct io_kiocb *req, req->ctx->compat);if (unlikely (ret <0 ))return ERR_PTR (ret); -return iovec; +if (iter->nr_segs !=1 ) +return iovec; +/* +* Convert to non-vectored request if we have a single segment. If we +* need to defer the request, then we no longer have to allocate and +* maintain a struct io_async_rw. Additionally, we won't have cleanup +* to do at completion time +*/ + rw->addr = (unsigned long ) iter->iov[0 ].iov_base; + rw->len = iter->iov[0 ].iov_len; +iov_iter_ubuf (iter, ddir, iter->iov[0 ].iov_base, rw->len); +/* readv -> read distance is the same as writev -> write */ +BUILD_BUG_ON ((IORING_OP_READ -IORING_OP_READV ) != + (IORING_OP_WRITE -IORING_OP_WRITEV )); + req->opcode += (IORING_OP_READ -IORING_OP_READV ); +return NULL ; } static inline int io_import_iovec(int rw, struct io_kiocb *req,
Signature · 1 line
Jens Axboe
Quoted 3 lines · Jens Axboe
It's very common to have applications that use vectored reads or writes, even if they only pass in a single segment. Obviously they should be using read/write at that point, but...
Yeah, it is like fixing application issue in kernel side, :-)
Quoted 53 lines
Vectored IO comes with the downside of needing to retain iovec state, and hence they require and allocation and state copy if they end up getting deferred. Additionally, they also require extra cleanup when completed as the memory as the allocated state memory has to be freed.
Automatically transform single segment IORING_OP_{READV,WRITEV} into IORING_OP_{READ,WRITE}, and hence into an
ITER_UBUF. Outside of being more efficient if needing deferral,ITER_UBUFis also more efficient for normal processing compared toITER_IOVEC, as they don't require iteration. The latter is apparent when running peak testing, where usingIORING_OP_READVto randomly read 24 drives previously scored:IOPS=72.54M, BW=35.42GiB/s, IOS/call=32/31
IOPS=73.35M, BW=35.81GiB/s, IOS/call=32/31
IOPS=72.71M, BW=35.50GiB/s, IOS/call=32/31
IOPS=73.29M, BW=35.78GiB/s, IOS/call=32/32
IOPS=73.45M, BW=35.86GiB/s, IOS/call=32/32
IOPS=73.19M, BW=35.74GiB/s, IOS/call=31/32
IOPS=72.89M, BW=35.59GiB/s, IOS/call=32/31
IOPS=73.07M, BW=35.68GiB/s, IOS/call=32/32and after the change we get:
IOPS=77.31M, BW=37.75GiB/s, IOS/call=32/31
IOPS=77.32M, BW=37.75GiB/s, IOS/call=32/32
IOPS=77.45M, BW=37.81GiB/s, IOS/call=31/31
IOPS=77.47M, BW=37.83GiB/s, IOS/call=32/32
IOPS=77.14M, BW=37.67GiB/s, IOS/call=32/32
IOPS=77.14M, BW=37.66GiB/s, IOS/call=31/31
IOPS=77.37M, BW=37.78GiB/s, IOS/call=32/32
IOPS=77.25M, BW=37.72GiB/s, IOS/call=32/32which is a nice win as well.
Signed-off-by Jens Axboe <[email protected]>---
io_uring/rw.c +5
-1@@ -402,7 +402,22 @@ static struct iovec *__io_import_iovec(int ddir, struct io_kiocb *req, req->ctx->compat);if (unlikely (ret <0 ))return ERR_PTR (ret); -return iovec; +if (iter->nr_segs !=1 ) +return iovec; +/* +* Convert to non-vectored request if we have a single segment. If we +* need to defer the request, then we no longer have to allocate and
io_uring/rw.c static struct iovec *__io_import_iovec(int ddir, struct io_kiocb *req, · line 419
Quoted 10 lines
io_uring/rw.c +10
-0+* maintain a struct io_async_rw. Additionally, we won't have cleanup +* to do at completion time +*/ + rw->addr = (unsigned long ) iter->iov[0 ].iov_base; + rw->len = iter->iov[0 ].iov_len; +iov_iter_ubuf (iter, ddir, iter->iov[0 ].iov_base, rw->len); +/* readv -> read distance is the same as writev -> write */ +BUILD_BUG_ON ((IORING_OP_READ -IORING_OP_READV ) != + (IORING_OP_WRITE -IORING_OP_WRITEV )); + req->opcode += (IORING_OP_READ -IORING_OP_READV );
It is a bit fragile to change ->opcode, which may need matched callbacks for the two OPs, also cause inconsistent opcode in traces.
I am wondering why not play the magic in io_prep_rw() from beginning?
Thanks,
Ming
Jens Axboe
maintainerQuoted 6 lines · Ming Lei
Quoted 3 lines · Jens Axboe
It's very common to have applications that use vectored reads or writes, even if they only pass in a single segment. Obviously they should be using read/write at that point, but...
Yeah, it is like fixing application issue in kernel side, :-)
It totally is, the same thing happens all of the time for readv as well. No amount of informing or documenting will ever fix that...
Also see:
https://lore.kernel.org/linux-fsdevel/[email protected]/
with which I think I'll change this one to just be:
if (iter->iter_type == ITER_UBUF) {
rw->addr = iter->ubuf;
rw->len = iter->count;
/* readv -> read distance is the same as writev -> write */
BUILD_BUG_ON((IORING_OP_READ - IORING_OP_READV) !=
(IORING_OP_WRITE - IORING_OP_WRITEV));
req->opcode += (IORING_OP_READ - IORING_OP_READV);
}instead.
We could also just skip it completely and just have liburing do the right thing if io_uring_prep_readv/writev is called with nr_segs == 1. Just turn it into a READ/WRITE at that point. If we do that, and with the above generic change, it's probably Good Enough. If you use READV/WRITEV and you're using the raw interface, then you're on your own.
Quoted 11 lines
io_uring/rw.c static struct iovec *__io_import_iovec(int ddir, struct io_kiocb *req, · line 419
Quoted 7 lines
io_uring/rw.c +7
-0+ rw->addr = (unsigned long ) iter->iov[0 ].iov_base; + rw->len = iter->iov[0 ].iov_len; +iov_iter_ubuf (iter, ddir, iter->iov[0 ].iov_base, rw->len); +/* readv -> read distance is the same as writev -> write */ +BUILD_BUG_ON ((IORING_OP_READ -IORING_OP_READV ) != + (IORING_OP_WRITE -IORING_OP_WRITEV )); + req->opcode += (IORING_OP_READ -IORING_OP_READV );It is a bit fragile to change ->opcode, which may need matched callbacks for the two OPs, also cause inconsistent opcode in traces.
Quoted 1 line
I am wondering why not play the magic in
io_prep_rw()from beginning?
It has to be done when importing the vec, we cannot really do it in prep... Well we could, but that'd be adding a bunch more code and duplicating part of the vec import.
Signature · 1 line
Jens Axboe
Quoted 12 lines · Jens Axboe
io_uring/rw.c +7
-1@@ -402,7 +402,22 @@ static struct iovec *__io_import_iovec(int ddir, struct io_kiocb *req, req->ctx->compat);if (unlikely (ret <0 ))return ERR_PTR (ret); -return iovec; +if (iter->nr_segs !=1 ) +return iovec; +/* +* Convert to non-vectored request if we have a single segment. If we +* need to defer the request, then we no longer have to allocate and +* maintain a struct io_async_rw. Additionally, we won't have cleanup +* to do at completion time
io_uring/rw.c static struct iovec *__io_import_iovec(int ddir, struct io_kiocb *req, · line 421
Quoted 10 lines
io_uring/rw.c +9
-0+*/ + rw->addr = (unsigned long ) iter->iov[0 ].iov_base; + rw->len = iter->iov[0 ].iov_len; +iov_iter_ubuf (iter, ddir, iter->iov[0 ].iov_base, rw->len); +/* readv -> read distance is the same as writev -> write */ +BUILD_BUG_ON ((IORING_OP_READ -IORING_OP_READV ) != + (IORING_OP_WRITE -IORING_OP_WRITEV )); + req->opcode += (IORING_OP_READ -IORING_OP_READV ); +return NULL ; }
This may break anyone using io_uring with those bizarre drivers that have entirely different readv semantics from normal read. I think we can safely say no one's using io_uring for such interfaces, so probably a moot point. If you wanted to be extra cautious though, you may want to skip this transformation if the file->f_op implements both .read+read_iter and .write+.write_iter.
Quoted 46 lines · Jens Axboe
Quoted 6 lines · Ming Lei
Quoted 3 lines · Jens Axboe
It's very common to have applications that use vectored reads or writes, even if they only pass in a single segment. Obviously they should be using read/write at that point, but...
Yeah, it is like fixing application issue in kernel side, :-)
It totally is, the same thing happens all of the time for readv as well. No amount of informing or documenting will ever fix that...
Also see:
https://lore.kernel.org/linux-fsdevel/[email protected]/
with which I think I'll change this one to just be:
if (iter->iter_type == ITER_UBUF) { rw->addr = iter->ubuf; rw->len = iter->count; /* readv -> read distance is the same as writev -> write */ BUILD_BUG_ON((IORING_OP_READ - IORING_OP_READV) != (IORING_OP_WRITE - IORING_OP_WRITEV)); req->opcode += (IORING_OP_READ - IORING_OP_READV); }instead.
We could also just skip it completely and just have liburing do the right thing if io_uring_prep_readv/writev is called with
nr_segs== 1. Just turn it into a READ/WRITE at that point. If we do that, and with the above generic change, it's probably Good Enough. If you use READV/WRITEV and you're using the raw interface, then you're on your own.Quoted 11 lines
Review
Quoted 7 lines
(quoted diff) +7
-0+ rw->addr = (unsigned long) iter->iov[0].iov_base; + rw->len = iter->iov[0].iov_len; + iov_iter_ubuf(iter, ddir, iter->iov[0].iov_base, rw->len); + /* readv -> read distance is the same as writev -> write */ + BUILD_BUG_ON((IORING_OP_READ - IORING_OP_READV) != + (IORING_OP_WRITE - IORING_OP_WRITEV)); + req->opcode += (IORING_OP_READ - IORING_OP_READV);It is a bit fragile to change ->opcode, which may need matched callbacks for the two OPs, also cause inconsistent opcode in traces.
Quoted 5 lines
Quoted 1 line
I am wondering why not play the magic in
io_prep_rw()from beginning?It has to be done when importing the vec, we cannot really do it in prep... Well we could, but that'd be adding a bunch more code and duplicating part of the vec import.
I meant something like the following(un-tested), which at least guarantees that op_code, rw->addr/len are finalized since ->prep().
io_uring/rw.c +19 -0
@@ -120,6 +120,25 @@ int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe)return ret; } +if (req->opcode ==IORING_OP_READV && req->opcode ==IORING_OP_WRITEV && + rw->len ==1 ) { +struct iovec iov; +struct iovec *iovp; + + iovp =iovec_from_user (u64_to_user_ptr (rw->addr),1 ,1 , &iov, + req->ctx->compat); +if (IS_ERR (iovp)) +return PTR_ERR (iovp); + + rw->addr = (unsigned long ) iovp->iov_base; + rw->len = iovp->iov_len; + +/* readv -> read distance is the same as writev -> write */ +BUILD_BUG_ON ((IORING_OP_READ -IORING_OP_READV ) != + (IORING_OP_WRITE -IORING_OP_WRITEV )); + req->opcode += (IORING_OP_READ -IORING_OP_READV ); + } +return 0 ; }
Thanks,
Ming
Quoted 21 lines · Keith Busch
Quoted 20 lines · Jens Axboe
io_uring/rw.c +15
-1@@ -402,7 +402,22 @@ static struct iovec *__io_import_iovec(int ddir, struct io_kiocb *req, req->ctx->compat);if (unlikely (ret <0 ))return ERR_PTR (ret); -return iovec; +if (iter->nr_segs !=1 ) +return iovec; +/* +* Convert to non-vectored request if we have a single segment. If we +* need to defer the request, then we no longer have to allocate and +* maintain a struct io_async_rw. Additionally, we won't have cleanup +* to do at completion time +*/ + rw->addr = (unsigned long ) iter->iov[0 ].iov_base; + rw->len = iter->iov[0 ].iov_len; +iov_iter_ubuf (iter, ddir, iter->iov[0 ].iov_base, rw->len); +/* readv -> read distance is the same as writev -> write */ +BUILD_BUG_ON ((IORING_OP_READ -IORING_OP_READV ) != + (IORING_OP_WRITE -IORING_OP_WRITEV )); + req->opcode += (IORING_OP_READ -IORING_OP_READV );
Quoted 8 lines
io_uring/rw.c static struct iovec *__io_import_iovec(int ddir, struct io_kiocb *req, · line 421
Quoted 2 lines
io_uring/rw.c +1
-0+return NULL ; }This may break anyone using
io_uringwith those bizarre drivers that have entirely different readv semantics from normal read. I think we can safely say no one's usingio_uringfor such interfaces, so probably a moot point. If you wanted to be extra cautious though, you may want to skip this transformation if thefile->f_opimplements both .read+read_iter and .write+.write_iter.
Sorry, scratch that; those drivers are already broken if they implement both with different semantics, so we can definitely say none of those are using io_uring. This new patch should be safe then.
Quoted 27 lines · Ming Lei
Quoted 26 lines · Jens Axboe
Quoted 6 lines · Ming Lei
Quoted 3 lines · Jens Axboe
It's very common to have applications that use vectored reads or writes, even if they only pass in a single segment. Obviously they should be using read/write at that point, but...
Yeah, it is like fixing application issue in kernel side, :-)
It totally is, the same thing happens all of the time for readv as well. No amount of informing or documenting will ever fix that...
Also see:
https://lore.kernel.org/linux-fsdevel/[email protected]/
with which I think I'll change this one to just be:
if (iter->iter_type == ITER_UBUF) { rw->addr = iter->ubuf; rw->len = iter->count; /* readv -> read distance is the same as writev -> write */ BUILD_BUG_ON((IORING_OP_READ - IORING_OP_READV) != (IORING_OP_WRITE - IORING_OP_WRITEV)); req->opcode += (IORING_OP_READ - IORING_OP_READV); }
Quoted 8 lines
Quoted 8 lines
instead.
We could also just skip it completely and just have liburing do the right thing if io_uring_prep_readv/writev is called with
nr_segs== 1. Just turn it into a READ/WRITE at that point. If we do that, and with the above generic change, it's probably Good Enough. If you use READV/WRITEV and you're using the raw interface, then you're on your own.
I like this option but sendmsg and recvmsg probably do need the same fix up, which is more justified as they can't get converted to send/recv as this.
Another option is to internally detangle opcodes from iter types.
import() {
if (req->op == READV)
import_iovec();
else
import_buf();
}would get replaced with:
prep() {
if (req->op == READV)
req->flags = REQ_F_IOVEC;
}import() {
if (req->flags & REQ_F_IOVEC)
import_iovec();
else
import_buf();
}Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
io_uring/rw.c int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) · line 139
Quoted 7 lines
io_uring/rw.c +7
-0+ rw->addr = (unsigned long ) iter->iov[0 ].iov_base; + rw->len = iter->iov[0 ].iov_len; +iov_iter_ubuf (iter, ddir, iter->iov[0 ].iov_base, rw->len); +/* readv -> read distance is the same as writev -> write */ +BUILD_BUG_ON ((IORING_OP_READ -IORING_OP_READV ) != + (IORING_OP_WRITE -IORING_OP_WRITEV )); + req->opcode += (IORING_OP_READ -IORING_OP_READV );It is a bit fragile to change ->opcode, which may need matched callbacks for the two OPs, also cause inconsistent opcode in traces.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I am wondering why not play the magic in
io_prep_rw()from beginning?It has to be done when importing the vec, we cannot really do it in prep... Well we could, but that'd be adding a bunch more code and duplicating part of the vec import.
I meant something like the following(un-tested), which at least guarantees that
op_code, rw->addr/len are finalized since ->prep().
It sounds like a better approach. With opcode machinations it's easy to forget about some kind of state that could be fatal.
Take IOSQE_ASYNC for example. The core code will allocate async_data and do io_readv_prep_async() -> import_iovec(), which inside changes the opcode. It'll be a problem if io_readv_prep_async() forgets that it might a different opcode with a slightly different req layout, or even non-vectored read would do sth weird with ->async_data or mishandle REQ_F_NEED_CLEANUP.
fwiw, needs compat handling, i.e. leave as iovec if compat
Quoted 29 lines
io_uring/rw.c +19
-0@@ -120,6 +120,25 @@ int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe)return ret; } +if (req->opcode ==IORING_OP_READV && req->opcode ==IORING_OP_WRITEV && + rw->len ==1 ) { +struct iovec iov; +struct iovec *iovp; + + iovp =iovec_from_user (u64_to_user_ptr (rw->addr),1 ,1 , &iov, + req->ctx->compat); +if (IS_ERR (iovp)) +return PTR_ERR (iovp); + + rw->addr = (unsigned long ) iovp->iov_base; + rw->len = iovp->iov_len; + +/* readv -> read distance is the same as writev -> write */ +BUILD_BUG_ON ((IORING_OP_READ -IORING_OP_READV ) != + (IORING_OP_WRITE -IORING_OP_WRITEV )); + req->opcode += (IORING_OP_READ -IORING_OP_READV ); + } +return 0 ; }
Signature · 1 line
Pavel Begunkov