io_uring: avoid req->flags store-to-load forwarding failures
cover io_uring: avoid req->flags store-to-load forwarding failures
clang likes to set/clear req->flags bits using the narrowest possible or/and instructions (e.g. an 8-bit or if all flag bits are in the same byte). This causes store-to-load forwarding failures on subsequent wider req->flags loads.
For example, clang compiles the req->flags |= REQ_F_ASYNC_DATA in io_uring_cmd_prep() -> io_uring_alloc_cached_async_data() on x86_64 into an orb $0x40, 0x4a(%rdi) instruction, since REQ_F_ASYNC_DATA = 0x40 << 16 and offsetof(struct io_kiocb, flags) = 0x48. The subsequent 64-bit req->flags load for unlikely(req->flags & (IO_REQ_LINK_FLAGS | REQ_F_FORCE_ASYNC | REQ_F_FAIL)) in io_submit_sqe() is extremely hot because it can't be served from the store buffer and must wait for the REQ_F_ASYNC_DATA set to hit the L1 cache. On a thread issuing ublk and NVMe passthrough commands, that one req->flags load consumes around 1% of all CPU cycles.
Introduce io_req_set_flags() and io_req_clear_flags() helpers to set req->flags using volatile writes so clang won't emit a store to only part of the field. The downside of volatile writes is that consecutive req->flags updates must store to req->flags multiple times. Where possible, convert these updates to pass multiple flags to io_req_set_flags(), io_req_clear_flags(), or an extra helper io_req_switch_flags() which can do both. An extra store would still be much cheaper than a store-to-load forwarding stall.
Alternatively, -mllvm -combiner-reduce-load-op-store-width=false could be used to disable the clang "optimization". It seemed ugly to refer to such an internal optimization pass, and I wasn't sure what other impacts this flag might have (even if restricted to io_uring). But I'm happy to implement that approach if it's preferred.
Caleb Sander Mateos (4):
io_uring: return io_req_flags_t from io_slot_flags()
io_uring: add multishot_flags constant in io_wq_submit_work()
io_uring: add io_req_{set,clear,switch}_flags()
io_uring: use io_req_{set,clear,switch}_flags()
include/linux/io_uring.h | 35 +++++++++++++++++++++++++----- io_uring/advise.c | 4 ++-- io_uring/cancel.c | 2 +- io_uring/dma-buf.h | 4 ++-- io_uring/filetable.h | 2 +- io_uring/fs.c | 25 +++++++++------------- io_uring/futex.c | 4 ++-- io_uring/io_uring.c | 41 +++++++++++++++++------------------ io_uring/io_uring.h | 8 +++---- io_uring/kbuf.c | 31 +++++++++++++++------------ io_uring/kbuf.h | 4 ++-- io_uring/msg_ring.c | 4 ++-- io_uring/net.c | 46 ++++++++++++++++++++-------------------- io_uring/nop.c | 2 +- io_uring/openclose.c | 6 +++--- io_uring/poll.c | 20 +++++++++-------- io_uring/refs.h | 4 ++-- io_uring/rsrc.c | 10 ++++----- io_uring/rw.c | 28 ++++++++++++------------ io_uring/splice.c | 4 ++-- io_uring/statx.c | 3 +-- io_uring/sync.c | 6 +++--- io_uring/timeout.c | 8 +++---- io_uring/truncate.c | 2 +- io_uring/uring_cmd.c | 8 +++---- io_uring/xattr.c | 8 +++---- 26 files changed, 170 insertions(+), 149 deletions(-)
base-commit: ddcc50dd41d331dc66e0e5793492c3fa09941ff9
Signature · 1 line
2.56.1
PATCH 1/4 io_uring: return io_req_flags_t from io_slot_flags()
The flag bits (29 and 30) returned from io_slot_flags() happen to fit in a 32-bit unsigned int, but not all flag bits would. io_req_flags_t (an alias for u64) is the canonical flags type. It's already returned by io_file_get_flags(), so return it from io_slot_flags() for consistency.
--- io_uring/filetable.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
io_uring/filetable.h +1 -1
@@ -35,11 +35,11 @@ static inline void io_file_bitmap_set(struct io_file_table *table, int bit)#define FFS_NOWAIT 0x1UL#define FFS_ISREG 0x2UL#define FFS_MASK ~(FFS_NOWAIT|FFS_ISREG) -static inline unsigned int io_slot_flags (struct io_rsrc_node *node) +static inline io_req_flags_t io_slot_flags (struct io_rsrc_node *node) {return (node->file_ptr & ~FFS_MASK ) <<REQ_F_SUPPORT_NOWAIT_BIT ; }
base-commit: ddcc50dd41d331dc66e0e5793492c3fa09941ff9
Signature · 1 line
2.56.1
PATCH 2/4 io_uring: add multishot_flags constant in io_wq_submit_work()
In reply to Caleb Sander MateosREQ_F_MULTISHOT|REQ_F_APOLL_MULTISHOT is repeated twice in io_wq_submit_work(). Introduce a constant variable multishot_flags for it to reduce the repetition.
--- io_uring/io_uring.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
io_uring/io_uring.c +3 -2
@@ -1617,10 +1617,11 @@ void io_wq_free_work(struct io_wq_work *work)void io_wq_submit_work (struct io_wq_work *work) { struct io_kiocb *req = container_of(work, struct io_kiocb, work);const struct io_issue_def *def = &io_issue_defs[req->opcode];unsigned int issue_flags =IO_URING_F_UNLOCKED |IO_URING_F_IOWQ ; +io_req_flags_t multishot_flags =REQ_F_MULTISHOT |REQ_F_APOLL_MULTISHOT ;bool needs_poll = false;int ret =0 , err = -ECANCELED ;/* one will be dropped by io_wq_put_work() after returning to io-wq */ if (!(req->flags &REQ_F_REFCOUNT ))
@@ -1647,22 +1648,22 @@ void io_wq_submit_work(struct io_wq_work *work)* right context, however this is not the case of auxiliary CQEs, * which is the main mean of operation for multishot requests. * Don't allow any multishot execution from io-wq. It's more restrictive * than necessary and also cleaner. */ -if (req->flags & (REQ_F_MULTISHOT |REQ_F_APOLL_MULTISHOT )) { +if (req->flags & multishot_flags) { err = -EBADFD ;if (!io_file_can_poll (req)) goto fail;if (req->file->f_flags &O_NONBLOCK || req->file->f_mode &FMODE_NOWAIT ) { err = -ECANCELED ;if (io_arm_poll_handler (req, issue_flags) !=IO_APOLL_OK ) goto fail;return ; }else { - req->flags &= ~(REQ_F_APOLL_MULTISHOT |REQ_F_MULTISHOT ); + req->flags &= ~multishot_flags; } }if (req->flags &REQ_F_FORCE_ASYNC ) {bool opcode_poll = def->pollin || def->pollout;
Signature · 1 line
2.56.1
PATCH 3/4 io_uring: add io_req_{set,clear,switch}_flags()
In reply to Caleb Sander Mateosclang likes to set/clear flag bits using the narrowest possible or/and instructions (e.g. an 8-bit or if all flag bits are in the same byte). This causes store-to-load forwarding failures on subsequent wider loads of req->flags.
Introduce helpers io_req_set_flags() and io_req_clear_flags() to set and clear bits in req->flags. Use a volatile write to prevent clang from emitting a narrower store to part of req->flags. The downside of volatile writes is that consecutive req->flags updates must store to req->flags multiple times. So provide io_req_switch_flags() as well, which can both set and clear flags in a single volatile write.
--- include/linux/io_uring.h | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
include/linux/io_uring.h +25 -0
@@ -5,10 +5,35 @@#include <linux/io_uring_types.h> #include <linux/sched.h> #include <linux/xarray.h> #include <uapi/linux/io_uring.h> +/* +* clang likes to set/clear flag bits using the narrowest possible or/and +* instructions (e.g. an 8-bit or if all flag bits are in the same byte). +* This causes store-to-load forwarding failures on subsequent wider loads +* of req->flags. +* Store to req->flags using volatile writes to prevent this "optimization". +*/ +static inline void io_req_switch_flags (struct io_kiocb *req, +io_req_flags_t clear_flags, +io_req_flags_t set_flags) +{ +WRITE_ONCE (req->flags, (req->flags | set_flags) & ~clear_flags); +} + +static inline void io_req_set_flags (struct io_kiocb *req,io_req_flags_t flags) +{ +io_req_switch_flags (req,0 , flags); +} + +static inline void io_req_clear_flags (struct io_kiocb *req, +io_req_flags_t flags) +{ +io_req_switch_flags (req, flags,0 ); +} +static inline void req_set_fail (struct io_kiocb *req) { req->flags |=REQ_F_FAIL ;if (req->flags &REQ_F_CQE_SKIP ) { req->flags &= ~REQ_F_CQE_SKIP ;
Signature · 1 line
2.56.1
PATCH 4/4 io_uring: use io_req_{set,clear,switch}_flags()
In reply to Caleb Sander MateosReplace manual req->flags |= flags and req->flags &= ~flags with calls to io_req_set_flags() and io_req_clear_flags(), respectively. This ensures the flags field is written with a 64-bit store to prevent store-to-load forwarding failures.
Since io_req_set_flags() and io_req_clear_flags() store to req->flags with volatile writes, consecutive req->flags updates won't be able to be combined by the compiler. So combine setting/clearing multiple flags into a single io_req_set_flags(), io_req_clear_flags(), or io_req_switch_flags() call where possible.
--- include/linux/io_uring.h | 10 ++++----- io_uring/advise.c | 4 ++-- io_uring/cancel.c | 2 +- io_uring/dma-buf.h | 4 ++-- io_uring/fs.c | 25 +++++++++------------- io_uring/futex.c | 4 ++-- io_uring/io_uring.c | 38 ++++++++++++++++----------------- io_uring/io_uring.h | 8 +++---- io_uring/kbuf.c | 31 +++++++++++++++------------ io_uring/kbuf.h | 4 ++-- io_uring/msg_ring.c | 4 ++-- io_uring/net.c | 46 ++++++++++++++++++++-------------------- io_uring/nop.c | 2 +- io_uring/openclose.c | 6 +++--- io_uring/poll.c | 20 +++++++++-------- io_uring/refs.h | 4 ++-- io_uring/rsrc.c | 10 ++++----- io_uring/rw.c | 28 ++++++++++++------------ io_uring/splice.c | 4 ++-- io_uring/statx.c | 3 +-- io_uring/sync.c | 6 +++--- io_uring/timeout.c | 8 +++---- io_uring/truncate.c | 2 +- io_uring/uring_cmd.c | 8 +++---- io_uring/xattr.c | 8 +++---- 25 files changed, 142 insertions(+), 147 deletions(-)
include/linux/io_uring.h +5 -5
@@ -32,15 +32,15 @@ static inline void io_req_clear_flags(struct io_kiocb *req,io_req_switch_flags (req, flags,0 ); }static inline void req_set_fail (struct io_kiocb *req) { - req->flags |=REQ_F_FAIL ; -if (req->flags &REQ_F_CQE_SKIP ) { - req->flags &= ~REQ_F_CQE_SKIP ; - req->flags |=REQ_F_SKIP_LINK_CQES ; - } +io_req_flags_t flags =REQ_F_FAIL ; + +if (req->flags &REQ_F_CQE_SKIP ) + flags |=REQ_F_SKIP_LINK_CQES ; +io_req_switch_flags (req,REQ_F_CQE_SKIP , flags); }static inline void io_req_set_res (struct io_kiocb *req,s32 res,u32 cflags) { req->cqe.res = res;
io_uring/advise.c +2 -2
@@ -39,11 +39,11 @@ int io_madvise_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) ma->addr =READ_ONCE (sqe->addr); ma->len =READ_ONCE (sqe->off);if (!ma->len) ma->len =READ_ONCE (sqe->len); ma->advice =READ_ONCE (sqe->fadvise_advice); - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_FORCE_ASYNC );return 0 ;#else return -EOPNOTSUPP ;#endif }
@@ -87,11 +87,11 @@ int io_fadvise_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) fa->len =READ_ONCE (sqe->addr);if (!fa->len) fa->len =READ_ONCE (sqe->len); fa->advice =READ_ONCE (sqe->fadvise_advice);if (io_fadvise_force_async (fa)) - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_FORCE_ASYNC );return 0 ; }int io_fadvise (struct io_kiocb *req,unsigned int issue_flags) {
io_uring/cancel.c +1 -1
@@ -245,11 +245,11 @@ int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags)int ret;if (cd.flags &IORING_ASYNC_CANCEL_FD ) {if (req->flags &REQ_F_FIXED_FILE || cd.flags &IORING_ASYNC_CANCEL_FD_FIXED ) { - req->flags |=REQ_F_FIXED_FILE ; +io_req_set_flags (req,REQ_F_FIXED_FILE ); req->file =io_file_get_fixed (req, cancel->fd, issue_flags); }else { req->file =io_file_get_normal (req, cancel->fd); }
io_uring/dma-buf.h +2 -2
@@ -1,11 +1,11 @@// SPDX-License-Identifier: GPL-2.0 #ifndef IOU_DMA_BUF_H #define IOU_DMA_BUF_H #include <linux/dma-buf-io.h> -#include <linux/io_uring_types.h> +#include <linux/io_uring.h> #include "rsrc.h" struct io_buf_dma { struct dma_buf_io_ctx *ctx;
@@ -57,11 +57,11 @@ static inline int io_import_dmabuf_map(struct io_kiocb *req,if (!IS_ENABLED (CONFIG_DMA_SHARED_BUFFER ))return -EOPNOTSUPP ;if (req->file != bd->target_file)return -EBADF ; - req->flags |=REQ_F_DMABUF ; +io_req_set_flags (req,REQ_F_DMABUF );iov_iter_dmabuf_map (iter, ddir,NULL , offset, len);return 0 ; } static inline int io_register_dmabuf(struct io_ring_ctx *ctx,
io_uring/fs.c +10 -15
@@ -72,12 +72,11 @@ int io_renameat_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)if (unlikely (err)) {dismiss_delayed_filename (&ren->oldpath);return err; } - req->flags |=REQ_F_NEED_CLEANUP ; - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP |REQ_F_FORCE_ASYNC );return 0 ; }int io_renameat (struct io_kiocb *req,unsigned int issue_flags) {
@@ -89,11 +88,11 @@ int io_renameat(struct io_kiocb *req, unsigned int issue_flags)WARN_ON_ONCE (issue_flags &IO_URING_F_NONBLOCK ); ret =filename_renameat2 (ren->old_dfd, old, ren->new_dfd, new, ren->flags); - req->flags &= ~REQ_F_NEED_CLEANUP ; +io_req_clear_flags (req,REQ_F_NEED_CLEANUP );io_req_set_res (req, ret,0 );return IOU_COMPLETE ; }void io_renameat_cleanup (struct io_kiocb *req)
@@ -125,12 +124,11 @@ int io_unlinkat_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) fname =u64_to_user_ptr (READ_ONCE (sqe->addr)); err =delayed_getname (&un->filename, fname);if (unlikely (err))return err; - req->flags |=REQ_F_NEED_CLEANUP ; - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP |REQ_F_FORCE_ASYNC );return 0 ; }int io_unlinkat (struct io_kiocb *req,unsigned int issue_flags) {
@@ -143,11 +141,11 @@ int io_unlinkat(struct io_kiocb *req, unsigned int issue_flags)if (un->flags &AT_REMOVEDIR ) ret =filename_rmdir (un->dfd, name);else ret =filename_unlinkat (un->dfd, name); - req->flags &= ~REQ_F_NEED_CLEANUP ; +io_req_clear_flags (req,REQ_F_NEED_CLEANUP );io_req_set_res (req, ret,0 );return IOU_COMPLETE ; }void io_unlinkat_cleanup (struct io_kiocb *req)
@@ -174,12 +172,11 @@ int io_mkdirat_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) fname =u64_to_user_ptr (READ_ONCE (sqe->addr)); err =delayed_getname (&mkd->filename, fname);if (unlikely (err))return err; - req->flags |=REQ_F_NEED_CLEANUP ; - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP |REQ_F_FORCE_ASYNC );return 0 ; }int io_mkdirat (struct io_kiocb *req,unsigned int issue_flags) {
@@ -189,11 +186,11 @@ int io_mkdirat(struct io_kiocb *req, unsigned int issue_flags)WARN_ON_ONCE (issue_flags &IO_URING_F_NONBLOCK ); ret =filename_mkdirat (mkd->dfd, name, mkd->mode); - req->flags &= ~REQ_F_NEED_CLEANUP ; +io_req_clear_flags (req,REQ_F_NEED_CLEANUP );io_req_set_res (req, ret,0 );return IOU_COMPLETE ; }void io_mkdirat_cleanup (struct io_kiocb *req)
@@ -226,12 +223,11 @@ int io_symlinkat_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)if (unlikely (err)) {dismiss_delayed_filename (&sl->oldpath);return err; } - req->flags |=REQ_F_NEED_CLEANUP ; - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP |REQ_F_FORCE_ASYNC );return 0 ; }int io_symlinkat (struct io_kiocb *req,unsigned int issue_flags) {
@@ -242,11 +238,11 @@ int io_symlinkat(struct io_kiocb *req, unsigned int issue_flags)WARN_ON_ONCE (issue_flags &IO_URING_F_NONBLOCK ); ret =filename_symlinkat (old, sl->new_dfd, new); - req->flags &= ~REQ_F_NEED_CLEANUP ; +io_req_clear_flags (req,REQ_F_NEED_CLEANUP );io_req_set_res (req, ret,0 );return IOU_COMPLETE ; }int io_linkat_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe)
@@ -274,12 +270,11 @@ int io_linkat_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)if (unlikely (err)) {dismiss_delayed_filename (&lnk->oldpath);return err; } - req->flags |=REQ_F_NEED_CLEANUP ; - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP |REQ_F_FORCE_ASYNC );return 0 ; }int io_linkat (struct io_kiocb *req,unsigned int issue_flags) {
@@ -290,11 +285,11 @@ int io_linkat(struct io_kiocb *req, unsigned int issue_flags)WARN_ON_ONCE (issue_flags &IO_URING_F_NONBLOCK ); ret =filename_linkat (lnk->old_dfd, old, lnk->new_dfd, new, lnk->flags); - req->flags &= ~REQ_F_NEED_CLEANUP ; +io_req_clear_flags (req,REQ_F_NEED_CLEANUP );io_req_set_res (req, ret,0 );return IOU_COMPLETE ; }void io_link_cleanup (struct io_kiocb *req)
io_uring/futex.c +2 -2
@@ -240,11 +240,11 @@ int io_futexv_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)break ; } } iof->futexv_unqueued =0 ; - req->flags |=REQ_F_ASYNC_DATA ; +io_req_set_flags (req,REQ_F_ASYNC_DATA ); req->async_data = ifd;return 0 ; }static void io_futex_wake_fn (struct wake_q_head *wake_q,struct futex_q *q)
@@ -331,11 +331,11 @@ int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags)if (!io_cache_alloc (&ctx->futex_cache,GFP_NOWAIT , (void **)&ifd)) { ret = -ENOMEM ; goto done_unlock; } - req->flags |=REQ_F_ASYNC_DATA ; +io_req_set_flags (req,REQ_F_ASYNC_DATA ); req->async_data = ifd; ifd->q = futex_q_init; ifd->q.bitset = iof->futex_mask; ifd->q.wake = io_futex_wake_fn; ifd->req = req;
io_uring/io_uring.c +18 -20
@@ -326,33 +326,32 @@ static void io_clean_op(struct io_kiocb *req)put_cred (req->creds);if (req->flags &REQ_F_ASYNC_DATA ) {kfree (req->async_data); req->async_data =NULL ; } - req->flags &= ~IO_REQ_CLEAN_FLAGS ; +io_req_clear_flags (req,IO_REQ_CLEAN_FLAGS ); }/* * Mark the request as inflight, so that file cancelation will find it. * Can be used if the file is an io_uring instance, or if the request itself * relies on ->mm being alive for the duration of the request. */ inline void io_req_track_inflight (struct io_kiocb *req) {if (!(req->flags &REQ_F_INFLIGHT )) { - req->flags |=REQ_F_INFLIGHT ; +io_req_set_flags (req,REQ_F_INFLIGHT );atomic_inc (&req->tctx->inflight_tracked); } }static struct io_kiocb *__io_prep_linked_timeout (struct io_kiocb *req) {if (WARN_ON_ONCE (!req->link))return NULL ; - req->flags &= ~REQ_F_ARM_LTIMEOUT ; - req->flags |=REQ_F_LINK_TIMEOUT ; +io_req_switch_flags (req,REQ_F_ARM_LTIMEOUT ,REQ_F_LINK_TIMEOUT );/* linked timeouts should have two refs once prep'ed */ io_req_set_refcount (req);__io_req_set_refcount (req->link,2 );return req->link;
@@ -361,21 +360,21 @@ static struct io_kiocb *__io_prep_linked_timeout(struct io_kiocb *req)static void io_prep_async_work (struct io_kiocb *req) {const struct io_issue_def *def = &io_issue_defs[req->opcode];if (!(req->flags &REQ_F_CREDS )) { - req->flags |=REQ_F_CREDS ; +io_req_set_flags (req,REQ_F_CREDS ); req->creds =get_current_cred (); } req->work.list.next =NULL ;atomic_set (&req->work.flags,0 );if (req->flags &REQ_F_FORCE_ASYNC )atomic_or (IO_WQ_WORK_CONCURRENT , &req->work.flags);if (req->file && !(req->flags &REQ_F_FIXED_FILE )) - req->flags |=io_file_get_flags (req->file); +io_req_set_flags (req,io_file_get_flags (req->file));if (req->file && (req->flags &REQ_F_ISREG )) {bool should_hash = def->hash_reg_file;/* don't serialize this request if the fs doesn't need it */
@@ -1035,11 +1034,11 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags)/* * Request not freed here because we know it's called from io-wq only, * which holds a reference. Hence it can't be the last put. The CQE * has been posted, last put frees it. */ - req->flags |=REQ_F_CQE_SKIP ; +io_req_set_flags (req,REQ_F_CQE_SKIP );req_ref_put (req); } void io_req_defer_failed(struct io_kiocb *req, s32 res) __must_hold(&ctx->uring_lock)
@@ -1088,11 +1087,11 @@ __cold bool __io_alloc_req_refill(struct io_ring_ctx *ctx) }__cold void io_free_req (struct io_kiocb *req) {/* refs were already put, restore them for io_req_task_complete() */ - req->flags &= ~REQ_F_REFCOUNT ; +io_req_clear_flags (req,REQ_F_REFCOUNT ); req->io_task_work.func = io_req_task_complete;io_req_task_work_add (req); }static void __io_req_find_next_prep (struct io_kiocb *req)
@@ -1185,28 +1184,28 @@ static void io_free_batch_list(struct io_ring_ctx *ctx, comp_list);if (unlikely (req->flags &IO_REQ_CLEAN_SLOW_FLAGS )) {if (req->flags &REQ_F_REISSUE ) { node = req->comp_list.next; - req->flags &= ~REQ_F_REISSUE ; +io_req_clear_flags (req,REQ_F_REISSUE );io_queue_iowq (req);continue ; }if (req->flags &REQ_F_REFCOUNT ) { node = req->comp_list.next;/* CQE posted, the last put only frees */ - req->flags |=REQ_F_CQE_SKIP ; +io_req_set_flags (req,REQ_F_CQE_SKIP );if (!req_ref_put_and_test (req))continue ; }if ((req->flags &REQ_F_POLLED ) && req->apoll) {struct async_poll *apoll = req->apoll;if (apoll->double_poll)kfree (apoll->double_poll);io_cache_free (&ctx->apoll_cache, apoll); - req->flags &= ~REQ_F_POLLED ; +io_req_clear_flags (req,REQ_F_POLLED ); }if (req->flags &IO_REQ_LINK_FLAGS )io_queue_next (req);if (unlikely (req->flags &IO_REQ_CLEAN_FLAGS ))io_clean_op (req);
@@ -1659,11 +1658,11 @@ void io_wq_submit_work(struct io_wq_work *work) err = -ECANCELED ;if (io_arm_poll_handler (req, issue_flags) !=IO_APOLL_OK ) goto fail;return ; }else { - req->flags &= ~multishot_flags; +io_req_clear_flags (req, multishot_flags); } }if (req->flags &REQ_F_FORCE_ASYNC ) {bool opcode_poll = def->pollin || def->pollout;
@@ -1724,11 +1723,11 @@ inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd,io_ring_submit_lock (ctx, issue_flags); node =io_rsrc_node_lookup (&ctx->file_table.data, fd);if (node) { node->refs++; req->file_node = node; - req->flags |=io_slot_flags (node); +io_req_set_flags (req,io_slot_flags (node)); file =io_slot_file (node); }io_ring_submit_unlock (ctx, issue_flags);return file; }
@@ -1749,11 +1748,11 @@ static int io_req_sqe_copy(struct io_kiocb *req, unsigned int issue_flags) {const struct io_cold_def *def = &io_cold_defs[req->opcode];if (req->flags &REQ_F_SQE_COPIED )return 0 ; - req->flags |=REQ_F_SQE_COPIED ; +io_req_set_flags (req,REQ_F_SQE_COPIED );if (!def->sqe_copy)return 0 ;if (WARN_ON_ONCE (!(issue_flags &IO_URING_F_INLINE )))return -EFAULT ; def->sqe_copy (req);
@@ -1808,12 +1807,11 @@ static void io_queue_sqe_fallback(struct io_kiocb *req)if (unlikely (req->flags &REQ_F_FAIL )) {/* * We don't submit, fail them all, for that replace hardlinks * with normal links. Extra REQ_F_LINK is tolerated. */ - req->flags &= ~REQ_F_HARDLINK ; - req->flags |=REQ_F_LINK ; +io_req_switch_flags (req,REQ_F_HARDLINK ,REQ_F_LINK );io_req_defer_failed (req, req->cqe.res); }else {/* can't fail with IO_URING_F_INLINE */ io_req_sqe_copy (req,IO_URING_F_INLINE );if (unlikely (req->ctx->int_flags &IO_RING_F_DRAIN_ACTIVE ))
@@ -1859,11 +1857,11 @@ static void io_init_drain(struct io_ring_ctx *ctx)* the head request and the next request/link after the current * link. Considering sequential execution of links, * REQ_F_IO_DRAIN will be maintained for every request of our * link. */ - head->flags |=REQ_F_IO_DRAIN |REQ_F_FORCE_ASYNC ; +io_req_set_flags (head,REQ_F_IO_DRAIN |REQ_F_FORCE_ASYNC ); ctx->int_flags |=IO_RING_F_DRAIN_NEXT ; } }static __cold int io_init_fail_req (struct io_kiocb *req,int err)
@@ -1947,16 +1945,16 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req,if (unlikely (ctx->int_flags & (IO_RING_F_OP_RESTRICTED |IO_RING_F_DRAIN_ACTIVE |IO_RING_F_DRAIN_NEXT ))) {if (!io_check_restriction (ctx, req, sqe_flags))return io_init_fail_req (req, -EACCES );/* knock it to the slow queue path, will be drained there */ if (ctx->int_flags &IO_RING_F_DRAIN_ACTIVE ) - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_FORCE_ASYNC );/* if there is no link, we're at "next" request and need to drain */ if (unlikely (ctx->int_flags &IO_RING_F_DRAIN_NEXT ) && !ctx->submit_state.link.head) { ctx->int_flags &= ~IO_RING_F_DRAIN_NEXT ; ctx->int_flags |=IO_RING_F_DRAIN_ACTIVE ; - req->flags |=REQ_F_IO_DRAIN |REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_IO_DRAIN |REQ_F_FORCE_ASYNC ); } }if (!def->ioprio && sqe->ioprio)return io_init_fail_req (req, -EINVAL );
@@ -1990,11 +1988,11 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, ret =security_uring_override_creds (req->creds);if (ret) {put_cred (req->creds);return io_init_fail_req (req, ret); } - req->flags |=REQ_F_CREDS ; +io_req_set_flags (req,REQ_F_CREDS ); }return def->prep (req, sqe); }
io_uring/io_uring.h +4 -4
@@ -343,22 +343,22 @@ io_uring_alloc_cached_async_data(struct io_alloc_cache *cache, struct io_kiocb *req) {if (!io_cache_alloc (cache,GFP_KERNEL , &req->async_data))return false; - req->flags |=REQ_F_ASYNC_DATA ; +io_req_set_flags (req,REQ_F_ASYNC_DATA );return true; }static inline void *io_uring_alloc_async_data (struct io_kiocb *req) {const struct io_issue_def *def = &io_issue_defs[req->opcode];WARN_ON_ONCE (!def->async_size); req->async_data =kmalloc (def->async_size,GFP_KERNEL );if (req->async_data) - req->flags |=REQ_F_ASYNC_DATA ; +io_req_set_flags (req,REQ_F_ASYNC_DATA );return req->async_data; }static inline bool req_has_async_data (struct io_kiocb *req) {
@@ -366,11 +366,11 @@ static inline bool req_has_async_data(struct io_kiocb *req) }static inline void io_req_async_data_clear (struct io_kiocb *req,io_req_flags_t extra_flags) { - req->flags &= ~(REQ_F_ASYNC_DATA |extra_flags); +io_req_clear_flags (req,REQ_F_ASYNC_DATA | extra_flags); req->async_data =NULL ; }static inline void io_req_async_data_free (struct io_kiocb *req) {
@@ -569,11 +569,11 @@ static inline void io_req_queue_tw_complete(struct io_kiocb *req, s32 res)static inline bool io_file_can_poll (struct io_kiocb *req) {if (req->flags &REQ_F_CAN_POLL )return true;if (req->file &&file_can_poll (req->file)) { - req->flags |=REQ_F_CAN_POLL ; +io_req_set_flags (req,REQ_F_CAN_POLL );return true; }return false; }
io_uring/kbuf.c +17 -14
@@ -64,11 +64,11 @@ bool io_kbuf_commit(struct io_kiocb *req, struct io_buffer_list *bl, int len, int nr) { if (unlikely(!(req->flags & REQ_F_BUFFERS_COMMIT))) return true; - req->flags &= ~REQ_F_BUFFERS_COMMIT ; +io_req_clear_flags (req,REQ_F_BUFFERS_COMMIT );if (unlikely (len <0 ))return true;if (bl->flags &IOBL_INC )return io_kbuf_inc_commit (bl, len);
@@ -99,11 +99,11 @@ static int io_buffer_add_list(struct io_ring_ctx *ctx,void io_kbuf_drop_legacy (struct io_kiocb *req) {if (WARN_ON_ONCE (!(req->flags &REQ_F_BUFFER_SELECTED )))return ; - req->flags &= ~REQ_F_BUFFER_SELECTED ; +io_req_clear_flags (req,REQ_F_BUFFER_SELECTED );kfree (req->kbuf); req->kbuf =NULL ; }bool io_kbuf_recycle_legacy (struct io_kiocb *req,unsigned issue_flags)
@@ -124,31 +124,32 @@ bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags)list_add (&buf->list, &bl->buf_list); bl->nbufs++; }else {kfree (buf); } - req->flags &= ~REQ_F_BUFFER_SELECTED ; +io_req_clear_flags (req,REQ_F_BUFFER_SELECTED ); req->kbuf =NULL ;io_ring_submit_unlock (ctx, issue_flags);return true; }static void __user *io_provided_buffer_select (struct io_kiocb *req,size_t *len,struct io_buffer_list *bl) {if (!list_empty (&bl->buf_list)) { +io_req_flags_t flags =REQ_F_BUFFER_SELECTED ;struct io_buffer *kbuf; kbuf = list_first_entry(&bl->buf_list, struct io_buffer, list);list_del (&kbuf->list); bl->nbufs--;if (*len ==0 || *len > kbuf->len) *len = kbuf->len;if (list_empty (&bl->buf_list)) - req->flags |=REQ_F_BL_EMPTY ; - req->flags |=REQ_F_BUFFER_SELECTED ; + flags |=REQ_F_BL_EMPTY ; +io_req_set_flags (req, flags); req->kbuf = kbuf; req->buf_index = kbuf->bid;return u64_to_user_ptr (kbuf->addr); }return NULL ;
@@ -203,28 +204,28 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, tail =smp_load_acquire (&br->tail);if (unlikely (tail == head))return sel;if ((__u16)(head +1 ) == tail) - req->flags |=REQ_F_BL_EMPTY ; +io_req_set_flags (req,REQ_F_BL_EMPTY ); buf =io_ring_head_to_buf (br, head, bl->mask); buf_len =READ_ONCE (buf->len);if (*len ==0 || *len > buf_len) *len = buf_len; sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr));if (unlikely (!access_ok (sel.addr, *len))) { sel.addr =NULL ;return sel; } - req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; +io_req_set_flags (req,REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ); req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl;if (io_should_commit (req, issue_flags)) {if (!io_kbuf_commit (req, sel.buf_list, *len,1 )) - req->flags |=REQ_F_BUF_MORE ; +io_req_set_flags (req,REQ_F_BUF_MORE ); sel.buf_list =NULL ; }return sel; }
@@ -253,10 +254,11 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,static int io_ring_buffers_peek (struct io_kiocb *req,struct buf_sel_arg *arg,struct io_buffer_list *bl) {struct io_uring_buf_ring *br = bl->buf_ring; +io_req_flags_t flags =REQ_F_BUFFER_RING ;struct iovec *org_iovs = arg->iovs;struct iovec *iov = arg->iovs;int nr_iovs = arg->nr_iovs;__u16 nr_avail, tail, head;struct io_uring_buf *buf;
@@ -336,13 +338,13 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg,if (arg->iovs != org_iovs && (arg->mode &KBUF_MODE_FREE ))kfree (org_iovs);if (head == tail) - req->flags |=REQ_F_BL_EMPTY ; + flags |=REQ_F_BL_EMPTY ; - req->flags |=REQ_F_BUFFER_RING ; +io_req_set_flags (req, flags);return iov - arg->iovs; }int io_buffers_select (struct io_kiocb *req,struct buf_sel_arg *arg,struct io_br_sel *sel,unsigned int issue_flags)
@@ -363,13 +365,14 @@ int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg,* buffers for a send/write operation would happen on the app * side anyway with normal buffers. Besides, we already * committed them, they cannot be put back in the queue. */ if (ret >0 ) { - req->flags |=REQ_F_BUFFERS_COMMIT |REQ_F_BL_NO_RECYCLE ; +io_req_set_flags (req,REQ_F_BUFFERS_COMMIT | +REQ_F_BL_NO_RECYCLE );if (!io_kbuf_commit (req, sel->buf_list, arg->out_len, ret)) - req->flags |=REQ_F_BUF_MORE ; +io_req_set_flags (req,REQ_F_BUF_MORE ); } }else { ret =io_provided_buffers_select (req, &arg->out_len, sel->buf_list, arg->iovs); } out_unlock:
@@ -394,11 +397,11 @@ int io_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg,return -ENOENT ;if (bl->flags &IOBL_BUF_RING ) { ret =io_ring_buffers_peek (req, arg, bl);if (ret >0 ) - req->flags |=REQ_F_BUFFERS_COMMIT ; +io_req_set_flags (req,REQ_F_BUFFERS_COMMIT ); sel->buf_list = bl;return ret; }/* don't support multiple buffer selections for legacy */
@@ -414,11 +417,11 @@ static inline bool __io_put_kbuf_ring(struct io_kiocb *req,if (bl) ret =io_kbuf_commit (req, bl, len, nr);if (ret && (req->flags &REQ_F_BUF_MORE )) ret = false; - req->flags &= ~(REQ_F_BUFFER_RING |REQ_F_BUF_MORE ); +io_req_clear_flags (req,REQ_F_BUFFER_RING |REQ_F_BUF_MORE );return ret; }unsigned int __io_put_kbufs (struct io_kiocb *req,struct io_buffer_list *bl,int len,int nbufs)
io_uring/kbuf.h +2 -2
@@ -1,11 +1,11 @@// SPDX-License-Identifier: GPL-2.0 #ifndef IOU_KBUF_H #define IOU_KBUF_H #include <uapi/linux/io_uring.h> -#include <linux/io_uring_types.h> +#include <linux/io_uring.h> enum {/* ring mapped provided buffers */ IOBL_BUF_RING =1 , /* buffers are consumed incrementally rather than always fully */
@@ -96,11 +96,11 @@ struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx,static inline bool io_kbuf_recycle_ring (struct io_kiocb *req,struct io_buffer_list *bl) {if (bl) { - req->flags &= ~(REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ); +io_req_clear_flags (req,REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT );return true; }return false; }
io_uring/msg_ring.c +2 -2
@@ -184,11 +184,11 @@ static int io_msg_grab_file(struct io_kiocb *req, unsigned int issue_flags) node =io_rsrc_node_lookup (&ctx->file_table.data, msg->src_fd);if (node) { msg->src_file =io_slot_file (node);if (msg->src_file)get_file (msg->src_file); - req->flags |=REQ_F_NEED_CLEANUP ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP ); ret =0 ; }io_ring_submit_unlock (ctx, issue_flags);return ret; }
@@ -206,11 +206,11 @@ static int io_msg_install_complete(struct io_kiocb *req, unsigned int issue_flag ret =__io_fixed_fd_install (target_ctx, src_file, msg->dst_fd);if (ret <0 ) goto out_unlock; msg->src_file =NULL ; - req->flags &= ~REQ_F_NEED_CLEANUP ; +io_req_clear_flags (req,REQ_F_NEED_CLEANUP );if (msg->flags &IORING_MSG_RING_CQE_SKIP ) goto out_unlock;/* * If this fails, the target still received the file descriptor but
io_uring/net.c +23 -23
253 lines not on this page · raw message
io_uring/nop.c +1 -1
@@ -39,11 +39,11 @@ int io_nop_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)if (nop->flags &IORING_NOP_FILE ) nop->fd =READ_ONCE (sqe->fd);else nop->fd =-1 ;if (nop->flags &IORING_NOP_FIXED_FILE ) - req->flags |=REQ_F_FIXED_FILE ; +io_req_set_flags (req,REQ_F_FIXED_FILE );if (nop->flags &IORING_NOP_FIXED_BUFFER ) req->buf_index =READ_ONCE (sqe->buf_index);if (nop->flags &IORING_NOP_CQE32 ) {struct io_ring_ctx *ctx = req->ctx;
io_uring/openclose.c +3 -3
@@ -68,19 +68,19 @@ static int __io_openat_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe open->dfd =READ_ONCE (sqe->fd); fname =u64_to_user_ptr (READ_ONCE (sqe->addr)); ret =delayed_getname (&open->filename, fname);if (unlikely (ret))return ret; - req->flags |=REQ_F_NEED_CLEANUP ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP ); open->file_slot =READ_ONCE (sqe->file_index);if (open->file_slot && (open->how.flags &O_CLOEXEC ))return -EINVAL ; open->nofile =rlimit (RLIMIT_NOFILE );if (io_openat_force_async (open)) - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_FORCE_ASYNC );return 0 ; }void io_openat_bpf_populate (struct io_uring_bpf_ctx *bctx,struct io_kiocb *req) {
@@ -176,11 +176,11 @@ int io_openat2(struct io_kiocb *req, unsigned int issue_flags)fd_install (ret, file); else ret =io_fixed_fd_install (req, issue_flags, file, open->file_slot); err: - req->flags &= ~REQ_F_NEED_CLEANUP ; +io_req_clear_flags (req,REQ_F_NEED_CLEANUP );if (ret <0 )req_set_fail (req);io_req_set_res (req, ret,0 );return IOU_COMPLETE ; }
io_uring/poll.c +11 -9
@@ -446,13 +446,13 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync,/* optional, saves extra locking for removal in tw handler */ if (mask && poll->events &EPOLLONESHOT ) {io_poll_remove_waitq (poll);if (wqe_is_double (wait)) - req->flags &= ~REQ_F_DOUBLE_POLL ; +io_req_clear_flags (req,REQ_F_DOUBLE_POLL );else - req->flags &= ~REQ_F_SINGLE_POLL ; +io_req_clear_flags (req,REQ_F_SINGLE_POLL ); }__io_poll_execute (req, mask,IOU_F_TWQ_IN_WAKE ); }return 1 ; }
@@ -471,14 +471,16 @@ static bool io_poll_double_prepare(struct io_kiocb *req)* io_poll_wake(). There is only one poll entry queued, serialise with * it by taking its head lock. As we're still arming the tw hanlder * is not going to be run, so there are no races with it. */ if (head) { +io_req_flags_t flags =REQ_F_DOUBLE_POLL ; +spin_lock_irq (&head->lock); - req->flags |=REQ_F_DOUBLE_POLL ;if (req->opcode ==IORING_OP_POLL_ADD ) - req->flags |=REQ_F_ASYNC_DATA ; + flags |=REQ_F_ASYNC_DATA ; +io_req_set_flags (req, flags);spin_unlock_irq (&head->lock); }rcu_read_unlock ();return !!head; }
@@ -524,11 +526,11 @@ static void __io_queue_proc(struct io_poll *poll, struct io_poll_table *pt,return ; } *poll_ptr = poll; }else {/* fine to modify, there is no poll queued to race with us */ - req->flags |=REQ_F_SINGLE_POLL ; +io_req_set_flags (req,REQ_F_SINGLE_POLL ); } pt->nr_entries++; poll->head = head; poll->wait.private = (void *) wqe_private;
@@ -605,11 +607,11 @@ static int __io_arm_poll_handler(struct io_kiocb *req,* wake if someone does wait(events > 1). Ensure we don't do * lazy wake for those, as we need to process each one as they * come in. */ if (poll->events &EPOLLEXCLUSIVE ) - req->flags |=REQ_F_POLL_NO_LAZY ; +io_req_set_flags (req,REQ_F_POLL_NO_LAZY ); mask =vfs_poll (req->file, &ipt->pt) & poll->events;if (unlikely (ipt->error || !ipt->nr_entries)) {io_poll_remove_entries (req);
@@ -726,12 +728,12 @@ int io_arm_apoll(struct io_kiocb *req, unsigned issue_flags, __poll_t mask) mask |=EPOLLONESHOT ; apoll =io_req_alloc_apoll (req, issue_flags);if (!apoll)return IO_APOLL_ABORTED ; - req->flags &= ~(REQ_F_SINGLE_POLL |REQ_F_DOUBLE_POLL ); - req->flags |=REQ_F_POLLED ; +io_req_switch_flags (req,REQ_F_SINGLE_POLL |REQ_F_DOUBLE_POLL , +REQ_F_POLLED ); ipt.pt._qproc = io_async_queue_proc; ret =__io_arm_poll_handler (req, &apoll->poll, &ipt, mask, issue_flags);if (ret)return ret >0 ?IO_APOLL_READY :IO_APOLL_ABORTED ;
@@ -983,11 +985,11 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags)* the entries of the previous round like io_arm_apoll() does, a stale * second entry would be taken for a third wait queue. */ if (req_has_async_data (req))io_req_async_data_free (req); - req->flags &= ~(REQ_F_SINGLE_POLL |REQ_F_DOUBLE_POLL ); +io_req_clear_flags (req,REQ_F_SINGLE_POLL |REQ_F_DOUBLE_POLL ); ret =__io_arm_poll_handler (req, poll, &ipt, poll->events, issue_flags);if (ret >0 ) {io_req_set_res (req, ipt.result_mask,0 );return IOU_COMPLETE ;
io_uring/refs.h +2 -2
@@ -1,11 +1,11 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef IOU_REQ_REF_H #define IOU_REQ_REF_H #include <linux/atomic.h> -#include <linux/io_uring_types.h> +#include <linux/io_uring.h> /* * Shamelessly stolen from the mm implementation of page reference checking, * see commit f958d7b528b1 for details. */
@@ -49,11 +49,11 @@ static inline void req_ref_put(struct io_kiocb *req) }static inline void __io_req_set_refcount (struct io_kiocb *req,int nr) {if (!(req->flags &REQ_F_REFCOUNT )) { - req->flags |=REQ_F_REFCOUNT ; +io_req_set_flags (req,REQ_F_REFCOUNT );atomic_set (&req->refs, nr); } }/*
io_uring/rsrc.c +5 -5
@@ -1357,21 +1357,21 @@ inline struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,struct io_ring_ctx *ctx = req->ctx;struct io_rsrc_node *node;if (req->flags &REQ_F_BUF_NODE )return req->buf_node; - req->flags |=REQ_F_BUF_NODE ; +io_req_set_flags (req,REQ_F_BUF_NODE );io_ring_submit_lock (ctx, issue_flags); node =io_rsrc_node_lookup (&ctx->buf_table, req->buf_index);if (node) { node->refs++; req->buf_node = node;io_ring_submit_unlock (ctx, issue_flags);return node; } - req->flags &= ~REQ_F_BUF_NODE ; +io_req_clear_flags (req,REQ_F_BUF_NODE );io_ring_submit_unlock (ctx, issue_flags);return NULL ; } int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
@@ -1852,11 +1852,11 @@ int __io_import_reg_vec(int ddir, struct iov_iter *iter,memcpy (tmp_vec.iovec + iovec_off, iov,sizeof (*iov) * nr_iovs);io_vec_free (vec); *vec = tmp_vec; iov = vec->iovec + iovec_off; - req->flags |=REQ_F_NEED_CLEANUP ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP ); }if (imu->flags &IO_REGBUF_F_KBUF )return io_vec_fill_kern_bvec (ddir, iter, imu, iov, nr_iovs, vec);
@@ -1872,19 +1872,19 @@ int io_prep_reg_iovec(struct io_kiocb *req, struct iou_vec *iv,if (uvec_segs > iv->nr) { ret =io_vec_realloc (iv, uvec_segs);if (ret)return ret; - req->flags |=REQ_F_NEED_CLEANUP ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP ); }/* pad iovec to the right */ iovec_off = iv->nr - uvec_segs; iov = iv->iovec + iovec_off; res =iovec_from_user (uvec, uvec_segs, uvec_segs, iov,io_is_compat (req->ctx));if (IS_ERR (res))return PTR_ERR (res); - req->flags |=REQ_F_IMPORT_BUFFER ; +io_req_set_flags (req,REQ_F_IMPORT_BUFFER );return 0 ; }
io_uring/rw.c +14 -14
@@ -87,11 +87,11 @@ static int io_import_vec(int ddir, struct io_kiocb *req, ret =__import_iovec (ddir, uvec, uvec_segs, nr_segs, &iov, &io->iter,io_is_compat (req->ctx));if (unlikely (ret <0 ))return ret;if (iov) { - req->flags |=REQ_F_NEED_CLEANUP ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP );io_vec_reset_iovec (&io->vec, iov, io->iter.nr_segs); }return 0 ; }
@@ -178,11 +178,11 @@ static void io_req_rw_cleanup(struct io_kiocb *req, unsigned int issue_flags)* path should assume that a successful (or -EIOCBQUEUED) return can * mean that the underlying data can be gone at any time. But that * should be fixed separately, and then this check could be killed. */ if (!(req->flags & (REQ_F_REISSUE |REQ_F_REFCOUNT ))) { - req->flags &= ~REQ_F_NEED_CLEANUP ; +io_req_clear_flags (req,REQ_F_NEED_CLEANUP );if (!io_rw_recycle (req, issue_flags)) {struct io_async_rw *rw = req->async_data;io_vec_free (&rw->vec); }
@@ -196,11 +196,11 @@ static int io_rw_alloc_async(struct io_kiocb *req)if (!io_uring_alloc_cached_async_data (&ctx->rw_cache, req))return -ENOMEM ; rw = req->async_data;if (rw->vec.iovec) - req->flags |=REQ_F_NEED_CLEANUP ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP ); rw->bytes_done =0 ;return 0 ; }static inline void io_meta_save_state (struct io_async_rw *io)
@@ -237,11 +237,11 @@ static int io_prep_rw_pi(struct io_kiocb *req, struct io_rw *rw, int ddir, io->meta.seed = pi_attr.seed; ret =import_ubuf (ddir,u64_to_user_ptr (pi_attr.addr), pi_attr.len, &io->meta.iter);if (unlikely (ret <0 ))return ret; - req->flags |=REQ_F_HAS_METADATA ; +io_req_set_flags (req,REQ_F_HAS_METADATA );io_meta_save_state (io);return ret; } static int __io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe,
@@ -394,11 +394,11 @@ static int io_rw_import_reg_vec(struct io_kiocb *req, ret =io_import_reg_vec (ddir, &io->iter, req, &io->vec, uvec_segs, issue_flags);if (unlikely (ret))return ret;iov_iter_save_state (&io->iter, &io->iter_state); - req->flags &= ~REQ_F_IMPORT_BUFFER ; +io_req_clear_flags (req,REQ_F_IMPORT_BUFFER );return 0 ; }static int io_rw_prep_reg_vec (struct io_kiocb *req) {
@@ -448,11 +448,11 @@ int io_read_mshot_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)return ret;if (rw->addr || rw->len)return -EINVAL ; - req->flags |=REQ_F_APOLL_MULTISHOT ; +io_req_set_flags (req,REQ_F_APOLL_MULTISHOT );return 0 ; }void io_readv_writev_cleanup (struct io_kiocb *req) {
@@ -469,11 +469,11 @@ static inline loff_t *io_kiocb_update_pos(struct io_kiocb *req)if (rw->kiocb.ki_pos !=-1 )return &rw->kiocb.ki_pos;if (!(req->file->f_mode &FMODE_STREAM )) { - req->flags |=REQ_F_CUR_POS ; +io_req_set_flags (req,REQ_F_CUR_POS ); rw->kiocb.ki_pos = req->file->f_pos;return &rw->kiocb.ki_pos; } rw->kiocb.ki_pos =0 ;
@@ -542,11 +542,11 @@ static void io_req_io_end(struct io_kiocb *req)static void __io_complete_rw_common (struct io_kiocb *req,long res) {if (res == req->cqe.res)return ;if ((res == -EOPNOTSUPP || res == -EAGAIN ) &&io_rw_should_reissue (req)) { - req->flags |=REQ_F_REISSUE |REQ_F_BL_NO_RECYCLE ; +io_req_set_flags (req,REQ_F_REISSUE |REQ_F_BL_NO_RECYCLE ); }else {req_set_fail (req); req->cqe.res = res; } }
@@ -603,11 +603,11 @@ static void io_complete_rw_iopoll(struct kiocb *kiocb, long res)if (kiocb->ki_flags &IOCB_WRITE )io_req_end_write (req);if (res == -EAGAIN &&io_rw_should_reissue (req)) - req->flags |=REQ_F_REISSUE |REQ_F_BL_NO_RECYCLE ; +io_req_set_flags (req,REQ_F_REISSUE |REQ_F_BL_NO_RECYCLE );else if (unlikely (final_res != req->cqe.res)) req->cqe.res = final_res;/* order with io_iopoll_complete() checking ->iopoll_completed */ smp_store_release (&req->iopoll_completed,1 );
@@ -840,11 +840,11 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode, int rw_type)if (unlikely (!(file->f_mode & mode)))return -EBADF ;if (!(req->flags &REQ_F_FIXED_FILE )) - req->flags |=io_file_get_flags (file); +io_req_set_flags (req,io_file_get_flags (file)); kiocb->ki_flags = file->f_iocb_flags; ret =kiocb_set_rw_flags (kiocb, rw->flags, rw_type);if (unlikely (ret))return ret;
@@ -854,23 +854,23 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode, int rw_type)* supports async. Otherwise it's impossible to use O_NONBLOCK files * reliably. If not, or it IOCB_NOWAIT is set, don't retry. */ if (kiocb->ki_flags &IOCB_NOWAIT || ((file->f_flags &O_NONBLOCK && !(req->flags &REQ_F_SUPPORT_NOWAIT )))) - req->flags |=REQ_F_NOWAIT ; +io_req_set_flags (req,REQ_F_NOWAIT );if (ctx->flags &IORING_SETUP_IOPOLL ) {if (!(kiocb->ki_flags &IOCB_DIRECT ) || !file->f_op->iopoll)return -EOPNOTSUPP ; - req->flags |=REQ_F_IOPOLL ; +io_req_set_flags (req,REQ_F_IOPOLL ); kiocb->private =NULL ; kiocb->ki_flags |=IOCB_HIPRI ; req->iopoll_completed =0 ; req->cqe.flags =0 ;if (ctx->flags &IORING_SETUP_HYBRID_IOPOLL ) {/* make sure every req only blocks once*/ - req->flags &= ~REQ_F_IOPOLL_STATE ; +io_req_clear_flags (req,REQ_F_IOPOLL_STATE ); req->iopoll_start =ktime_get_ns (); } }else {if (kiocb->ki_flags &IOCB_HIPRI )return -EINVAL ;
@@ -1277,11 +1277,11 @@ static u64 io_hybrid_iopoll_delay(struct io_ring_ctx *ctx, struct io_kiocb *req)/* Using half the running time to do schedule */ sleep_time = ctx->hybrid_poll_time /2 ; kt =ktime_set (0 , sleep_time); - req->flags |=REQ_F_IOPOLL_STATE ; +io_req_set_flags (req,REQ_F_IOPOLL_STATE ); mode =HRTIMER_MODE_REL ;hrtimer_setup_sleeper_on_stack (&timer,CLOCK_MONOTONIC , mode);hrtimer_set_expires (&timer.timer, kt);set_current_state (TASK_INTERRUPTIBLE );
io_uring/splice.c +2 -2
@@ -35,11 +35,11 @@ static int __io_splice_prep(struct io_kiocb *req, sp->flags =READ_ONCE (sqe->splice_flags);if (unlikely (sp->flags & ~valid_flags))return -EINVAL ; sp->splice_fd_in =READ_ONCE (sqe->splice_fd_in); sp->rsrc_node =NULL ; - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_FORCE_ASYNC );return 0 ; }int io_tee_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
@@ -71,11 +71,11 @@ static struct file *io_splice_get_file(struct io_kiocb *req, node =io_rsrc_node_lookup (&ctx->file_table.data, sp->splice_fd_in);if (node) { node->refs++; sp->rsrc_node = node; file =io_slot_file (node); - req->flags |=REQ_F_NEED_CLEANUP ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP ); }io_ring_submit_unlock (ctx, issue_flags);return file; }
io_uring/statx.c +1 -2
@@ -40,12 +40,11 @@ int io_statx_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) ret =delayed_getname_uflags (&sx->filename, path, sx->flags);if (unlikely (ret))return ret; - req->flags |=REQ_F_NEED_CLEANUP ; - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP |REQ_F_FORCE_ASYNC );return 0 ; }int io_statx (struct io_kiocb *req,unsigned int issue_flags) {
io_uring/sync.c +3 -3
@@ -30,11 +30,11 @@ int io_sfr_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)return -EINVAL ; sync->off =READ_ONCE (sqe->off); sync->len =READ_ONCE (sqe->len); sync->flags =READ_ONCE (sqe->sync_range_flags); - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_FORCE_ASYNC );return 0 ; }int io_sync_file_range (struct io_kiocb *req,unsigned int issue_flags)
@@ -63,11 +63,11 @@ int io_fsync_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sync->off =READ_ONCE (sqe->off);if (sync->off <0 )return -EINVAL ; sync->len =READ_ONCE (sqe->len); - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_FORCE_ASYNC );return 0 ; }int io_fsync (struct io_kiocb *req,unsigned int issue_flags) {
@@ -92,11 +92,11 @@ int io_fallocate_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)return -EINVAL ; sync->off =READ_ONCE (sqe->off); sync->len =READ_ONCE (sqe->addr); sync->mode =READ_ONCE (sqe->len); - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_FORCE_ASYNC );return 0 ; }int io_fallocate (struct io_kiocb *req,unsigned int issue_flags) {
io_uring/timeout.c +4 -4
@@ -232,13 +232,13 @@ static void io_fail_links(struct io_kiocb *req)if (!link)return ;while (link) {if (ignore_cqes) - link->flags |=REQ_F_CQE_SKIP ; +io_req_set_flags (link,REQ_F_CQE_SKIP );else - link->flags &= ~REQ_F_CQE_SKIP ; +io_req_clear_flags (link,REQ_F_CQE_SKIP );trace_io_uring_fail_link (req, link); link = link->link; } link = req->link;
@@ -260,11 +260,11 @@ void io_disarm_next(struct io_kiocb *req) {struct io_kiocb *link =NULL ;if (req->flags &REQ_F_ARM_LTIMEOUT ) { link = req->link; - req->flags &= ~REQ_F_ARM_LTIMEOUT ; +io_req_clear_flags (req,REQ_F_ARM_LTIMEOUT );if (link && link->opcode ==IORING_OP_LINK_TIMEOUT ) {io_remove_next_linked (req);io_req_queue_tw_complete (link, -ECANCELED ); } }else if (req->flags &REQ_F_LINK_TIMEOUT ) {
@@ -658,11 +658,11 @@ static int __io_timeout_prep(struct io_kiocb *req,if (!link->head)return -EINVAL ;if (link->last->opcode ==IORING_OP_LINK_TIMEOUT )return -EINVAL ; timeout->head = link->last; - link->last->flags |=REQ_F_ARM_LTIMEOUT ; +io_req_set_flags (link->last,REQ_F_ARM_LTIMEOUT );hrtimer_setup (&data->timer, io_link_timeout_fn,io_timeout_get_clock (data), data->mode); }else {hrtimer_setup (&data->timer, io_timeout_fn,io_timeout_get_clock (data), data->mode); }
io_uring/truncate.c +1 -1
@@ -28,11 +28,11 @@ int io_ftruncate_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sqe->splice_fd_in || sqe->addr3)return -EINVAL ; ft->len =READ_ONCE (sqe->off); - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_FORCE_ASYNC );return 0 ; }int io_ftruncate (struct io_kiocb *req,unsigned int issue_flags) {
io_uring/uring_cmd.c +4 -4
@@ -192,11 +192,11 @@ int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)if (!io_uring_alloc_cached_async_data (&req->ctx->cmd_cache, req))return -ENOMEM ; ac = req->async_data;if (ac->vec.iovec) - req->flags |=REQ_F_NEED_CLEANUP ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP ); ioucmd->sqe = sqe;return 0 ; }/*
@@ -243,16 +243,16 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags)if (ctx->flags & (IORING_SETUP_CQE32 |IORING_SETUP_CQE_MIXED )) issue_flags |=IO_URING_F_CQE32 ;if (io_is_compat (ctx)) issue_flags |=IO_URING_F_COMPAT ;if (ctx->flags &IORING_SETUP_IOPOLL && file->f_op->uring_cmd_iopoll) { - req->flags |=REQ_F_IOPOLL ; +io_req_set_flags (req,REQ_F_IOPOLL ); issue_flags |=IO_URING_F_IOPOLL ; req->iopoll_completed =0 ;if (ctx->flags &IORING_SETUP_HYBRID_IOPOLL ) {/* make sure every req only blocks once */ - req->flags &= ~REQ_F_IOPOLL_STATE ; +io_req_clear_flags (req,REQ_F_IOPOLL_STATE ); req->iopoll_start =ktime_get_ns (); } } ret = file->f_op->uring_cmd (ioucmd, issue_flags);
@@ -319,11 +319,11 @@ int io_cmd_poll_multishot(struct io_uring_cmd *cmd,int ret;if (likely (req->flags &REQ_F_APOLL_MULTISHOT ))return 0 ; - req->flags |=REQ_F_APOLL_MULTISHOT ; +io_req_set_flags (req,REQ_F_APOLL_MULTISHOT ); mask &= ~EPOLLONESHOT ; ret =io_arm_apoll (req, issue_flags, mask);return ret ==IO_APOLL_OK ? -EIOCBQUEUED : -ECANCELED ; }
io_uring/xattr.c +3 -5
@@ -31,11 +31,11 @@ void io_xattr_cleanup(struct io_kiocb *req)kvfree (ix->ctx.kvalue); }static void io_xattr_finish (struct io_kiocb *req,int ret) { - req->flags &= ~REQ_F_NEED_CLEANUP ; +io_req_clear_flags (req,REQ_F_NEED_CLEANUP );io_xattr_cleanup (req);io_req_set_res (req, ret,0 ); }
@@ -64,12 +64,11 @@ static int __io_getxattr_prep(struct io_kiocb *req,if (ret) {kfree (ix->ctx.kname);return ret; } - req->flags |=REQ_F_NEED_CLEANUP ; - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP |REQ_F_FORCE_ASYNC );return 0 ; }int io_fgetxattr_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
@@ -141,12 +140,11 @@ static int __io_setxattr_prep(struct io_kiocb *req,if (ret) {kfree (ix->ctx.kname);return ret; } - req->flags |=REQ_F_NEED_CLEANUP ; - req->flags |=REQ_F_FORCE_ASYNC ; +io_req_set_flags (req,REQ_F_NEED_CLEANUP |REQ_F_FORCE_ASYNC );return 0 ; }int io_setxattr_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
Signature · 1 line
2.56.1
Quoted 14 lines · Caleb Sander Mateos
clang likes to set/clear
req->flagsbits using the narrowest possible or/and instructions (e.g. an 8-bit or if all flag bits are in the same byte). This causes store-to-load forwarding failures on subsequent widerreq->flagsloads.For example, clang compiles the
req->flags|=REQ_F_ASYNC_DATAinio_uring_cmd_prep()->io_uring_alloc_cached_async_data()onx86_64into an orb $0x40, 0x4a(%rdi) instruction, sinceREQ_F_ASYNC_DATA=0x40<< 16 andoffsetof(struct io_kiocb, flags)=0x48. The subsequent 64-bitreq->flagsload forunlikely(req->flags & (IO_REQ_LINK_FLAGS | REQ_F_FORCE_ASYNC | REQ_F_FAIL))inio_submit_sqe()is extremely hot because it can't be served from the store buffer and must wait for theREQ_F_ASYNC_DATAset to hit the L1 cache. On a thread issuing ublk and NVMe passthrough commands, that onereq->flagsload consumes around 1% of all CPU cycles.
Quoted 1 line
[...]
Applied, thanks!
[1/4] io_uring: return io_req_flags_t from io_slot_flags()
commit: 4d81c9390d0a326f2beadea2503d3e3929c739e1
[2/4] io_uring: add multishot_flags constant in io_wq_submit_work()
commit: 91801f8eab5e5963befb1358efa6832e9aa78499
[3/4] io_uring: add io_req_{set,clear,switch}_flags()
commit: 3e5dd5a145214bd8996744e8a828ceb249570301
[4/4] io_uring: use io_req_{set,clear,switch}_flags()
commit: 33e97a6c608064795dc5e36643a9cc66d98bfa99Best regards,
Signature · 1 line
Jens Axboe