io_uring: kill io_cqring_ev_posted() and __io_cq_unlock_post()
Jens Axboe
maintainerPATCH 1/1 io_uring: kill io_cqring_ev_posted() and __io_cq_unlock_post()
__io_cq_unlock_post() is identical to io_cq_unlock_post(), and io_cqring_ev_posted() has a single caller so migth as well just inline it there.
---
io_uring/io_uring.c +4 -13
@@ -581,23 +581,14 @@ void __io_commit_cqring_flush(struct io_ring_ctx *ctx)io_eventfd_flush_signal (ctx); } -static inline void io_cqring_ev_posted (struct io_ring_ctx *ctx) -{ -io_commit_cqring_flush (ctx); -io_cqring_wake (ctx); -} - -static inline void __io_cq_unlock_post (struct io_ring_ctx *ctx) +void io_cq_unlock_post (struct io_ring_ctx *ctx) __releases(ctx->completion_lock) {io_commit_cqring (ctx);spin_unlock (&ctx->completion_lock); -io_cqring_ev_posted (ctx); -} -void io_cq_unlock_post (struct io_ring_ctx *ctx) -{ -__io_cq_unlock_post (ctx); +io_commit_cqring_flush (ctx); +io_cqring_wake (ctx); }/* Returns true if there are no backlogged entries after the flush */
@@ -1346,7 +1337,7 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx)if (!(req->flags &REQ_F_CQE_SKIP ))__io_fill_cqe_req (ctx, req); } -__io_cq_unlock_post (ctx); +io_cq_unlock_post (ctx);io_free_batch_list (ctx, state->compl_reqs.first);INIT_WQ_LIST (&state->compl_reqs);
Signature · 1 line
Jens Axboe
Quoted 3 lines · Jens Axboe
__io_cq_unlock_post()is identical toio_cq_unlock_post(), andio_cqring_ev_posted()has a single caller so migth as well just inline it there.
It was there for one purpose, to inline it in the hottest path, i.e. __io_submit_flush_completions(). I'll be reverting it back
Quoted 45 lines
Signed-off-by Jens Axboe <[email protected]>---
io_uring/io_uring.c +4
-13@@ -581,23 +581,14 @@ void __io_commit_cqring_flush(struct io_ring_ctx *ctx)io_eventfd_flush_signal (ctx); } -static inline void io_cqring_ev_posted (struct io_ring_ctx *ctx) -{ -io_commit_cqring_flush (ctx); -io_cqring_wake (ctx); -} - -static inline void __io_cq_unlock_post (struct io_ring_ctx *ctx) +void io_cq_unlock_post (struct io_ring_ctx *ctx) __releases(ctx->completion_lock) {io_commit_cqring (ctx);spin_unlock (&ctx->completion_lock); -io_cqring_ev_posted (ctx); -} -void io_cq_unlock_post (struct io_ring_ctx *ctx) -{ -__io_cq_unlock_post (ctx); +io_commit_cqring_flush (ctx); +io_cqring_wake (ctx); }/* Returns true if there are no backlogged entries after the flush */ @@ -1346,7 +1337,7 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx)if (!(req->flags &REQ_F_CQE_SKIP ))__io_fill_cqe_req (ctx, req); } -__io_cq_unlock_post (ctx); +io_cq_unlock_post (ctx);io_free_batch_list (ctx, state->compl_reqs.first);INIT_WQ_LIST (&state->compl_reqs);
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 7 lines · Pavel Begunkov
Quoted 3 lines · Jens Axboe
__io_cq_unlock_post()is identical toio_cq_unlock_post(), andio_cqring_ev_posted()has a single caller so migth as well just inline it there.It was there for one purpose, to inline it in the hottest path, i.e.
__io_submit_flush_completions(). I'll be reverting it back
The compiler is most certainly already doing that, in fact even __io_submit_flush_completions() is inlined in io_submit_flush_completions() for me here.
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
Quoted 7 lines · Pavel Begunkov
Quoted 3 lines · Jens Axboe
__io_cq_unlock_post()is identical toio_cq_unlock_post(), andio_cqring_ev_posted()has a single caller so migth as well just inline it there.It was there for one purpose, to inline it in the hottest path, i.e.
__io_submit_flush_completions(). I'll be reverting it backThe compiler is most certainly already doing that, in fact even
.L1493:
# io_uring/io_uring.c:631: io_cq_unlock_post(ctx); movq %r15, %rdi # ctx, call io_cq_unlock_post #
Even more, after IORING_SETUP_CQE32 was added I didn't see once __io_fill_cqe_req actually inlined even though it's marked so.
Quoted 2 lines
__io_submit_flush_completions()is inlined inio_submit_flush_completions()for me here.
And io_submit_flush_completions is inlined as well, right? That would be quite odd, __io_submit_flush_completions() is not small by any means and there are 3 call sites.
Signature · 1 line
Pavel Begunkov
Quoted 10 lines · Pavel Begunkov
Quoted 9 lines · Jens Axboe
Quoted 7 lines · Pavel Begunkov
Quoted 3 lines · Jens Axboe
__io_cq_unlock_post()is identical toio_cq_unlock_post(), andio_cqring_ev_posted()has a single caller so migth as well just inline it there.It was there for one purpose, to inline it in the hottest path, i.e.
__io_submit_flush_completions(). I'll be reverting it back
Quoted 6 lines
Quoted 1 line
The compiler is most certainly already doing that, in fact even
.L1493:
# io_uring/io_uring.c:631:io_cq_unlock_post(ctx);
movq %r15, %rdi # ctx,
call io_cq_unlock_post #
wrong one,
__io_submit_flush_completions:
pushq %rbp # ... .L1793: # io_uring/io_uring.c:1394: io_cq_unlock_post(ctx); movq %r12, %rdi # ctx, call io_cq_unlock_post #
Quoted 11 lines
Even more, after
IORING_SETUP_CQE32was added I didn't see once__io_fill_cqe_reqactually inlined even though it's marked so.Quoted 2 lines
__io_submit_flush_completions()is inlined inio_submit_flush_completions()for me here.And
io_submit_flush_completionsis inlined as well, right? That would be quite odd,__io_submit_flush_completions()is not small by any means and there are 3 call sites.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 10 lines · Pavel Begunkov
Quoted 9 lines · Jens Axboe
Quoted 7 lines · Pavel Begunkov
Quoted 3 lines · Jens Axboe
__io_cq_unlock_post()is identical toio_cq_unlock_post(), andio_cqring_ev_posted()has a single caller so migth as well just inline it there.It was there for one purpose, to inline it in the hottest path, i.e.
__io_submit_flush_completions(). I'll be reverting it back
Quoted 6 lines
Quoted 1 line
The compiler is most certainly already doing that, in fact even
.L1493:
#io_uring/io_uring.c:631:????io_cq_unlock_post(ctx); ????movq??? %r15, %rdi??? # ctx,
????call???io_cq_unlock_post??? #
Doubled checked here, and you're actually right:
55bc: 94000000 bl 4760 <io_cq_unlock_post>
Huh, that's very odd that it doesn't inline it. It doesn't even it I mark it inline, __always_inline gets it done.
Quoted 3 lines
Even more, after
IORING_SETUP_CQE32was added I didn't see once__io_fill_cqe_reqactually inlined even though it's marked so.
Doesn't seem to be inlined here either. Compiler:
gcc (Debian 12.2.0-9) 12.2.0
Quoted 6 lines
Quoted 2 lines
__io_submit_flush_completions()is inlined inio_submit_flush_completions()for me here.And
io_submit_flush_completionsis inlined as well, right? That would be quite odd,__io_submit_flush_completions()is not small by any means and there are 3 call sites.
io_submit_flush_completions() doesn't get inlined, __io_submit_flush_completions() gets inlined in io_submit_flush_completions().
Signature · 1 line
Jens Axboe
Quoted 18 lines · Jens Axboe
Quoted 10 lines · Pavel Begunkov
Quoted 9 lines · Jens Axboe
Quoted 7 lines · Pavel Begunkov
Quoted 3 lines · Jens Axboe
__io_cq_unlock_post()is identical toio_cq_unlock_post(), andio_cqring_ev_posted()has a single caller so migth as well just inline it there.It was there for one purpose, to inline it in the hottest path, i.e.
__io_submit_flush_completions(). I'll be reverting it backQuoted 6 lines
Quoted 1 line
The compiler is most certainly already doing that, in fact even
.L1493:
#io_uring/io_uring.c:631:????io_cq_unlock_post(ctx); ????movq??? %r15, %rdi??? # ctx,
????call???io_cq_unlock_post??? #
Quoted 6 lines
Doubled checked here, and you're actually right:
55bc: 94000000 bl 4760 <io_cq_unlock_post>Huh, that's very odd that it doesn't inline it. It doesn't even it I mark it inline,
__always_inlinegets it done.
That's odd as well for a function of this size
Quoted 11 lines
Quoted 3 lines
Even more, after
IORING_SETUP_CQE32was added I didn't see once__io_fill_cqe_reqactually inlined even though it's marked so.Doesn't seem to be inlined here either. Compiler:
gcc (Debian 12.2.0-9) 12.2.0
Quoted 3 lines
Quoted 2 lines
__io_submit_flush_completions()is inlined inio_submit_flush_completions()for me here.
Quoted 7 lines
Quoted 3 lines
And
io_submit_flush_completionsis inlined as well, right? That would be quite odd,__io_submit_flush_completions()is not small by any means and there are 3 call sites.
io_submit_flush_completions()doesn't get inlined,__io_submit_flush_completions()gets inlined inio_submit_flush_completions().
Then the compiler is drunk. It doesn't inline the function explicitly marked inline but does it for a non-inline one. Unless it's PGO'ed I can't think of a sane reason for it.
Signature · 1 line
Pavel Begunkov