tw/wakeup tweaks
Jens Axboe
maintainercover tw/wakeup tweaks
Hi,
One patch tweaks when we think we have task_work pending to be more inclusive rather than gate on TIF_NOTIFY_SIGNAL, patch two ensures we don't do extra loops io_cqring_wait() - and just as importantly, that the expected run path for task_work will not have the task itself adding to the ctx->cq_wait waitqueue.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/2 io_uring: don't use TIF_NOTIFY_SIGNAL to test for availability of task_work
Use task_work_pending() as a better test for whether we have task_work or not, TIF_NOTIFY_SIGNAL is only valid if the any of the task_work items had been queued with TWA_SIGNAL as the notification mechanism. Hence task_work_pending() is a more reliable check.
--- io_uring/io_uring.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
io_uring/io_uring.h +1 -2
@@ -266,8 +266,7 @@ static inline int io_run_task_work(void)static inline bool io_task_work_pending (struct io_ring_ctx *ctx) { -return test_thread_flag (TIF_NOTIFY_SIGNAL ) || - !wq_list_empty (&ctx->work_llist); +return task_work_pending (current) || !wq_list_empty (&ctx->work_llist); }static inline int io_run_task_work_ctx (struct io_ring_ctx *ctx)
Signature · 1 line
2.35.1
Jens Axboe
maintainerPATCH 2/2 io_uring: include task_work run after scheduling in wait for events
In reply to Jens AxboeIt's quite possible that we got woken up because task_work was queued, and we need to process this task_work to generate the events waited for. If we return to the wait loop without running task_work, we'll end up adding the task to the waitqueue again, only to call io_cqring_wait_schedule() again which will run the task_work. This is less efficient than it could be, as it requires adding to the cq_wait queue again. It also triggers the wakeup path for completions as cq_wait is now non-empty with the task itself, and it'll require another lock grab and deletion to remove ourselves from the waitqueue.
--- io_uring/io_uring.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
io_uring/io_uring.c +3 -1
@@ -2481,7 +2481,7 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, }if (!schedule_hrtimeout (&timeout,HRTIMER_MODE_ABS ))return -ETIME ; -return 1 ; +return io_run_task_work_sig (ctx); }/*
@@ -2546,6 +2546,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events,prepare_to_wait_exclusive (&ctx->cq_wait, &iowq.wq,TASK_INTERRUPTIBLE ); ret =io_cqring_wait_schedule (ctx, &iowq, timeout); +if (__io_cqring_events_user (ctx) >= min_events) +break ;cond_resched (); }while (ret >0 );
Signature · 1 line
2.35.1
Jens Axboe
maintainerPATCH v2 2/2 io_uring: include task_work run after scheduling in wait for events
It's quite possible that we got woken up because task_work was queued, and we need to process this task_work to generate the events waited for. If we return to the wait loop without running task_work, we'll end up adding the task to the waitqueue again, only to call io_cqring_wait_schedule() again which will run the task_work. This is less efficient than it could be, as it requires adding to the cq_wait queue again. It also triggers the wakeup path for completions as cq_wait is now non-empty with the task itself, and it'll require another lock grab and deletion to remove ourselves from the waitqueue.
---
v2: tweak return value so we don't potentially return early from
waiting on events, if we had nothing to do post returning from schedule.
io_uring/io_uring.c +10 -1
@@ -2481,7 +2481,14 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, }if (!schedule_hrtimeout (&timeout,HRTIMER_MODE_ABS ))return -ETIME ; -return 1 ; + +/* +* Run task_work after scheduling. If we got woken because of +* task_work being processed, run it now rather than let the caller +* do another wait loop. +*/ + ret =io_run_task_work_sig (ctx); +return ret <0 ? ret :1 ; }/*
@@ -2546,6 +2553,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events,prepare_to_wait_exclusive (&ctx->cq_wait, &iowq.wq,TASK_INTERRUPTIBLE ); ret =io_cqring_wait_schedule (ctx, &iowq, timeout); +if (__io_cqring_events_user (ctx) >= min_events) +break ;cond_resched (); }while (ret >0 );
Signature · 1 line
Jens Axboe