Minor SQPOLL thread fix and improvement
cover Minor SQPOLL thread fix and improvement
I have been investigated a deadlock situation in my io_uring usage where the SQPOLL thread was going to sleep and my user threads were waiting inside io_uring_enter() for completions.
https://github.com/axboe/liburing/issues/367
This patch serie is the result of my investigation.
Olivier Langlois (2):
io_uring: Fix race condition when sqp thread goes to sleep
io_uring: Create define to modify a SQPOLL parameter
fs/io_uring.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
Signature · 1 line
2.32.0
PATCH 1/2 io_uring: Fix race condition when sqp thread goes to sleep
If an asynchronous completion happens before the task is preparing itself to wait and set its state to TASK_INTERRUPTABLE, the completion will not wake up the sqp thread.
--- fs/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io_uring.c +1 -1
@@ -6902,7 +6902,7 @@ static int io_sq_thread(void *data) }prepare_to_wait (&sqd->wait, &wait,TASK_INTERRUPTIBLE ); -if (!io_sqd_events_pending (sqd)) { +if (!io_sqd_events_pending (sqd) && !current->task_works) { needs_sched = true;list_for_each_entry (ctx, &sqd->ctx_list, sqd_list) {io_ring_set_wakeup_flag (ctx);
Signature · 1 line
2.32.0
PATCH 2/2 io_uring: Create define to modify a SQPOLL parameter
In reply to Olivier LangloisThe magic number used to cap the number of entries extracted from an io_uring instance SQ before moving to the other instances is an interesting parameter to experiment with.
A define has been created to make it easy to change its value from a single location.
--- fs/io_uring.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/io_uring.c +3 -2
@@ -89,6 +89,7 @@#define IORING_MAX_ENTRIES 32768#define IORING_MAX_CQ_ENTRIES (2 * IORING_MAX_ENTRIES) +#define IORING_SQPOLL_CAP_ENTRIES_VALUE 8/* * Shift of 9 is 512 entries, or exactly one page on 64-bit archs
@@ -6797,8 +6798,8 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries) to_submit =io_sqring_entries (ctx);/* if we're handling multiple rings, cap submit size for fairness */ -if (cap_entries && to_submit >8 ) - to_submit =8 ; +if (cap_entries && to_submit >IORING_SQPOLL_CAP_ENTRIES_VALUE ) + to_submit =IORING_SQPOLL_CAP_ENTRIES_VALUE ;if (!list_empty (&ctx->iopoll_list) || to_submit) {unsigned nr_events =0 ;
Signature · 1 line
2.32.0
Re: [PATCH 1/2] io_uring: Fix race condition when sqp thread goes to sleep
In reply to Olivier LangloisQuoted 5 lines · Olivier Langlois
If an asynchronous completion happens before the task is preparing itself to wait and set its state to
TASK_INTERRUPTABLE, the completion
will not wake up the sqp thread.
I have just noticed that I made a typo in the description. I will send a v2 of that patch.
Sorry about that. I was too excited to share my discovery...
Re: [PATCH 1/2] io_uring: Fix race condition when sqp thread goes to sleep
Quoted 10 lines · Olivier Langlois
Quoted 5 lines · Olivier Langlois
If an asynchronous completion happens before the task is preparing itself to wait and set its state to
TASK_INTERRUPTABLE, the completion
will not wake up the sqp thread.I have just noticed that I made a typo in the description. I will send a v2 of that patch.
Sorry about that. I was too excited to share my discovery...
git format-patch --cover-letter --thread=shallow ...
would be even better, but the fix looks right
Signature · 1 line
Pavel Begunkov
Re: [PATCH 1/2] io_uring: Fix race condition when sqp thread goes to sleep
Quoted 12 lines · Pavel Begunkov
Quoted 11 lines · Olivier Langlois
Quoted 6 lines · Olivier Langlois
If an asynchronous completion happens before the task is preparing
itself to wait and set its state toTASK_INTERRUPTABLE, the completion
will not wake up the sqp thread.I have just noticed that I made a typo in the description. I will send
a v2 of that patch.
Quoted 6 lines
Quoted 1 line
Sorry about that. I was too excited to share my discovery...
git format-patch --cover-letter --thread=shallow ...
would be even better, but the fix looks right
You are too good... I thought that I could get away from hacking manually the patch file for such a minor change...
It seems that I got caught...
Let me know if you need me to redo it the right way...
Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep
In reply to a message not in the archiveQuoted 14 lines · Olivier Langlois
If an asynchronous completion happens before the task is preparing itself to wait and set its state to
TASK_INTERRUPTIBLE, the completion will not wake up the sqp thread.Signed-off-by Olivier Langlois <[email protected]>--- fs/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/io_uring.c +0
-0
fs/io_uring.c static int io_sq_thread(void *data) · line 6905
Quoted 5 lines
fs/io_uring.c +1
-1@@ -6902,7 +6902,7 @@ static int io_sq_thread(void *data) }prepare_to_wait (&sqd->wait, &wait,TASK_INTERRUPTIBLE ); -if (!io_sqd_events_pending (sqd)) { +if (!io_sqd_events_pending (sqd) && !current->task_works) {
Agree that it should be here, but we also lack a good enough task_work_run() around, and that may send the task burn CPU for a while in some cases. Let's do
if (!io_sqd_events_pending(sqd) && !io_run_task_work())
...
fwiw, no need to worry about TASK_INTERRUPTIBLE as io_run_task_work() sets it to TASK_RUNNING.
Quoted 4 lines
fs/io_uring.c +0
-0needs_sched = true;list_for_each_entry (ctx, &sqd->ctx_list, sqd_list) {io_ring_set_wakeup_flag (ctx);
Signature · 1 line
Pavel Begunkov
Re: [PATCH 2/2] io_uring: Create define to modify a SQPOLL parameter
In reply to Olivier LangloisQuoted 3 lines · Olivier Langlois
The magic number used to cap the number of entries extracted from an
io_uringinstance SQ before moving to the other instances is an interesting parameter to experiment with.
Not particularly related to this patch, but the problem with this capping is that there is no reliable way to do request linking using shared sqpoll. It may break a link in half (or in N parts for long links) and submit them separately and in parallel.
Quoted 32 lines
A define has been created to make it easy to change its value from a single location.
Signed-off-by Olivier Langlois <[email protected]>--- fs/io_uring.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/io_uring.c +3
-2@@ -89,6 +89,7 @@#define IORING_MAX_ENTRIES 32768#define IORING_MAX_CQ_ENTRIES (2 * IORING_MAX_ENTRIES) +#define IORING_SQPOLL_CAP_ENTRIES_VALUE 8/* * Shift of 9 is 512 entries, or exactly one page on 64-bit archs @@ -6797,8 +6798,8 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries) to_submit =io_sqring_entries (ctx);/* if we're handling multiple rings, cap submit size for fairness */ -if (cap_entries && to_submit >8 ) - to_submit =8 ; +if (cap_entries && to_submit >IORING_SQPOLL_CAP_ENTRIES_VALUE ) + to_submit =IORING_SQPOLL_CAP_ENTRIES_VALUE ;if (!list_empty (&ctx->iopoll_list) || to_submit) {unsigned nr_events =0 ;
Signature · 1 line
Pavel Begunkov
Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep
In reply to Pavel BegunkovQuoted 22 lines · Pavel Begunkov
Quoted 21 lines · Olivier Langlois
If an asynchronous completion happens before the task is preparing itself to wait and set its state to
TASK_INTERRUPTIBLE, the completion
will not wake up the sqp thread.Signed-off-by Olivier Langlois <[email protected]>---
fs/io_uring.c | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)fs/io_uring.c +0
-0@@ -6902,7 +6902,7 @@ static int io_sq_thread(void *data) }
prepare_to_wait(&sqd->wait, &wait,TASK_INTERRUPTIBLE);
- if (!io_sqd_events_pending(sqd)) { + if (!io_sqd_events_pending(sqd) && !current-
Quoted 8 lines
Quoted 1 line
Quoted 1 line
task_works) {Agree that it should be here, but we also lack a good enough
task_work_run()around, and that may send the task burn CPU for a while in some cases. Let's doif (!io_sqd_events_pending(sqd) && !io_run_task_work())
...
I can do that if you want but considering that the function is inline and the race condition is a relatively rare occurence, is the cost coming with inline expansion really worth it in this case?
Quoted 3 lines
fwiw, no need to worry about
TASK_INTERRUPTIBLEasio_run_task_work()sets it toTASK_RUNNING.
I wasn't worried about that as I believe that finish_wait() is taking care the state as well.
What I wasn't sure about was if the patch was sufficient to totally eliminate the race condition.
I had to educate myself about how schedule() works to appreciate its design and convince myself that the patch was good.
Quoted 7 lines
Quoted 6 lines
needs_sched = true;
list_for_each_entry(ctx, &sqd->ctx_list, sqd_list){
io_ring_set_wakeup_flag(ctx);
Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep
Quoted 30 lines · Olivier Langlois
Quoted 29 lines · Pavel Begunkov
Quoted 17 lines · Olivier Langlois
If an asynchronous completion happens before the task is preparing
itself to wait and set its state toTASK_INTERRUPTIBLE, the completion
will not wake up the sqp thread.Signed-off-by Olivier Langlois <[email protected]>---
fs/io_uring.c | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)fs/io_uring.c +0
-0@@ -6902,7 +6902,7 @@ static int io_sq_thread(void *data) }Quoted 6 lines
prepare_to_wait(&sqd->wait, &wait,TASK_INTERRUPTIBLE);
- if (!io_sqd_events_pending(sqd)) { + if (!io_sqd_events_pending(sqd) && !current-Quoted 1 line
task_works) {Agree that it should be here, but we also lack a good enough
task_work_run()around, and that may send the task burn CPU for a while in some cases. Let's do
Quoted 7 lines
Quoted 2 lines
if (!io_sqd_events_pending(sqd) && !io_run_task_work())
...I can do that if you want but considering that the function is inline and the race condition is a relatively rare occurence, is the cost coming with inline expansion really worth it in this case?
Quoted 1 line
On hand, there is the inline expansion concern.
OTOH, the benefit of going with your suggestion is that completions generally precedes new submissions so yes, it might be better that way.
I'm really unsure about this. I'm just raising the concern and I'll let you make the final decision...
Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep
Quoted 38 lines · Olivier Langlois
Quoted 30 lines · Olivier Langlois
Quoted 29 lines · Pavel Begunkov
Quoted 17 lines · Olivier Langlois
If an asynchronous completion happens before the task is preparing
itself to wait and set its state toTASK_INTERRUPTIBLE, the completion
will not wake up the sqp thread.Signed-off-by Olivier Langlois <[email protected]>---
fs/io_uring.c | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)fs/io_uring.c +0
-0@@ -6902,7 +6902,7 @@ static int io_sq_thread(void *data) }Quoted 6 lines
prepare_to_wait(&sqd->wait, &wait,TASK_INTERRUPTIBLE);
- if (!io_sqd_events_pending(sqd)) { + if (!io_sqd_events_pending(sqd) && !current-Quoted 1 line
task_works) {Agree that it should be here, but we also lack a good enough
task_work_run()around, and that may send the task burn CPU for a while in some cases. Let's doQuoted 7 lines
Quoted 2 lines
if (!io_sqd_events_pending(sqd) && !io_run_task_work())
...I can do that if you want but considering that the function is inline and the race condition is a relatively rare occurence, is the cost coming with inline expansion really worth it in this case?
Quoted 1 line
Quoted 7 lines
On hand, there is the inline expansion concern.
OTOH, the benefit of going with your suggestion is that completions generally precedes new submissions so yes, it might be better that way.
I'm really unsure about this. I'm just raising the concern and I'll let you make the final decision...
It seems it may actually loop infinitely until it gets a signal, so yes. And even if not, rare stalls are nasty, they will ruin some 9s of latency and hard to catch.
That part is quite cold anyway, would generate some extra cold instructions, meh
Signature · 1 line
Pavel Begunkov
Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep
Quoted 21 lines · Pavel Begunkov
Quoted 12 lines · Olivier Langlois
Quoted 9 lines · Olivier Langlois
On Tue, 2021-06-22 at 21:45 +0100, Pavel Begunkov wrote:
I can do that if you want but considering that the function is inline
and the race condition is a relatively rare occurence, is the cost
coming with inline expansion really worth it in this case?Quoted 1 line
On hand, there is the inline expansion concern.
Quoted 7 lines
OTOH, the benefit of going with your suggestion is that completions generally precedes new submissions so yes, it might be better that way.
I'm really unsure about this. I'm just raising the concern and I'll let
you make the final decision...
Quoted 7 lines
It seems it may actually loop infinitely until it gets a signal, so yes. And even if not, rare stalls are nasty, they will ruin some 9s of latency and hard to catch.
That part is quite cold anyway, would generate some extra cold instructions, meh
I'm not 100% sure to see the infinite loop possibility but I guess that with some badly placed preemptions, it could take few iterations before entering the block:
if (sqt_spin || !time_after(jiffies, timeout)) {So I will go ahead with your suggestion.
I'll retest the new patch version (it should be a formality) and I'll resend an update once done.
Greetings,
Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep
Quoted 26 lines · Olivier Langlois
Quoted 25 lines · Pavel Begunkov
Quoted 12 lines · Olivier Langlois
Quoted 9 lines · Olivier Langlois
On Tue, 2021-06-22 at 21:45 +0100, Pavel Begunkov wrote:
I can do that if you want but considering that the function is inline
and the race condition is a relatively rare occurence, is the cost
coming with inline expansion really worth it in this case?Quoted 1 line
On hand, there is the inline expansion concern.
Quoted 7 lines
OTOH, the benefit of going with your suggestion is that completions generally precedes new submissions so yes, it might be better that way.
I'm really unsure about this. I'm just raising the concern and I'll let
you make the final decision...It seems it may actually loop infinitely until it gets a signal, so yes. And even if not, rare stalls are nasty, they will ruin some 9s of latency and hard to catch.
Quoted 8 lines
Quoted 3 lines
That part is quite cold anyway, would generate some extra cold instructions, meh
I'm not 100% sure to see the infinite loop possibility but I guess that with some badly placed preemptions, it could take few iterations before entering the block:
if (sqt_spin || !time_after(jiffies, timeout)) {
Had a case in mind, but looking through the branches it can't really happen. Agree that won't be infinite in real life, until we start using (and there was an RFC) finer grained timeouts.
In any case for several reasons think it's the right thing to do.
Quoted 4 lines
So I will go ahead with your suggestion.
I'll retest the new patch version (it should be a formality) and I'll resend an update once done.
Perfect
Signature · 1 line
Pavel Begunkov