Skip to contentSkip to conversation

Minor SQPOLL thread fix and improvement

PATCH Superseded Olivier Langlois 2 participants 13 messages Jun 22, 2021 – Jun 23, 2021 1 message not in the archive

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Minor SQPOLL thread fix and improvement

I have been investigated a deadlock situation in my io_uring usage where the SQPOLL thread was going to sleep and my user threads were waiting inside io_uring_enter() for completions.

https://github.com/axboe/liburing/issues/367

This patch serie is the result of my investigation.

Olivier Langlois (2):
io_uring: Fix race condition when sqp thread goes to sleep
io_uring: Create define to modify a SQPOLL parameter

 fs/io_uring.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
Signature · 1 line
2.32.0
#1 lore raw

PATCH 1/2 io_uring: Fix race condition when sqp thread goes to sleep

If an asynchronous completion happens before the task is preparing itself to wait and set its state to TASK_INTERRUPTABLE, the completion will not wake up the sqp thread.

---
 fs/io_uring.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io_uring.c +1 -1
index fc8637f591a6..02f789e07d4c 100644
@@ -6902,7 +6902,7 @@ static int io_sq_thread(void *data)
 		}
 
 		prepare_to_wait(&sqd->wait, &wait, TASK_INTERRUPTIBLE);
-		if (!io_sqd_events_pending(sqd)) {
+		if (!io_sqd_events_pending(sqd) && !current->task_works) {
 			needs_sched = true;
 			list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
 				io_ring_set_wakeup_flag(ctx);
Signature · 1 line
2.32.0
#2 lore raw

Re: [PATCH 1/2] io_uring: Fix race condition when sqp thread goes to sleep

Quoted 5 lines · Olivier Langlois

If an asynchronous completion happens before the task is preparing itself to wait and set its state to TASK_INTERRUPTABLE, the completion
will not wake up the sqp thread.

I have just noticed that I made a typo in the description. I will send a v2 of that patch.

Sorry about that. I was too excited to share my discovery...

#3 lore raw

Re: [PATCH 1/2] io_uring: Fix race condition when sqp thread goes to sleep

Quoted 10 lines · Olivier Langlois
Quoted 5 lines · Olivier Langlois

If an asynchronous completion happens before the task is preparing itself to wait and set its state to TASK_INTERRUPTABLE, the completion
will not wake up the sqp thread.

I have just noticed that I made a typo in the description. I will send a v2 of that patch.

Sorry about that. I was too excited to share my discovery...

git format-patch --cover-letter --thread=shallow ...

would be even better, but the fix looks right

Signature · 1 line
Pavel Begunkov
#4 lore raw

Re: [PATCH 1/2] io_uring: Fix race condition when sqp thread goes to sleep

Quoted 12 lines · Pavel Begunkov
Quoted 11 lines · Olivier Langlois
Quoted 6 lines · Olivier Langlois

If an asynchronous completion happens before the task is preparing
itself to wait and set its state to TASK_INTERRUPTABLE, the completion
will not wake up the sqp thread.

I have just noticed that I made a typo in the description. I will send
a v2 of that patch.

Quoted 6 lines
Quoted 1 line

Sorry about that. I was too excited to share my discovery...

git format-patch --cover-letter --thread=shallow ...

would be even better, but the fix looks right

You are too good... I thought that I could get away from hacking manually the patch file for such a minor change...

It seems that I got caught...

Let me know if you need me to redo it the right way...

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep

Quoted 14 lines · Olivier Langlois

If an asynchronous completion happens before the task is preparing itself to wait and set its state to TASK_INTERRUPTIBLE, the completion will not wake up the sqp thread.

---
 fs/io_uring.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io_uring.c +0 -0
index fc8637f591a6..02f789e07d4c 100644

fs/io_uring.c static int io_sq_thread(void *data) · line 6905

Quoted 5 lines
fs/io_uring.c +1 -1
@@ -6902,7 +6902,7 @@ static int io_sq_thread(void *data)
 		}
 
 		prepare_to_wait(&sqd->wait, &wait, TASK_INTERRUPTIBLE);
-		if (!io_sqd_events_pending(sqd)) {
+		if (!io_sqd_events_pending(sqd) && !current->task_works) {

Agree that it should be here, but we also lack a good enough task_work_run() around, and that may send the task burn CPU for a while in some cases. Let's do

if (!io_sqd_events_pending(sqd) && !io_run_task_work())
...

fwiw, no need to worry about TASK_INTERRUPTIBLE as io_run_task_work() sets it to TASK_RUNNING.

Quoted 4 lines
fs/io_uring.c +0 -0
 			needs_sched = true;
 			list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
 				io_ring_set_wakeup_flag(ctx);
Signature · 1 line
Pavel Begunkov
#7 lore raw

Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep

Quoted 22 lines · Pavel Begunkov
Quoted 21 lines · Olivier Langlois

If an asynchronous completion happens before the task is preparing itself to wait and set its state to TASK_INTERRUPTIBLE, the completion
will not wake up the sqp thread.

---
 fs/io_uring.c | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io_uring.c +0 -0
index fc8637f591a6..02f789e07d4c 100644
@@ -6902,7 +6902,7 @@ static int io_sq_thread(void *data)
                 }


                prepare_to_wait(&sqd->wait, &wait, TASK_INTERRUPTIBLE);
-               if (!io_sqd_events_pending(sqd)) { +               if (!io_sqd_events_pending(sqd) && !current-

Quoted 8 lines
Quoted 1 line
Quoted 1 line

task_works) {

Agree that it should be here, but we also lack a good enough task_work_run() around, and that may send the task burn CPU for a while in some cases. Let's do

if (!io_sqd_events_pending(sqd) && !io_run_task_work())
   ...

I can do that if you want but considering that the function is inline and the race condition is a relatively rare occurence, is the cost coming with inline expansion really worth it in this case?

Quoted 3 lines

fwiw, no need to worry about TASK_INTERRUPTIBLE as io_run_task_work() sets it to TASK_RUNNING.

I wasn't worried about that as I believe that finish_wait() is taking care the state as well.

What I wasn't sure about was if the patch was sufficient to totally eliminate the race condition.

I had to educate myself about how schedule() works to appreciate its design and convince myself that the patch was good.

Quoted 7 lines
Quoted 6 lines

                        needs_sched = true; list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
                                io_ring_set_wakeup_flag(ctx);

#8 lore raw

Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep

Quoted 30 lines · Olivier Langlois
Quoted 29 lines · Pavel Begunkov
Quoted 17 lines · Olivier Langlois

If an asynchronous completion happens before the task is preparing
itself to wait and set its state to TASK_INTERRUPTIBLE, the completion
will not wake up the sqp thread.

---
 fs/io_uring.c | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io_uring.c +0 -0
index fc8637f591a6..02f789e07d4c 100644
@@ -6902,7 +6902,7 @@ static int io_sq_thread(void *data)
                 }
Quoted 6 lines


                prepare_to_wait(&sqd->wait, &wait, TASK_INTERRUPTIBLE);
-               if (!io_sqd_events_pending(sqd)) { +               if (!io_sqd_events_pending(sqd) && !current-

Quoted 1 line

task_works) {

Agree that it should be here, but we also lack a good enough task_work_run() around, and that may send the task burn CPU for a while in some cases. Let's do

Quoted 7 lines
Quoted 2 lines

if (!io_sqd_events_pending(sqd) && !io_run_task_work())
   ...

I can do that if you want but considering that the function is inline and the race condition is a relatively rare occurence, is the cost coming with inline expansion really worth it in this case?

Quoted 1 line

On hand, there is the inline expansion concern.

OTOH, the benefit of going with your suggestion is that completions generally precedes new submissions so yes, it might be better that way.

I'm really unsure about this. I'm just raising the concern and I'll let you make the final decision...

#9 lore raw

Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep

Quoted 38 lines · Olivier Langlois
Quoted 30 lines · Olivier Langlois
Quoted 29 lines · Pavel Begunkov
Quoted 17 lines · Olivier Langlois

If an asynchronous completion happens before the task is preparing
itself to wait and set its state to TASK_INTERRUPTIBLE, the completion
will not wake up the sqp thread.

---
 fs/io_uring.c | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io_uring.c +0 -0
index fc8637f591a6..02f789e07d4c 100644
@@ -6902,7 +6902,7 @@ static int io_sq_thread(void *data)
                 }
Quoted 6 lines


                prepare_to_wait(&sqd->wait, &wait, TASK_INTERRUPTIBLE);
-               if (!io_sqd_events_pending(sqd)) { +               if (!io_sqd_events_pending(sqd) && !current-

Quoted 1 line

task_works) {

Agree that it should be here, but we also lack a good enough task_work_run() around, and that may send the task burn CPU for a while in some cases. Let's do

Quoted 7 lines
Quoted 2 lines

if (!io_sqd_events_pending(sqd) && !io_run_task_work())
   ...

I can do that if you want but considering that the function is inline and the race condition is a relatively rare occurence, is the cost coming with inline expansion really worth it in this case?

Quoted 1 line
Quoted 7 lines

On hand, there is the inline expansion concern.

OTOH, the benefit of going with your suggestion is that completions generally precedes new submissions so yes, it might be better that way.

I'm really unsure about this. I'm just raising the concern and I'll let you make the final decision...

It seems it may actually loop infinitely until it gets a signal, so yes. And even if not, rare stalls are nasty, they will ruin some 9s of latency and hard to catch.

That part is quite cold anyway, would generate some extra cold instructions, meh

Signature · 1 line
Pavel Begunkov
#10 lore raw

Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep

Quoted 21 lines · Pavel Begunkov
Quoted 12 lines · Olivier Langlois
Quoted 9 lines · Olivier Langlois

On Tue, 2021-06-22 at 21:45 +0100, Pavel Begunkov wrote:

I can do that if you want but considering that the function is inline
and the race condition is a relatively rare occurence, is the cost
coming with inline expansion really worth it in this case?

Quoted 1 line

On hand, there is the inline expansion concern.

Quoted 7 lines

OTOH, the benefit of going with your suggestion is that completions generally precedes new submissions so yes, it might be better that way.

I'm really unsure about this. I'm just raising the concern and I'll let
you make the final decision...

Quoted 7 lines

It seems it may actually loop infinitely until it gets a signal, so yes. And even if not, rare stalls are nasty, they will ruin some 9s of latency and hard to catch.

That part is quite cold anyway, would generate some extra cold instructions, meh

I'm not 100% sure to see the infinite loop possibility but I guess that with some badly placed preemptions, it could take few iterations before entering the block:

		if (sqt_spin || !time_after(jiffies, timeout)) {

So I will go ahead with your suggestion.

I'll retest the new patch version (it should be a formality) and I'll resend an update once done.

Greetings,

#11 lore raw

Re: [PATCH 1/2 v2] io_uring: Fix race condition when sqp thread goes to sleep

Quoted 26 lines · Olivier Langlois
Quoted 25 lines · Pavel Begunkov
Quoted 12 lines · Olivier Langlois
Quoted 9 lines · Olivier Langlois

On Tue, 2021-06-22 at 21:45 +0100, Pavel Begunkov wrote:

I can do that if you want but considering that the function is inline
and the race condition is a relatively rare occurence, is the cost
coming with inline expansion really worth it in this case?

Quoted 1 line

On hand, there is the inline expansion concern.

Quoted 7 lines

OTOH, the benefit of going with your suggestion is that completions generally precedes new submissions so yes, it might be better that way.

I'm really unsure about this. I'm just raising the concern and I'll let
you make the final decision...

It seems it may actually loop infinitely until it gets a signal, so yes. And even if not, rare stalls are nasty, they will ruin some 9s of latency and hard to catch.

Quoted 8 lines
Quoted 3 lines

That part is quite cold anyway, would generate some extra cold instructions, meh

I'm not 100% sure to see the infinite loop possibility but I guess that with some badly placed preemptions, it could take few iterations before entering the block:

		if (sqt_spin || !time_after(jiffies, timeout)) {

Had a case in mind, but looking through the branches it can't really happen. Agree that won't be infinite in real life, until we start using (and there was an RFC) finer grained timeouts.

In any case for several reasons think it's the right thing to do.

Quoted 4 lines

So I will go ahead with your suggestion.

I'll retest the new patch version (it should be a formality) and I'll resend an update once done.

Perfect

Signature · 1 line
Pavel Begunkov
#12 lore raw

PATCH 2/2 io_uring: Create define to modify a SQPOLL parameter

In reply to Olivier Langlois

The magic number used to cap the number of entries extracted from an io_uring instance SQ before moving to the other instances is an interesting parameter to experiment with.

A define has been created to make it easy to change its value from a single location.

---
 fs/io_uring.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
fs/io_uring.c +3 -2
index 02f789e07d4c..3f271bd7726b 100644
@@ -89,6 +89,7 @@
 
 #define IORING_MAX_ENTRIES	32768
 #define IORING_MAX_CQ_ENTRIES	(2 * IORING_MAX_ENTRIES)
+#define IORING_SQPOLL_CAP_ENTRIES_VALUE 8
 
 /*
  * Shift of 9 is 512 entries, or exactly one page on 64-bit archs
@@ -6797,8 +6798,8 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries)
 
 	to_submit = io_sqring_entries(ctx);
 	/* if we're handling multiple rings, cap submit size for fairness */
-	if (cap_entries && to_submit > 8)
-		to_submit = 8;
+	if (cap_entries && to_submit > IORING_SQPOLL_CAP_ENTRIES_VALUE)
+		to_submit = IORING_SQPOLL_CAP_ENTRIES_VALUE;
 
 	if (!list_empty(&ctx->iopoll_list) || to_submit) {
 		unsigned nr_events = 0;
Signature · 1 line
2.32.0
#13 lore raw

Re: [PATCH 2/2] io_uring: Create define to modify a SQPOLL parameter

Quoted 3 lines · Olivier Langlois

The magic number used to cap the number of entries extracted from an io_uring instance SQ before moving to the other instances is an interesting parameter to experiment with.

Not particularly related to this patch, but the problem with this capping is that there is no reliable way to do request linking using shared sqpoll. It may break a link in half (or in N parts for long links) and submit them separately and in parallel.

Quoted 32 lines

A define has been created to make it easy to change its value from a single location.

---
 fs/io_uring.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
fs/io_uring.c +3 -2
index 02f789e07d4c..3f271bd7726b 100644
@@ -89,6 +89,7 @@
 
 #define IORING_MAX_ENTRIES	32768
 #define IORING_MAX_CQ_ENTRIES	(2 * IORING_MAX_ENTRIES)
+#define IORING_SQPOLL_CAP_ENTRIES_VALUE 8
 
 /*
  * Shift of 9 is 512 entries, or exactly one page on 64-bit archs
@@ -6797,8 +6798,8 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries)
 
 	to_submit = io_sqring_entries(ctx);
 	/* if we're handling multiple rings, cap submit size for fairness */
-	if (cap_entries && to_submit > 8)
-		to_submit = 8;
+	if (cap_entries && to_submit > IORING_SQPOLL_CAP_ENTRIES_VALUE)
+		to_submit = IORING_SQPOLL_CAP_ENTRIES_VALUE;
 
 	if (!list_empty(&ctx->iopoll_list) || to_submit) {
 		unsigned nr_events = 0;
Signature · 1 line
Pavel Begunkov