Skip to contentSkip to conversation

io_uring: fix memory ordering when SQPOLL thread goes to sleep

PATCH Superseded Almog Khaikin 2 participants 4 messages Mar 6, 2022

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 io_uring: fix memory ordering when SQPOLL thread goes to sleep

Without a full memory barrier between the store to the flags and the load of the SQ tail the two operations can be reordered and this can lead to a situation where the SQPOLL thread goes to sleep while the application writes to the SQ tail and doesn't see the wakeup flag. This memory barrier pairs with a full memory barrier in the application between its store to the SQ tail and its load of the flags.

---
 fs/io_uring.c | 7 +++++++
 1 file changed, 7 insertions(+)
fs/io_uring.c +7 -0
index 4715980e9015..221961be9213 100644
@@ -7613,6 +7613,13 @@ static int io_sq_thread(void *data)
 					needs_sched = false;
 					break;
 				}
+
+				/*
+				 * Ensure the store of the wakeup flag is not
+				 * reordered with the load of the SQ tail
+				 */
+				smp_mb();
+				
 				if (io_sqring_entries(ctx)) {
 					needs_sched = false;
 					break;
Signature · 1 line
2.35.1
#1 lore raw
Quoted 6 lines · Almog Khaikin

Without a full memory barrier between the store to the flags and the load of the SQ tail the two operations can be reordered and this can lead to a situation where the SQPOLL thread goes to sleep while the application writes to the SQ tail and doesn't see the wakeup flag. This memory barrier pairs with a full memory barrier in the application between its store to the SQ tail and its load of the flags.

The IOPOLL list is internal to the kernel, userspace doesn't interact with it. AFAICT it can't cause any races with userspace so the check if the list is empty seems unnecessary. The flags and the SQ tail are the only things that are shared that can cause any problems when the kernel thread goes to sleep so I think it's safe to remove that check.

The race here can result in a situation where the kernel thread goes to sleep while the application updates the SQ tail and doesn't see the NEED_WAKEUP flag. Checking the SQ tail after setting the wakeup flag along with the full barrier would ensure that either we see the tail update or the application sees the wakeup flag. The IOPOLL list doesn't tie into any of this.

 fs/io_uring.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
fs/io_uring.c +6 -5
index 4715980e9015..99af6607b770 100644
@@ -7608,11 +7608,12 @@ static int io_sq_thread(void *data)
 			list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
 				io_ring_set_wakeup_flag(ctx);
 
-				if ((ctx->flags & IORING_SETUP_IOPOLL) &&
-				    !wq_list_empty(&ctx->iopoll_list)) {
-					needs_sched = false;
-					break;
-				}
+				/*
+				 * Ensure the store of the wakeup flag is not
+				 * reordered with the load of the SQ tail
+				 */
+				smp_mb();
+
 				if (io_sqring_entries(ctx)) {
 					needs_sched = false;
 					break;
Signature · 1 line
2.35.1
maintainer
#2 lore raw
Quoted 14 lines · Almog Khaikin
Quoted 6 lines · Almog Khaikin

Without a full memory barrier between the store to the flags and the load of the SQ tail the two operations can be reordered and this can lead to a situation where the SQPOLL thread goes to sleep while the application writes to the SQ tail and doesn't see the wakeup flag. This memory barrier pairs with a full memory barrier in the application between its store to the SQ tail and its load of the flags.

The IOPOLL list is internal to the kernel, userspace doesn't interact with it. AFAICT it can't cause any races with userspace so the check if the list is empty seems unnecessary. The flags and the SQ tail are the only things that are shared that can cause any problems when the kernel thread goes to sleep so I think it's safe to remove that check.

Quoted 6 lines

The race here can result in a situation where the kernel thread goes to sleep while the application updates the SQ tail and doesn't see the NEED_WAKEUP flag. Checking the SQ tail after setting the wakeup flag along with the full barrier would ensure that either we see the tail update or the application sees the wakeup flag. The IOPOLL list doesn't tie into any of this.

I think you're mixing up two different things, and even if not, the IOPOLL change should be a separate change.

The iopoll list check isn't about synchronizing with userspace, it's about not going to sleep if we have entries to reap. If we're running with IOPOLL|SQPOLL, then it's the sq poll thread that does the polling and reaping.

Signature · 1 line
Jens Axboe
#3 lore raw
Quoted 22 lines · Jens Axboe
Quoted 14 lines · Almog Khaikin
Quoted 6 lines · Almog Khaikin

Without a full memory barrier between the store to the flags and the load of the SQ tail the two operations can be reordered and this can lead to a situation where the SQPOLL thread goes to sleep while the application writes to the SQ tail and doesn't see the wakeup flag. This memory barrier pairs with a full memory barrier in the application between its store to the SQ tail and its load of the flags.

The IOPOLL list is internal to the kernel, userspace doesn't interact with it. AFAICT it can't cause any races with userspace so the check if the list is empty seems unnecessary. The flags and the SQ tail are the only things that are shared that can cause any problems when the kernel thread goes to sleep so I think it's safe to remove that check.

Quoted 6 lines

The race here can result in a situation where the kernel thread goes to sleep while the application updates the SQ tail and doesn't see the NEED_WAKEUP flag. Checking the SQ tail after setting the wakeup flag along with the full barrier would ensure that either we see the tail update or the application sees the wakeup flag. The IOPOLL list doesn't tie into any of this.

Quoted 7 lines

I think you're mixing up two different things, and even if not, the IOPOLL change should be a separate change.

The iopoll list check isn't about synchronizing with userspace, it's about not going to sleep if we have entries to reap. If we're running with IOPOLL|SQPOLL, then it's the sq poll thread that does the polling and reaping.

I understand that, but the iopoll list check is already done earlier in the function and if the list isn't empty, the timer is reset. Checking again just a little later in the function and after writing the NEED_WAKEUP flag seems unnecessary but regardless, I guess it's not really relevant to this patch as it's not a bug that the check is there. The patch in the original message along with the liburing pull request should fix the race.