Skip to contentSkip to conversation

io_uring: minor sqpoll code refactoring

PATCH Applied Olivier Langlois 3 participants 8 messages Jul 30, 2024 – Aug 2, 2024 io_uring

lore
Share & tools
lore
#0 lore raw

cover io_uring: minor sqpoll code refactoring

the first patch is minor micro-optimization that attempts to avoid a memory access if by testing a variable to is very likely already in a register

the second patch is also minor but this is much more serious. Without it, it is possible to have a ring that is configured to enable NAPI busy polling to NOT perform busy polling in specific conditions.

Olivier Langlois (2):
io_uring: micro optimization of __io_sq_thread() condition
io_uring: do the sqpoll napi busy poll outside the submission block

 io_uring/napi.h   | 9 +++++++++
 io_uring/sqpoll.c | 9 +++++----
 2 files changed, 14 insertions(+), 4 deletions(-)
Signature · 1 line
2.45.2
#1 lore raw

PATCH 1/2 io_uring: micro optimization of __io_sq_thread() condition

reverse the order of the element evaluation in an if statement.

for many users that are not using iopoll, the iopoll_list will always evaluate to false after having made a memory access whereas to_submit is very likely already loaded in a register.

---
 io_uring/sqpoll.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
io_uring/sqpoll.c +1 -1
index b3722e5275e7..cc4a25136030 100644
@@ -176,7 +176,7 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries)
 	if (cap_entries && to_submit > IORING_SQPOLL_CAP_ENTRIES_VALUE)
 		to_submit = IORING_SQPOLL_CAP_ENTRIES_VALUE;
 
-	if (!wq_list_empty(&ctx->iopoll_list) || to_submit) {
+	if (to_submit || !wq_list_empty(&ctx->iopoll_list)) {
 		const struct cred *creds = NULL;
 
 		if (ctx->sq_creds != current_cred())
Signature · 1 line
2.45.2
#3 lore raw

PATCH 2/2 io_uring: do the sqpoll napi busy poll outside the submission block

In reply to Olivier Langlois

there are many small reasons justifying this change.

1. busy poll must be performed even on rings that have no iopoll and no new sqe. It is quite possible that a ring configured for inbound traffic with multishot be several hours without receiving new request submissions
2. NAPI busy poll does not perform any credential validation
3. If the thread is awaken by task work, processing the task work is prioritary over NAPI busy loop. This is why a second loop has been created after the io_sq_tw() call instead of doing the busy loop in __io_sq_thread() outside its credential acquisition block.

---
 io_uring/napi.h   | 9 +++++++++
 io_uring/sqpoll.c | 7 ++++---
 2 files changed, 13 insertions(+), 3 deletions(-)
io_uring/napi.h +9 -0
index 88f1c21d5548..5506c6af1ff5 100644
@@ -101,4 +101,13 @@ static inline int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx)
 }
 #endif /* CONFIG_NET_RX_BUSY_POLL */
 
+static inline int io_do_sqpoll_napi(struct io_ring_ctx *ctx)
+{
+	int ret = 0;
+
+	if (io_napi(ctx))
+		ret = io_napi_sqpoll_busy_poll(ctx);
+	return ret;
+}
+
 #endif
io_uring/sqpoll.c +4 -3
index cc4a25136030..ec558daa0331 100644
@@ -195,9 +195,6 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries)
 			ret = io_submit_sqes(ctx, to_submit);
 		mutex_unlock(&ctx->uring_lock);
 
-		if (io_napi(ctx))
-			ret += io_napi_sqpoll_busy_poll(ctx);
-
 		if (to_submit && wq_has_sleeper(&ctx->sqo_sq_wait))
 			wake_up(&ctx->sqo_sq_wait);
 		if (creds)
@@ -322,6 +319,10 @@ static int io_sq_thread(void *data)
 		if (io_sq_tw(&retry_list, IORING_TW_CAP_ENTRIES_VALUE))
 			sqt_spin = true;
 
+		list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
+			if (io_do_sqpoll_napi(ctx))
+				sqt_spin = true;
+		}
 		if (sqt_spin || !time_after(jiffies, timeout)) {
 			if (sqt_spin) {
 				io_sq_update_worktime(sqd, &start);
Signature · 1 line
2.45.2
#4 lore raw

Re: [PATCH 2/2] io_uring: do the sqpoll napi busy poll outside the submission block

Quoted 11 lines · Olivier Langlois

there are many small reasons justifying this change.

1. busy poll must be performed even on rings that have no iopoll and no new sqe. It is quite possible that a ring configured for inbound traffic with multishot be several hours without receiving new request submissions
2. NAPI busy poll does not perform any credential validation
3. If the thread is awaken by task work, processing the task work is prioritary over NAPI busy loop. This is why a second loop has been created after the io_sq_tw() call instead of doing the busy loop in __io_sq_thread() outside its credential acquisition block.

That patch should be first as it's a fix we care to backport. It's also

Fixes 8d0c12a80cdeb ("io-uring: add napi busy poll support")

And a comment below

Quoted 40 lines
---
  io_uring/napi.h   | 9 +++++++++
  io_uring/sqpoll.c | 7 ++++---
  2 files changed, 13 insertions(+), 3 deletions(-)
io_uring/napi.h +9 -0
index 88f1c21d5548..5506c6af1ff5 100644
@@ -101,4 +101,13 @@ static inline int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx)
  }
  #endif /* CONFIG_NET_RX_BUSY_POLL */
 
+static inline int io_do_sqpoll_napi(struct io_ring_ctx *ctx)
+{
+	int ret = 0;
+
+	if (io_napi(ctx))
+		ret = io_napi_sqpoll_busy_poll(ctx);
+	return ret;
+}
+
  #endif
io_uring/sqpoll.c +0 -3
index cc4a25136030..ec558daa0331 100644
@@ -195,9 +195,6 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries)
  			ret = io_submit_sqes(ctx, to_submit);
  		mutex_unlock(&ctx->uring_lock);
 
-		if (io_napi(ctx))
-			ret += io_napi_sqpoll_busy_poll(ctx);
-
  		if (to_submit && wq_has_sleeper(&ctx->sqo_sq_wait))
  			wake_up(&ctx->sqo_sq_wait);
  		if (creds)

io_uring/sqpoll.c static int io_sq_thread(void *data) · line 324

Quoted 6 lines
io_uring/sqpoll.c +3 -0
@@ -322,6 +319,10 @@ static int io_sq_thread(void *data)
  		if (io_sq_tw(&retry_list, IORING_TW_CAP_ENTRIES_VALUE))
  			sqt_spin = true;
 
+		list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
+			if (io_do_sqpoll_napi(ctx))
+				sqt_spin = true;

io_do_sqpoll_napi() returns 1 as long as there are napis in the list, iow even if there is no activity it'll spin almost forever (60s is forever) bypassing sq_thread_idle.

Let's not update sqt_spin here, if the user wants it to poll for longer it can pass a larger SQPOLL idle timeout value.

Quoted 4 lines
io_uring/sqpoll.c +1 -0
+		}
  		if (sqt_spin || !time_after(jiffies, timeout)) {
  			if (sqt_spin) {
  				io_sq_update_worktime(sqd, &start);
Signature · 1 line
Pavel Begunkov
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] io_uring: micro optimization of __io_sq_thread() condition

In reply to Olivier Langlois
Quoted 5 lines · Olivier Langlois

reverse the order of the element evaluation in an if statement.

for many users that are not using iopoll, the iopoll_list will always evaluate to false after having made a memory access whereas to_submit is very likely already loaded in a register.

doubt it'd make any difference, but it might be useful if sqpoll submits requests often enough.

Quoted 18 lines
---
  io_uring/sqpoll.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
io_uring/sqpoll.c +1 -1
index b3722e5275e7..cc4a25136030 100644
@@ -176,7 +176,7 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries)
  	if (cap_entries && to_submit > IORING_SQPOLL_CAP_ENTRIES_VALUE)
  		to_submit = IORING_SQPOLL_CAP_ENTRIES_VALUE;
 
-	if (!wq_list_empty(&ctx->iopoll_list) || to_submit) {
+	if (to_submit || !wq_list_empty(&ctx->iopoll_list)) {
  		const struct cred *creds = NULL;
 
  		if (ctx->sq_creds != current_cred())
Signature · 1 line
Pavel Begunkov
maintainer Applied
#7 lore raw

Re: (subset) [PATCH 0/2] io_uring: minor sqpoll code refactoring

In reply to Olivier Langlois
Quoted 8 lines · Olivier Langlois

the first patch is minor micro-optimization that attempts to avoid a memory access if by testing a variable to is very likely already in a register

the second patch is also minor but this is much more serious. Without it, it is possible to have a ring that is configured to enable NAPI busy polling to NOT perform busy polling in specific conditions.

[...]

Applied, thanks!

[1/2] io_uring: micro optimization of __io_sq_thread() condition

      commit: 5fa7a249d5bc847876e04b91133d6b18d5c17140

Best regards,

Signature · 1 line
Jens Axboe
#5 lore raw

Re: [PATCH 2/2] io_uring: do the sqpoll napi busy poll outside the submission block

In reply to Pavel Begunkov
Quoted 10 lines · Pavel Begunkov

io_do_sqpoll_napi() returns 1 as long as there are napis in the list, iow even if there is no activity it'll spin almost forever (60s is forever) bypassing sq_thread_idle.

Let's not update sqt_spin here, if the user wants it to poll for longer it can pass a larger SQPOLL idle timeout value.

fair enough...

in that case, maybe the man page SQPOLL idle timeout description should include the mention that if NAPI busy loop is used, the idle timeout should be at least as large as gro_flush_timeout to meet NAPI requirement to not generate interrupts as described in

Documentation/networking/napi.rst
section "Software IRQ coalescing"

I have discovered this fact the hard way by having spent days to figure out how to do busy poll the right way.

this simple mention could save the trouble to many new users of the feature.

I'll rework the patch and send a new version in the few days.

Greetings,

#6 lore raw

Re: [PATCH 2/2] io_uring: do the sqpoll napi busy poll outside the submission block

Quoted 11 lines · Olivier Langlois
Quoted 10 lines · Pavel Begunkov

io_do_sqpoll_napi() returns 1 as long as there are napis in the list, iow even if there is no activity it'll spin almost forever (60s is forever) bypassing sq_thread_idle.

Let's not update sqt_spin here, if the user wants it to poll for longer it can pass a larger SQPOLL idle timeout value.

Quoted 6 lines

fair enough...

in that case, maybe the man page SQPOLL idle timeout description should include the mention that if NAPI busy loop is used, the idle timeout should be at least as large as gro_flush_timeout to meet NAPI requirement to not generate interrupts as described in

Would be great to have, I agree. We might also need to start a tips and tricks document, not like many people are looking at documentation.

Quoted 10 lines

Documentation/networking/napi.rst
section "Software IRQ coalescing"

I have discovered this fact the hard way by having spent days to figure out how to do busy poll the right way.

this simple mention could save the trouble to many new users of the feature.

I'll rework the patch and send a new version in the few days.

Awesome, thanks

Signature · 1 line
Pavel Begunkov