Skip to contentSkip to conversation

napi tracking strategy

PATCH Applied Olivier Langlois 2 participants 9 messages Oct 13, 2024 – Nov 4, 2024 io_uring

lore
Share & tools
lore
#0 lore raw

v4 cover napi tracking strategy

the actual napi tracking strategy is inducing a non-negligeable overhead. Everytime a multishot poll is triggered or any poll armed, if the napi is enabled on the ring a lookup is performed to either add a new napi id into the napi_list or its timeout value is updated.

For many scenarios, this is overkill as the napi id list will be pretty
much static most of the time. To address this common scenario, the concept of io_uring_napi_tracking_strategy has been created. the tracking strategy can be specified when io_register_napi() is called.

To keep backward compatibility, the legacy strategy IO_URING_NAPI_TRACKING_DYNAMIC is assigned the value 0 so that existing code using io_uring napi busy polling continue working as before. If IO_URING_NAPI_TRACKING_STATIC is provided, io_napi_add() becomes a noop function and the responsability to update the napi devices list is given to the user by calling io_register_napi() with the opcode value of IO_URING_NAPI_STATIC_ADD_ID or IO_URING_NAPI_STATIC_DEL_ID.

the NAPI ids used by a process can be discovered by calling getsockopt(fd, SOL_SOCKET, SO_INCOMING_NAPI_ID, &napi_id, &len)

the patch serie consist of very minor fixes followed by the core of the changes to implement the new feature.

v4 changes:
- improve cover letter text
- rebase patch on for-6.13/io_uring
- create a prep-patch for the __io_napi_add refactoring change
- create a prep-patch for the Scope-Based Resource Management refactoring
- create a prep-patch for the __io_napi_do_busy_loop cleanup
- adress io_napi_add() code review comment

v3 changes:
- address minor comments in patch #3
- replace the double for loop hash macro with the single loop list macro to iterate the napi elements in patch #2
- add __cold attribute to common_tracking_show_fdinfo() and napi_show_fdinfo()

v2 changes:
- extract small changes from the core changes to ease minor fixes backport
- totally remove the io_napi_tracking_ops interface

Olivier Langlois (6):
io_uring/napi: protect concurrent io_napi_entry timeout accesses io_uring/napi: fix io_napi_entry RCU accesses io_uring/napi: improve __io_napi_add
io_uring/napi: Use lock guards
io_uring/napi: clean up __io_napi_do_busy_loop io_uring/napi: add static napi tracking strategy

 include/linux/io_uring_types.h |   2 +-
 include/uapi/linux/io_uring.h  |  32 +++++-
 io_uring/fdinfo.c              |  54 +++++++---
 io_uring/napi.c                | 184 +++++++++++++++++++++++----------
 io_uring/napi.h                |   8 +-
 5 files changed, 207 insertions(+), 73 deletions(-)
Signature · 1 line
2.47.0
#1 lore raw

PATCH v4 1/6 io_uring/napi: protect concurrent io_napi_entry timeout accesses

io_napi_entry timeout value can be updated while accessed from the poll functions.

Its concurrent accesses are wrapped with READ_ONCE()/WRITE_ONCE() macros to avoid incorrect compiler optimizations.

---
 io_uring/napi.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io_uring/napi.c +3 -3
index d0cf694d0172..dda2e083fb5d 100644
@@ -60,7 +60,7 @@ void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock)
 	rcu_read_lock();
 	e = io_napi_hash_find(hash_list, napi_id);
 	if (e) {
-		e->timeout = jiffies + NAPI_TIMEOUT;
+		WRITE_ONCE(e->timeout, jiffies + NAPI_TIMEOUT);
 		rcu_read_unlock();
 		return;
 	}
@@ -92,7 +92,7 @@ static void __io_napi_remove_stale(struct io_ring_ctx *ctx)
 
 	spin_lock(&ctx->napi_lock);
 	hash_for_each(ctx->napi_ht, i, e, node) {
-		if (time_after(jiffies, e->timeout)) {
+		if (time_after(jiffies, READ_ONCE(e->timeout))) {
 			list_del(&e->list);
 			hash_del_rcu(&e->node);
 			kfree_rcu(e, rcu);
@@ -150,7 +150,7 @@ static bool __io_napi_do_busy_loop(struct io_ring_ctx *ctx,
 		napi_busy_loop_rcu(e->napi_id, loop_end, loop_end_arg,
 				   ctx->napi_prefer_busy_poll, BUSY_POLL_BUDGET);
 
-		if (time_after(jiffies, e->timeout))
+		if (time_after(jiffies, READ_ONCE(e->timeout)))
 			is_stale = true;
 	}
 
Signature · 1 line
2.47.0
#2 lore raw

PATCH v4 2/6 io_uring/napi: fix io_napi_entry RCU accesses

In reply to Olivier Langlois

correct 3 RCU structures modifications that were not using the RCU functions to make their update.

---
 io_uring/napi.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
io_uring/napi.c +11 -6
index dda2e083fb5d..921de9de8d75 100644
@@ -81,19 +81,24 @@ void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock)
 	}
 
 	hlist_add_tail_rcu(&e->node, hash_list);
-	list_add_tail(&e->list, &ctx->napi_list);
+	list_add_tail_rcu(&e->list, &ctx->napi_list);
 	spin_unlock(&ctx->napi_lock);
 }
 
 static void __io_napi_remove_stale(struct io_ring_ctx *ctx)
 {
 	struct io_napi_entry *e;
-	unsigned int i;
 
 	spin_lock(&ctx->napi_lock);
-	hash_for_each(ctx->napi_ht, i, e, node) {
+	/*
+	 * list_for_each_entry_safe() is not required as long as:
+	 * 1. list_del_rcu() does not reset the deleted node next pointer
+	 * 2. kfree_rcu() delays the memory freeing until the next quiescent
+	 *    state
+	 */
+	list_for_each_entry(e, &ctx->napi_list, list) {
 		if (time_after(jiffies, READ_ONCE(e->timeout))) {
-			list_del(&e->list);
+			list_del_rcu(&e->list);
 			hash_del_rcu(&e->node);
 			kfree_rcu(e, rcu);
 		}
@@ -204,13 +209,13 @@ void io_napi_init(struct io_ring_ctx *ctx)
 void io_napi_free(struct io_ring_ctx *ctx)
 {
 	struct io_napi_entry *e;
-	unsigned int i;
 
 	spin_lock(&ctx->napi_lock);
-	hash_for_each(ctx->napi_ht, i, e, node) {
+	list_for_each_entry(e, &ctx->napi_list, list) {
 		hash_del_rcu(&e->node);
 		kfree_rcu(e, rcu);
 	}
+	INIT_LIST_HEAD_RCU(&ctx->napi_list);
 	spin_unlock(&ctx->napi_lock);
 }
 
Signature · 1 line
2.47.0
#3 lore raw

PATCH v4 3/6 io_uring/napi: improve __io_napi_add

In reply to Olivier Langlois

1. move the sock->sk pointer validity test outside the function to avoid the function call overhead and to make the function more more reusable
2. change its name to __io_napi_add_id to be more precise about it is doing
3. return an error code to report errors

---
 io_uring/napi.c | 19 ++++++-------------
 io_uring/napi.h |  6 +++---
 2 files changed, 9 insertions(+), 16 deletions(-)
io_uring/napi.c +6 -13
index 921de9de8d75..5e2299e7ff8e 100644
@@ -38,22 +38,14 @@ static inline ktime_t net_to_ktime(unsigned long t)
 	return ns_to_ktime(t << 10);
 }
 
-void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock)
+int __io_napi_add_id(struct io_ring_ctx *ctx, unsigned int napi_id)
 {
 	struct hlist_head *hash_list;
-	unsigned int napi_id;
-	struct sock *sk;
 	struct io_napi_entry *e;
 
-	sk = sock->sk;
-	if (!sk)
-		return;
-
-	napi_id = READ_ONCE(sk->sk_napi_id);
-
 	/* Non-NAPI IDs can be rejected. */
 	if (napi_id < MIN_NAPI_ID)
-		return;
+		return -EINVAL;
 
 	hash_list = &ctx->napi_ht[hash_min(napi_id, HASH_BITS(ctx->napi_ht))];
 
@@ -62,13 +54,13 @@ void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock)
 	if (e) {
 		WRITE_ONCE(e->timeout, jiffies + NAPI_TIMEOUT);
 		rcu_read_unlock();
-		return;
+		return -EEXIST;
 	}
 	rcu_read_unlock();
 
 	e = kmalloc(sizeof(*e), GFP_NOWAIT);
 	if (!e)
-		return;
+		return -ENOMEM;
 
 	e->napi_id = napi_id;
 	e->timeout = jiffies + NAPI_TIMEOUT;
@@ -77,12 +69,13 @@ void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock)
 	if (unlikely(io_napi_hash_find(hash_list, napi_id))) {
 		spin_unlock(&ctx->napi_lock);
 		kfree(e);
-		return;
+		return -EEXIST;
 	}
 
 	hlist_add_tail_rcu(&e->node, hash_list);
 	list_add_tail_rcu(&e->list, &ctx->napi_list);
 	spin_unlock(&ctx->napi_lock);
+	return 0;
 }
 
 static void __io_napi_remove_stale(struct io_ring_ctx *ctx)
io_uring/napi.h +3 -3
index fd275ef0456d..4ae622f37b30 100644
@@ -15,7 +15,7 @@ void io_napi_free(struct io_ring_ctx *ctx);
 int io_register_napi(struct io_ring_ctx *ctx, void __user *arg);
 int io_unregister_napi(struct io_ring_ctx *ctx, void __user *arg);
 
-void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock);
+int __io_napi_add_id(struct io_ring_ctx *ctx, unsigned int napi_id);
 
 void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq);
 int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx);
@@ -48,8 +48,8 @@ static inline void io_napi_add(struct io_kiocb *req)
 		return;
 
 	sock = sock_from_file(req->file);
-	if (sock)
-		__io_napi_add(ctx, sock);
+	if (sock && sock->sk)
+		__io_napi_add_id(ctx, READ_ONCE(sock->sk->sk_napi_id));
 }
 
 #else
Signature · 1 line
2.47.0
#4 lore raw

PATCH v4 4/6 io_uring/napi: Use lock guards

In reply to Olivier Langlois

Convert napi locks to use the shiny new Scope-Based Resource Management machinery.

---
 io_uring/napi.c | 40 +++++++++++++++++++++-------------------
 1 file changed, 21 insertions(+), 19 deletions(-)
io_uring/napi.c +21 -19
index 5e2299e7ff8e..6d5fdd397f2f 100644
@@ -49,14 +49,13 @@ int __io_napi_add_id(struct io_ring_ctx *ctx, unsigned int napi_id)
 
 	hash_list = &ctx->napi_ht[hash_min(napi_id, HASH_BITS(ctx->napi_ht))];
 
-	rcu_read_lock();
-	e = io_napi_hash_find(hash_list, napi_id);
-	if (e) {
-		WRITE_ONCE(e->timeout, jiffies + NAPI_TIMEOUT);
-		rcu_read_unlock();
-		return -EEXIST;
+	scoped_guard(rcu) {
+		e = io_napi_hash_find(hash_list, napi_id);
+		if (e) {
+			WRITE_ONCE(e->timeout, jiffies + NAPI_TIMEOUT);
+			return -EEXIST;
+		}
 	}
-	rcu_read_unlock();
 
 	e = kmalloc(sizeof(*e), GFP_NOWAIT);
 	if (!e)
@@ -65,6 +64,10 @@ int __io_napi_add_id(struct io_ring_ctx *ctx, unsigned int napi_id)
 	e->napi_id = napi_id;
 	e->timeout = jiffies + NAPI_TIMEOUT;
 
+	/*
+	 * guard(spinlock) is not used to manually unlock it before calling
+	 * kfree()
+	 */
 	spin_lock(&ctx->napi_lock);
 	if (unlikely(io_napi_hash_find(hash_list, napi_id))) {
 		spin_unlock(&ctx->napi_lock);
@@ -82,7 +85,7 @@ static void __io_napi_remove_stale(struct io_ring_ctx *ctx)
 {
 	struct io_napi_entry *e;
 
-	spin_lock(&ctx->napi_lock);
+	guard(spinlock)(&ctx->napi_lock);
 	/*
 	 * list_for_each_entry_safe() is not required as long as:
 	 * 1. list_del_rcu() does not reset the deleted node next pointer
@@ -96,7 +99,6 @@ static void __io_napi_remove_stale(struct io_ring_ctx *ctx)
 			kfree_rcu(e, rcu);
 		}
 	}
-	spin_unlock(&ctx->napi_lock);
 }
 
 static inline void io_napi_remove_stale(struct io_ring_ctx *ctx, bool is_stale)
@@ -168,11 +170,12 @@ static void io_napi_blocking_busy_loop(struct io_ring_ctx *ctx,
 	if (list_is_singular(&ctx->napi_list))
 		loop_end_arg = iowq;
 
-	rcu_read_lock();
-	do {
-		is_stale = __io_napi_do_busy_loop(ctx, loop_end_arg);
-	} while (!io_napi_busy_loop_should_end(iowq, start_time) && !loop_end_arg);
-	rcu_read_unlock();
+	scoped_guard(rcu) {
+		do {
+			is_stale = __io_napi_do_busy_loop(ctx, loop_end_arg);
+		} while (!io_napi_busy_loop_should_end(iowq, start_time) &&
+			 !loop_end_arg);
+	}
 
 	io_napi_remove_stale(ctx, is_stale);
 }
@@ -203,13 +206,12 @@ void io_napi_free(struct io_ring_ctx *ctx)
 {
 	struct io_napi_entry *e;
 
-	spin_lock(&ctx->napi_lock);
+	guard(spinlock)(&ctx->napi_lock);
 	list_for_each_entry(e, &ctx->napi_list, list) {
 		hash_del_rcu(&e->node);
 		kfree_rcu(e, rcu);
 	}
 	INIT_LIST_HEAD_RCU(&ctx->napi_list);
-	spin_unlock(&ctx->napi_lock);
 }
 
 /*
@@ -305,9 +307,9 @@ int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx)
 	if (list_empty_careful(&ctx->napi_list))
 		return 0;
 
-	rcu_read_lock();
-	is_stale = __io_napi_do_busy_loop(ctx, NULL);
-	rcu_read_unlock();
+	scoped_guard(rcu) {
+		is_stale = __io_napi_do_busy_loop(ctx, NULL);
+	}
 
 	io_napi_remove_stale(ctx, is_stale);
 	return 1;
Signature · 1 line
2.47.0
#5 lore raw

PATCH v4 5/6 io_uring/napi: clean up __io_napi_do_busy_loop

In reply to Olivier Langlois

__io_napi_do_busy_loop now requires to have loop_end in its parameters. This makes the code cleaner and also has the benefit of removing a branch since the only caller not passing NULL for loop_end_arg is also setting the value conditionally.

---
 io_uring/napi.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
io_uring/napi.c +8 -7
index 6d5fdd397f2f..1de1543d8034 100644
@@ -137,15 +137,12 @@ static bool io_napi_busy_loop_should_end(void *data,
 }
 
 static bool __io_napi_do_busy_loop(struct io_ring_ctx *ctx,
+				   bool (*loop_end)(void *, unsigned long),
 				   void *loop_end_arg)
 {
 	struct io_napi_entry *e;
-	bool (*loop_end)(void *, unsigned long) = NULL;
 	bool is_stale = false;
 
-	if (loop_end_arg)
-		loop_end = io_napi_busy_loop_should_end;
-
 	list_for_each_entry_rcu(e, &ctx->napi_list, list) {
 		napi_busy_loop_rcu(e->napi_id, loop_end, loop_end_arg,
 				   ctx->napi_prefer_busy_poll, BUSY_POLL_BUDGET);
@@ -161,18 +158,22 @@ static void io_napi_blocking_busy_loop(struct io_ring_ctx *ctx,
 				       struct io_wait_queue *iowq)
 {
 	unsigned long start_time = busy_loop_current_time();
+	bool (*loop_end)(void *, unsigned long) = NULL;
 	void *loop_end_arg = NULL;
 	bool is_stale = false;
 
 	/* Singular lists use a different napi loop end check function and are
 	 * only executed once.
 	 */
-	if (list_is_singular(&ctx->napi_list))
+	if (list_is_singular(&ctx->napi_list)) {
+		loop_end = io_napi_busy_loop_should_end;
 		loop_end_arg = iowq;
+	}
 
 	scoped_guard(rcu) {
 		do {
-			is_stale = __io_napi_do_busy_loop(ctx, loop_end_arg);
+			is_stale = __io_napi_do_busy_loop(ctx, loop_end,
+							  loop_end_arg);
 		} while (!io_napi_busy_loop_should_end(iowq, start_time) &&
 			 !loop_end_arg);
 	}
@@ -308,7 +309,7 @@ int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx)
 		return 0;
 
 	scoped_guard(rcu) {
-		is_stale = __io_napi_do_busy_loop(ctx, NULL);
+		is_stale = __io_napi_do_busy_loop(ctx, NULL, NULL);
 	}
 
 	io_napi_remove_stale(ctx, is_stale);
Signature · 1 line
2.47.0
#6 lore raw

PATCH v4 6/6 io_uring/napi: add static napi tracking strategy

In reply to Olivier Langlois

Add the static napi tracking strategy. That allows the user to manually manage the napi ids list for busy polling, and eliminate the overhead of dynamically updating the list from the fast path.

---
 include/linux/io_uring_types.h |  2 +-
 include/uapi/linux/io_uring.h  | 32 ++++++++++-
 io_uring/fdinfo.c              | 54 ++++++++++++++-----
 io_uring/napi.c                | 97 ++++++++++++++++++++++++++++++----
 io_uring/napi.h                |  2 +-
 5 files changed, 160 insertions(+), 27 deletions(-)
include/linux/io_uring_types.h +1 -1
index 4b9ba523978d..f435433f29a3 100644
@@ -409,7 +409,7 @@ struct io_ring_ctx {
 	/* napi busy poll default timeout */
 	ktime_t			napi_busy_poll_dt;
 	bool			napi_prefer_busy_poll;
-	bool			napi_enabled;
+	u8			napi_track_mode;
 
 	DECLARE_HASHTABLE(napi_ht, 4);
 #endif
include/uapi/linux/io_uring.h +30 -2
index 86cb385fe0b5..99a7a082421e 100644
@@ -771,12 +771,40 @@ struct io_uring_buf_status {
 	__u32	resv[8];
 };
 
+enum io_uring_napi_op {
+	/* register/ungister backward compatible opcode */
+	IO_URING_NAPI_REGISTER_OP = 0,
+
+	/* opcodes to update napi_list when static tracking is used */
+	IO_URING_NAPI_STATIC_ADD_ID = 1,
+	IO_URING_NAPI_STATIC_DEL_ID = 2
+};
+
+enum io_uring_napi_tracking_strategy {
+	/* value must be 0 for backward compatibility */
+	IO_URING_NAPI_TRACKING_DYNAMIC = 0,
+	IO_URING_NAPI_TRACKING_STATIC = 1,
+	IO_URING_NAPI_TRACKING_INACTIVE = 255
+};
+
 /* argument for IORING_(UN)REGISTER_NAPI */
 struct io_uring_napi {
 	__u32	busy_poll_to;
 	__u8	prefer_busy_poll;
-	__u8	pad[3];
-	__u64	resv;
+
+	/* a io_uring_napi_op value */
+	__u8	opcode;
+	__u8	pad[2];
+
+	/*
+	 * for IO_URING_NAPI_REGISTER_OP, it is a
+	 * io_uring_napi_tracking_strategy value.
+	 *
+	 * for IO_URING_NAPI_STATIC_ADD_ID/IO_URING_NAPI_STATIC_DEL_ID
+	 * it is the napi id to add/del from napi_list.
+	 */
+	__u32	op_param;
+	__u32	resv;
 };
 
 /*
io_uring/fdinfo.c +41 -13

65 lines not on this page · raw message

io_uring/napi.c +87 -10

150 lines not on this page · raw message

io_uring/napi.h +1 -1
index 4ae622f37b30..fa742f42e09b 100644
@@ -44,7 +44,7 @@ static inline void io_napi_add(struct io_kiocb *req)
 	struct io_ring_ctx *ctx = req->ctx;
 	struct socket *sock;
 
-	if (!READ_ONCE(ctx->napi_enabled))
+	if (READ_ONCE(ctx->napi_track_mode) != IO_URING_NAPI_TRACKING_DYNAMIC)
 		return;
 
 	sock = sock_from_file(req->file);
Signature · 1 line
2.47.0
maintainer Applied
#7 lore raw
In reply to Olivier Langlois
Quoted 10 lines · Olivier Langlois

the actual napi tracking strategy is inducing a non-negligeable overhead. Everytime a multishot poll is triggered or any poll armed, if the napi is enabled on the ring a lookup is performed to either add a new napi id into the napi_list or its timeout value is updated.

For many scenarios, this is overkill as the napi id list will be pretty
much static most of the time. To address this common scenario, the concept of io_uring_napi_tracking_strategy has been created. the tracking strategy can be specified when io_register_napi() is called.

[...]

Applied, thanks!

[1/6] io_uring/napi: protect concurrent io_napi_entry timeout accesses

      commit: d54db33e51090f68645fecb252a3ad22f28512cf
[2/6] io_uring/napi: fix io_napi_entry RCU accesses
      commit: 613dbde4863699fe88e601ddd7315f04c1aa3239
[3/6] io_uring/napi: improve __io_napi_add
      commit: e17bd6f1106d8c45e186a52d3ac0412f17e657c3
[4/6] io_uring/napi: Use lock guards
      commit: 6710c043c8e9d8fa9649fffd8855e3ad883bf001
[5/6] io_uring/napi: clean up __io_napi_do_busy_loop
      commit: c596060fbe5a1c094d46d8f7191a866879fe6672
[6/6] io_uring/napi: add static napi tracking strategy
      commit: cc909543d239912669b14250e796bbd877f8128a

Best regards,

Signature · 1 line
Jens Axboe
maintainer
#8 lore raw
Quoted 16 lines · Jens Axboe
Quoted 10 lines · Olivier Langlois

the actual napi tracking strategy is inducing a non-negligeable overhead. Everytime a multishot poll is triggered or any poll armed, if the napi is enabled on the ring a lookup is performed to either add a new napi id into the napi_list or its timeout value is updated.

For many scenarios, this is overkill as the napi id list will be pretty
much static most of the time. To address this common scenario, the concept of io_uring_napi_tracking_strategy has been created. the tracking strategy can be specified when io_register_napi() is called.

[...]

Applied, thanks!

[1/6] io_uring/napi: protect concurrent io_napi_entry timeout accesses

Quoted 11 lines
      commit: d54db33e51090f68645fecb252a3ad22f28512cf
[2/6] io_uring/napi: fix io_napi_entry RCU accesses
      commit: 613dbde4863699fe88e601ddd7315f04c1aa3239
[3/6] io_uring/napi: improve __io_napi_add
      commit: e17bd6f1106d8c45e186a52d3ac0412f17e657c3
[4/6] io_uring/napi: Use lock guards
      commit: 6710c043c8e9d8fa9649fffd8855e3ad883bf001
[5/6] io_uring/napi: clean up __io_napi_do_busy_loop
      commit: c596060fbe5a1c094d46d8f7191a866879fe6672
[6/6] io_uring/napi: add static napi tracking strategy
      commit: cc909543d239912669b14250e796bbd877f8128a

Finally got around to this one, apologies for the delay. It looks really nice at this point.

I'm assuming you have a liburing branch too, with the new register parts documented, and probably helpers for setting it up? Would be nice to have an addition to the napi test case as well, so this can get exercised as part of the usual testing.

Signature · 1 line
Jens Axboe