napi tracking strategy
v3 cover napi tracking strategy
the actual napi tracking strategy is inducing a non-negligeable overhead. Everytime a multishot poll is triggered or any poll armed, if the napi is enabled on the ring a lookup is performed to either add a new napi id into the napi_list or its timeout value is updated.
For many scenarios, this is overkill as the napi id list will be pretty much static most of the time. To address this common scenario, a new abstraction has been created following the common Linux kernel idiom of creating an abstract interface with a struct filled with function pointers.
the patch serie consist of very minor fixes followed by the core of the changes to implement the new feature.
v3 changes:
- address minor comments in patch #3
- replace the double for loop hash macro with the single loop list macro to iterate the napi elements in patch #2
- add __cold attribute to common_tracking_show_fdinfo() and napi_show_fdinfo()
v2 changes:
- extract small changes from the core changes to ease minor fixes backport
- totally remove the io_napi_tracking_ops interface
Olivier Langlois (3):
io_uring/napi: protect concurrent io_napi_entry timeout accesses io_uring/napi: fix io_napi_entry RCU accesses io_uring/napi: add static napi tracking strategy
include/linux/io_uring_types.h | 2 +- include/uapi/linux/io_uring.h | 32 +++++++- io_uring/fdinfo.c | 41 ++++++++++ io_uring/napi.c | 140 +++++++++++++++++++++++++-------- io_uring/napi.h | 15 +++- 5 files changed, 192 insertions(+), 38 deletions(-)
Signature · 1 line
2.46.1
PATCH v3 1/3 io_uring/napi: protect concurrent io_napi_entry timeout accesses
io_napi_entry timeout value can be updated while accessed from the poll functions.
Its concurrent accesses are wrapped with READ_ONCE()/WRITE_ONCE() macros to avoid incorrect compiler optimizations.
--- io_uring/napi.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
io_uring/napi.c +3 -3
@@ -60,7 +60,7 @@ void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock)rcu_read_lock (); e =io_napi_hash_find (hash_list, napi_id);if (e) { - e->timeout = jiffies +NAPI_TIMEOUT ; +WRITE_ONCE (e->timeout, jiffies +NAPI_TIMEOUT );rcu_read_unlock ();return ; }
@@ -92,7 +92,7 @@ static void __io_napi_remove_stale(struct io_ring_ctx *ctx)spin_lock (&ctx->napi_lock);hash_for_each (ctx->napi_ht, i, e, node) { -if (time_after (jiffies, e->timeout)) { +if (time_after (jiffies,READ_ONCE (e->timeout))) {list_del (&e->list);hash_del_rcu (&e->node);kfree_rcu (e, rcu);
@@ -150,7 +150,7 @@ static bool __io_napi_do_busy_loop(struct io_ring_ctx *ctx,napi_busy_loop_rcu (e->napi_id, loop_end, loop_end_arg, ctx->napi_prefer_busy_poll,BUSY_POLL_BUDGET ); -if (time_after (jiffies, e->timeout)) +if (time_after (jiffies,READ_ONCE (e->timeout))) is_stale = true; }
Signature · 1 line
2.46.1
PATCH v3 2/3 io_uring/napi: fix io_napi_entry RCU accesses
In reply to Olivier Langloiscorrect 3 RCU structures modifications that were not using the RCU functions to make their update.
--- io_uring/napi.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
io_uring/napi.c +11 -6
@@ -81,19 +81,24 @@ void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock) }hlist_add_tail_rcu (&e->node, hash_list); -list_add_tail (&e->list, &ctx->napi_list); +list_add_tail_rcu (&e->list, &ctx->napi_list);spin_unlock (&ctx->napi_lock); }static void __io_napi_remove_stale (struct io_ring_ctx *ctx) {struct io_napi_entry *e; -unsigned int i;spin_lock (&ctx->napi_lock); -hash_for_each (ctx->napi_ht, i, e, node) { +/* +* list_for_each_entry_safe() is not required as long as: +* 1. list_del_rcu() does not reset the deleted node next pointer +* 2. kfree_rcu() delays the memory freeing until the next quiescent +* state +*/ +list_for_each_entry (e, &ctx->napi_list, list) {if (time_after (jiffies,READ_ONCE (e->timeout))) { -list_del (&e->list); +list_del_rcu (&e->list);hash_del_rcu (&e->node);kfree_rcu (e, rcu); }
@@ -204,13 +209,13 @@ void io_napi_init(struct io_ring_ctx *ctx)void io_napi_free (struct io_ring_ctx *ctx) {struct io_napi_entry *e; -unsigned int i;spin_lock (&ctx->napi_lock); -hash_for_each (ctx->napi_ht, i, e, node) { +list_for_each_entry (e, &ctx->napi_list, list) {hash_del_rcu (&e->node);kfree_rcu (e, rcu); } +INIT_LIST_HEAD_RCU (&ctx->napi_list);spin_unlock (&ctx->napi_lock); }
Signature · 1 line
2.46.1
PATCH v3 3/3 io_uring/napi: add static napi tracking strategy
In reply to Olivier Langloisadd the static napi tracking strategy that allows the user to manually manage the napi ids list to busy poll and offload the ring from dynamically update the list.
--- include/linux/io_uring_types.h | 2 +- include/uapi/linux/io_uring.h | 32 ++++++++- io_uring/fdinfo.c | 41 ++++++++++++ io_uring/napi.c | 117 ++++++++++++++++++++++++++------- io_uring/napi.h | 15 +++-- 5 files changed, 178 insertions(+), 29 deletions(-)
include/linux/io_uring_types.h +1 -1
@@ -406,7 +406,7 @@ struct io_ring_ctx {/* napi busy poll default timeout */ ktime_t napi_busy_poll_dt;bool napi_prefer_busy_poll; -bool napi_enabled; +u8 napi_track_mode;DECLARE_HASHTABLE (napi_ht,4 );#endif
include/uapi/linux/io_uring.h +30 -2
@@ -728,12 +728,40 @@ struct io_uring_buf_status {__u32 resv[8 ]; }; +enum io_uring_napi_op { +/* register/ungister backward compatible opcode */ +IO_URING_NAPI_REGISTER_OP =0 , + +/* opcodes to update napi_list when static tracking is used */ +IO_URING_NAPI_STATIC_ADD_ID =1 , +IO_URING_NAPI_STATIC_DEL_ID =2 +}; + +enum io_uring_napi_tracking_strategy { +/* value must be 0 for backward compatibility */ +IO_URING_NAPI_TRACKING_DYNAMIC =0 , +IO_URING_NAPI_TRACKING_STATIC =1 , +IO_URING_NAPI_TRACKING_INACTIVE =255 +}; +/* argument for IORING_(UN)REGISTER_NAPI */ struct io_uring_napi {__u32 busy_poll_to;__u8 prefer_busy_poll; -__u8 pad[3 ]; -__u64 resv; + +/* a io_uring_napi_op value */ +__u8 opcode; +__u8 pad[2 ]; + +/* +* for IO_URING_NAPI_REGISTER_OP, it is a +* io_uring_napi_tracking_strategy value. +* +* for IO_URING_NAPI_STATIC_ADD_ID/IO_URING_NAPI_STATIC_DEL_ID +* it is the napi id to add/del from napi_list. +*/ +__u32 op_param; +__u32 resv; };/*
io_uring/fdinfo.c +41 -0
52 lines not on this page · raw message
io_uring/napi.c +95 -22
198 lines not on this page · raw message
io_uring/napi.h +11 -4
@@ -15,7 +15,7 @@ void io_napi_free(struct io_ring_ctx *ctx);int io_register_napi (struct io_ring_ctx *ctx,void __user *arg);int io_unregister_napi (struct io_ring_ctx *ctx,void __user *arg); -void __io_napi_add (struct io_ring_ctx *ctx,struct socket *sock); +int __io_napi_add_id (struct io_ring_ctx *ctx,unsigned int napi_id);void __io_napi_adjust_timeout (struct io_ring_ctx *ctx,struct io_wait_queue *iowq,ktime_t to_wait);
@@ -54,13 +54,20 @@ static inline void io_napi_add(struct io_kiocb *req) {struct io_ring_ctx *ctx = req->ctx;struct socket *sock; +struct sock *sk; -if (!READ_ONCE (ctx->napi_enabled)) +if (READ_ONCE (ctx->napi_track_mode) !=IO_URING_NAPI_TRACKING_DYNAMIC )return ; sock =sock_from_file (req->file); -if (sock) -__io_napi_add (ctx, sock); +if (!sock) +return ; + + sk = sock->sk; +if (!sk) +return ; + +__io_napi_add_id (ctx,READ_ONCE (sk->sk_napi_id)); }#else
Signature · 1 line
2.46.1
Jens Axboe
maintainerRe: [PATCH v3 3/3] io_uring/napi: add static napi tracking strategy
Quoted 3 lines · Olivier Langlois
add the static napi tracking strategy that allows the user to manually manage the napi ids list to busy poll and offload the ring from dynamically update the list.
Add the static napi tracking strategy. That allows the user to manually manage the napi ids list for busy polling, and eliminate the overhead of dynamically updating the list from the fast path.
Maybe?
Quoted 36 lines
index
b1e0e0d85349..6f0e40e1469c100644io_uring/fdinfo.c +29
-0@@ -46,6 +46,46 @@ static __cold int io_uring_show_cred(struct seq_file *m, unsigned int id,return 0 ; } +#ifdef CONFIG_NET_RX_BUSY_POLL +static __cold void common_tracking_show_fdinfo (struct io_ring_ctx *ctx, +struct seq_file *m, +const char *tracking_strategy) +{ +seq_puts (m,"NAPI:\tenabled\n" ); +seq_printf (m,"napi tracking:\t%s\n" , tracking_strategy); +seq_printf (m,"napi_busy_poll_dt:\t%llu\n" , ctx->napi_busy_poll_dt); +if (ctx->napi_prefer_busy_poll) +seq_puts (m,"napi_prefer_busy_poll:\ttrue\n" ); +else +seq_puts (m,"napi_prefer_busy_poll:\tfalse\n" ); +} + +static __cold void napi_show_fdinfo (struct io_ring_ctx *ctx, +struct seq_file *m) +{ +unsigned int mode =READ_ONCE (ctx->napi_track_mode); + +switch (mode) { +case IO_URING_NAPI_TRACKING_INACTIVE : +seq_puts (m,"NAPI:\tdisabled\n" ); +break ; +case IO_URING_NAPI_TRACKING_DYNAMIC : +common_tracking_show_fdinfo (ctx, m,"dynamic" ); +break ; +case IO_URING_NAPI_TRACKING_STATIC : +common_tracking_show_fdinfo (ctx, m,"static" ); +break ;
io_uring/fdinfo.c static __cold int io_uring_show_cred(struct seq_file *m, unsigned int id, · line 87
Quoted 10 lines
io_uring/fdinfo.c +10
-0+default : + seq_printf(m, "NAPI:\tunknown mode (%u)\n", mode); + } +} +#else +static inline void napi_show_fdinfo (struct io_ring_ctx *ctx, +struct seq_file *m) +{ +} +#endif
I think this code should go in napi.c, with the stub CONFIG_NET_RX_BUSY_POLL in napi.h. Not a huge deal.
This also conflicts with your previous napi patch adding fdinfo support. What kernel is this patchset based on? You should rebase it for-6.12/io_uring, then it should apply to the development branch going forward too.
Quoted 46 lines
io_uring/napi.c +4
-12@@ -38,22 +38,14 @@ static inline ktime_t net_to_ktime(unsigned long t)return ns_to_ktime (t <<10 ); } -void __io_napi_add (struct io_ring_ctx *ctx,struct socket *sock) +int __io_napi_add_id (struct io_ring_ctx *ctx,unsigned int napi_id) {struct hlist_head *hash_list; -unsigned int napi_id; -struct sock *sk;struct io_napi_entry *e; - sk = sock->sk; -if (!sk) -return ; - - napi_id =READ_ONCE (sk->sk_napi_id); -/* Non-NAPI IDs can be rejected. */ if (napi_id <MIN_NAPI_ID ) -return ; +return -EINVAL ; hash_list = &ctx->napi_ht[hash_min (napi_id,HASH_BITS (ctx->napi_ht))];@@ -62,13 +54,13 @@ void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock)if (e) {WRITE_ONCE (e->timeout, jiffies +NAPI_TIMEOUT );rcu_read_unlock (); -return ; +return -EEXIST ; }rcu_read_unlock (); e =kmalloc (sizeof (*e),GFP_NOWAIT );if (!e) -return ; +return -ENOMEM ; e->napi_id = napi_id; e->timeout = jiffies +NAPI_TIMEOUT ;
io_uring/napi.c void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock) · line 73
Quoted 6 lines
io_uring/napi.c +1
-1@@ -77,12 +69,37 @@ void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock)if (unlikely (io_napi_hash_find (hash_list, napi_id))) {spin_unlock (&ctx->napi_lock);kfree (e); -return ; +return -EEXIST ; }
You could abstract this out to a prep patch, having __io_napi_add() return an error value. That would leave the meat of your patch simpler and easier to review.
Quoted 13 lines
io_uring/napi.c +13
-0+static int __io_napi_del_id (struct io_ring_ctx *ctx,unsigned int napi_id) +{ +struct hlist_head *hash_list; +struct io_napi_entry *e; + +/* Non-NAPI IDs can be rejected. */ +if (napi_id <MIN_NAPI_ID ) +return -EINVAL ; + + hash_list = &ctx->napi_ht[hash_min (napi_id,HASH_BITS (ctx->napi_ht))]; +spin_lock (&ctx->napi_lock); + e =io_napi_hash_find (hash_list, napi_id); +if (unlikely (!e)) {
io_uring/napi.c void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock) · line 103
Quoted 10 lines
io_uring/napi.c +9
-0+spin_unlock (&ctx->napi_lock); +return -ENOENT ; + } + +list_del_rcu (&e->list); +hash_del_rcu (&e->node); +kfree_rcu (e, rcu); +spin_unlock (&ctx->napi_lock); +return 0 ; }
For new code, not a bad idea to use:
guard(spinlock)(&ctx->napi_lock);
Only one cleanup path here, but...
Quoted 20 lines
io_uring/napi.c +13
-2static void __io_napi_remove_stale (struct io_ring_ctx *ctx)@@ -141,8 +158,26 @@ static bool io_napi_busy_loop_should_end(void *data,return false; } -static bool __io_napi_do_busy_loop (struct io_ring_ctx *ctx, -void *loop_end_arg) +/* +* never report stale entries +*/ +static bool static_tracking_do_busy_loop (struct io_ring_ctx *ctx, +void *loop_end_arg) +{ +struct io_napi_entry *e; +bool (*loop_end)(void *,unsigned long ) =NULL ; + +if (loop_end_arg) + loop_end = io_napi_busy_loop_should_end; + +list_for_each_entry_rcu (e, &ctx->napi_list, list)
io_uring/napi.c static bool io_napi_busy_loop_should_end(void *data, · line 183
Quoted 10 lines
io_uring/napi.c +7
-0+napi_busy_loop_rcu (e->napi_id, loop_end, loop_end_arg, + ctx->napi_prefer_busy_poll,BUSY_POLL_BUDGET ); +return false; +} + +static bool dynamic_tracking_do_busy_loop (struct io_ring_ctx *ctx, +void *loop_end_arg) {struct io_napi_entry *e;bool (*loop_end)(void *,unsigned long ) =NULL ;
This is somewhat convoluted, but I think it'd be cleaner to have a prep patch that just passes in both loop_end and loop_end_arg to the caller? Some of this predates your changes here, but seems there's room for cleaning this up. What do you think?
Quoted 16 lines
io_uring/napi.h +2
-2@@ -54,13 +54,20 @@ static inline void io_napi_add(struct io_kiocb *req) {struct io_ring_ctx *ctx = req->ctx;struct socket *sock; +struct sock *sk; -if (!READ_ONCE (ctx->napi_enabled)) +if (READ_ONCE (ctx->napi_track_mode) !=IO_URING_NAPI_TRACKING_DYNAMIC )return ; sock =sock_from_file (req->file); -if (sock)
io_uring/napi.h static inline void io_napi_add(struct io_kiocb *req) · line 71
Quoted 10 lines
io_uring/napi.h +8
-1-__io_napi_add (ctx, sock); +if (!sock) +return ; + + sk = sock->sk; +if (!sk) +return ; + +__io_napi_add_id (ctx,READ_ONCE (sk->sk_napi_id)); }
I like having this follow the expected outcome, which is that sock and sk are valid.
sock = sock_from_file(req->file); if (sock && sock->sk) __io_napi_add_id(ctx, READ_ONCE(sock->sk->sk_napi_id));
or something like that. At least to me that's more readable.
Signature · 1 line
Jens Axboe
Re: [PATCH v3 3/3] io_uring/napi: add static napi tracking strategy
Quoted 6 lines · Jens Axboe
Quoted 4 lines · Olivier Langlois
add the static napi tracking strategy that allows the user to manually
manage the napi ids list to busy poll and offload the ring from dynamically update the list.
Quoted 7 lines
Add the static napi tracking strategy. That allows the user to manually
manage the napi ids list for busy polling, and eliminate the overhead of
dynamically updating the list from the fast path.Maybe?
ok
Quoted 55 lines
Quoted 41 lines
index
b1e0e0d85349..6f0e40e1469c100644io_uring/fdinfo.c +0
-0@@ -46,6 +46,46 @@ static __cold int io_uring_show_cred(struct seq_file *m, unsigned int id, return 0; }io_uring/fdinfo.c +2
-0+#ifdef CONFIG_NET_RX_BUSY_POLL +static __cold void common_tracking_show_fdinfo(struct io_ring_ctx*ctx,
io_uring/fdinfo.c +2
-0+ struct seq_file *m, + const char
*tracking_strategy)io_uring/fdinfo.c +4
-0+{ +seq_puts (m,"NAPI:\tenabled\n" ); +seq_printf (m,"napi tracking:\t%s\n" , tracking_strategy); + seq_printf(m, "napi_busy_poll_dt:\t%llu\n", ctx-Quoted 1 line
napi_busy_poll_dt);io_uring/fdinfo.c +21
-0+if (ctx->napi_prefer_busy_poll) +seq_puts (m,"napi_prefer_busy_poll:\ttrue\n" ); +else +seq_puts (m,"napi_prefer_busy_poll:\tfalse\n" ); +} + +static __cold void napi_show_fdinfo (struct io_ring_ctx *ctx, + struct seq_file *m) +{ +unsigned int mode =READ_ONCE (ctx->napi_track_mode); + +switch (mode) { +case IO_URING_NAPI_TRACKING_INACTIVE : +seq_puts (m,"NAPI:\tdisabled\n" ); +break ; +case IO_URING_NAPI_TRACKING_DYNAMIC : +common_tracking_show_fdinfo (ctx, m,"dynamic" ); +break ; +case IO_URING_NAPI_TRACKING_STATIC : +common_tracking_show_fdinfo (ctx, m,"static" ); +break ;io_uring/fdinfo.c static __cold int io_uring_show_cred(struct seq_file *m, unsigned int id, · line 87
Quoted 10 lines
io_uring/fdinfo.c +10
-0+default : + seq_printf(m, "NAPI:\tunknown mode (%u)\n", mode); + } +} +#else +static inline void napi_show_fdinfo (struct io_ring_ctx *ctx, + struct seq_file *m) +{ +} +#endif I think this code should go in
napi.c, with the stubCONFIG_NET_RX_BUSY_POLLinnapi.h. Not a huge deal.
Quoted 6 lines
This also conflicts with your previous napi patch adding fdinfo support.
What kernel is this patchset based on? You should rebase it for-6.12/io_uring, then it should apply to the development branch going
forward too.
The repo used is Linus head. Yes I was surprised to not see my own patch there. Kernel dev process is still a bit obscure to me... but I figured that the conflict would be very trivial to solve. Basically, simply drop the previous patch to replace it with this one.
but yes. I can rebase the patch into your repo. Can you provide me the URL of your repo?
About your suggestion to move stuff in napi.h. I did though about it. In fact, the code was there at some point before submission (or in v1). but I did conclude the exact opposite. That it would be cleaner to keep the fdinfo stuff together instead of polluting napi TU with it...
with this argument, if you still prefer to see that code in napi TU, I will move it there.
Quoted 55 lines
Quoted 55 lines
io_uring/napi.c +0
-0@@ -38,22 +38,14 @@ static inline ktime_t net_to_ktime(unsigned long t) return ns_to_ktime(t << 10); }io_uring/napi.c +1
-1-void __io_napi_add (struct io_ring_ctx *ctx,struct socket *sock) +int __io_napi_add_id(struct io_ring_ctx *ctx, unsigned int
napi_id)
{
structhlist_head*hash_list;io_uring/napi.c +0
-2-unsigned int napi_id; -struct sock *sk;struct
io_napi_entry*e;io_uring/napi.c +0
-6- sk = sock->sk; -if (!sk) -return ; - - napi_id =READ_ONCE (sk->sk_napi_id); -/* Non-NAPI IDs can be rejected. */
if (napi_id<MIN_NAPI_ID)io_uring/napi.c +1
-1-return ; +return -EINVAL ;
hash_list= &ctx->napi_ht[hash_min(napi_id, HASH_BITS(ctx-Quoted 1 line
napi_ht))];io_uring/napi.c +1
-1@@ -62,13 +54,13 @@ void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock) if (e) { WRITE_ONCE(e->timeout, jiffies + NAPI_TIMEOUT); rcu_read_unlock(); -return ; +return -EEXIST ; } rcu_read_unlock();
e =kmalloc(sizeof(*e), GFP_NOWAIT);
if (!e)io_uring/napi.c +1
-1-return ; +return -ENOMEM ;
e->napi_id=napi_id;
e->timeout= jiffies +NAPI_TIMEOUT;io_uring/napi.c +0
-0@@ -77,12 +69,37 @@ void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock) if (unlikely(io_napi_hash_find(hash_list, napi_id))) { spin_unlock(&ctx->napi_lock); kfree(e);
Quoted 8 lines
io_uring/napi.c void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock) · line 58
Quoted 3 lines
io_uring/napi.c +1
-1-return ; +return -EEXIST ; }You could abstract this out to a prep patch, having
__io_napi_add()return an error value. That would leave the meat of your patch simpler
and easier to review.
ok
Quoted 29 lines
Quoted 16 lines
io_uring/napi.c +13
-0+static int __io_napi_del_id (struct io_ring_ctx *ctx,unsigned int napi_id) +{ +struct hlist_head *hash_list; +struct io_napi_entry *e; + +/* Non-NAPI IDs can be rejected. */ +if (napi_id <MIN_NAPI_ID ) +return -EINVAL ; + + hash_list = &ctx->napi_ht[hash_min (napi_id,HASH_BITS (ctx- >napi_ht))]; +spin_lock (&ctx->napi_lock); + e =io_napi_hash_find (hash_list, napi_id); +if (unlikely (!e)) {io_uring/napi.c void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock) · line 103
Quoted 10 lines
io_uring/napi.c +9
-0+spin_unlock (&ctx->napi_lock); +return -ENOENT ; + } + +list_del_rcu (&e->list); +hash_del_rcu (&e->node); +kfree_rcu (e, rcu); +spin_unlock (&ctx->napi_lock); +return 0 ; }For new code, not a bad idea to use:
Quoted 3 lines
guard(spinlock)(&ctx->napi_lock);Only one cleanup path here, but...
first time I hear about this... Some sort of RAII where the compiler adds automatically unlock function call at every exit points?
ok, I'll look into it
Quoted 33 lines
Quoted 33 lines
io_uring/napi.c +0
-0static void __io_napi_remove_stale(struct io_ring_ctx *ctx)@@ -141,8 +158,26 @@ static bool io_napi_busy_loop_should_end(void *data, return false; }io_uring/napi.c +14
-2-static bool __io_napi_do_busy_loop (struct io_ring_ctx *ctx, - void *loop_end_arg) +/* +* never report stale entries +*/ +static bool static_tracking_do_busy_loop (struct io_ring_ctx *ctx, +void *loop_end_arg) +{ +struct io_napi_entry *e; +bool (*loop_end)(void *,unsigned long ) =NULL ; + +if (loop_end_arg) + loop_end = io_napi_busy_loop_should_end; + +list_for_each_entry_rcu (e, &ctx->napi_list, list) + napi_busy_loop_rcu(e->napi_id, loop_end,
loop_end_arg,io_uring/napi.c +1
-0+ ctx->napi_prefer_busy_poll,
BUSY_POLL_BUDGET);io_uring/napi.c +5
-0+return false; +} + +static bool dynamic_tracking_do_busy_loop (struct io_ring_ctx *ctx, + void *loop_end_arg){
structio_napi_entry*e;
Quoted 8 lines
Quoted 1 line
bool (*loop_end)(void *, unsigned long) = NULL;
This is somewhat convoluted, but I think it'd be cleaner to have a prep
patch that just passes in bothloop_endandloop_end_argto the caller?
Some of this predates your changes here, but seems there's room for cleaning this up. What do you think?
I am not sure that I see your point that it would make the code cleaner. TBH, I did not fully understood what was the point of the whole loop_end business thing but since it was unrelated to my changes, I did leave that thing undisturbed.
I'll take a second look at it to see if I can clean this up a bit...
Quoted 34 lines
Quoted 28 lines
io_uring/napi.h +1
-0@@ -54,13 +54,20 @@ static inline void io_napi_add(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; struct socket *sock; +struct sock *sk;io_uring/napi.h +1
-1-if (!READ_ONCE (ctx->napi_enabled)) + if (READ_ONCE(ctx->napi_track_mode) !=
IO_URING_NAPI_TRACKING_DYNAMIC)
return;
sock =sock_from_file(req->file);io_uring/napi.h +8
-2-if (sock) -__io_napi_add (ctx, sock); +if (!sock) +return ; + + sk = sock->sk; +if (!sk) +return ; + +__io_napi_add_id (ctx,READ_ONCE (sk->sk_napi_id));Quoted 1 line
}
I like having this follow the expected outcome, which is that sock and
sk are valid.
Quoted 7 lines
sock = sock_from_file(req->file); if (sock && sock->sk) __io_napi_add_id(ctx, READ_ONCE(sock->sk-Quoted 1 line
sk_napi_id));or something like that. At least to me that's more readable.
ok.
All those points will be addressed in v4
Jens Axboe
maintainerQuoted 9 lines · Olivier Langlois
the actual napi tracking strategy is inducing a non-negligeable overhead. Everytime a multishot poll is triggered or any poll armed, if the napi is enabled on the ring a lookup is performed to either add a new napi id into the
napi_listor its timeout value is updated.For many scenarios, this is overkill as the napi id list will be pretty much static most of the time. To address this common scenario, a new abstraction has been created following the common Linux kernel idiom of creating an abstract interface with a struct filled with function pointers.
This paragraph seems to be a remnant of the v1 implementation?
Signature · 1 line
Jens Axboe
Quoted 18 lines · Jens Axboe
Quoted 8 lines · Olivier Langlois
the actual napi tracking strategy is inducing a non-negligeable overhead.
Everytime a multishot poll is triggered or any poll armed, if the napi is
enabled on the ring a lookup is performed to either add a new napi id into
thenapi_listor its timeout value is updated.Quoted 8 lines
For many scenarios, this is overkill as the napi id list will be pretty
much static most of the time. To address this common scenario, a new
abstraction has been created following the common Linux kernel idiom of
creating an abstract interface with a struct filled with function pointers.
Quoted 2 lines
This paragraph seems to be a remnant of the v1 implementation?
you are right. At least the last sentence is. Is the cover letter injected somewhere in any way once the patch is accepted? Would that detail alone justify the creation of a v4?
I am taking note of the detail and I will correct this if there is a need for a v4.
Jens Axboe
maintainerQuoted 19 lines · Olivier Langlois
Quoted 18 lines · Jens Axboe
Quoted 16 lines · Olivier Langlois
the actual napi tracking strategy is inducing a non-negligeable overhead.
Everytime a multishot poll is triggered or any poll armed, if the napi is
enabled on the ring a lookup is performed to either add a new napi id into
thenapi_listor its timeout value is updated.For many scenarios, this is overkill as the napi id list will be pretty
much static most of the time. To address this common scenario, a new
abstraction has been created following the common Linux kernel idiom of
creating an abstract interface with a struct filled with function pointers.
Quoted 8 lines
Quoted 2 lines
This paragraph seems to be a remnant of the v1 implementation?
you are right. At least the last sentence is. Is the cover letter injected somewhere in any way once the patch is accepted? Would that detail alone justify the creation of a v4?
I am taking note of the detail and I will correct this if there is a need for a v4.
Right just make a note of it if you need to send a v4 for other reasons, cover letter doesn't go anywhere in the git log - but is, however, linked off the git commits in terms of the Link in there. But since we now have this discussion in there as well clarifying it, that is fine.
Doing conferences in Europe until end of next week, I'll see if I can do some thorough looking at this version during that, or worst case end of that.
Signature · 1 line
Jens Axboe