io_uring: fix bug in slow unregistering of nodes
PATCH 1/1 io_uring: fix bug in slow unregistering of nodes
In some cases io_rsrc_ref_quiesce will call io_rsrc_node_switch_start, and then immediately flush the delayed work queue &ctx->rsrc_put_work.
However the percpu_ref_put does not immediately destroy the node, it will be called asynchronously via RCU. That ends up with io_rsrc_node_ref_zero only being called after rsrc_put_work has been flushed, and so the process ends up sleeping for 1 second unnecessarily.
This patch executes the put code immediately if we are busy quiescing.
--- fs/io_uring.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
fs/io_uring.c +6 -1
@@ -7822,10 +7822,15 @@ static __cold void io_rsrc_node_ref_zero(struct percpu_ref *ref)struct io_ring_ctx *ctx = node->rsrc_data->ctx;unsigned long flags;bool first_add = false; +unsigned long delay =HZ ;spin_lock_irqsave (&ctx->rsrc_ref_lock, flags); node->done = true; +/* if we are mid-quiesce then do not delay */ +if (node->rsrc_data->quiesce) + delay =0 ; +while (!list_empty (&ctx->rsrc_ref_list)) { node =list_first_entry (&ctx->rsrc_ref_list, struct io_rsrc_node, node);
@@ -7838,7 +7843,7 @@ static __cold void io_rsrc_node_ref_zero(struct percpu_ref *ref)spin_unlock_irqrestore (&ctx->rsrc_ref_lock, flags);if (first_add) -mod_delayed_work (system_wq, &ctx->rsrc_put_work,HZ ); +mod_delayed_work (system_wq, &ctx->rsrc_put_work, delay); }static struct io_rsrc_node *io_rsrc_node_alloc (struct io_ring_ctx *ctx)
Signature · 1 line
2.30.2
Jens Axboe
maintainerQuoted 10 lines · Dylan Yudaken
In some cases
io_rsrc_ref_quiescewill callio_rsrc_node_switch_start, and then immediately flush the delayed work queue&ctx->rsrc_put_work.However the
percpu_ref_putdoes not immediately destroy the node, it will be called asynchronously via RCU. That ends up withio_rsrc_node_ref_zeroonly being called afterrsrc_put_workhas been flushed, and so the process ends up sleeping for 1 second unnecessarily.This patch executes the put code immediately if we are busy quiescing.
Looks good to me, and as far as I can tell, this bug was introduced by:
commit 4a38aed2a0a729ccecd84dca5b76d827b9e1294d
Author: Jens Axboe <[email protected]> Date: Thu May 14 17:21:15 2020 -0600
io_uring: batch reap of dead file registrations
so I'll add a fixes line to that effect to the commit. Thanks!
Signature · 1 line
Jens Axboe
Quoted 9 lines · Dylan Yudaken
In some cases
io_rsrc_ref_quiescewill callio_rsrc_node_switch_start, and then immediately flush the delayed work queue&ctx->rsrc_put_work.However the
percpu_ref_putdoes not immediately destroy the node, it will be called asynchronously via RCU. That ends up withio_rsrc_node_ref_zeroonly being called afterrsrc_put_workhas been flushed, and so the process ends up sleeping for 1 second unnecessarily.[...]
Applied, thanks!
[1/1] io_uring: fix bug in slow unregistering of nodes
(no commit info)
Best regards,
Signature · 1 line
Jens Axboe