io_uring: add schedule points to request freeing loops
cover io_uring: add schedule points to request freeing loops
An I/O stress test on a production v6.6-based debug kernel (stress-ng -a 1 --class io -t 7h on a 384-CPU machine) hit the soft lockup watchdog twice. The kernel was built with KASAN, kmemleak, lockdep and CONFIG_PREEMPT_NONE=y. Both lockups occurred while a CPU was busy in an io_uring request freeing loop that has no scheduling point:
1) A stress-ng worker looped in io_free_batch_list(), reached from exit-time task_work, until the machine panicked on "softlockup:
hung tasks".
2) A completion fallback work item held uring_lock for 671 seconds in that same free loop, until the watchdog fired again ("BUG: soft lockup - CPU#61 stuck for 671s").
The field kernel is v6.6-based, so to check whether the same weakness exists in current mainline, I tried to reproduce the pattern on v7.3-rc7 on a 192-CPU debug kernel (KASAN, kmemleak, lockdep; preempt=none - see the note below), with a test that submits 32K requests per ring and then kills the rings. I don't know whether real workloads hit these exact loops on mainline as hard as our test does, but the loops clearly have the same unbounded shape:
- A single io_free_batch_list() call freeing a 32K completion batch took 443ms with no scheduling point in between.
- Killing 48 rings at once made one __io_req_caches_free() call drain a single cache for up to 13 seconds, with 8 kworkers each stuck for over 12 seconds; a separate run with 96 rings produced 97 soft lockup warnings during teardown (the stack is in patch 2).
With this series applied, the same 96-ring run produces zero kworker soft lockup warnings: all 97 drain-path warnings are gone. This series targets the request freeing and cache draining paths; the submission loop itself may need a separate treatment and is intentionally left out of this series.
The free loops are the remaining gap: the CQ overflow flush already drops the lock and reschedules (eac2ca2d682f), and io_add_buffers(), io_remove_buffers_legacy(), the fdinfo dump loops and the tctx teardown all iterate with a cond_resched() in the loop. These two request freeing loops have none. The debug options amplify the per-request cost, but the loops are otherwise bounded only by the number of completed or cached requests. No functional change is intended; the patches only add scheduling points to long loops, and cond_resched() only invokes the scheduler if need_resched() is set. Unlike the overflow flush resched point, these do not drop the lock or leave any critical section, so they introduce no reordering concerns.
Note: the mainline test was booted with preempt=none to match the production kernel above. On x86 this required a local Kconfig adjustment to disable PREEMPT_LAZY; this is a test-environment detail and does not change the fix. cond_resched() is the standard schedule point on PREEMPT_NONE/VOLUNTARY kernels.
Tested with CONFIG_DEBUG_ATOMIC_SLEEP and CONFIG_PROVE_LOCKING enabled; no might_sleep or scheduling-while-atomic warnings during the reproduction runs.
Yuanhe Shu (2):
io_uring: add a schedule point in io_free_batch_list()
io_uring: add a schedule point in __io_req_caches_free()
io_uring/io_uring.c | 2 ++ 1 file changed, 2 insertions(+)
base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
Signature · 1 line
2.43.7
PATCH 1/2 io_uring: add a schedule point in io_free_batch_list()
Completed requests accumulate in submit_state.compl_reqs - deferred completions queued by task_work handlers via io_req_complete_defer(), plus everything io_do_iopoll() reaps - and are then all freed in one io_free_batch_list() pass under ctx->uring_lock. The loop has no scheduling point.
Each iteration does real work: io_clean_op() freeing request payloads, file, task and rsrc puts. With debug features like KASAN or kmemleak enabled, that work gets expensive enough that a large batch can hold the CPU well beyond the soft lockup threshold.
Caught by io stress testing (stress-ng -a 1 --class io -t 7h) on a 384 CPU machine running a v6.6-based debug kernel (KASAN, kmemleak, lockdep; CONFIG_PREEMPT_NONE=y): a stress-ng worker was looping here past the watchdog threshold, entered from exit-time task_work, until the machine panicked on "softlockup: hung tasks". Frames from the dump:
io_clean_op
io_free_batch_list
__io_submit_flush_completions
ctx_flush_and_put
tctx_task_work
task_work_run
get_signal
The same test later caught a completion fallback work item holding uring_lock for 671 seconds in this same loop ("BUG: soft lockup - CPU#61 stuck for 671s").
Also measured on mainline v7.3-rc7 (192-CPU debug kernel with the same debug features): a single pass freeing a 32K completion batch took 443ms with no scheduling point in between.
Insert a cond_resched() in the loop, like other potentially long io_uring loops already have. The loop runs in process context with ctx->uring_lock held, which is a mutex, so scheduling is allowed.
--- io_uring/io_uring.c | 1 + 1 file changed, 1 insertion(+)
io_uring/io_uring.c +1 -0
@@ -1123,6 +1123,7 @@ static void io_free_batch_list(struct io_ring_ctx *ctx, node = req->comp_list.next;io_req_add_to_cache (req, ctx); +cond_resched (); }while (node); }
Signature · 1 line
2.43.7
PATCH 2/2 io_uring: add a schedule point in __io_req_caches_free()
In reply to Yuanhe ShuThe request cache can grow to the workload's in-flight high-water mark. __io_req_caches_free() drains it with one kmem_cache_free() per request under ctx->uring_lock, with no scheduling point in between. With debug features like KASAN or kmemleak enabled, each free gets expensive, and draining a large cache can hold the CPU well beyond the soft lockup threshold.
Reproduced on mainline v7.3-rc7 on a 192-CPU debug kernel (preempt=none), with a test that submits 32K requests per ring and then kills the rings. Killing 48 rings at once made one drain take up to 13 seconds with no scheduling point, with 8 kworkers each stuck for over 12 seconds; a separate run with 96 rings produced 97 soft lockup warnings during teardown, of the form:
watchdog: BUG: soft lockup - CPU#2 stuck for 12s! [kworker/u769:5]
Workqueue: iou_exit io_ring_exit_work
Call Trace:
stack_trace_save
kasan_save_stack
kmem_cache_free+0x3cf/0x6a0
__io_req_caches_free+0x196/0x4f0
io_ring_exit_work+0x287/0x880
process_one_workRegardless of the per-free cost, the loop has no bound other than the workload's in-flight high-water mark, so it belongs with the other potentially long io_uring loops.
Insert a cond_resched() in the drain loop. The function is __cold; all callers run in process context and hold only ctx->uring_lock, a mutex, so scheduling is allowed.
--- io_uring/io_uring.c | 1 + 1 file changed, 1 insertion(+)
io_uring/io_uring.c +1 -0
@@ -2142,6 +2142,7 @@ static __cold void __io_req_caches_free(struct io_ring_ctx *ctx)io_poison_req (req);kmem_cache_free (req_cachep, req); nr++; +cond_resched (); }if (nr) { ctx->nr_req_allocated -= nr;
Signature · 1 line
2.43.7