io_uring/io-wq: let workers exit when unused
cover io_uring/io-wq: let workers exit when unused
io_uring uses io-wq to offload regular file I/O. When that happens, the kernel creates per-task iou-wrk-<tgid> workers (PF_IO_WORKER) via create_io_thread(), so the worker is part of the process thread group and shows up under /proc/<pid>/task/.
io-wq shrinks the pool on idle, but it intentionally keeps the last worker around indefinitely as a keepalive to avoid churn. Combined with io_uring's per-task context lifetime (tctx stays attached to the task until exit), a process may permanently retain an idle iou-wrk thread even after it has closed its last io_uring instance and has no active rings.
The keepalive behavior is a reasonable default(I guess): workloads may have bursty I/O patterns, and always tearing down the last worker would add thread churn and latency. Creating io-wq workers goes through create_io_thread() (copy_process), which is not cheap to do repeatedly.
However, CRIU currently doesn't cope well with such workers being part of the checkpointed thread group. The iou-wrk thread is a kernel-managed worker (PF_IO_WORKER) running io_wq_worker() on a kernel stack, rather than a normal userspace thread executing application code. In our setup, if the iou-wrk thread remains present after quiescing and closing the last io_uring instance, criu dump may hang while trying to stop and dump the thread group.
Besides the resource overhead and surprising userspace-visible threads, this is a problem for checkpoint/restore. CRIU needs to freeze and dump all threads in the thread group. With a lingering iou-wrk thread, we observed criu dump can hang even after the ring has been quiesced and the io_uring fd closed, e.g.:
criu dump -t $PID -D images -o dump.log -v4 --shell-job ps -T -p $PID -o pid,tid,comm | grep iou-wrk
This series is a kernel-side enabler for checkpoint/restore in the current reality where userspace needs to quiesce and close io_uring rings before dump. It is not trying to make io_uring rings checkpointable, nor does it change what CRIU can or cannot restore (e.g. in-flight SQEs/CQEs, SQPOLL, SQE128/CQE32, registered resources). Even with userspace gaining limited io_uring support, this series only targets the specific "no active io_uring contexts left, but an idle iou-wrk keepalive thread remains" case.
This series adds an explicit exit-on-idle mode to io-wq, and toggles it from io_uring task context when the task has no active io_uring contexts (xa_empty(&tctx->xa)). The mode is cleared on subsequent io_uring usage, so the default behavior for active io_uring users is unchanged.
Tested on x86_64 with CRIU 4.2.
With this series applied, after closing the ring iou-wrk exited within ~200ms and criu dump completed.
Li Chen (2):
io-wq: add exit-on-idle mode
io_uring: allow io-wq workers to exit when unused
io_uring/io-wq.c | 31 +++++++++++++++++++++++++++++++ io_uring/io-wq.h | 1 + io_uring/tctx.c | 11 +++++++++++ 3 files changed, 43 insertions(+)
Signature · 1 line
2.52.0
PATCH 1/2 io-wq: add exit-on-idle mode
io-wq uses an idle timeout to shrink the pool, but keeps the last worker around indefinitely to avoid churn.
For tasks that used io_uring for file I/O and then stop using io_uring, this can leave an iou-wrk-* thread behind even after all io_uring instances are gone. This is unnecessary overhead and also gets in the way of process checkpoint/restore.
Add an exit-on-idle mode that makes all io-wq workers exit as soon as they become idle, and provide io_wq_set_exit_on_idle() to toggle it.
--- io_uring/io-wq.c | 31 +++++++++++++++++++++++++++++++ io_uring/io-wq.h | 1 + 2 files changed, 32 insertions(+)
io_uring/io-wq.c +31 -0
@@ -35,6 +35,7 @@ enum {enum {IO_WQ_BIT_EXIT =0 ,/* wq exiting */ +IO_WQ_BIT_EXIT_ON_IDLE =1 ,/* allow all workers to exit on idle */ };enum {
@@ -655,6 +656,18 @@ static int io_wq_worker(void *data)io_worker_handle_work (acct, worker);raw_spin_lock (&wq->lock); +/* +* If wq is marked idle-exit, drop this worker as soon as it +* becomes idle. This is used to avoid keeping io-wq worker +* threads around for tasks that no longer have any active +* io_uring instances. +*/ +if (test_bit (IO_WQ_BIT_EXIT_ON_IDLE , &wq->state)) { + acct->nr_workers--; +raw_spin_unlock (&wq->lock); +__set_current_state (TASK_RUNNING ); +break ; + }/* * Last sleep timed out. Exit if we're not the last worker, * or if someone modified our affinity.
@@ -894,6 +907,24 @@ static bool io_wq_worker_wake(struct io_worker *worker, void *data)return false; } +void io_wq_set_exit_on_idle (struct io_wq *wq,bool enable) +{ +if (!wq->task) +return ; + +if (!enable) { +clear_bit (IO_WQ_BIT_EXIT_ON_IDLE , &wq->state); +return ; + } + +if (test_and_set_bit (IO_WQ_BIT_EXIT_ON_IDLE , &wq->state)) +return ; + +rcu_read_lock (); +io_wq_for_each_worker (wq, io_wq_worker_wake,NULL ); +rcu_read_unlock (); +} +static void io_run_cancel (struct io_wq_work *work,struct io_wq *wq) { do {
io_uring/io-wq.h +1 -0
@@ -46,6 +46,7 @@ struct io_wq_data {struct io_wq *io_wq_create (unsigned bounded,struct io_wq_data *data);void io_wq_exit_start (struct io_wq *wq);void io_wq_put_and_exit (struct io_wq *wq); +void io_wq_set_exit_on_idle (struct io_wq *wq,bool enable);void io_wq_enqueue (struct io_wq *wq,struct io_wq_work *work);void io_wq_hash_work (struct io_wq_work *work,void *val);
Signature · 1 line
2.52.0
Jens Axboe
maintainerRe: [PATCH v1 1/2] io-wq: add exit-on-idle mode
Quoted 10 lines · Li Chen
io-wq uses an idle timeout to shrink the pool, but keeps the last worker around indefinitely to avoid churn.
For tasks that used
io_uringfor file I/O and then stop usingio_uring, this can leave an iou-wrk-* thread behind even after allio_uringinstances are gone. This is unnecessary overhead and also gets in the way of process checkpoint/restore.Add an exit-on-idle mode that makes all io-wq workers exit as soon as they become idle, and provide
io_wq_set_exit_on_idle()to toggle it.
Was going to say, rather than add a mode for this, why not just have the idle single worker exit when the last ring is closed? But that is indeed exactly what these two patches do. So I think this is fine, I just don't think using the word "mode" for it is correct. "state" would be a lot better - if we have all rings exited, then that's a state change in terms of yeah let's just dump that idle worker.
With that in mind, I think these two patches look fine. I'll give them a closer look. Could you perhaps write a test case for this?
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerRe: [PATCH v1 1/2] io-wq: add exit-on-idle mode
In reply to Li ChenQuoted 13 lines · Li Chen
io_uring/io-wq.c +5
-0@@ -655,6 +656,18 @@ static int io_wq_worker(void *data)io_worker_handle_work (acct, worker);raw_spin_lock (&wq->lock); +/* +* If wq is marked idle-exit, drop this worker as soon as it +* becomes idle. This is used to avoid keeping io-wq worker +* threads around for tasks that no longer have any active +* io_uring instances.
io_uring/io-wq.c static int io_wq_worker(void *data) · line 673
Quoted 10 lines
io_uring/io-wq.c +7
-0+*/ +if (test_bit (IO_WQ_BIT_EXIT_ON_IDLE , &wq->state)) { + acct->nr_workers--; +raw_spin_unlock (&wq->lock); +__set_current_state (TASK_RUNNING ); +break ; + }/* * Last sleep timed out. Exit if we're not the last worker, * or if someone modified our affinity.
One more note - just add this test_bit() to the check right below, then you avoid duplicating all of that exit logic. They do the exact same thing.
Signature · 1 line
Jens Axboe
PATCH 2/2 io_uring: allow io-wq workers to exit when unused
In reply to Li Chenio_uring keeps a per-task io-wq around, even when the task no longer has any io_uring instances.
If the task previously used io_uring for file I/O, this can leave an unrelated iou-wrk-* worker thread behind after the last io_uring instance is gone.
When the last io_uring ctx is removed from the task context, mark the io-wq exit-on-idle so workers can go away. Clear the flag on subsequent io_uring usage.
--- io_uring/tctx.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
io_uring/tctx.c +11 -0
@@ -124,6 +124,14 @@ int __io_uring_add_tctx_node(struct io_ring_ctx *ctx)return ret; } } + +/* +* Re-activate io-wq keepalive on any new io_uring usage. The wq may have +* been marked for idle-exit when the task temporarily had no active +* io_uring instances. +*/ +if (tctx->io_wq) +io_wq_set_exit_on_idle (tctx->io_wq, false);if (!xa_load (&tctx->xa, (unsigned long )ctx)) { node =kmalloc (sizeof (*node),GFP_KERNEL );if (!node)
@@ -185,6 +193,9 @@ __cold void io_uring_del_tctx_node(unsigned long index)if (tctx->last == node->ctx) tctx->last =NULL ;kfree (node); + +if (xa_empty (&tctx->xa) && tctx->io_wq) +io_wq_set_exit_on_idle (tctx->io_wq, true); }__cold void io_uring_clean_tctx (struct io_uring_task *tctx)
Signature · 1 line
2.52.0
Jens Axboe
maintainerQuoted 39 lines · Li Chen
io_uringuses io-wq to offload regular file I/O. When that happens, the kernel creates per-task iou-wrk-<tgid> workers (PF_IO_WORKER) viacreate_io_thread(), so the worker is part of the process thread group and shows up under /proc/<pid>/task/.io-wq shrinks the pool on idle, but it intentionally keeps the last worker around indefinitely as a keepalive to avoid churn. Combined with io_uring's per-task context lifetime (tctx stays attached to the task until exit), a process may permanently retain an idle iou-wrk thread even after it has closed its last
io_uringinstance and has no active rings.The keepalive behavior is a reasonable default(I guess): workloads may have bursty I/O patterns, and always tearing down the last worker would add thread churn and latency. Creating io-wq workers goes through
create_io_thread()(copy_process), which is not cheap to do repeatedly.However, CRIU currently doesn't cope well with such workers being part of the checkpointed thread group. The iou-wrk thread is a kernel-managed worker (
PF_IO_WORKER) runningio_wq_worker()on a kernel stack, rather than a normal userspace thread executing application code. In our setup, if the iou-wrk thread remains present after quiescing and closing the lastio_uringinstance, criu dump may hang while trying to stop and dump the thread group.Besides the resource overhead and surprising userspace-visible threads, this is a problem for checkpoint/restore. CRIU needs to freeze and dump all threads in the thread group. With a lingering iou-wrk thread, we observed criu dump can hang even after the ring has been quiesced and the
io_uringfd closed, e.g.:criu dump -t $PID -D images -o dump.log -v4 --shell-job ps -T -p $PID -o pid,tid,comm | grep iou-wrkThis series is a kernel-side enabler for checkpoint/restore in the current reality where userspace needs to quiesce and close
io_uringrings before dump. It is not trying to makeio_uringrings checkpointable, nor does it change what CRIU can or cannot restore (e.g. in-flight SQEs/CQEs, SQPOLL, SQE128/CQE32, registered resources). Even with userspace gaining limitedio_uringsupport, this series only targets the specific "no activeio_uringcontexts left, but an idle iou-wrk keepalive thread remains" case.
Quoted 8 lines
This series adds an explicit exit-on-idle mode to io-wq, and toggles it from
io_uringtask context when the task has no activeio_uringcontexts (xa_empty(&tctx->xa)). The mode is cleared on subsequentio_uringusage, so the default behavior for activeio_uringusers is unchanged.Tested on
x86_64with CRIU 4.2.
With this series applied, after closing the ring iou-wrk exited within ~200ms and criu dump completed.
Applied with the mentioned commit message and IO_WQ_BIT_EXIT_ON_IDLE test placement.
Signature · 1 line
Jens Axboe
Hi Jens,
---- On Mon, 02 Feb 2026 23:21:22 +0800 Jens Axboe <[email protected]> wrote ---
Quoted 45 lines · Li Chen
Quoted 44 lines
io_uringuses io-wq to offload regular file I/O. When that happens, the kernel creates per-task iou-wrk-<tgid> workers (PF_IO_WORKER) viacreate_io_thread(), so the worker is part of the process thread group and shows up under /proc/<pid>/task/.io-wq shrinks the pool on idle, but it intentionally keeps the last worker around indefinitely as a keepalive to avoid churn. Combined with io_uring's per-task context lifetime (tctx stays attached to the task until exit), a process may permanently retain an idle iou-wrk thread even after it has closed its last
io_uringinstance and has no active rings.The keepalive behavior is a reasonable default(I guess): workloads may have bursty I/O patterns, and always tearing down the last worker would add thread churn and latency. Creating io-wq workers goes through
create_io_thread()(copy_process), which is not cheap to do repeatedly.However, CRIU currently doesn't cope well with such workers being part of the checkpointed thread group. The iou-wrk thread is a kernel-managed worker (
PF_IO_WORKER) runningio_wq_worker()on a kernel stack, rather than a normal userspace thread executing application code. In our setup, if the iou-wrk thread remains present after quiescing and closing the lastio_uringinstance, criu dump may hang while trying to stop and dump the thread group.Besides the resource overhead and surprising userspace-visible threads, this is a problem for checkpoint/restore. CRIU needs to freeze and dump all threads in the thread group. With a lingering iou-wrk thread, we observed criu dump can hang even after the ring has been quiesced and the
io_uringfd closed, e.g.:criu dump -t $PID -D images -o dump.log -v4 --shell-job ps -T -p $PID -o pid,tid,comm | grep iou-wrkThis series is a kernel-side enabler for checkpoint/restore in the current reality where userspace needs to quiesce and close
io_uringrings before dump. It is not trying to makeio_uringrings checkpointable, nor does it change what CRIU can or cannot restore (e.g. in-flight SQEs/CQEs, SQPOLL, SQE128/CQE32, registered resources). Even with userspace gaining limitedio_uringsupport, this series only targets the specific "no activeio_uringcontexts left, but an idle iou-wrk keepalive thread remains" case.This series adds an explicit exit-on-idle mode to io-wq, and toggles it from
io_uringtask context when the task has no activeio_uringcontexts (xa_empty(&tctx->xa)). The mode is cleared on subsequentio_uringusage, so the default behavior for activeio_uringusers is unchanged.
Quoted 6 lines
Quoted 3 lines
Tested on
x86_64with CRIU 4.2.
With this series applied, after closing the ring iou-wrk exited within ~200ms and criu dump completed.Applied with the mentioned commit message and
IO_WQ_BIT_EXIT_ON_IDLEtest placement.
Thanks a lot for your review!
If you still want a test, I'm happy to write it. Since you've already tweaked/applied the v1 series, I can send the test as a standalone follow-up patch (no v2).
If kselftest is preferred, I'll base it on the same CRIU-style workload:
spawn iou-wrk-* via io_uring, quiesce/close the last ring, and check the worker exits within a short timeout.
Regards,
Li
Jens Axboe
maintainerQuoted 57 lines · Li Chen
Hi Jens,
---- On Mon, 02 Feb 2026 23:21:22 +0800 Jens Axboe <[email protected]> wrote ---
Quoted 45 lines · Li Chen
Quoted 44 lines
io_uringuses io-wq to offload regular file I/O. When that happens, the kernel creates per-task iou-wrk-<tgid> workers (PF_IO_WORKER) viacreate_io_thread(), so the worker is part of the process thread group and shows up under /proc/<pid>/task/.io-wq shrinks the pool on idle, but it intentionally keeps the last worker around indefinitely as a keepalive to avoid churn. Combined with io_uring's per-task context lifetime (tctx stays attached to the task until exit), a process may permanently retain an idle iou-wrk thread even after it has closed its last
io_uringinstance and has no active rings.The keepalive behavior is a reasonable default(I guess): workloads may have bursty I/O patterns, and always tearing down the last worker would add thread churn and latency. Creating io-wq workers goes through
create_io_thread()(copy_process), which is not cheap to do repeatedly.However, CRIU currently doesn't cope well with such workers being part of the checkpointed thread group. The iou-wrk thread is a kernel-managed worker (
PF_IO_WORKER) runningio_wq_worker()on a kernel stack, rather than a normal userspace thread executing application code. In our setup, if the iou-wrk thread remains present after quiescing and closing the lastio_uringinstance, criu dump may hang while trying to stop and dump the thread group.Besides the resource overhead and surprising userspace-visible threads, this is a problem for checkpoint/restore. CRIU needs to freeze and dump all threads in the thread group. With a lingering iou-wrk thread, we observed criu dump can hang even after the ring has been quiesced and the
io_uringfd closed, e.g.:criu dump -t $PID -D images -o dump.log -v4 --shell-job ps -T -p $PID -o pid,tid,comm | grep iou-wrkThis series is a kernel-side enabler for checkpoint/restore in the current reality where userspace needs to quiesce and close
io_uringrings before dump. It is not trying to makeio_uringrings checkpointable, nor does it change what CRIU can or cannot restore (e.g. in-flight SQEs/CQEs, SQPOLL, SQE128/CQE32, registered resources). Even with userspace gaining limitedio_uringsupport, this series only targets the specific "no activeio_uringcontexts left, but an idle iou-wrk keepalive thread remains" case.This series adds an explicit exit-on-idle mode to io-wq, and toggles it from
io_uringtask context when the task has no activeio_uringcontexts (xa_empty(&tctx->xa)). The mode is cleared on subsequentio_uringusage, so the default behavior for activeio_uringusers is unchanged.Quoted 6 lines
Quoted 3 lines
Tested on
x86_64with CRIU 4.2.
With this series applied, after closing the ring iou-wrk exited within ~200ms and criu dump completed.Applied with the mentioned commit message and
IO_WQ_BIT_EXIT_ON_IDLEtest placement.Thanks a lot for your review!
Quoted 7 lines
If you still want a test, I'm happy to write it. Since you've already tweaked/applied the v1 series, I can send the test as a standalone follow-up patch (no v2).
If kselftest is preferred, I'll base it on the same CRIU-style workload:
spawn iou-wrk-* viaio_uring, quiesce/close the last ring, and check the worker exits within a short timeout.
That sounds like the right way to do the test. Preferably a liburing test/ case would be better, we don't do a lot of in-kernel selftests so far. But liburing has everything.
Signature · 1 line
Jens Axboe
Hi Jens,
---- On Tue, 03 Feb 2026 10:29:50 +0800 Jens Axboe <[email protected]> wrote ---
Quoted 62 lines · Li Chen
Quoted 61 lines
Hi Jens,
---- On Mon, 02 Feb 2026 23:21:22 +0800 Jens Axboe <[email protected]> wrote ---
Quoted 45 lines · Li Chen
Quoted 44 lines
io_uringuses io-wq to offload regular file I/O. When that happens, the kernel creates per-task iou-wrk-<tgid> workers (PF_IO_WORKER) viacreate_io_thread(), so the worker is part of the process thread group and shows up under /proc/<pid>/task/.io-wq shrinks the pool on idle, but it intentionally keeps the last worker around indefinitely as a keepalive to avoid churn. Combined with io_uring's per-task context lifetime (tctx stays attached to the task until exit), a process may permanently retain an idle iou-wrk thread even after it has closed its last
io_uringinstance and has no active rings.The keepalive behavior is a reasonable default(I guess): workloads may have bursty I/O patterns, and always tearing down the last worker would add thread churn and latency. Creating io-wq workers goes through
create_io_thread()(copy_process), which is not cheap to do repeatedly.However, CRIU currently doesn't cope well with such workers being part of the checkpointed thread group. The iou-wrk thread is a kernel-managed worker (
PF_IO_WORKER) runningio_wq_worker()on a kernel stack, rather than a normal userspace thread executing application code. In our setup, if the iou-wrk thread remains present after quiescing and closing the lastio_uringinstance, criu dump may hang while trying to stop and dump the thread group.Besides the resource overhead and surprising userspace-visible threads, this is a problem for checkpoint/restore. CRIU needs to freeze and dump all threads in the thread group. With a lingering iou-wrk thread, we observed criu dump can hang even after the ring has been quiesced and the
io_uringfd closed, e.g.:criu dump -t $PID -D images -o dump.log -v4 --shell-job ps -T -p $PID -o pid,tid,comm | grep iou-wrkThis series is a kernel-side enabler for checkpoint/restore in the current reality where userspace needs to quiesce and close
io_uringrings before dump. It is not trying to makeio_uringrings checkpointable, nor does it change what CRIU can or cannot restore (e.g. in-flight SQEs/CQEs, SQPOLL, SQE128/CQE32, registered resources). Even with userspace gaining limitedio_uringsupport, this series only targets the specific "no activeio_uringcontexts left, but an idle iou-wrk keepalive thread remains" case.This series adds an explicit exit-on-idle mode to io-wq, and toggles it from
io_uringtask context when the task has no activeio_uringcontexts (xa_empty(&tctx->xa)). The mode is cleared on subsequentio_uringusage, so the default behavior for activeio_uringusers is unchanged.Quoted 6 lines
Quoted 3 lines
Tested on
x86_64with CRIU 4.2.
With this series applied, after closing the ring iou-wrk exited within ~200ms and criu dump completed.Applied with the mentioned commit message and
IO_WQ_BIT_EXIT_ON_IDLEtest placement.Thanks a lot for your review!
If you still want a test, I'm happy to write it. Since you've already tweaked/applied the v1 series, I can send the test as a standalone follow-up patch (no v2).
Quoted 7 lines
Quoted 3 lines
If kselftest is preferred, I'll base it on the same CRIU-style workload:
spawn iou-wrk-* viaio_uring, quiesce/close the last ring, and check the worker exits within a short timeout.That sounds like the right way to do the test. Preferably a liburing test/ case would be better, we don't do a lot of in-kernel selftests so far. But liburing has everything.
Thanks for your suggestion. I just adapted my local test program to liburing and posted the liburing
PR here: https://github.com/axboe/liburing/pull/1529
Regards,
Li
Jens Axboe
maintainerQuoted 10 lines · Li Chen
Hi Jens,
Quoted 8 lines
If you still want a test, I'm happy to write it. Since you've already
Quoted 6 lines
tweaked/applied the v1 series, I can send the test as a standalone follow-up patch (no v2).
If kselftest is preferred, I'll base it on the same CRIU-style workload:
spawn iou-wrk-* viaio_uring, quiesce/close the last ring, and check the worker exits within a short timeout.
Quoted 7 lines
Quoted 3 lines
That sounds like the right way to do the test. Preferably a liburing test/ case would be better, we don't do a lot of in-kernel selftests so far. But liburing has everything.
Thanks for your suggestion. I just adapted my local test program to liburing and posted the liburing PR here:
https://github.com/axboe/liburing/pull/1529
Thanks, merged!
Signature · 1 line
Jens Axboe