Avoid spurious syzbot induced hung task panics
Jens Axboe
maintainercover Avoid spurious syzbot induced hung task panics
Hi,
For details, see this saga:
https://lore.kernel.org/io-uring/[email protected]/
where the tldr is that there's no real bug here, it's just syzbot doing hundreds of 2GB /dev/msr* reads in a tiny vm and with a bunch of debugging enabled. That leads to triggering the hung task detector when we wait on io-wq workers to exit. I did queue a patch for 6.19 that makes this less likely to occur, as it'll only run the very first of the items before noticing the cancelation:
https://lore.kernel.org/io-uring/[email protected]/
but even that isn't quite enough due to how much syzbot overloads the system.
This will still throw a WARN_ON_ONCE(), which perhaps should just be a printk() of some sort as the trace isn't THAT interesting. But it will avoid hitting the hung task timeout detector, which for syzbot leads to a panic + reboot.
io_uring/io-wq.c | 22 +++++++++++++++++++++- io_uring/io_uring.c | 2 +- io_uring/io_uring.h | 6 ++++++ 3 files changed, 28 insertions(+), 2 deletions(-)
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/2 io_uring: add IO_URING_EXIT_WAIT_MAX definition
Add the timeout we normally wait before complaining about things being stuck waiting for cancelations to complete as a define, and use it in io_ring_exit_work().
--- io_uring/io_uring.c | 2 +- io_uring/io_uring.h | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-)
io_uring/io_uring.c +1 -1
@@ -2984,7 +2984,7 @@ static __cold void io_tctx_exit_cb(struct callback_head *cb)static __cold void io_ring_exit_work (struct work_struct *work) { struct io_ring_ctx *ctx = container_of(work, struct io_ring_ctx, exit_work); -unsigned long timeout = jiffies +HZ *60 *5 ; +unsigned long timeout = jiffies +IO_URING_EXIT_WAIT_MAX ;unsigned long interval =HZ /20 ;struct io_tctx_exit exit;struct io_tctx_node *node;
io_uring/io_uring.h +6 -0
@@ -89,6 +89,12 @@ struct io_ctx_config { IOSQE_BUFFER_SELECT |\ IOSQE_CQE_SKIP_SUCCESS) +/* +* Complaint timeout for io_uring cancelation exits, and for io-wq exit +* worker waiting. +*/ +#define IO_URING_EXIT_WAIT_MAX (HZ * 60 * 5) +enum {IOU_COMPLETE =0 ,
Signature · 1 line
2.51.0
Jens Axboe
maintainerPATCH 2/2 io_uring/io-wq: don't trigger hung task for syzbot craziness
In reply to Jens AxboeUse the same trick that blk_io_schedule() does to avoid triggering the hung task warning (and potential reboot/panic, depending on system settings), and only wait for half the hung task timeout at the time. If we exceed the default IO_URING_EXIT_WAIT_MAX period where we expect things to certainly have finished unless there's a bug, then throw a WARN_ON_ONCE() for that case.
--- io_uring/io-wq.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
io_uring/io-wq.c +21 -1
@@ -17,6 +17,7 @@#include <linux/task_work.h> #include <linux/audit.h> #include <linux/mmu_context.h> +#include <linux/sched/sysctl.h> #include <uapi/linux/io_uring.h> #include "io-wq.h"
@@ -1313,6 +1314,8 @@ static void io_wq_cancel_tw_create(struct io_wq *wq)static void io_wq_exit_workers (struct io_wq *wq) { +unsigned long timeout, warn_timeout; +if (!wq->task)return ;
@@ -1322,7 +1325,24 @@ static void io_wq_exit_workers(struct io_wq *wq)io_wq_for_each_worker (wq, io_wq_worker_wake,NULL );rcu_read_unlock ();io_worker_ref_put (wq); -wait_for_completion (&wq->worker_done); + +/* +* Shut up hung task complaint, see for example +* +* https://lore.kernel.org/all/[email protected]/ +* +* where completely overloading the system with tons of long running +* io-wq items can easily trigger the hung task timeout. Only sleep +* uninterruptibly for half that time, and warn if we exceeded end +* up waiting more than IO_URING_EXIT_WAIT_MAX. +*/ + timeout = sysctl_hung_task_timeout_secs *HZ /2 ; + warn_timeout = jiffies +IO_URING_EXIT_WAIT_MAX ; +do { +if (wait_for_completion_timeout (&wq->worker_done, timeout)) +break ; +WARN_ON_ONCE (time_after (jiffies, warn_timeout)); + }while (1 );spin_lock_irq (&wq->hash->wait.lock);list_del_init (&wq->wait.entry);
Signature · 1 line
2.51.0