io_uring/timeout: don't rearm a multishot timeout on cancel
PATCH 1/1 io_uring/timeout: don't rearm a multishot timeout on cancel
io_timeout_complete() does not check tw.cancel. When it runs from io_cancel_local_task_work(), a multishot timeout that has not reached its count still posts an IORING_CQE_F_MORE completion and rearms its timer. With a zero interval the timer expires again right away and queues new task work, so the drain keeps running it until posting a CQE fails.
Normally the CQ fills up quickly and the request then completes. But if userspace moves the CQ head far ahead of the tail, for example to cq.tail + 0x7fffffff, posting keeps succeeding and ring teardown is delayed for a long time.
Don't post another CQE or rearm the timer when tw.cancel is set, so the request completes through io_req_task_complete(), like the poll and linked timeout task work does when cancelled.
---
Report: https://lore.kernel.org/io-uring/[email protected]/
Runtime-tested on v7.3 with the reproducer from the report. io_timeout_complete() is the same in io_uring-7.4, where it is only build-tested.
io_uring/timeout.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
io_uring/timeout.c +2 -1
@@ -115,7 +115,8 @@ static void io_timeout_complete(struct io_tw_req tw_req, io_tw_token_t tw)struct io_ring_ctx *ctx = req->ctx;if (!io_timeout_finish (timeout, data)) { -if (io_req_post_cqe (req, -ETIME ,IORING_CQE_F_MORE )) { +if (!tw.cancel && +io_req_post_cqe (req, -ETIME ,IORING_CQE_F_MORE )) {/* re-arm timer */ raw_spin_lock_irq (&ctx->timeout_lock);list_add (&timeout->list, ctx->timeout_list.prev);
Signature · 1 line
2.34.1
Jens Axboe
maintainerQuoted 11 lines · Cheng Yongkang
io_timeout_complete()does not check tw.cancel. When it runs fromio_cancel_local_task_work(), a multishot timeout that has not reached its count still posts anIORING_CQE_F_MOREcompletion and rearms its timer. With a zero interval the timer expires again right away and queues new task work, so the drain keeps running it until posting a CQE fails.Normally the CQ fills up quickly and the request then completes. But if userspace moves the CQ head far ahead of the tail, for example to cq.tail +
0x7fffffff, posting keeps succeeding and ring teardown is delayed for a long time.
But why would the app do that, to shoot itself in the foot?
I think the fix is fine, but the premise of it is very odd.
Signature · 1 line
Jens Axboe
Quoted 3 lines · Jens Axboe
But why would the app do that, to shoot itself in the foot?
I think the fix is fine, but the premise of it is very odd.
Agreed, no sane app would do that. The only thing that made it look worth fixing to me is where the cost lands. The drain runs from io_ring_exit_work() after the task has closed the ring, so the time is spent in a kworker holding uring_lock, and the app can no longer stop it by being killed.
Without the bogus CQ head the timeout is still rearmed during cancel, it just stops once the CQ is full, so the change mostly makes the timeout handler skip the rearm like the other cancel paths do.
I can reword the commit message to lead with that instead, if you prefer.
Thanks,
Cheng Yongkang
Jens Axboe
maintainerQuoted 11 lines · Cheng Yongkang
Quoted 3 lines · Jens Axboe
But why would the app do that, to shoot itself in the foot?
I think the fix is fine, but the premise of it is very odd.
Agreed, no sane app would do that. The only thing that made it look worth fixing to me is where the cost lands. The drain runs from
io_ring_exit_work()after the task has closed the ring, so the time is spent in a kworker holdinguring_lock, and the app can no longer stop it by being killed.
Quoted 6 lines
Without the bogus CQ head the timeout is still rearmed during cancel, it just stops once the CQ is full, so the change mostly makes the timeout handler skip the rearm like the other cancel paths do.
I can reword the commit message to lead with that instead, if you prefer.
No it's fine, I already queued it up. But it really reeks of odd and bogus syzbot test case, it's most certainly not something that any normal application would see.
Signature · 1 line
Jens Axboe
Quoted 8 lines · Cheng Yongkang
io_timeout_complete()does not check tw.cancel. When it runs fromio_cancel_local_task_work(), a multishot timeout that has not reached its count still posts anIORING_CQE_F_MOREcompletion and rearms its timer. With a zero interval the timer expires again right away and queues new task work, so the drain keeps running it until posting a CQE fails.[...]
Applied, thanks!
[1/1] io_uring/timeout: don't rearm a multishot timeout on cancel
(no commit info)
Best regards,
Signature · 1 line
Jens Axboe