io_uring: fix io_wqe->work_list corruption
PATCH 1/1 io_uring: fix io_wqe->work_list corruption
For the first time a req punted to io-wq, we'll initialize io_wq_work's list to be NULL, then insert req to io_wqe->work_list. If this req is not inserted into tail of io_wqe->work_list, this req's io_wq_work list will point to another req's io_wq_work. For splitted bio case, this req maybe inserted to io_wqe->work_list repeatedly, once we insert it to tail of io_wqe->work_list for the second time, now io_wq_work->list->next will be invalid pointer, which then result in many strang error, panic, kernel soft-lockup, rcu stall, etc.
In my vm, kernel doest not have commit cc29e1bf0d63f7 ("block: disable iopoll for split bio"), below fio job can reproduce this bug steadily:
[global]
name=iouring-sqpoll-iopoll-1
ioengine=io_uring
iodepth=128
numjobs=1
thread
rw=randread
direct=1
registerfiles=1
hipri=1
bs=4m
size=100M
runtime=120time_basedgroup_reporting
randrepeat=0
[device]
directory=/home/feiman.wxg/mntpoint/ # an ext4 mount point
If we have commit cc29e1bf0d63f7 ("block: disable iopoll for split bio"), there will no splitted bio case for polled io, but I think we still to need to fix this list corruption, it also should maybe go to stable branchs.
To fix this corruption, if a req is inserted into tail of io_wqe->work_list, initialize req->io_wq_work->list->next to bu NULL.
--- fs/io-wq.h | 1 + 1 file changed, 1 insertion(+)
fs/io-wq.h +1 -0
@@ -59,6 +59,7 @@ static inline void wq_list_add_tail(struct io_wq_work_node *node, list->last->next = node; list->last = node; } + node->next =NULL ; } static inline void wq_list_cut(struct io_wq_work_list *list,
Signature · 1 line
2.17.2
Quoted 8 lines · Xiaoguang Wang
For the first time a req punted to io-wq, we'll initialize io_wq_work's list to be NULL, then insert req to
io_wqe->work_list. If this req is not inserted into tail ofio_wqe->work_list, this req'sio_wq_worklist will point to another req'sio_wq_work. For splitted bio case, this req maybe inserted toio_wqe->work_listrepeatedly, once we insert it to tail ofio_wqe->work_listfor the second time, nowio_wq_work->list->nextwill be invalid pointer, which then result in many strang error, panic, kernel soft-lockup, rcu stall, etc.
[...]
Quoted 2 lines
To fix this corruption, if a req is inserted into tail of
io_wqe->work_list, initializereq->io_wq_work->list->nextto bu NULL.
Looks fine, and the function is cold to not care about overhead.
Signature · 1 line
Pavel Begunkov
Quoted 31 lines · Xiaoguang Wang
For the first time a req punted to io-wq, we'll initialize io_wq_work's list to be NULL, then insert req to
io_wqe->work_list. If this req is not inserted into tail ofio_wqe->work_list, this req'sio_wq_worklist will point to another req'sio_wq_work. For splitted bio case, this req maybe inserted toio_wqe->work_listrepeatedly, once we insert it to tail ofio_wqe->work_listfor the second time, nowio_wq_work->list->nextwill be invalid pointer, which then result in many strang error, panic, kernel soft-lockup, rcu stall, etc.In my vm, kernel doest not have commit
cc29e1bf0d63f7("block: disable iopoll for split bio"), below fio job can reproduce this bug steadily:
[global]
name=iouring-sqpoll-iopoll-1
ioengine=io_uring
iodepth=128
numjobs=1
thread
rw=randread
direct=1
registerfiles=1
hipri=1
bs=4m
size=100M
runtime=120time_basedgroup_reporting
randrepeat=0[device]
directory=/home/feiman.wxg/mntpoint/ # an ext4 mount point
Quoted 6 lines
If we have commit
cc29e1bf0d63f7("block: disable iopoll for split bio"), there will no splitted bio case for polled io, but I think we still to need to fix this list corruption, it also should maybe go to stable branchs.To fix this corruption, if a req is inserted into tail of
io_wqe->work_list, initializereq->io_wq_work->list->nextto bu NULL.
Applied, and marked for stable.
Signature · 1 line
Jens Axboe