xfs: fix use-after-free of buf_log_item in xlog_cil_build_lv_chain
PATCH 1/1 xfs: fix use-after-free of buf_log_item in xlog_cil_build_lv_chain
xfs_buf_item_done() frees the buf_item via xfs_buf_item_relse() but does not remove the item from the CIL log_items list (li_cil). When the item is freed through an error/shutdown/abort path before the CIL push worker processes it, the freed memory remains linked in ctx->log_items.
The CIL push worker in xlog_cil_build_lv_chain() then dereferences the freed object via item->li_lv, triggering a KASAN slab-use-after-free. For details, see Link[1].
Add down_read() on xc_ctx_lock before list_del_init() in xfs_buf_item_done() to safely remove the item from the CIL list. This uses the same lock that protects CIL list operations: insertions are done under xc_ctx_lock read-side (xlog_cil_insert_items) and removals under write-side (xlog_cil_build_lv_chain). The read lock is safe here because xfs_buf_item_done() is always called in process context (workqueue or direct I/O wait) and cannot deadlock with the CIL push worker which holds the write lock during xlog_cil_build_lv_chain - the worker does not trigger metadata buffer I/O that would call xfs_buf_item_done().
--- fs/xfs/xfs_buf_item.c | 5 +++++ 1 file changed, 5 insertions(+)
fs/xfs/xfs_buf_item.c +5 -0
@@ -1067,6 +1067,11 @@ voidxfs_buf_item_done ( struct xfs_buf *bp) { +if (bp->b_log_item->bli_item.li_log->l_cilp) { +down_read (&bp->b_log_item->bli_item.li_log->l_cilp->xc_ctx_lock); +list_del_init (&bp->b_log_item->bli_item.li_cil); +up_read (&bp->b_log_item->bli_item.li_log->l_cilp->xc_ctx_lock); + }/* * If we are forcibly shutting down, this may well be off the AIL * already. That's because we simulate the log-committed callbacks to
Signature · 1 line
2.20.1
Quoted 8 lines · Gou Hao
xfs_buf_item_done()frees thebuf_itemviaxfs_buf_item_relse()but does not remove the item from the CILlog_itemslist (li_cil). When the item is freed through an error/shutdown/abort path before the CIL push worker processes it, the freed memory remains linked inctx->log_items.The CIL push worker in
xlog_cil_build_lv_chain()then dereferences the freed object viaitem->li_lv, triggering a KASAN slab-use-after-free. For details, see Link[1].
There's no reproducer there. Do you have a local one?
Quoted 9 lines
Add
down_read()onxc_ctx_lockbeforelist_del_init()inxfs_buf_item_done()to safely remove the item from the CIL list. This uses the same lock that protects CIL list operations: insertions are done underxc_ctx_lockread-side (xlog_cil_insert_items) and removals under write-side (xlog_cil_build_lv_chain). The read lock is safe here becausexfs_buf_item_done()is always called in process context (workqueue or direct I/O wait) and cannot deadlock with the CIL push worker which holds the write lock duringxlog_cil_build_lv_chain- the worker does not trigger metadata buffer I/O that would callxfs_buf_item_done().
This looks like a more general issue as we should never free anything that is still on the CIL. I.e. it looks like we have even more issues with the buf item state machine here :(
Quoted 31 lines
Reported-by [email protected]Fixes 816c330b605c ("xfs: factor out stale buffer item completion")Suggested-by Zhan Jun <[email protected]>Signed-off-by Gou Hao <[email protected]>--- fs/xfs/xfs_buf_item.c | 5 +++++ 1 file changed, 5 insertions(+)fs/xfs/xfs_buf_item.c +5
-0@@ -1067,6 +1067,11 @@ voidxfs_buf_item_done ( struct xfs_buf *bp) { +if (bp->b_log_item->bli_item.li_log->l_cilp) { +down_read (&bp->b_log_item->bli_item.li_log->l_cilp->xc_ctx_lock); +list_del_init (&bp->b_log_item->bli_item.li_cil); +up_read (&bp->b_log_item->bli_item.li_log->l_cilp->xc_ctx_lock); + }/* * If we are forcibly shutting down, this may well be off the AIL * already. That's because we simulate the log-committed callbacks to Signature · 1 line
2.20.1
---end quoted text---
Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Gou Hao
xfs_buf_item_done()frees thebuf_itemviaxfs_buf_item_relse()but does not remove the item from the CILlog_itemslist (li_cil). When the item is freed through an error/shutdown/abort path before the CIL push worker processes it, the freed memory remains linked inctx->log_items.The CIL push worker in
xlog_cil_build_lv_chain()then dereferences the freed object viaitem->li_lv, triggering a KASAN slab-use-after-free. For details, see Link[1].There's no reproducer there. Do you have a local one?
No, I don't have one either. It's just based on code analysis.
Quoted 45 lines
Quoted 9 lines
Add
down_read()onxc_ctx_lockbeforelist_del_init()inxfs_buf_item_done()to safely remove the item from the CIL list. This uses the same lock that protects CIL list operations: insertions are done underxc_ctx_lockread-side (xlog_cil_insert_items) and removals under write-side (xlog_cil_build_lv_chain). The read lock is safe here becausexfs_buf_item_done()is always called in process context (workqueue or direct I/O wait) and cannot deadlock with the CIL push worker which holds the write lock duringxlog_cil_build_lv_chain- the worker does not trigger metadata buffer I/O that would callxfs_buf_item_done().This looks like a more general issue as we should never free anything that is still on the CIL. I.e. it looks like we have even more issues with the buf item state machine here :(
Quoted 30 lines
Reported-by [email protected]Fixes 816c330b605c ("xfs: factor out stale buffer item completion")Suggested-by Zhan Jun <[email protected]>Signed-off-by Gou Hao <[email protected]>--- fs/xfs/xfs_buf_item.c | 5 +++++ 1 file changed, 5 insertions(+)fs/xfs/xfs_buf_item.c +5
-0@@ -1067,6 +1067,11 @@ voidxfs_buf_item_done ( struct xfs_buf *bp) { +if (bp->b_log_item->bli_item.li_log->l_cilp) { +down_read (&bp->b_log_item->bli_item.li_log->l_cilp->xc_ctx_lock); +list_del_init (&bp->b_log_item->bli_item.li_cil); +up_read (&bp->b_log_item->bli_item.li_log->l_cilp->xc_ctx_lock); + }/* * If we are forcibly shutting down, this may well be off the AIL * already. That's because we simulate the log-committed callbacks to Signature · 1 line
2.20.1---end quoted text---
Quoted 22 lines · Christoph Hellwig
Quoted 8 lines · Gou Hao
xfs_buf_item_done()frees thebuf_itemviaxfs_buf_item_relse()but does not remove the item from the CILlog_itemslist (li_cil). When the item is freed through an error/shutdown/abort path before the CIL push worker processes it, the freed memory remains linked inctx->log_items.The CIL push worker in
xlog_cil_build_lv_chain()then dereferences the freed object viaitem->li_lv, triggering a KASAN slab-use-after-free. For details, see Link[1].There's no reproducer there. Do you have a local one?
Quoted 9 lines
Add
down_read()onxc_ctx_lockbeforelist_del_init()inxfs_buf_item_done()to safely remove the item from the CIL list. This uses the same lock that protects CIL list operations: insertions are done underxc_ctx_lockread-side (xlog_cil_insert_items) and removals under write-side (xlog_cil_build_lv_chain). The read lock is safe here becausexfs_buf_item_done()is always called in process context (workqueue or direct I/O wait) and cannot deadlock with the CIL push worker which holds the write lock duringxlog_cil_build_lv_chain- the worker does not trigger metadata buffer I/O that would callxfs_buf_item_done().
Quoted 3 lines
This looks like a more general issue as we should never free anything that is still on the CIL. I.e. it looks like we have even more issues with the buf item state machine here :(
I finally found some time to look at this syzbot report and do some analysis of it. AFAICT, the BLI life cycle is solid.
Go have a look at the syzbot report. i.e. where KASAN reports that the BLI has been freed from. It is freed from buffer read IO completion. Now go and have a look at xfs_buf_ioend(): the read IO completion does not -ever- access attached BLIs - even on IO failure
- let alone free them. Only the write IO completion path (i.e. !XBF_READ) accesses the attached BLI.
IOWs, the KASAN trace is telling us we've got a read IO completion *without* XBF_READ being set on the buffer, and that freeing the BLI from this context is how we ended up with the CIL reference to the BLI being removed incorrectly leading to the UAF.
I spent some time trying to come up with ways we could have a read IO completion run without XBF_READ being set, and I cannot find it. No combination of corruption errors, reverifier, readahead, stale, and/or shutdown races appear to allow a READ IO without XBF_READ being set on the buffer through to IO completion, even in manual failure paths like xfs_buf_fail().
I can only conclude that this was caused by one of two things:
- semaphores got broken unexpectedly; or
- memory corruption from some other syzbot test that was running at the same time trashed bp->b_flags whilst the buffer was under IO.
Given that syzbot has only reported this 5 times in 12 hours only on a 7.1-rc3 kernel, never before and never since, an external memory corruption bug that has since been fixed seems like the most like cause here.
IOWs, I think there's nothing in XFS to fix here, and the syzbot report should simply be closed.
-Dave.
Signature · 2 lines
Dave Chinner [email protected]