jbd2: Fix null-ptr-deref when process reserved list in jbd2_journal_commit_transaction
PATCH 1/1 jbd2: Fix null-ptr-deref when process reserved list in jbd2_journal_commit_transaction
we got issue as follows:
[ 72.796117] EXT4-fs error (device sda): ext4_journal_check_start:83: comm fallocate: Detected aborted journal [ 72.826847] EXT4-fs (sda): Remounting filesystem read-only fallocate: fallocate failed: Read-only file system [ 74.791830] jbd2_journal_commit_transaction: jh=0xffff9cfefe725d90 bh=0x0000000000000000 end delay [ 74.793597] ------------[ cut here ]------------ [ 74.794203] kernel BUG at fs/jbd2/transaction.c:2063! [ 74.794886] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 74.795533] CPU: 4 PID: 2260 Comm: jbd2/sda-8 Not tainted 5.17.0-rc8-next-20220315-dirty #150 [ 74.798327] RIP: 0010:__jbd2_journal_unfile_buffer+0x3e/0x60 [ 74.801971] RSP: 0018:ffffa828c24a3cb8 EFLAGS: 00010202 [ 74.802694] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 74.803601] RDX: 0000000000000001 RSI: ffff9cfefe725d90 RDI: ffff9cfefe725d90 [ 74.804554] RBP: ffff9cfefe725d90 R08: 0000000000000000 R09: ffffa828c24a3b20 [ 74.805471] R10: 0000000000000001 R11: 0000000000000001 R12: ffff9cfefe725d90 [ 74.806385] R13: ffff9cfefe725d98 R14: 0000000000000000 R15: ffff9cfe833a4d00 [ 74.807301] FS: 0000000000000000(0000) GS:ffff9d01afb00000(0000) knlGS:0000000000000000 [ 74.808338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 74.809084] CR2: 00007f2b81bf4000 CR3: 0000000100056000 CR4: 00000000000006e0 [ 74.810047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 74.810981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 74.811897] Call Trace: [ 74.812241] <TASK> [ 74.812566] __jbd2_journal_refile_buffer+0x12f/0x180 [ 74.813246] jbd2_journal_refile_buffer+0x4c/0xa0 [ 74.813869] jbd2_journal_commit_transaction.cold+0xa1/0x148 [ 74.817550] kjournald2+0xf8/0x3e0 [ 74.819056] kthread+0x153/0x1c0 [ 74.819963] ret_from_fork+0x22/0x30
Above issue may happen as follows:
write truncate kjournald2generic_perform_write
ext4_write_begin
ext4_walk_page_buffers
do_journal_get_write_access ->add BJ_Reserved list
ext4_journalled_write_end
ext4_walk_page_buffers
write_end_fn
ext4_handle_dirty_metadata
***************JBD2 ABORT**************
jbd2_journal_dirty_metadata
-> return -EROFS, jh in reserved_list
jbd2_journal_commit_transaction
while (commit_transaction->t_reserved_list)
jh = commit_transaction->t_reserved_list;
truncate_pagecache_range
do_invalidatepage
ext4_journalled_invalidatepage
jbd2_journal_invalidatepage
journal_unmap_buffer
__dispose_buffer
__jbd2_journal_unfile_buffer
jbd2_journal_put_journal_head ->put last ref_count
__journal_remove_journal_head
bh->b_private = NULL;
jh->b_bh = NULL;
jbd2_journal_refile_buffer(journal, jh);
bh = jh2bh(jh);
->bh is NULL, later will trigger null-ptr-deref
journal_free_journal_head(jh);As after 96f1e0974575 commit, handle reserved list will not hold "journal->j_state_lock" when kjournald2 commit transaction. So journal_unmap_buffer maybe free journal_head when handle reserved list. And lead to null-ptr-deref or some strange errors.
As reserved list almost time is empty. Use "journal->j_state_lock" to protect handle reserved list can simply solve above issue.
--- fs/jbd2/commit.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/jbd2/commit.c +1 -1
@@ -488,7 +488,6 @@ void jbd2_journal_commit_transaction(journal_t *journal)jbd2_journal_wait_updates (journal); commit_transaction->t_state =T_SWITCH ; -write_unlock (&journal->j_state_lock);J_ASSERT (atomic_read (&commit_transaction->t_outstanding_credits) <= journal->j_max_transaction_buffers);
@@ -527,6 +526,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)jbd2_journal_refile_buffer (journal, jh); } +write_unlock (&journal->j_state_lock);/* * Now try to drop any written-back buffers from the journal's * checkpoint lists. We do this *before* commit because it potentially
Signature · 1 line
2.31.1
Quoted 69 lines · Ye Bin
we got issue as follows:
[ 72.796117] EXT4-fs error (device sda): ext4_journal_check_start:83: comm fallocate: Detected aborted journal [ 72.826847] EXT4-fs (sda): Remounting filesystem read-only fallocate: fallocate failed: Read-only file system [ 74.791830] jbd2_journal_commit_transaction: jh=0xffff9cfefe725d90 bh=0x0000000000000000 end delay [ 74.793597] ------------[ cut here ]------------ [ 74.794203] kernel BUG at fs/jbd2/transaction.c:2063! [ 74.794886] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 74.795533] CPU: 4 PID: 2260 Comm: jbd2/sda-8 Not tainted 5.17.0-rc8-next-20220315-dirty #150 [ 74.798327] RIP: 0010:__jbd2_journal_unfile_buffer+0x3e/0x60 [ 74.801971] RSP: 0018:ffffa828c24a3cb8 EFLAGS: 00010202 [ 74.802694] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 74.803601] RDX: 0000000000000001 RSI: ffff9cfefe725d90 RDI: ffff9cfefe725d90 [ 74.804554] RBP: ffff9cfefe725d90 R08: 0000000000000000 R09: ffffa828c24a3b20 [ 74.805471] R10: 0000000000000001 R11: 0000000000000001 R12: ffff9cfefe725d90 [ 74.806385] R13: ffff9cfefe725d98 R14: 0000000000000000 R15: ffff9cfe833a4d00 [ 74.807301] FS: 0000000000000000(0000) GS:ffff9d01afb00000(0000) knlGS:0000000000000000 [ 74.808338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 74.809084] CR2: 00007f2b81bf4000 CR3: 0000000100056000 CR4: 00000000000006e0 [ 74.810047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 74.810981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 74.811897] Call Trace: [ 74.812241] <TASK> [ 74.812566] __jbd2_journal_refile_buffer+0x12f/0x180 [ 74.813246] jbd2_journal_refile_buffer+0x4c/0xa0 [ 74.813869] jbd2_journal_commit_transaction.cold+0xa1/0x148 [ 74.817550] kjournald2+0xf8/0x3e0 [ 74.819056] kthread+0x153/0x1c0 [ 74.819963] ret_from_fork+0x22/0x30Above issue may happen as follows:
write truncate kjournald2generic_perform_write
ext4_write_begin
ext4_walk_page_buffers
do_journal_get_write_access->addBJ_Reservedlist
ext4_journalled_write_end
ext4_walk_page_buffers
write_end_fnext4_handle_dirty_metadata ***************JBD2 ABORT************** jbd2_journal_dirty_metadata -> return -EROFS, jh in reserved_list jbd2_journal_commit_transaction while (commit_transaction->t_reserved_list) jh = commit_transaction->t_reserved_list; truncate_pagecache_range do_invalidatepage ext4_journalled_invalidatepage jbd2_journal_invalidatepage journal_unmap_buffer __dispose_buffer __jbd2_journal_unfile_buffer jbd2_journal_put_journal_head ->put last ref_count __journal_remove_journal_head bh->b_private = NULL; jh->b_bh = NULL; jbd2_journal_refile_buffer(journal, jh); bh = jh2bh(jh); ->bh is NULL, later will trigger null-ptr-deref journal_free_journal_head(jh);As after
96f1e0974575commit, handle reserved list will not hold "journal->j_state_lock" when kjournald2 commit transaction. Sojournal_unmap_buffermaybe freejournal_headwhen handle reserved list. And lead to null-ptr-deref or some strange errors.
As reserved list almost time is empty. Use "journal->j_state_lock" to protect handle reserved list can simply solve above issue.
Quoted 2 lines
Fixes 96f1e0974575("jbd2: avoid long hold times of j_state_lock while committing a transaction")Signed-off-by Ye Bin <[email protected]>
Good spotting! Thanks for the analysis and the patch!
Quoted 13 lines
fs/jbd2/commit.c +0
-1@@ -488,7 +488,6 @@ void jbd2_journal_commit_transaction(journal_t *journal)jbd2_journal_wait_updates (journal); commit_transaction->t_state =T_SWITCH ; -write_unlock (&journal->j_state_lock);J_ASSERT (atomic_read (&commit_transaction->t_outstanding_credits) <= journal->j_max_transaction_buffers);
fs/jbd2/commit.c void jbd2_journal_commit_transaction(journal_t *journal) · line 532
Quoted 7 lines
fs/jbd2/commit.c +1
-0@@ -527,6 +526,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)jbd2_journal_refile_buffer (journal, jh); } +write_unlock (&journal->j_state_lock);/* * Now try to drop any written-back buffers from the journal's * checkpoint lists. We do this *before* commit because it potentially
Honestly, using j_state_lock seems like unnecessarily big hammer for this. I'd rather use journal->j_list_lock for this which is much more natural for list processing. And we grab it anyway from jbd2_journal_refile_buffer() so it is not like there will be any additional overhead anyway.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 99 lines · yebin
Quoted 96 lines · Jan Kara
Quoted 69 lines · Ye Bin
we got issue as follows:
[ 72.796117] EXT4-fs error (device sda): ext4_journal_check_start:83: comm fallocate: Detected aborted journal [ 72.826847] EXT4-fs (sda): Remounting filesystem read-only fallocate: fallocate failed: Read-only file system [ 74.791830] jbd2_journal_commit_transaction: jh=0xffff9cfefe725d90 bh=0x0000000000000000 end delay [ 74.793597] ------------[ cut here ]------------ [ 74.794203] kernel BUG at fs/jbd2/transaction.c:2063! [ 74.794886] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 74.795533] CPU: 4 PID: 2260 Comm: jbd2/sda-8 Not tainted 5.17.0-rc8-next-20220315-dirty #150 [ 74.798327] RIP: 0010:__jbd2_journal_unfile_buffer+0x3e/0x60 [ 74.801971] RSP: 0018:ffffa828c24a3cb8 EFLAGS: 00010202 [ 74.802694] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 74.803601] RDX: 0000000000000001 RSI: ffff9cfefe725d90 RDI: ffff9cfefe725d90 [ 74.804554] RBP: ffff9cfefe725d90 R08: 0000000000000000 R09: ffffa828c24a3b20 [ 74.805471] R10: 0000000000000001 R11: 0000000000000001 R12: ffff9cfefe725d90 [ 74.806385] R13: ffff9cfefe725d98 R14: 0000000000000000 R15: ffff9cfe833a4d00 [ 74.807301] FS: 0000000000000000(0000) GS:ffff9d01afb00000(0000) knlGS:0000000000000000 [ 74.808338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 74.809084] CR2: 00007f2b81bf4000 CR3: 0000000100056000 CR4: 00000000000006e0 [ 74.810047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 74.810981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 74.811897] Call Trace: [ 74.812241] <TASK> [ 74.812566] __jbd2_journal_refile_buffer+0x12f/0x180 [ 74.813246] jbd2_journal_refile_buffer+0x4c/0xa0 [ 74.813869] jbd2_journal_commit_transaction.cold+0xa1/0x148 [ 74.817550] kjournald2+0xf8/0x3e0 [ 74.819056] kthread+0x153/0x1c0 [ 74.819963] ret_from_fork+0x22/0x30Above issue may happen as follows:
write truncate kjournald2generic_perform_write
ext4_write_begin
ext4_walk_page_buffersdo_journal_get_write_access ->add BJ_Reserved list ext4_journalled_write_end ext4_walk_page_buffers write_end_fn ext4_handle_dirty_metadata ***************JBD2 ABORT************** jbd2_journal_dirty_metadata -> return -EROFS, jh in reserved_list jbd2_journal_commit_transaction while (commit_transaction->t_reserved_list) jh = commit_transaction->t_reserved_list; truncate_pagecache_range do_invalidatepage ext4_journalled_invalidatepage jbd2_journal_invalidatepage journal_unmap_buffer __dispose_buffer __jbd2_journal_unfile_buffer jbd2_journal_put_journal_head ->put last ref_count __journal_remove_journal_head bh->b_private = NULL; jh->b_bh = NULL; jbd2_journal_refile_buffer(journal, jh); bh = jh2bh(jh); ->bh is NULL, later will trigger null-ptr-deref journal_free_journal_head(jh);As after
96f1e0974575commit, handle reserved list will not hold "journal->j_state_lock" when kjournald2 commit transaction. Sojournal_unmap_buffermaybe freejournal_headwhen handle reserved list. And lead to null-ptr-deref or some strange errors.
As reserved list almost time is empty. Use "journal->j_state_lock" to protect handle reserved list can simply solve above issue.Quoted 2 lines
Fixes 96f1e0974575("jbd2: avoid long hold times of j_state_lock while committing a transaction")Signed-off-by Ye Bin <[email protected]>Good spotting! Thanks for the analysis and the patch!
Quoted 11 lines
fs/jbd2/commit.c +0
-1@@ -488,7 +488,6 @@ void jbd2_journal_commit_transaction(journal_t *journal)jbd2_journal_wait_updates (journal); commit_transaction->t_state =T_SWITCH ; -write_unlock (&journal->j_state_lock);J_ASSERT (atomic_read (&commit_transaction->t_outstanding_credits) <= journal->j_max_transaction_buffers);fs/jbd2/commit.c void jbd2_journal_commit_transaction(journal_t *journal) · line 531
Quoted 6 lines
fs/jbd2/commit.c +1
-0@@ -527,6 +526,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)jbd2_journal_refile_buffer (journal, jh); } +write_unlock (&journal->j_state_lock);/* * Now try to drop any written-back buffers from the journal's * checkpoint lists. We do this *before* commit because it potentially Honestly, using
j_state_lockseems like unnecessarily big hammer for this. I'd rather usejournal->j_list_lockfor this which is much more natural for list processing. And we grab it anyway fromjbd2_journal_refile_buffer()so it is not like there will be any additional overhead anyway.
Quoted 6 lines
Quoted 1 line
Honza
If use
journal->j_list_lockfor reserved list, we must change order ofjh->b_state_lockandjournal->j_list_lock. <../cgi-bin/global.cgi?pattern=j_list_lock&type=symbol>
I have another idea as follows. Do you think it is feasible?
Oh right, I forgot the lock ordering is not very convenient for these two locks. Ok, then I probably prefer your original solution to the one below. But please add a comment like:
We use journal->j_state_lock here to serialize processing of t_reserved_list with eviction of buffers from journal_unmap_buffer().
Thanks!
Honza
Quoted 29 lines
fs/jbd2/transaction.c +1
-0@@ -2282,6 +2282,7 @@ static int journal_unmap_buffer(journal_t *journal, struct buffer_head *bh, if (!jh) goto zap_buffer_unlocked; +repeat:/* OK, we have data buffer in journaled mode */ write_lock (&journal->j_state_lock);spin_lock(&jh->b_state_lock);fs/jbd2/transaction.c +5
-0@@ -2357,6 +2358,11 @@ static int journal_unmap_buffer(journal_t *journal,struct buffer_head *bh, goto zap_buffer; } } + }else if (transaction->t_state ==T_SWITCH ) { +spin_unlock (&journal->j_list_lock); +spin_unlock (&jh->b_state_lock); +wait_transaction_switching (journal); + goto repeat; }else if (transaction == journal->j_committing_transaction) {JBUFFER_TRACE (jh,"on committing transaction" );/*--
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR