Skip to contentSkip to conversation

jbd2: Fix null-ptr-deref when process reserved list in jbd2_journal_commit_transaction

PATCH Superseded Ye Bin 2 participants 3 messages Mar 17, 2022 1 message not in the archive Journalling layer for block devices (JBD2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Ye Bin

author
#0 lore raw

PATCH 1/1 jbd2: Fix null-ptr-deref when process reserved list in jbd2_journal_commit_transaction

we got issue as follows:

[   72.796117] EXT4-fs error (device sda): ext4_journal_check_start:83: comm fallocate: Detected aborted journal
[   72.826847] EXT4-fs (sda): Remounting filesystem read-only
fallocate: fallocate failed: Read-only file system
[   74.791830] jbd2_journal_commit_transaction: jh=0xffff9cfefe725d90 bh=0x0000000000000000 end delay
[   74.793597] ------------[ cut here ]------------
[   74.794203] kernel BUG at fs/jbd2/transaction.c:2063!
[   74.794886] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[   74.795533] CPU: 4 PID: 2260 Comm: jbd2/sda-8 Not tainted 5.17.0-rc8-next-20220315-dirty #150
[   74.798327] RIP: 0010:__jbd2_journal_unfile_buffer+0x3e/0x60
[   74.801971] RSP: 0018:ffffa828c24a3cb8 EFLAGS: 00010202
[   74.802694] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[   74.803601] RDX: 0000000000000001 RSI: ffff9cfefe725d90 RDI: ffff9cfefe725d90
[   74.804554] RBP: ffff9cfefe725d90 R08: 0000000000000000 R09: ffffa828c24a3b20
[   74.805471] R10: 0000000000000001 R11: 0000000000000001 R12: ffff9cfefe725d90
[   74.806385] R13: ffff9cfefe725d98 R14: 0000000000000000 R15: ffff9cfe833a4d00
[   74.807301] FS:  0000000000000000(0000) GS:ffff9d01afb00000(0000) knlGS:0000000000000000
[   74.808338] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   74.809084] CR2: 00007f2b81bf4000 CR3: 0000000100056000 CR4: 00000000000006e0
[   74.810047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   74.810981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   74.811897] Call Trace:
[   74.812241]  <TASK>
[   74.812566]  __jbd2_journal_refile_buffer+0x12f/0x180
[   74.813246]  jbd2_journal_refile_buffer+0x4c/0xa0
[   74.813869]  jbd2_journal_commit_transaction.cold+0xa1/0x148
[   74.817550]  kjournald2+0xf8/0x3e0
[   74.819056]  kthread+0x153/0x1c0
[   74.819963]  ret_from_fork+0x22/0x30

Above issue may happen as follows:
write truncate kjournald2
generic_perform_write
ext4_write_begin
ext4_walk_page_buffers
do_journal_get_write_access ->add BJ_Reserved list
ext4_journalled_write_end
ext4_walk_page_buffers
write_end_fn

    ext4_handle_dirty_metadata
                ***************JBD2 ABORT**************
     jbd2_journal_dirty_metadata
 -> return -EROFS, jh in reserved_list
                                                   jbd2_journal_commit_transaction
                                                    while (commit_transaction->t_reserved_list)
                                                      jh = commit_transaction->t_reserved_list;
                        truncate_pagecache_range
                         do_invalidatepage
			  ext4_journalled_invalidatepage
			   jbd2_journal_invalidatepage
			    journal_unmap_buffer
			     __dispose_buffer
			      __jbd2_journal_unfile_buffer
			       jbd2_journal_put_journal_head ->put last ref_count
			        __journal_remove_journal_head
				 bh->b_private = NULL;
				 jh->b_bh = NULL;
				                      jbd2_journal_refile_buffer(journal, jh);
							bh = jh2bh(jh);
							->bh is NULL, later will trigger null-ptr-deref
				 journal_free_journal_head(jh);

As after 96f1e0974575 commit, handle reserved list will not hold "journal->j_state_lock" when kjournald2 commit transaction. So journal_unmap_buffer maybe free journal_head when handle reserved list. And lead to null-ptr-deref or some strange errors.
As reserved list almost time is empty. Use "journal->j_state_lock" to protect handle reserved list can simply solve above issue.

Fixes 96f1e0974575("jbd2: avoid long hold times of j_state_lock while committing a transaction")
---
 fs/jbd2/commit.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/jbd2/commit.c +1 -1
index 5b9408e3b370..2b737b928d26 100644
@@ -488,7 +488,6 @@ void jbd2_journal_commit_transaction(journal_t *journal)
 	jbd2_journal_wait_updates(journal);
 
 	commit_transaction->t_state = T_SWITCH;
-	write_unlock(&journal->j_state_lock);
 
 	J_ASSERT (atomic_read(&commit_transaction->t_outstanding_credits) <=
 			journal->j_max_transaction_buffers);
@@ -527,6 +526,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)
 		jbd2_journal_refile_buffer(journal, jh);
 	}
 
+	write_unlock(&journal->j_state_lock);
 	/*
 	 * Now try to drop any written-back buffers from the journal's
 	 * checkpoint lists.  We do this *before* commit because it potentially
Signature · 1 line
2.31.1
#1 lore raw
Quoted 69 lines · Ye Bin

we got issue as follows:

[   72.796117] EXT4-fs error (device sda): ext4_journal_check_start:83: comm fallocate: Detected aborted journal
[   72.826847] EXT4-fs (sda): Remounting filesystem read-only
fallocate: fallocate failed: Read-only file system
[   74.791830] jbd2_journal_commit_transaction: jh=0xffff9cfefe725d90 bh=0x0000000000000000 end delay
[   74.793597] ------------[ cut here ]------------
[   74.794203] kernel BUG at fs/jbd2/transaction.c:2063!
[   74.794886] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[   74.795533] CPU: 4 PID: 2260 Comm: jbd2/sda-8 Not tainted 5.17.0-rc8-next-20220315-dirty #150
[   74.798327] RIP: 0010:__jbd2_journal_unfile_buffer+0x3e/0x60
[   74.801971] RSP: 0018:ffffa828c24a3cb8 EFLAGS: 00010202
[   74.802694] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[   74.803601] RDX: 0000000000000001 RSI: ffff9cfefe725d90 RDI: ffff9cfefe725d90
[   74.804554] RBP: ffff9cfefe725d90 R08: 0000000000000000 R09: ffffa828c24a3b20
[   74.805471] R10: 0000000000000001 R11: 0000000000000001 R12: ffff9cfefe725d90
[   74.806385] R13: ffff9cfefe725d98 R14: 0000000000000000 R15: ffff9cfe833a4d00
[   74.807301] FS:  0000000000000000(0000) GS:ffff9d01afb00000(0000) knlGS:0000000000000000
[   74.808338] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   74.809084] CR2: 00007f2b81bf4000 CR3: 0000000100056000 CR4: 00000000000006e0
[   74.810047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   74.810981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   74.811897] Call Trace:
[   74.812241]  <TASK>
[   74.812566]  __jbd2_journal_refile_buffer+0x12f/0x180
[   74.813246]  jbd2_journal_refile_buffer+0x4c/0xa0
[   74.813869]  jbd2_journal_commit_transaction.cold+0xa1/0x148
[   74.817550]  kjournald2+0xf8/0x3e0
[   74.819056]  kthread+0x153/0x1c0
[   74.819963]  ret_from_fork+0x22/0x30

Above issue may happen as follows:
write truncate kjournald2
generic_perform_write
ext4_write_begin
ext4_walk_page_buffers
do_journal_get_write_access ->add BJ_Reserved list
ext4_journalled_write_end
ext4_walk_page_buffers
write_end_fn

    ext4_handle_dirty_metadata
                ***************JBD2 ABORT**************
     jbd2_journal_dirty_metadata
 -> return -EROFS, jh in reserved_list
                                                   jbd2_journal_commit_transaction
                                                    while (commit_transaction->t_reserved_list)
                                                      jh = commit_transaction->t_reserved_list;
                        truncate_pagecache_range
                         do_invalidatepage
			  ext4_journalled_invalidatepage
			   jbd2_journal_invalidatepage
			    journal_unmap_buffer
			     __dispose_buffer
			      __jbd2_journal_unfile_buffer
			       jbd2_journal_put_journal_head ->put last ref_count
			        __journal_remove_journal_head
				 bh->b_private = NULL;
				 jh->b_bh = NULL;
				                      jbd2_journal_refile_buffer(journal, jh);
							bh = jh2bh(jh);
							->bh is NULL, later will trigger null-ptr-deref
				 journal_free_journal_head(jh);

As after 96f1e0974575 commit, handle reserved list will not hold "journal->j_state_lock" when kjournald2 commit transaction. So journal_unmap_buffer maybe free journal_head when handle reserved list. And lead to null-ptr-deref or some strange errors.
As reserved list almost time is empty. Use "journal->j_state_lock" to protect handle reserved list can simply solve above issue.

Quoted 2 lines
Fixes 96f1e0974575("jbd2: avoid long hold times of j_state_lock while committing a transaction")

Good spotting! Thanks for the analysis and the patch!

Quoted 13 lines
fs/jbd2/commit.c +0 -1
index 5b9408e3b370..2b737b928d26 100644
@@ -488,7 +488,6 @@ void jbd2_journal_commit_transaction(journal_t *journal)
 	jbd2_journal_wait_updates(journal);
 
 	commit_transaction->t_state = T_SWITCH;
-	write_unlock(&journal->j_state_lock);
 
 	J_ASSERT (atomic_read(&commit_transaction->t_outstanding_credits) <=
 			journal->j_max_transaction_buffers);

fs/jbd2/commit.c void jbd2_journal_commit_transaction(journal_t *journal) · line 532

Quoted 7 lines
fs/jbd2/commit.c +1 -0
@@ -527,6 +526,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)
 		jbd2_journal_refile_buffer(journal, jh);
 	}
 
+	write_unlock(&journal->j_state_lock);
 	/*
 	 * Now try to drop any written-back buffers from the journal's
 	 * checkpoint lists.  We do this *before* commit because it potentially

Honestly, using j_state_lock seems like unnecessarily big hammer for this. I'd rather use journal->j_list_lock for this which is much more natural for list processing. And we grab it anyway from jbd2_journal_refile_buffer() so it is not like there will be any additional overhead anyway.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 99 lines · yebin
Quoted 96 lines · Jan Kara
Quoted 69 lines · Ye Bin

we got issue as follows:

[   72.796117] EXT4-fs error (device sda): ext4_journal_check_start:83: comm fallocate: Detected aborted journal
[   72.826847] EXT4-fs (sda): Remounting filesystem read-only
fallocate: fallocate failed: Read-only file system
[   74.791830] jbd2_journal_commit_transaction: jh=0xffff9cfefe725d90 bh=0x0000000000000000 end delay
[   74.793597] ------------[ cut here ]------------
[   74.794203] kernel BUG at fs/jbd2/transaction.c:2063!
[   74.794886] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[   74.795533] CPU: 4 PID: 2260 Comm: jbd2/sda-8 Not tainted 5.17.0-rc8-next-20220315-dirty #150
[   74.798327] RIP: 0010:__jbd2_journal_unfile_buffer+0x3e/0x60
[   74.801971] RSP: 0018:ffffa828c24a3cb8 EFLAGS: 00010202
[   74.802694] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[   74.803601] RDX: 0000000000000001 RSI: ffff9cfefe725d90 RDI: ffff9cfefe725d90
[   74.804554] RBP: ffff9cfefe725d90 R08: 0000000000000000 R09: ffffa828c24a3b20
[   74.805471] R10: 0000000000000001 R11: 0000000000000001 R12: ffff9cfefe725d90
[   74.806385] R13: ffff9cfefe725d98 R14: 0000000000000000 R15: ffff9cfe833a4d00
[   74.807301] FS:  0000000000000000(0000) GS:ffff9d01afb00000(0000) knlGS:0000000000000000
[   74.808338] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   74.809084] CR2: 00007f2b81bf4000 CR3: 0000000100056000 CR4: 00000000000006e0
[   74.810047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   74.810981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   74.811897] Call Trace:
[   74.812241]  <TASK>
[   74.812566]  __jbd2_journal_refile_buffer+0x12f/0x180
[   74.813246]  jbd2_journal_refile_buffer+0x4c/0xa0
[   74.813869]  jbd2_journal_commit_transaction.cold+0xa1/0x148
[   74.817550]  kjournald2+0xf8/0x3e0
[   74.819056]  kthread+0x153/0x1c0
[   74.819963]  ret_from_fork+0x22/0x30

Above issue may happen as follows:
write truncate kjournald2
generic_perform_write
ext4_write_begin
ext4_walk_page_buffers

    do_journal_get_write_access ->add BJ_Reserved list
  ext4_journalled_write_end
   ext4_walk_page_buffers
    write_end_fn
     ext4_handle_dirty_metadata
                 ***************JBD2 ABORT**************
      jbd2_journal_dirty_metadata
  -> return -EROFS, jh in reserved_list
                                                    jbd2_journal_commit_transaction
                                                     while (commit_transaction->t_reserved_list)
                                                       jh = commit_transaction->t_reserved_list;
                         truncate_pagecache_range
                          do_invalidatepage
			  ext4_journalled_invalidatepage
			   jbd2_journal_invalidatepage
			    journal_unmap_buffer
			     __dispose_buffer
			      __jbd2_journal_unfile_buffer
			       jbd2_journal_put_journal_head ->put last ref_count
			        __journal_remove_journal_head
				 bh->b_private = NULL;
				 jh->b_bh = NULL;
				                      jbd2_journal_refile_buffer(journal, jh);
							bh = jh2bh(jh);
							->bh is NULL, later will trigger null-ptr-deref
				 journal_free_journal_head(jh);

As after 96f1e0974575 commit, handle reserved list will not hold "journal->j_state_lock" when kjournald2 commit transaction. So journal_unmap_buffer maybe free journal_head when handle reserved list. And lead to null-ptr-deref or some strange errors.
As reserved list almost time is empty. Use "journal->j_state_lock" to protect handle reserved list can simply solve above issue.

Quoted 2 lines
Fixes 96f1e0974575("jbd2: avoid long hold times of j_state_lock while committing a transaction")

Good spotting! Thanks for the analysis and the patch!

Quoted 11 lines
fs/jbd2/commit.c +0 -1
index 5b9408e3b370..2b737b928d26 100644
@@ -488,7 +488,6 @@ void jbd2_journal_commit_transaction(journal_t *journal)
  	jbd2_journal_wait_updates(journal);
  	commit_transaction->t_state = T_SWITCH;
-	write_unlock(&journal->j_state_lock);
  	J_ASSERT (atomic_read(&commit_transaction->t_outstanding_credits) <=
  			journal->j_max_transaction_buffers);

fs/jbd2/commit.c void jbd2_journal_commit_transaction(journal_t *journal) · line 531

Quoted 6 lines
fs/jbd2/commit.c +1 -0
@@ -527,6 +526,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)
  		jbd2_journal_refile_buffer(journal, jh);
  	}
+	write_unlock(&journal->j_state_lock);
  	/*
  	 * Now try to drop any written-back buffers from the journal's
  	 * checkpoint lists.  We do this *before* commit because it potentially

Honestly, using j_state_lock seems like unnecessarily big hammer for this. I'd rather use journal->j_list_lock for this which is much more natural for list processing. And we grab it anyway from jbd2_journal_refile_buffer() so it is not like there will be any additional overhead anyway.

Quoted 6 lines
Quoted 1 line

Honza

If use journal->j_list_lock for reserved list, we must change order of jh->b_state_lock and
journal->j_list_lock. <../cgi-bin/global.cgi?pattern=j_list_lock&type=symbol>
I have another idea as follows. Do you think it is feasible?

Oh right, I forgot the lock ordering is not very convenient for these two locks. Ok, then I probably prefer your original solution to the one below. But please add a comment like:

We use journal->j_state_lock here to serialize processing of t_reserved_list with eviction of buffers from journal_unmap_buffer().

Thanks!

Honza

Quoted 29 lines
fs/jbd2/transaction.c +1 -0
index 73ed02f061e1..00c51a92a5cc 100644
@@ -2282,6 +2282,7 @@ static int journal_unmap_buffer(journal_t *journal,
 struct buffer_head *bh,
        if (!jh)
                goto zap_buffer_unlocked;
 
+repeat:
        /* OK, we have data buffer in journaled mode */
        write_lock(&journal->j_state_lock);
        spin_lock(&jh->b_state_lock);
fs/jbd2/transaction.c +5 -0
@@ -2357,6 +2358,11 @@ static int journal_unmap_buffer(journal_t *journal,
 struct buffer_head *bh,
                                goto zap_buffer;
                        }
                }
+       } else if (transaction->t_state == T_SWITCH) {
+               spin_unlock(&journal->j_list_lock);
+               spin_unlock(&jh->b_state_lock);
+               wait_transaction_switching(journal);
+               goto repeat;
        } else if (transaction == journal->j_committing_transaction) {
                JBUFFER_TRACE(jh, "on committing transaction");
                /*

--

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR