jbd: Fix a race between checkpointing code and journal_get_write_access()
PATCH 1/1 jbd: Fix a race between checkpointing code and journal_get_write_access()
The following race can happen:
CPU1 CPU2
checkpointing code checks the buffer, adds
it to an array for writebackdo_get_write_access()
...
lock_buffer()
unlock_buffer()
flush_batch() submits the buffer for IO
__jbd_journal_file_buffer()
So a buffer under writeout is returned from do_get_write_access(). Since
the filesystem code relies on the fact that journaled buffers cannot be written out, it does not take the buffer lock and so it can modify buffer while it is under writeout. That can lead to a filesystem corruption if we crash at the right moment.
We fix the problem by clearing the buffer dirty bit under buffer_lock
even if the buffer is on BJ_None list. Actually, we clear the dirty bit regardless the list the buffer is in and warn about the fact if the buffer is already journalled.
Thanks for spotting the problem goes to dingdinghua <[email protected]>.
--- fs/jbd/transaction.c | 52 +++++++++++++++++++------------------------------ 1 files changed, 20 insertions(+), 32 deletions(-)
fs/jbd/transaction.c +11 -31
@@ -489,34 +489,15 @@ void journal_unlock_updates (journal_t *journal)wake_up (&journal->j_wait_transaction_locked); } -/* -* Report any unexpected dirty buffers which turn up. Normally those -* indicate an error, but they can occur if the user is running (say) -* tune2fs to modify the live filesystem, so we need the option of -* continuing as gracefully as possible. # -* -* The caller should already hold the journal lock and -* j_list_lock spinlock: most callers will need those anyway -* in order to probe the buffer's journaling state safely. -*/ -static void jbd_unexpected_dirty_buffer (struct journal_head *jh) +static void warn_dirty_buffer (struct buffer_head *bh) { -int jlist; - -/* If this buffer is one which might reasonably be dirty -* --- ie. data, or not part of this journal --- then -* we're OK to leave it alone, but otherwise we need to -* move the dirty bit to the journal's own internal -* JBDDirty bit. */ - jlist = jh->b_jlist; - -if (jlist == BJ_Metadata || jlist == BJ_Reserved || - jlist == BJ_Shadow || jlist == BJ_Forget) { -struct buffer_head *bh =jh2bh (jh); +char b[BDEVNAME_SIZE ]; -if (test_clear_buffer_dirty (bh)) -set_buffer_jbddirty (bh); - } +printk (KERN_WARNING +"JBD: Spotted dirty metadata buffer (dev = %s, blocknr = %llu). " +"There's a risk of filesystem corruption in case of system " +"crash.\n" , +bdevname (bh->b_bdev, b), (unsigned long long )bh->b_blocknr); }/*
@@ -583,14 +564,16 @@ repeat:if (jh->b_next_transaction)J_ASSERT_JH (jh, jh->b_next_transaction == transaction); +warn_dirty_buffer (bh); }/* * In any case we need to clean the dirty flag and we must * do it under the buffer lock to be sure we don't race * with running write-out. */ -JBUFFER_TRACE (jh,"Unexpected dirty buffer" ); -jbd_unexpected_dirty_buffer (jh); +JBUFFER_TRACE (jh,"Journalling dirty buffer" ); +clear_buffer_dirty (bh); +set_buffer_jbddirty (bh); }unlock_buffer (bh);
@@ -2041,12 +2024,17 @@ void __journal_file_buffer(struct journal_head *jh,if (jh->b_transaction && jh->b_jlist == jlist)return ; -/* The following list of buffer states needs to be consistent -* with __jbd_unexpected_dirty_buffer()'s handling of dirty -* state. */
PATCH 1/1 jbd2: Fix a race between checkpointing code and journal_get_write_access()
The following race can happen:
CPU1 CPU2
checkpointing code checks the buffer, adds
it to an array for writebackdo_get_write_access()
...
lock_buffer()
unlock_buffer()
flush_batch() submits the buffer for IO
__jbd2_journal_file_buffer()
So a buffer under writeout is returned from do_get_write_access(). Since
the filesystem code relies on the fact that journaled buffers cannot be written out, it does not take the buffer lock and so it can modify buffer while it is under writeout. That can lead to a filesystem corruption if we crash at the right moment.
We fix the problem by clearing the buffer dirty bit under buffer_lock
even if the buffer is on BJ_None list. Actually, we clear the dirty bit regardless the list the buffer is in and warn about the fact if the buffer is already journalled.
Thanks for spotting the problem goes to dingdinghua <[email protected]>.
--- fs/jbd2/transaction.c | 52 ++++++++++++++++++------------------------------ 1 files changed, 20 insertions(+), 32 deletions(-)
fs/jbd2/transaction.c +11 -31
@@ -499,34 +499,15 @@ void jbd2_journal_unlock_updates (journal_t *journal)wake_up (&journal->j_wait_transaction_locked); } -/* -* Report any unexpected dirty buffers which turn up. Normally those -* indicate an error, but they can occur if the user is running (say) -* tune2fs to modify the live filesystem, so we need the option of -* continuing as gracefully as possible. # -* -* The caller should already hold the journal lock and -* j_list_lock spinlock: most callers will need those anyway -* in order to probe the buffer's journaling state safely. -*/ -static void jbd_unexpected_dirty_buffer (struct journal_head *jh) +static void warn_dirty_buffer (struct buffer_head *bh) { -int jlist; - -/* If this buffer is one which might reasonably be dirty -* --- ie. data, or not part of this journal --- then -* we're OK to leave it alone, but otherwise we need to -* move the dirty bit to the journal's own internal -* JBDDirty bit. */ - jlist = jh->b_jlist; - -if (jlist == BJ_Metadata || jlist == BJ_Reserved || - jlist == BJ_Shadow || jlist == BJ_Forget) { -struct buffer_head *bh =jh2bh (jh); +char b[BDEVNAME_SIZE ]; -if (test_clear_buffer_dirty (bh)) -set_buffer_jbddirty (bh); - } +printk (KERN_WARNING +"JBD: Spotted dirty metadata buffer (dev = %s, blocknr = %llu). " +"There's a risk of filesystem corruption in case of system " +"crash.\n" , +bdevname (bh->b_bdev, b), (unsigned long long )bh->b_blocknr); }/*
@@ -593,14 +574,16 @@ repeat:if (jh->b_next_transaction)J_ASSERT_JH (jh, jh->b_next_transaction == transaction); +warn_dirty_buffer (bh); }/* * In any case we need to clean the dirty flag and we must * do it under the buffer lock to be sure we don't race * with running write-out. */ -JBUFFER_TRACE (jh,"Unexpected dirty buffer" ); -jbd_unexpected_dirty_buffer (jh); +JBUFFER_TRACE (jh,"Journalling dirty buffer" ); +clear_buffer_dirty (bh); +set_buffer_jbddirty (bh); }unlock_buffer (bh);
@@ -1896,12 +1879,17 @@ void __jbd2_journal_file_buffer(struct journal_head *jh,if (jh->b_transaction && jh->b_jlist == jlist)return ; -/* The following list of buffer states needs to be consistent -* with __jbd_unexpected_dirty_buffer()'s handling of dirty -* state. */
Re: [PATCH] jbd2: Fix a race between checkpointing code and journal_get_write_access()
Quoted 22 lines · Jan Kara
The following race can happen:
CPU1 CPU2
checkpointing code checks the buffer, adds
it to an array for writebackdo_get_write_access()
...
lock_buffer()
unlock_buffer()
flush_batch()submits the buffer for IO
__jbd2_journal_file_buffer()So a buffer under writeout is returned from
do_get_write_access(). Since
the filesystem code relies on the fact that journaled buffers cannot be written out, it does not take the buffer lock and so it can modify buffer while it is under writeout. That can lead to a filesystem corruption if we crash at the right moment.
We fix the problem by clearing the buffer dirty bit underbuffer_lock
even if the buffer is onBJ_Nonelist. Actually, we clear the dirty bit regardless the list the buffer is in and warn about the fact if the buffer is already journalled.
Quoted 4 lines
Thanks for spotting the problem goes to dingdinghua <[email protected]>.
Reported-by dingdinghua <[email protected]>Signed-off-by Jan Kara <[email protected]>
Thanks, applied to the ext4 patch queue.
- Ted
Theodore Tso
maintainerRe: [PATCH] jbd2: Fix a race between checkpointing code and journal_get_write_access()
Quoted 13 lines · Theodore Tso
Quoted 12 lines · Jan Kara
The following race can happen:
CPU1 CPU2
checkpointing code checks the buffer, adds
it to an array for writebackdo_get_write_access()
...
lock_buffer()
unlock_buffer()
flush_batch()submits the buffer for IO
__jbd2_journal_file_buffer()
Quoted 9 lines
Quoted 9 lines
So a buffer under writeout is returned from
do_get_write_access(). Since
the filesystem code relies on the fact that journaled buffers cannot be written out, it does not take the buffer lock and so it can modify buffer while it is under writeout. That can lead to a filesystem corruption if we crash at the right moment.
We fix the problem by clearing the buffer dirty bit underbuffer_lock
even if the buffer is onBJ_Nonelist. Actually, we clear the dirty bit regardless the list the buffer is in and warn about the fact if the buffer is already journalled.
When running fsstress, we get the "Spotted dirty metadata buffer; there's a risk of filesystem corruption in csae of a system crash" at least half a dozen times or so. That sounds like we have a problem. Were you expecting that this was a "this should never happen" situation, or is there a known bug that we need to fix here?
- Ted
Re: [PATCH] jbd2: Fix a race between checkpointing code and journal_get_write_access()
Quoted 24 lines · Theodore Tso
Quoted 13 lines · Theodore Tso
Quoted 12 lines · Jan Kara
The following race can happen:
CPU1 CPU2
checkpointing code checks the buffer, adds
it to an array for writebackdo_get_write_access()
...
lock_buffer()
unlock_buffer()
flush_batch()submits the buffer for IO
__jbd2_journal_file_buffer()Quoted 9 lines
Quoted 9 lines
So a buffer under writeout is returned from
do_get_write_access(). Since
the filesystem code relies on the fact that journaled buffers cannot be written out, it does not take the buffer lock and so it can modify buffer while it is under writeout. That can lead to a filesystem corruption if we crash at the right moment.
We fix the problem by clearing the buffer dirty bit underbuffer_lock
even if the buffer is onBJ_Nonelist. Actually, we clear the dirty bit regardless the list the buffer is in and warn about the fact if the buffer is already journalled.
Quoted 5 lines
When running fsstress, we get the "Spotted dirty metadata buffer; there's a risk of filesystem corruption in csae of a system crash" at least half a dozen times or so. That sounds like we have a problem. Were you expecting that this was a "this should never happen" situation, or is there a known bug that we need to fix here?
Yes, it should be "this should never happen", unless you run something
like tune2fs while the filesystem is mounted. But looking at the code, I have missed that buffer could be dirty also in jbd2_journal_get_create_access() because jbd2_journal_forget() does not clear the dirty bit in case the buffer is just being committed. Attached patch should fix it. Thanks for report.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 1 line
From
0ab2ad057f96f49ba443f317df8e6368dd76def7Mon Sep 17 00:00:00 2001
From: Jan Kara <[email protected]>
Date: Fri, 10 Jul 2009 11:56:37 +0200
Subject: [PATCH] jbd2: Clear dirty bit in jbd2_journal_get_create_access()
In case we reallocate previously freed buffer, it can happen that the buffer we reallocate is dirty (jbd2_journal_forget() got called while the previous transaction was still committing). At the time we reallocate the buffer, the transaction freeing the buffer must be already committed so we can just remove the dirty bit. This silences the warning in jbd2_journal_file_buffer() about metadata buffer being dirty.
--- fs/jbd2/transaction.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)
fs/jbd2/transaction.c +9 -0
@@ -826,6 +826,15 @@ int jbd2_journal_get_create_access(handle_t *handle, struct buffer_head *bh)J_ASSERT_JH (jh,buffer_locked (jh2bh (jh)));if (jh->b_transaction ==NULL ) { +/* +* Previous jbd2_journal_forget() could have left the buffer +* with jbddirty bit set because it was being committed. When +* the commit finished, we've filed the buffer for +* checkpointing and marked it dirty. Now we are reallocating +* the buffer so the transaction freeing it must have +* committed and so it's safe to clear the dirty bit. +*/ +clear_buffer_dirty (jh2bh (jh)); jh->b_transaction = transaction;/* first access by this transaction */
Signature · 1 line
1.6.0.2
Theodore Tso
maintainerRe: [PATCH] jbd2: Fix a race between checkpointing code and journal_get_write_access()
Quoted 6 lines · Jan Kara
Yes, it should be "this should never happen", unless you run something
like tune2fs while the filesystem is mounted. But looking at the code, I have missed that buffer could be dirty also injbd2_journal_get_create_access()becausejbd2_journal_forget()does not clear the dirty bit in case the buffer is just being committed. Attached patch should fix it. Thanks for report.
Thanks, added to the patch queue.
- Ted
Re: [PATCH] jbd2: Fix a race between checkpointing code and journal_get_write_access()
Quoted 9 lines · Theodore Tso
Quoted 6 lines · Jan Kara
Yes, it should be "this should never happen", unless you run something
like tune2fs while the filesystem is mounted. But looking at the code, I have missed that buffer could be dirty also injbd2_journal_get_create_access()becausejbd2_journal_forget()does not clear the dirty bit in case the buffer is just being committed. Attached patch should fix it. Thanks for report.Thanks, added to the patch queue.
I've been doing some more tests here and there's one more path which emits
the warning when deleting symlinks (falsely this time). I'll send you the fix when I verify that I cannot trigger the warning anymore.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Theodore Tso
maintainerRe: [PATCH] jbd2: Fix a race between checkpointing code and journal_get_write_access()
Quoted 3 lines · Jan Kara
I've been doing some more tests here and there's one more path which emits
the warning when deleting symlinks (falsely this time). I'll send you the fix when I verify that I cannot trigger the warning anymore.
Ah, that would explain why I'm stil seeing the warning when fsstress runs. Was it doing it on long symlinks or short symlinks, out of curiosity?
- Ted
Re: [PATCH] jbd2: Fix a race between checkpointing code and journal_get_write_access()
Quoted 8 lines · Theodore Tso
Quoted 3 lines · Jan Kara
I've been doing some more tests here and there's one more path which emits
the warning when deleting symlinks (falsely this time). I'll send you the fix when I verify that I cannot trigger the warning anymore.Ah, that would explain why I'm stil seeing the warning when fsstress runs. Was it doing it on long symlinks or short symlinks, out of curiosity?
Long symlinks - the problem was that __dispose_buffer() called fromext4_invalidatepage() did __jbd2_journal_unfile_buffer() (jbddirty bit was moved to dirty bit) and then __jbd2_journal_file_buffer() which complained. As we cleared the jbddirty bit immediately afterwards it was quite straightforward to fix. I've audited all other occurences of jbd2_journal_unfile_buffer() and we discard the buffer head after all other calls...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR