Fix data race with transaction->state
v2 cover Fix data race with transaction->state
v1: https://lore.kernel.org/linux-btrfs/[email protected]/
v1->v2:
- I'm rusty and forgot READ_ONCE/WRITE_ONCE doesn't mean smp consistency, fixed the race with a proper locked check.
- Updated the smp_mb usage in start_transaction to use the proper helper.
I want to note that this isn't actually an observed hang, there was a problem with MMIO based block IO in my version of QEMU that was making IO just stop. I happened to notice this because the hung tasks looked very much like a deadlock. This fixes a real data race, and we would for sure miss wakeups without these fixes, but I don't actually have a reproducer for any sort of deadlock in this area.
=== Original email ===
I've been setting up Claude to setup fstests and run vms automatically and I kept hitting hangs. This turned out to be a bug with qemu's microvm, but at some point I was convinced there was a deadlock with running out of block tags and ordered extent completion and transaction commit. This actually wasn't the case, however this data race is in fact real. We can easily miss wakeups if we have to wait on transaction state to change because we do it outside of a lock and we do not have proper barriers around transaction->state. I suspect this explains the random hangs that I would see in production while at Meta that would clear up eventually (we do call wakeup on the transaction wait thing a lot). In any case this is a data race, even if it wasn't my particular bug, we should fix it. I've run it through fstests a few times, but obviously spot check it since I'm a little rusty with this stuff at the moment. Thanks,
Josef
Josef Bacik (2):
btrfs: fix data race on transaction->state
btrfs: remove useless smp_mb in start_transaction
fs/btrfs/transaction.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
Signature · 1 line
2.51.1
PATCH v2 1/2 btrfs: fix data race on transaction->state
Debugging a hang with btrfs on QEMU I discovered a data race with transaction->state. In wait_current_trans we do
wait_event(fs_info->transaction_wait,
cur_trans->state>=TRANS_STATE_UNBLOCKED || TRANS_ABORTED(cur_trans));
however we're doing this outside of the fs_info->trans_lock. This generally isn't super problematic because we hit wake_up(fs_info->transaction_wait) quite a bit, but it could lead to latencies where we miss wakeups, or in the worst case (like the compiler re-orders the load of the ->state outside of the wait_event loop) we could hang completely.
Fix this by using a helper that takes the fs_info->trans_lock to do the check safely.
I've added a lockdep_assert for the other helper to make sure nobody uses that one without holding the lock.
--- fs/btrfs/transaction.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
fs/btrfs/transaction.c +15 -3
@@ -509,11 +509,25 @@ int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans,static inline int is_transaction_blocked (struct btrfs_transaction *trans) { +lockdep_assert_held (&trans->fs_info->trans_lock); +return (trans->state >=TRANS_STATE_COMMIT_START && trans->state <TRANS_STATE_UNBLOCKED && !TRANS_ABORTED (trans)); } +/* Helper to check transaction state under lock for wait_event */ +static bool trans_unblocked (struct btrfs_transaction *trans) +{ +struct btrfs_fs_info *fs_info = trans->fs_info; +bool ret; + +spin_lock (&fs_info->trans_lock); + ret = trans->state >=TRANS_STATE_UNBLOCKED ||TRANS_ABORTED (trans); +spin_unlock (&fs_info->trans_lock); +return ret; +} +/* wait for commit against the current transaction to become unblocked * when this is done, it is safe to start a new transaction, but the current * transaction might not be fully on disk.
@@ -529,9 +543,7 @@ static void wait_current_trans(struct btrfs_fs_info *fs_info)spin_unlock (&fs_info->trans_lock);btrfs_might_wait_for_state (fs_info,BTRFS_LOCKDEP_TRANS_UNBLOCKED ); -wait_event (fs_info->transaction_wait, - cur_trans->state >=TRANS_STATE_UNBLOCKED || -TRANS_ABORTED (cur_trans)); +wait_event (fs_info->transaction_wait,trans_unblocked (cur_trans));btrfs_put_transaction (cur_trans); }else {spin_unlock (&fs_info->trans_lock);
Signature · 1 line
2.51.1
PATCH v2 2/2 btrfs: remove useless smp_mb in start_transaction
In reply to Josef Baciktrans->state is protected by a spin_lock() and there's no mb when we set it, simply the spin_unlock(). The correct pairing with that is smp_load_acquire(), so replace the smp_mb with the appropriate helper.
--- fs/btrfs/transaction.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/btrfs/transaction.c +1 -2
@@ -737,8 +737,7 @@ start_transaction(struct btrfs_root *root, unsigned int num_items,INIT_LIST_HEAD (&h->new_bgs);btrfs_init_metadata_block_rsv (fs_info, &h->delayed_rsv,BTRFS_BLOCK_RSV_DELOPS ); -smp_mb (); -if (cur_trans->state >=TRANS_STATE_COMMIT_START && +if (smp_load_acquire (&cur_trans->state) >=TRANS_STATE_COMMIT_START &&may_wait_transaction (fs_info, type)) { current->journal_info = h;btrfs_commit_transaction (h);
Signature · 1 line
2.51.1
Re: [PATCH v2 1/2] btrfs: fix data race on transaction->state
In reply to Josef BacikQuoted 31 lines · Josef Bacik
Debugging a hang with btrfs on QEMU I discovered a data race with
transaction->state. Inwait_current_transwe dowait_event(fs_info->transaction_wait,
cur_trans->state>=TRANS_STATE_UNBLOCKED || TRANS_ABORTED(cur_trans));however we're doing this outside of the
fs_info->trans_lock. This generally isn't super problematic because we hitwake_up(fs_info->transaction_wait)quite a bit, but it could lead to latencies where we miss wakeups, or in the worst case (like the compiler re-orders the load of the ->state outside of thewait_eventloop) we could hang completely.Fix this by using a helper that takes the
fs_info->trans_lockto do the check safely.I've added a
lockdep_assertfor the other helper to make sure nobody uses that one without holding the lock.Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/transaction.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)fs/btrfs/transaction.c +0
-0
fs/btrfs/transaction.c int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans, · line 513
Quoted 5 lines
fs/btrfs/transaction.c +2
-0@@ -509,11 +509,25 @@ int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans,static inline int is_transaction_blocked (struct btrfs_transaction *trans) { +lockdep_assert_held (&trans->fs_info->trans_lock); +
It seems odd to sneak this in when no other change in the patch introduces a call to this function.
I would make this a separate patch.
fs/btrfs/transaction.c int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans, · line 520
Quoted 7 lines
fs/btrfs/transaction.c +2
-0return (trans->state >=TRANS_STATE_COMMIT_START && trans->state <TRANS_STATE_UNBLOCKED && !TRANS_ABORTED (trans)); } +/* Helper to check transaction state under lock for wait_event */ +static bool trans_unblocked (struct btrfs_transaction *trans)
This could have a name that is closer to the other helper:is_transaction_unblocked()
Quoted 14 lines
fs/btrfs/transaction.c +10
-0+{ +struct btrfs_fs_info *fs_info = trans->fs_info; +bool ret; + +spin_lock (&fs_info->trans_lock); + ret = trans->state >=TRANS_STATE_UNBLOCKED ||TRANS_ABORTED (trans); +spin_unlock (&fs_info->trans_lock); +return ret; +} +/* wait for commit against the current transaction to become unblocked * when this is done, it is safe to start a new transaction, but the current * transaction might not be fully on disk.
fs/btrfs/transaction.c static void wait_current_trans(struct btrfs_fs_info *fs_info) · line 546
Quoted 7 lines
fs/btrfs/transaction.c +1
-3@@ -529,9 +543,7 @@ static void wait_current_trans(struct btrfs_fs_info *fs_info)spin_unlock (&fs_info->trans_lock);btrfs_might_wait_for_state (fs_info,BTRFS_LOCKDEP_TRANS_UNBLOCKED ); -wait_event (fs_info->transaction_wait, - cur_trans->state >=TRANS_STATE_UNBLOCKED || -TRANS_ABORTED (cur_trans)); +wait_event (fs_info->transaction_wait,trans_unblocked (cur_trans));
Instead of adding an helper and locking, couldn't this be simply:
wait_event(fs_info->transaction_wait, smp_load_acquire(cur_trans->state) >= TRANS_STATE_UNBLOCKED || TRANS_ABORTED(cur_trans));
Thanks.
Quoted 7 lines
fs/btrfs/transaction.c +0
-1btrfs_put_transaction (cur_trans); }else {spin_unlock (&fs_info->trans_lock); --2.51.1
Quoted 7 lines · Josef Bacik
v1: https://lore.kernel.org/linux-btrfs/[email protected]/
v1->v2:
- I'm rusty and forgot READ_ONCE/WRITE_ONCE doesn't mean smp consistency, fixed the race with a proper locked check.
- Updated thesmp_mbusage instart_transactionto use the proper helper.
Quoted 6 lines
I want to note that this isn't actually an observed hang, there was a problem with MMIO based block IO in my version of QEMU that was making IO just stop. I happened to notice this because the hung tasks looked very much like a deadlock. This fixes a real data race, and we would for sure miss wakeups without these fixes, but I don't actually have a reproducer for any sort of deadlock in this area.
Thanks,
Qu
Quoted 25 lines
=== Original email ===
I've been setting up Claude to setup fstests and run vms automatically and I kept hitting hangs. This turned out to be a bug with qemu's microvm, but at some point I was convinced there was a deadlock with running out of block tags and ordered extent completion and transaction commit. This actually wasn't the case, however this data race is in fact real. We can easily miss wakeups if we have to wait on transaction state to change because we do it outside of a lock and we do not have proper barriers around
transaction->state. I suspect this explains the random hangs that I would see in production while at Meta that would clear up eventually (we do call wakeup on the transaction wait thing a lot). In any case this is a data race, even if it wasn't my particular bug, we should fix it. I've run it through fstests a few times, but obviously spot check it since I'm a little rusty with this stuff at the moment. Thanks,Josef
Josef Bacik (2):
btrfs: fix data race ontransaction->state
btrfs: remove uselesssmp_mbinstart_transactionfs/btrfs/transaction.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
Re: [PATCH v2 1/2] btrfs: fix data race on transaction->state
In reply to Filipe MananaQuoted 77 lines · Filipe Manana
Quoted 31 lines · Josef Bacik
Debugging a hang with btrfs on QEMU I discovered a data race with
transaction->state. Inwait_current_transwe dowait_event(fs_info->transaction_wait,
cur_trans->state>=TRANS_STATE_UNBLOCKED || TRANS_ABORTED(cur_trans));however we're doing this outside of the
fs_info->trans_lock. This generally isn't super problematic because we hitwake_up(fs_info->transaction_wait)quite a bit, but it could lead to latencies where we miss wakeups, or in the worst case (like the compiler re-orders the load of the ->state outside of thewait_eventloop) we could hang completely.Fix this by using a helper that takes the
fs_info->trans_lockto do the check safely.I've added a
lockdep_assertfor the other helper to make sure nobody uses that one without holding the lock.Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/transaction.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)fs/btrfs/transaction.c +0
-0fs/btrfs/transaction.c int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans, · line 513
Quoted 5 lines
fs/btrfs/transaction.c +2
-0@@ -509,11 +509,25 @@ int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans,static inline int is_transaction_blocked (struct btrfs_transaction *trans) { +lockdep_assert_held (&trans->fs_info->trans_lock); +It seems odd to sneak this in when no other change in the patch introduces a call to this function.
I would make this a separate patch.fs/btrfs/transaction.c int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans, · line 520
Quoted 7 lines
fs/btrfs/transaction.c +2
-0return (trans->state >=TRANS_STATE_COMMIT_START && trans->state <TRANS_STATE_UNBLOCKED && !TRANS_ABORTED (trans)); } +/* Helper to check transaction state under lock for wait_event */ +static bool trans_unblocked (struct btrfs_transaction *trans)This could have a name that is closer to the other helper:
is_transaction_unblocked()Quoted 14 lines
fs/btrfs/transaction.c +10
-0+{ +struct btrfs_fs_info *fs_info = trans->fs_info; +bool ret; + +spin_lock (&fs_info->trans_lock); + ret = trans->state >=TRANS_STATE_UNBLOCKED ||TRANS_ABORTED (trans); +spin_unlock (&fs_info->trans_lock); +return ret; +} +/* wait for commit against the current transaction to become unblocked * when this is done, it is safe to start a new transaction, but the current * transaction might not be fully on disk. fs/btrfs/transaction.c static void wait_current_trans(struct btrfs_fs_info *fs_info) · line 546
Quoted 7 lines
fs/btrfs/transaction.c +1
-3@@ -529,9 +543,7 @@ static void wait_current_trans(struct btrfs_fs_info *fs_info)spin_unlock (&fs_info->trans_lock);btrfs_might_wait_for_state (fs_info,BTRFS_LOCKDEP_TRANS_UNBLOCKED ); -wait_event (fs_info->transaction_wait, - cur_trans->state >=TRANS_STATE_UNBLOCKED || -TRANS_ABORTED (cur_trans)); +wait_event (fs_info->transaction_wait,trans_unblocked (cur_trans));Instead of adding an helper and locking, couldn't this be simply:
Quoted 3 lines
wait_event(fs_info->transaction_wait,
smp_load_acquire(cur_trans->state)>=TRANS_STATE_UNBLOCKED||TRANS_ABORTED(cur_trans));
Not an expert on memory barriers, but isn't the key point here that we're accessing two different variables in a non-atomic way?
And to be honest, I really do not like introducing low level memory barrier callers, it's really hard to get it right.
Thanks,
Qu
Quoted 11 lines
Thanks.
Quoted 7 lines
fs/btrfs/transaction.c +0
-1btrfs_put_transaction (cur_trans); }else {spin_unlock (&fs_info->trans_lock); --2.51.1
Re: [PATCH v2 1/2] btrfs: fix data race on transaction->state
Quoted 81 lines · Qu Wenruo
Quoted 77 lines · Filipe Manana
Quoted 31 lines · Josef Bacik
Debugging a hang with btrfs on QEMU I discovered a data race with
transaction->state. Inwait_current_transwe dowait_event(fs_info->transaction_wait,
cur_trans->state>=TRANS_STATE_UNBLOCKED || TRANS_ABORTED(cur_trans));however we're doing this outside of the
fs_info->trans_lock. This generally isn't super problematic because we hitwake_up(fs_info->transaction_wait)quite a bit, but it could lead to latencies where we miss wakeups, or in the worst case (like the compiler re-orders the load of the ->state outside of thewait_eventloop) we could hang completely.Fix this by using a helper that takes the
fs_info->trans_lockto do the check safely.I've added a
lockdep_assertfor the other helper to make sure nobody uses that one without holding the lock.Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/transaction.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)fs/btrfs/transaction.c +0
-0fs/btrfs/transaction.c int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans, · line 513
Quoted 5 lines
fs/btrfs/transaction.c +2
-0@@ -509,11 +509,25 @@ int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans,static inline int is_transaction_blocked (struct btrfs_transaction *trans) { +lockdep_assert_held (&trans->fs_info->trans_lock); +It seems odd to sneak this in when no other change in the patch introduces a call to this function.
I would make this a separate patch.fs/btrfs/transaction.c int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans, · line 520
Quoted 7 lines
fs/btrfs/transaction.c +2
-0return (trans->state >=TRANS_STATE_COMMIT_START && trans->state <TRANS_STATE_UNBLOCKED && !TRANS_ABORTED (trans)); } +/* Helper to check transaction state under lock for wait_event */ +static bool trans_unblocked (struct btrfs_transaction *trans)This could have a name that is closer to the other helper:
is_transaction_unblocked()Quoted 14 lines
fs/btrfs/transaction.c +10
-0+{ +struct btrfs_fs_info *fs_info = trans->fs_info; +bool ret; + +spin_lock (&fs_info->trans_lock); + ret = trans->state >=TRANS_STATE_UNBLOCKED ||TRANS_ABORTED (trans); +spin_unlock (&fs_info->trans_lock); +return ret; +} +/* wait for commit against the current transaction to become unblocked * when this is done, it is safe to start a new transaction, but the current * transaction might not be fully on disk. fs/btrfs/transaction.c static void wait_current_trans(struct btrfs_fs_info *fs_info) · line 546
Quoted 7 lines
fs/btrfs/transaction.c +1
-3@@ -529,9 +543,7 @@ static void wait_current_trans(struct btrfs_fs_info *fs_info)spin_unlock (&fs_info->trans_lock);btrfs_might_wait_for_state (fs_info,BTRFS_LOCKDEP_TRANS_UNBLOCKED ); -wait_event (fs_info->transaction_wait, - cur_trans->state >=TRANS_STATE_UNBLOCKED || -TRANS_ABORTED (cur_trans)); +wait_event (fs_info->transaction_wait,trans_unblocked (cur_trans));Instead of adding an helper and locking, couldn't this be simply:
Quoted 6 lines
Quoted 3 lines
wait_event(fs_info->transaction_wait,
smp_load_acquire(cur_trans->state)>=TRANS_STATE_UNBLOCKED||TRANS_ABORTED(cur_trans));Not an expert on memory barriers, but isn't the key point here that we're accessing two different variables in a non-atomic way?
So what?
We can continue accessing the trans aborted case as we did because it's not expected to happen, so any latency in the waiting due to a transaction abort is fine.
Quoted 3 lines
And to be honest, I really do not like introducing low level memory barrier callers, it's really hard to get it right.
I generally don't like it either. But here it saves quite some code and it's for a very rare case anyway.
Plus there's also this inconsistency with the second patch, which favors using smp_load_acquire() instead of taking the lock.
__nfs_lookup_revalidate() uses smp_load_acquire() precisely in the same situation,
in a wait_var_event() condition to read dentry->d_fsdata, which is updated under the protection of dentry->d_lock.
Quoted 16 lines
Thanks,
QuQuoted 11 lines
Thanks.
Quoted 7 lines
fs/btrfs/transaction.c +0
-1btrfs_put_transaction (cur_trans); }else {spin_unlock (&fs_info->trans_lock); --2.51.1
Re: [PATCH v2 2/2] btrfs: remove useless smp_mb in start_transaction
In reply to Josef BacikQuoted 14 lines · Josef Bacik
trans->stateis protected by aspin_lock()and there's no mb when we set it, simply thespin_unlock(). The correct pairing with that issmp_load_acquire(), so replace thesmp_mbwith the appropriate helper.Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/transaction.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)fs/btrfs/transaction.c +0
-0
fs/btrfs/transaction.c start_transaction(struct btrfs_root *root, unsigned int num_items, · line 740
Quoted 6 lines
fs/btrfs/transaction.c +1
-2@@ -737,8 +737,7 @@ start_transaction(struct btrfs_root *root, unsigned int num_items,INIT_LIST_HEAD (&h->new_bgs);btrfs_init_metadata_block_rsv (fs_info, &h->delayed_rsv,BTRFS_BLOCK_RSV_DELOPS ); -smp_mb (); - if (cur_trans->state >= TRANS_STATE_COMMIT_START && + if (smp_load_acquire(&cur_trans->state) >= TRANS_STATE_COMMIT_START &&
This is a memory barrier so it should be documented.