btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
PATCH v4 1/1 btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
The commit e804861bd4e6 ("btrfs: fix deadlock between quota disable and qgroup rescan worker") by Kawasaki resolves deadlock between quota disable and qgroup rescan worker. But also there is a deadlock case like it. It's about enabling or disabling quota and creating or removing qgroup. It can be reproduced in simple script below.
for i in {1..100}
do
btrfs quota enable /mnt &
btrfs qgroup create 1/0 /mnt &
btrfs qgroup destroy 1/0 /mnt &
btrfs quota disable /mnt &
done
Here's why the deadlock happens:
1) The quota rescan task is running.
2) Task A calls btrfs_quota_disable(), locks the qgroup_ioctl_lock mutex, and then calls btrfs_qgroup_wait_for_completion(), to wait for the quota rescan task to complete.
3) Task B calls btrfs_remove_qgroup() and it blocks when trying to lock the qgroup_ioctl_lock mutex, because it's being held by task A. At that point task B is holding a transaction handle for the current transaction.
4) The quota rescan task calls btrfs_commit_transaction(). This results in it waiting for all other tasks to release their handles on the transaction, but task B is blocked on the qgroup_ioctl_lock mutex while holding a handle on the transaction, and that mutex is being held by task A, which is waiting for the quota rescan task to complete, resulting in a deadlock between these 3 tasks.
To resolve this issue, the thread disabling quota should unlock qgroup_ioctl_lock before waiting rescan completion. Move btrfs_qgroup_wait_for_completion() after unlock of qgroup_ioctl_lock.
qgroup rescan worker")
---
v4: fix typos, changelog.
v3: fix comments, typos, changelog.
v2: add comments, move locking before clear_bit.
--- fs/btrfs/qgroup.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
fs/btrfs/qgroup.c +8 -1
@@ -1213,6 +1213,14 @@ int btrfs_quota_disable(struct btrfs_fs_info *fs_info)if (!fs_info->quota_root) goto out; +/* +* Unlock the qgroup_ioctl_lock mutex before waiting for the rescan worker to +* complete. Otherwise we can deadlock because btrfs_remove_qgroup() needs +* to lock that mutex while holding a transaction handle and the rescan +* worker needs to commit a transaction. +*/ +mutex_unlock (&fs_info->qgroup_ioctl_lock); +/* * Request qgroup rescan worker to complete and wait for it. This wait * must be done before transaction start for quota disable since it may
@@ -1220,7 +1228,6 @@ int btrfs_quota_disable(struct btrfs_fs_info *fs_info)*/ clear_bit (BTRFS_FS_QUOTA_ENABLED , &fs_info->flags);btrfs_qgroup_wait_for_completion (fs_info, false); -mutex_unlock (&fs_info->qgroup_ioctl_lock);/* * 1 For the root item
Signature · 1 line
2.25.1
Quoted 33 lines · Sidong Yang
The commit
e804861bd4e6("btrfs: fix deadlock between quota disable and qgroup rescan worker") by Kawasaki resolves deadlock between quota disable and qgroup rescan worker. But also there is a deadlock case like it. It's about enabling or disabling quota and creating or removing qgroup. It can be reproduced in simple script below.for i in {1..100}
do
btrfs quota enable /mnt &
btrfs qgroup create 1/0 /mnt &
btrfs qgroup destroy 1/0 /mnt &
btrfs quota disable /mnt &
doneHere's why the deadlock happens:
1) The quota rescan task is running.
2) Task A calls
btrfs_quota_disable(), locks theqgroup_ioctl_lockmutex, and then callsbtrfs_qgroup_wait_for_completion(), to wait for the quota rescan task to complete.3) Task B calls
btrfs_remove_qgroup()and it blocks when trying to lock theqgroup_ioctl_lockmutex, because it's being held by task A. At that point task B is holding a transaction handle for the current transaction.4) The quota rescan task calls
btrfs_commit_transaction(). This results in it waiting for all other tasks to release their handles on the transaction, but task B is blocked on theqgroup_ioctl_lockmutex while holding a handle on the transaction, and that mutex is being held by task A, which is waiting for the quota rescan task to complete, resulting in a deadlock between these 3 tasks.
Quoted 8 lines
To resolve this issue, the thread disabling quota should unlock
qgroup_ioctl_lockbefore waiting rescan completion. Movebtrfs_qgroup_wait_for_completion()after unlock ofqgroup_ioctl_lock.Fixes e804861bd4e6 ("btrfs: fix deadlock between quota disable andqgroup rescan worker")
Reviewed-by Filipe Manana <[email protected]>Signed-off-by Sidong Yang <[email protected]>
Thanks. Looks good to me.
Signature · 2 lines
Best Regards, Shin'ichiro Kawasaki
Quoted 37 lines · Sidong Yang
The commit
e804861bd4e6("btrfs: fix deadlock between quota disable and qgroup rescan worker") by Kawasaki resolves deadlock between quota disable and qgroup rescan worker. But also there is a deadlock case like it. It's about enabling or disabling quota and creating or removing qgroup. It can be reproduced in simple script below.for i in {1..100}
do
btrfs quota enable /mnt &
btrfs qgroup create 1/0 /mnt &
btrfs qgroup destroy 1/0 /mnt &
btrfs quota disable /mnt &
doneHere's why the deadlock happens:
1) The quota rescan task is running.
2) Task A calls
btrfs_quota_disable(), locks theqgroup_ioctl_lockmutex, and then callsbtrfs_qgroup_wait_for_completion(), to wait for the quota rescan task to complete.3) Task B calls
btrfs_remove_qgroup()and it blocks when trying to lock theqgroup_ioctl_lockmutex, because it's being held by task A. At that point task B is holding a transaction handle for the current transaction.4) The quota rescan task calls
btrfs_commit_transaction(). This results in it waiting for all other tasks to release their handles on the transaction, but task B is blocked on theqgroup_ioctl_lockmutex while holding a handle on the transaction, and that mutex is being held by task A, which is waiting for the quota rescan task to complete, resulting in a deadlock between these 3 tasks.To resolve this issue, the thread disabling quota should unlock
qgroup_ioctl_lockbefore waiting rescan completion. Movebtrfs_qgroup_wait_for_completion()after unlock ofqgroup_ioctl_lock.
Quoted 6 lines
Fixes e804861bd4e6 ("btrfs: fix deadlock between quota disable andqgroup rescan worker")
Reviewed-by Filipe Manana <[email protected]>Signed-off-by Sidong Yang <[email protected]>---
v4: fix typos, changelog.
Perfect, thanks.