Skip to contentSkip to conversation

btrfs: qgroup: Fix deadlock where btrfs_qgroup_wait_for_completion() waits for never-queued work

PATCH In review Qu Wenruo 3 participants 6 messages Feb 7, 2020 – Feb 25, 2020 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

v2 cover btrfs: qgroup: Fix deadlock where btrfs_qgroup_wait_for_completion() waits for never-queued work

There is a long existing report about btrfs hangs at unmount time, waiting for qgroup.

Jeff has submitted a patch for that, but never merged.
https://patchwork.kernel.org/patch/10376585/

After re-digging the case, although Jeff's fix can solve the problem, the racy cause doesn't look correct to me.

After all, close_ctree() wait for qgroup rescan before destroying related work queues. Thus as long as the work is queued, we can finish the wait without problem.

Further digging into the bug, it looks like the deadlock is possible, and Jeff is right about the wait-for-never-queued-work part. But the racy part doesn't look possible, thus it should only happen when something wrong happened.

Now with a proper cause analyse, we can craft a much smaller thus better fix (anyway I'm the guy to backport, smaller is always better).

Changelog:
v2:
- Change the subject
It's not about race. I got confused by the initial patch.

- Change the cause analyse
No need for any race. Also add analyse for all qgroup_rescan_init() callers to ensure no missing fixes.
BTW, qgroup_rescan_init() uses BTRFS_QGROUP_STATUS_FLAG_RESCAN flag to determine if there is a conflicting rescan, thus it's not affected by the timing change.

- Split the spinlock cleanup into another patch

Qu Wenruo (2):
btrfs: qgroup: Ensure qgroup_rescan_running is only set when the
worker is at least queued
btrfs: qgroup: Remove the unnecesaary spin lock for
qgroup_rescan_running|queued

 fs/btrfs/qgroup.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
Signature · 1 line
2.25.0
author
#1 lore raw

PATCH v2 1/2 btrfs: qgroup: Ensure qgroup_rescan_running is only set when the worker is at least queued

[BUG]
There are some reports about btrfs wait forever to unmount itself, with the following call trace:

  INFO: task umount:4631 blocked for more than 491 seconds.
        Tainted: G               X  5.3.8-2-default #1
  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  umount          D    0  4631   3337 0x00000000
  Call Trace:
  ([<00000000174adf7a>] __schedule+0x342/0x748)
   [<00000000174ae3ca>] schedule+0x4a/0xd8
   [<00000000174b1f08>] schedule_timeout+0x218/0x420
   [<00000000174af10c>] wait_for_common+0x104/0x1d8
   [<000003ff804d6994>] btrfs_qgroup_wait_for_completion+0x84/0xb0 [btrfs]
   [<000003ff8044a616>] close_ctree+0x4e/0x380 [btrfs]
   [<0000000016fa3136>] generic_shutdown_super+0x8e/0x158
   [<0000000016fa34d6>] kill_anon_super+0x26/0x40
   [<000003ff8041ba88>] btrfs_kill_super+0x28/0xc8 [btrfs]
   [<0000000016fa39f8>] deactivate_locked_super+0x68/0x98
   [<0000000016fcb198>] cleanup_mnt+0xc0/0x140
   [<0000000016d6a846>] task_work_run+0xc6/0x110
   [<0000000016d04f76>] do_notify_resume+0xae/0xb8
   [<00000000174b30ae>] system_call+0xe2/0x2c8

[CAUSE]
The problem happens when we have called qgroup_rescan_init(), but doesn't queue the worker. It can be caused mostly by error handling.

Qgroup ioctl thread | Unmount thread
----------------------------------------+-----------------------------------

					|
btrfs_qgroup_rescan()			|
|- qgroup_rescan_init()			|
|  |- qgroup_rescan_running = true;	|
|					|
|- trans = btrfs_join_transaction()	|
|  Some error happened			|
|					|
|- btrfs_qgroup_rescan() returns error	|
   But qgroup_rescan_running == true;	|
					| close_ctree()
					| |- btrfs_qgroup_wait_for_completion()
					|    |- running == true;
					|    |- wait_for_completion();

btrfs_qgroup_rescan_worker is never queued, thus no one is going to wake up close_ctree() and we get a deadlock.

All involved qgroup_rescan_init() callers are:
- btrfs_qgroup_rescan()
The example above. It's possible to trigger the deadlock when error happened.

- btrfs_quota_enable()
Not possible. Just after qgroup_rescan_init() we queue the work.

- btrfs_read_qgroup_config()
It's possible to trigger the deadlock. It only init the work, the work queueing happens in btrfs_qgroup_rescan_resume(). Thus if error happened between, deadlock is possible.

We shouldn't set fs_info->qgroup_rescan_running just in qgroup_rescan_init(), as at that stage we haven't yet submit qgroup rescan worker to run.

[FIX]
This patch fixes the problem by setting qgroup_rescan_running before queueing the work, so that we ensure the rescan work is queued when we wait for it.

Fixes 8d9eddad194 (Btrfs: fix qgroup rescan worker initialization)
[ Change subject and cause analyse, use a smaller fix ]

---
Changelog:
v2:
- Change the subject
It's not about race. I got confused by the initial patch.

- Change the cause analyse
No need for any race. Also add analyse for all qgroup_rescan_init() callers to ensure no missing fixes.
BTW, qgroup_rescan_init() uses BTRFS_QGROUP_STATUS_FLAG_RESCAN flag to determine if there is a conflicting rescan, thus it's not affected by the timing change.

- Split the spinlock cleanup into another patch

---
 fs/btrfs/qgroup.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
fs/btrfs/qgroup.c +9 -2
index d4282e12f2a6..812f51f67903 100644
@@ -1030,6 +1030,7 @@ int btrfs_quota_enable(struct btrfs_fs_info *fs_info)
 	ret = qgroup_rescan_init(fs_info, 0, 1);
 	if (!ret) {
 	        qgroup_rescan_zero_tracking(fs_info);
+		fs_info->qgroup_rescan_running = true;
 	        btrfs_queue_work(fs_info->qgroup_rescan_workers,
 	                         &fs_info->qgroup_rescan_work);
 	}
@@ -3272,7 +3273,6 @@ qgroup_rescan_init(struct btrfs_fs_info *fs_info, u64 progress_objectid,
 		sizeof(fs_info->qgroup_rescan_progress));
 	fs_info->qgroup_rescan_progress.objectid = progress_objectid;
 	init_completion(&fs_info->qgroup_rescan_completion);
-	fs_info->qgroup_rescan_running = true;
 
 	spin_unlock(&fs_info->qgroup_lock);
 	mutex_unlock(&fs_info->qgroup_rescan_lock);
@@ -3335,8 +3335,11 @@ btrfs_qgroup_rescan(struct btrfs_fs_info *fs_info)
 
 	qgroup_rescan_zero_tracking(fs_info);
 
+	mutex_lock(&fs_info->qgroup_rescan_lock);
+	fs_info->qgroup_rescan_running = true;
 	btrfs_queue_work(fs_info->qgroup_rescan_workers,
 			 &fs_info->qgroup_rescan_work);
+	mutex_unlock(&fs_info->qgroup_rescan_lock);
 
 	return 0;
 }
@@ -3372,9 +3375,13 @@ int btrfs_qgroup_wait_for_completion(struct btrfs_fs_info *fs_info,
 void
 btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info)
 {
-	if (fs_info->qgroup_flags & BTRFS_QGROUP_STATUS_FLAG_RESCAN)
+	if (fs_info->qgroup_flags & BTRFS_QGROUP_STATUS_FLAG_RESCAN) {
+		mutex_lock(&fs_info->qgroup_rescan_lock);
+		fs_info->qgroup_rescan_running = true;
 		btrfs_queue_work(fs_info->qgroup_rescan_workers,
 				 &fs_info->qgroup_rescan_work);
+		mutex_unlock(&fs_info->qgroup_rescan_lock);
+	}
 }
 
 /*
Signature · 1 line
2.25.0
author
#2 lore raw

PATCH v2 2/2 btrfs: qgroup: Remove the unnecesaary spin lock for qgroup_rescan_running|queued

In reply to Qu Wenruo

Those two members are all protected by btrfs_fs_info::qgroup_rescan_lock, thus no need for the extra spinlock.

---
Changelog:
v2:
- New patch split in v2

---
 fs/btrfs/qgroup.c | 6 ------
 1 file changed, 6 deletions(-)
fs/btrfs/qgroup.c +0 -6
index 812f51f67903..e07d6a6b2049 100644
@@ -3247,7 +3247,6 @@ qgroup_rescan_init(struct btrfs_fs_info *fs_info, u64 progress_objectid,
 	}
 
 	mutex_lock(&fs_info->qgroup_rescan_lock);
-	spin_lock(&fs_info->qgroup_lock);
 
 	if (init_flags) {
 		if (fs_info->qgroup_flags & BTRFS_QGROUP_STATUS_FLAG_RESCAN) {
@@ -3262,7 +3261,6 @@ qgroup_rescan_init(struct btrfs_fs_info *fs_info, u64 progress_objectid,
 		}
 
 		if (ret) {
-			spin_unlock(&fs_info->qgroup_lock);
 			mutex_unlock(&fs_info->qgroup_rescan_lock);
 			return ret;
 		}
@@ -3273,8 +3271,6 @@ qgroup_rescan_init(struct btrfs_fs_info *fs_info, u64 progress_objectid,
 		sizeof(fs_info->qgroup_rescan_progress));
 	fs_info->qgroup_rescan_progress.objectid = progress_objectid;
 	init_completion(&fs_info->qgroup_rescan_completion);
-
-	spin_unlock(&fs_info->qgroup_lock);
 	mutex_unlock(&fs_info->qgroup_rescan_lock);
 
 	btrfs_init_work(&fs_info->qgroup_rescan_work,
@@ -3351,9 +3347,7 @@ int btrfs_qgroup_wait_for_completion(struct btrfs_fs_info *fs_info,
 	int ret = 0;
 
 	mutex_lock(&fs_info->qgroup_rescan_lock);
-	spin_lock(&fs_info->qgroup_lock);
 	running = fs_info->qgroup_rescan_running;
-	spin_unlock(&fs_info->qgroup_lock);
 	mutex_unlock(&fs_info->qgroup_rescan_lock);
 
 	if (!running)
Signature · 1 line
2.25.0
#3 lore raw

Re: [PATCH v2 2/2] btrfs: qgroup: Remove the unnecesaary spin lock for qgroup_rescan_running|queued

Quoted 2 lines · Qu Wenruo

Those two members are all protected by btrfs_fs_info::qgroup_rescan_lock, thus no need for the extra spinlock.

Two members refers to btrfs_fs_info::qgroup_rescan_lock and what else? Byt the subject it's something 'queued' but I can't find what it's referring to.

#4 lore raw

Re: [PATCH v2 2/2] btrfs: qgroup: Remove the unnecesaary spin lock for qgroup_rescan_running|queued

Quoted 8 lines · David Sterba
Quoted 2 lines · Qu Wenruo

Those two members are all protected by btrfs_fs_info::qgroup_rescan_lock, thus no need for the extra spinlock.

Two members refers to btrfs_fs_info::qgroup_rescan_lock and what else? Byt the subject it's something 'queued' but I can't find what it's referring to.

My bad, with the latest version, there is only qgroup_rescan_running, no qgroup_rescan_queued.

Just one member now.

Do I need to resend the patch with commit message updated?

Thanks,
Qu

#5 lore raw

Re: [PATCH v2 2/2] btrfs: qgroup: Remove the unnecesaary spin lock for qgroup_rescan_running|queued

Quoted 11 lines · Qu Wenruo
Quoted 8 lines · David Sterba
Quoted 2 lines · Qu Wenruo

Those two members are all protected by btrfs_fs_info::qgroup_rescan_lock, thus no need for the extra spinlock.

Two members refers to btrfs_fs_info::qgroup_rescan_lock and what else? Byt the subject it's something 'queued' but I can't find what it's referring to.

Quoted 6 lines

My bad, with the latest version, there is only qgroup_rescan_running, no qgroup_rescan_queued.

Just one member now.

Do I need to resend the patch with commit message updated?

Not needed, I only wanted to clarify if I'm not missing something. I'll update changelog and push to misc-next.