btrfs: add test for enable/disable quota and create/destroy qgroup repeatedly
PATCH 1/1 btrfs: add test for enable/disable quota and create/destroy qgroup repeatedly
Test enabling/disable quota and creating/destroying qgroup repeatedly in asynchronous and confirm it does not cause kernel hang. This is a regression test for the problem reported to linux-btrfs list [1].
The hang was recreated using the test case and fixed by kernel patch titled
btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
[1] https://lore.kernel.org/linux-btrfs/[email protected]/
--- tests/btrfs/262 | 54 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/262.out | 2 ++ 2 files changed, 56 insertions(+) create mode 100755 tests/btrfs/262 create mode 100644 tests/btrfs/262.out
tests/btrfs/262 new file +54 -0
@@ -0,0 +1,54 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 YOUR NAME HERE. All Rights Reserved. +# +# FS QA Test 262 +# +# Test the deadlock between qgroup and quota commands +# +. ./common/preamble +_begin_fstest auto qgroup + +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs + +_require_scratch + +# Run command that enable/disable quota and create/destroy qgroup asynchronously +qgroup_deadlock_test() +{ + _scratch_mkfs > /dev/null 2>&1 + _scratch_mount + echo "=== qgroup deadlock test ===" >> $seqres.full + + pids=() + for ((i = 0; i < 200; i++)); do + $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG qgroup create 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG qgroup destroy 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG quota disable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + done + + for pid in "${pids[@]}"; do + wait $pid + done + + _scratch_unmount + _check_scratch_fs +} + +qgroup_deadlock_test + +# success, all done +echo "Silence is golden" +status=0 +exit
tests/btrfs/262.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 262 +Silence is golden
Signature · 1 line
2.25.1
Hi Sidong,
I tried this patch and observed that it recreates the hang and confirms the fix. Thanks. Here's my comments for improvements.
Quoted 1 line · Sidong Yang
Test enabling/disable quota and creating/destroying qgroup repeatedly
nit: gerund (...ing) and base form are mixed. Base form would be the better to be same as the code comment.
Quoted 24 lines
in asynchronous and confirm it does not cause kernel hang. This is a regression test for the problem reported to linux-btrfs list [1].
The hang was recreated using the test case and fixed by kernel patch titled
btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
[1] https://lore.kernel.org/linux-btrfs/[email protected]/
Signed-off-by Sidong Yang <[email protected]>--- tests/btrfs/262 | 54 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/262.out | 2 ++ 2 files changed, 56 insertions(+) create mode 100755 tests/btrfs/262 create mode 100644 tests/btrfs/262.outtests/btrfs/262 new file +0
-0
tests/btrfs/262 line 3
Quoted 3 lines
tests/btrfs/262 new file +3
-0@@ -0,0 +1,54 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 YOUR NAME HERE. All Rights Reserved.
It's the better to put your name there :)
Quoted 4 lines
tests/btrfs/262 new file +4
-0+# +# FS QA Test 262 +# +# Test the deadlock between qgroup and quota commands
tests/btrfs/262 line 17
Quoted 10 lines
tests/btrfs/262 new file +10
-0+# +. ./common/preamble +_begin_fstest auto qgroup + +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate.
Can we remove this comment?
Quoted 11 lines
tests/btrfs/262 new file +11
-0+_supported_fs btrfs + +_require_scratch + +# Run command that enable/disable quota and create/destroy qgroup asynchronously +qgroup_deadlock_test() +{ + _scratch_mkfs > /dev/null 2>&1 + _scratch_mount + echo "=== qgroup deadlock test ===" >> $seqres.full +
tests/btrfs/262 line 38
Quoted 10 lines
tests/btrfs/262 new file +10
-0+ pids=() + for ((i = 0; i < 200; i++)); do + $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG qgroup create 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG qgroup destroy 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG quota disable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!)
Trailing tabs in the line above.
tests/btrfs/262 line 43
Quoted 5 lines
tests/btrfs/262 new file +5
-0+ done + + for pid in "${pids[@]}"; do + wait $pid + done
I think simple "wait" command does what the for loop does.
tests/btrfs/262 line 46
Quoted 3 lines
tests/btrfs/262 new file +3
-0+ + _scratch_unmount + _check_scratch_fs
I think _scratch_unmount and _check_scratch_fs are required only when the test case repeats scratch mount and unmount. This test case looks simple for me and the qgroup_deadlock_test function may not be necessary.
Quoted 19 lines
tests/btrfs/262 new file +8
-0+} + +qgroup_deadlock_test + +# success, all done +echo "Silence is golden" +status=0 +exittests/btrfs/262.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 262 +Silence is goldenSignature · 1 line
2.25.1
Signature · 2 lines
Best Regards, Shin'ichiro Kawasaki
On Wed, Mar 02, 2022 at 04:43:35AM +0000, Shinichiro Kawasaki wrote:
Hi, Shinichiro.
Thanks for reply!
Quoted 10 lines
Hi Sidong,
I tried this patch and observed that it recreates the hang and confirms the fix. Thanks. Here's my comments for improvements.
Quoted 1 line · Sidong Yang
Test enabling/disable quota and creating/destroying qgroup repeatedly
nit: gerund (...ing) and base form are mixed. Base form would be the better to be same as the code comment.
Yeah, 'disable' should be disabling.
Quoted 19 lines
Quoted 19 lines
in asynchronous and confirm it does not cause kernel hang. This is a regression test for the problem reported to linux-btrfs list [1].
The hang was recreated using the test case and fixed by kernel patch titled
btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
[1] https://lore.kernel.org/linux-btrfs/[email protected]/
Signed-off-by Sidong Yang <[email protected]>--- tests/btrfs/262 | 54 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/262.out | 2 ++ 2 files changed, 56 insertions(+) create mode 100755 tests/btrfs/262 create mode 100644 tests/btrfs/262.out
Quoted 11 lines
tests/btrfs/262 line 3
Quoted 9 lines
tests/btrfs/262 new file +3
-0@@ -0,0 +1,54 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 YOUR NAME HERE. All Rights Reserved.It's the better to put your name there :)
Oops,
Quoted 17 lines
Quoted 5 lines
tests/btrfs/262 new file +4
-0+# +# FS QA Test 262 +# +# Test the deadlock between qgroup and quota commandstests/btrfs/262 line 17
Quoted 10 lines
tests/btrfs/262 new file +10
-0+# +. ./common/preamble +_begin_fstest auto qgroup + +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate.Can we remove this comment?
Sure!
Quoted 16 lines
Quoted 16 lines
tests/btrfs/262 new file +15
-0+_supported_fs btrfs + +_require_scratch + +# Run command that enable/disable quota and create/destroy qgroup asynchronously +qgroup_deadlock_test() +{ + _scratch_mkfs > /dev/null 2>&1 + _scratch_mount + echo "=== qgroup deadlock test ===" >> $seqres.full + + pids=() + for ((i = 0; i < 200; i++)); do + $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!)
Quoted 8 lines
tests/btrfs/262 line 38
Quoted 6 lines
tests/btrfs/262 new file +6
-0+ $BTRFS_UTIL_PROG qgroup create 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG qgroup destroy 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG quota disable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!)Trailing tabs in the line above.
Oops..
Quoted 8 lines
tests/btrfs/262 line 43
Quoted 6 lines
tests/btrfs/262 new file +5
-0+ done + + for pid in "${pids[@]}"; do + wait $pid + doneI think simple "wait" command does what the for loop does.
I didn't know that "wait" command with no parameter waits all background processes to finish. So it seems that we don't need pids it can be deleted. Thanks.
Actually I've been agony about this. Does it needs timeout? When I tried to command like this "timeout 10s wait", This command couldn't be executed becase "wait" command is not binary. How can I insert timeout?
Quoted 8 lines
tests/btrfs/262 line 46
Quoted 4 lines
tests/btrfs/262 new file +3
-0+ + _scratch_unmount + _check_scratch_fsI think
_scratch_unmountand_check_scratch_fsare required only when the test case repeats scratch mount and unmount. This test case looks simple for me and theqgroup_deadlock_testfunction may not be necessary.
I agree. This case is very simple. We don't need to use function. It would be better that it's written in simple.
Quoted 24 lines
Quoted 20 lines
tests/btrfs/262 new file +8
-0+} + +qgroup_deadlock_test + +# success, all done +echo "Silence is golden" +status=0 +exittests/btrfs/262.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 262 +Silence is goldenSignature · 1 line
2.25.1Signature · 2 lines
Best Regards, Shin'ichiro Kawasaki
Quoted 31 lines · Sidong Yang
On Wed, Mar 02, 2022 at 04:43:35AM +0000, Shinichiro Kawasaki wrote:
Hi, Shinichiro.
Thanks for reply!
Quoted 10 lines
Hi Sidong,
I tried this patch and observed that it recreates the hang and confirms the fix. Thanks. Here's my comments for improvements.
Quoted 1 line · Sidong Yang
Test enabling/disable quota and creating/destroying qgroup repeatedly
nit: gerund (...ing) and base form are mixed. Base form would be the better to be same as the code comment.
Yeah, 'disable' should be disabling.
Quoted 13 lines
Quoted 13 lines
in asynchronous and confirm it does not cause kernel hang. This is a regression test for the problem reported to linux-btrfs list [1].
The hang was recreated using the test case and fixed by kernel patch titled
btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
[1] https://lore.kernel.org/linux-btrfs/[email protected]/
Signed-off-by Sidong Yang <[email protected]>---
(snip)
Quoted 8 lines
Quoted 7 lines
tests/btrfs/262 line 43
Quoted 5 lines
tests/btrfs/262 new file +5
-0+ done + + for pid in "${pids[@]}"; do + wait $pid + doneI think simple "wait" command does what the for loop does.
Quoted 7 lines
I didn't know that "wait" command with no parameter waits all background processes to finish. So it seems that we don't need pids it can be deleted. Thanks.
Actually I've been agony about this. Does it needs timeout? When I tried to command like this "timeout 10s wait", This command couldn't be executed becase "wait" command is not binary. How can I insert timeout?
I think recent discussion on the list is a good reference [1]. A patch was posted to add timeout to btrfs/255.
More importantly, it was discussed that such timeout of user space program will not help. Eryu pointed out that once "the kernel already deadlocked, and filesystem and/or device can't be used by next test either". IMHO, your new case will not require timeout either with same reasoning.
[1] https://lore.kernel.org/fstests/[email protected]/T/#me349d62ff367a0a6a28076bdd5b89263fc8109c0
Signature · 2 lines
Best Regards, Shin'ichiro Kawasaki
Quoted 54 lines · Shinichiro Kawasaki
Quoted 31 lines · Sidong Yang
On Wed, Mar 02, 2022 at 04:43:35AM +0000, Shinichiro Kawasaki wrote:
Hi, Shinichiro.
Thanks for reply!
Quoted 10 lines
Hi Sidong,
I tried this patch and observed that it recreates the hang and confirms the fix. Thanks. Here's my comments for improvements.
Quoted 1 line · Sidong Yang
Test enabling/disable quota and creating/destroying qgroup repeatedly
nit: gerund (...ing) and base form are mixed. Base form would be the better to be same as the code comment.
Yeah, 'disable' should be disabling.
Quoted 13 lines
Quoted 13 lines
in asynchronous and confirm it does not cause kernel hang. This is a regression test for the problem reported to linux-btrfs list [1].
The hang was recreated using the test case and fixed by kernel patch titled
btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
[1] https://lore.kernel.org/linux-btrfs/[email protected]/
Signed-off-by Sidong Yang <[email protected]>---
(snip)
Quoted 8 lines
Quoted 7 lines
tests/btrfs/262 line 43
Quoted 5 lines
tests/btrfs/262 new file +5
-0+ done + + for pid in "${pids[@]}"; do + wait $pid + doneI think simple "wait" command does what the for loop does.
Quoted 7 lines
I didn't know that "wait" command with no parameter waits all background processes to finish. So it seems that we don't need pids it can be deleted. Thanks.
Actually I've been agony about this. Does it needs timeout? When I tried to command like this "timeout 10s wait", This command couldn't be executed becase "wait" command is not binary. How can I insert timeout?
I think recent discussion on the list is a good reference [1]. A patch was posted to add timeout to btrfs/255.
Quoted 7 lines
More importantly, it was discussed that such timeout of user space program will not help. Eryu pointed out that once "the kernel already deadlocked, and filesystem and/or device can't be used by next test either". IMHO, your new case will not require timeout either with same reasoning.
[1] https://lore.kernel.org/fstests/[email protected]/T/#me349d62ff367a0a6a28076bdd5b89263fc8109c0
Thanks for a good example. I understand that there is no help in userspace for the kernel deadlocked. In this case, Just "wait" is enough.
Thanks,
Sidong
Quoted 3 lines
Signature · 2 lines
Best Regards, Shin'ichiro Kawasaki
Quoted 2 lines · Sidong Yang
Test enabling/disable quota and creating/destroying qgroup repeatedly in asynchronous and confirm it does not cause kernel hang. This is a
in asynchronous -> in parallel
Quoted 1 line
regression test for the problem reported to linux-btrfs list [1].
It's worth mentioning the deadlock only happens starting with kernel 5.17-rc3.
Quoted 9 lines
The hang was recreated using the test case and fixed by kernel patch titled
btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
[1] https://lore.kernel.org/linux-btrfs/[email protected]/
Signed-off-by Sidong Yang <[email protected]>
In addition to Shinichiro's comments...
Quoted 13 lines
--- tests/btrfs/262 | 54 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/262.out | 2 ++ 2 files changed, 56 insertions(+) create mode 100755 tests/btrfs/262 create mode 100644 tests/btrfs/262.outtests/btrfs/262 new file +0
-0
tests/btrfs/262 line 7
Quoted 7 lines
tests/btrfs/262 new file +7
-0@@ -0,0 +1,54 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 YOUR NAME HERE. All Rights Reserved. +# +# FS QA Test 262 +# +# Test the deadlock between qgroup and quota commands
The test description should be a lot more clear.
"the deadlock" is vague a gives the wrong idea we only ever had a single deadlock related to qgroups. "qgroup and quota commands" is confusing, and "qgroup" and "quota" are pretty much synonyms, and it should mention which commands.
Plus what we want to test is that we can run some qgroup operations in parallel without triggering a deadlock, crash, etc.
Perhaps something like:
"""
Test that running qgroup enable, create, destroy and disable commands in parallel does not result in a deadlock, a crash or any filesystem inconsistency.
"""
tests/btrfs/262 line 10
Quoted 3 lines
tests/btrfs/262 new file +3
-0+# +. ./common/preamble +_begin_fstest auto qgroup
Can also be added to the "quick" group. It takes 1 second in my slowest vm.
tests/btrfs/262 line 22
Quoted 12 lines
tests/btrfs/262 new file +12
-0+ +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs + +_require_scratch + +# Run command that enable/disable quota and create/destroy qgroup asynchronously
With the more clear test description above, this can go away.
tests/btrfs/262 line 27
Quoted 5 lines
tests/btrfs/262 new file +5
-0+qgroup_deadlock_test() +{ + _scratch_mkfs > /dev/null 2>&1 + _scratch_mount + echo "=== qgroup deadlock test ===" >> $seqres.full
There's no point in echoing this message to the .full file, it provides no value at all, as testing that is all that this testcase does.
Quoted 6 lines
tests/btrfs/262 new file +6
-0+ + pids=() + for ((i = 0; i < 200; i++)); do + $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG qgroup create 1/0 $SCRATCH_MNT 2>> $seqres.full &
tests/btrfs/262 line 43
Quoted 10 lines
tests/btrfs/262 new file +10
-0+ pids+=($!) + $BTRFS_UTIL_PROG qgroup destroy 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG quota disable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + done + + for pid in "${pids[@]}"; do + wait $pid + done
As pointed before by Shinichiro, a simple 'wait' here is enough, no need to keep track of the PIDs.
tests/btrfs/262 line 46
Quoted 3 lines
tests/btrfs/262 new file +3
-0+ + _scratch_unmount + _check_scratch_fs
Not needed, the fstests framework automatically runs 'btrfs check' when a test finishes. Doing this explicitly is only necessary when we need to do several mount/unmount operations and want to check the fs is fine after each unmount and before the next mount.
tests/btrfs/262 line 49
Quoted 3 lines
tests/btrfs/262 new file +3
-0+} + +qgroup_deadlock_test
There's no point in putting all the test code in a function, as the function is only called once.
Otherwise it looks good, and the test works as advertised, it triggers a deadlock on 5.17-rc3+ kernel and passes on a patched kernel.
Thanks for converting the reproducer into a test case.
Quoted 16 lines
tests/btrfs/262 new file +5
-0+ +# success, all done +echo "Silence is golden" +status=0 +exittests/btrfs/262.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 262 +Silence is goldenSignature · 1 line
2.25.1
On Wed, Mar 02, 2022 at 12:10:08PM +0000, Filipe Manana wrote:
Hi, Filipe!
Thanks for review.
Quoted 5 lines · Sidong Yang
Quoted 2 lines
Test enabling/disable quota and creating/destroying qgroup repeatedly in asynchronous and confirm it does not cause kernel hang. This is a
in asynchronous -> in parallel
Sure, Thanks!
Quoted 4 lines
Quoted 2 lines
regression test for the problem reported to linux-btrfs list [1].
It's worth mentioning the deadlock only happens starting with kernel 5.17-rc3.
It only happens in 5.17-rc3 version? I didn't know about it. I'll add mention about this.
Quoted 44 lines
Quoted 10 lines
The hang was recreated using the test case and fixed by kernel patch titled
btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
[1] https://lore.kernel.org/linux-btrfs/[email protected]/
Signed-off-by Sidong Yang <[email protected]>In addition to Shinichiro's comments...
Quoted 13 lines
--- tests/btrfs/262 | 54 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/262.out | 2 ++ 2 files changed, 56 insertions(+) create mode 100755 tests/btrfs/262 create mode 100644 tests/btrfs/262.outtests/btrfs/262 new file +0
-0tests/btrfs/262 line 7
Quoted 7 lines
tests/btrfs/262 new file +7
-0@@ -0,0 +1,54 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 YOUR NAME HERE. All Rights Reserved. +# +# FS QA Test 262 +# +# Test the deadlock between qgroup and quota commandsThe test description should be a lot more clear.
"the deadlock" is vague a gives the wrong idea we only ever had a single deadlock related to qgroups. "qgroup and quota commands" is confusing, and "qgroup" and "quota" are pretty much synonyms, and it should mention which commands.
Plus what we want to test is that we can run some qgroup operations in parallel without triggering a deadlock, crash, etc.
Quoted 8 lines
Perhaps something like:
"""
Test that running qgroup enable, create, destroy and disable commands in parallel does not result in a deadlock, a crash or any filesystem inconsistency.
"""
Yeah, It was not clear. I found that this test is not only for checking deadlock. But it checks that test runs without any problem.
Quoted 6 lines
tests/btrfs/262 line 10
Quoted 4 lines
tests/btrfs/262 new file +3
-0+# +. ./common/preamble +_begin_fstest auto qgroupCan also be added to the "quick" group. It takes 1 second in my slowest vm.
Okay, Thanks!
Quoted 15 lines
tests/btrfs/262 line 22
Quoted 13 lines
tests/btrfs/262 new file +12
-0+ +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs + +_require_scratch + +# Run command that enable/disable quota and create/destroy qgroup asynchronouslyWith the more clear test description above, this can go away.
Sure!
Quoted 9 lines
tests/btrfs/262 line 27
Quoted 6 lines
tests/btrfs/262 new file +5
-0+qgroup_deadlock_test() +{ + _scratch_mkfs > /dev/null 2>&1 + _scratch_mount + echo "=== qgroup deadlock test ===" >> $seqres.fullThere's no point in echoing this message to the .full file, it provides no value at all, as testing that is all that this testcase does.
I agree. This is pointless because it's simple test.
Quoted 12 lines
Quoted 12 lines
tests/btrfs/262 new file +11
-0+ + pids=() + for ((i = 0; i < 200; i++)); do + $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG qgroup create 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG qgroup destroy 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG quota disable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!)
Quoted 8 lines
tests/btrfs/262 line 43
Quoted 5 lines
tests/btrfs/262 new file +5
-0+ done + + for pid in "${pids[@]}"; do + wait $pid + doneAs pointed before by Shinichiro, a simple 'wait' here is enough, no need to keep track of the PIDs.
Yeah, I don't have to go hard way.
Quoted 10 lines
tests/btrfs/262 line 46
Quoted 4 lines
tests/btrfs/262 new file +3
-0+ + _scratch_unmount + _check_scratch_fsNot needed, the fstests framework automatically runs 'btrfs check' when a test finishes. Doing this explicitly is only necessary when we need to do several mount/unmount operations and want to check the fs is fine after each unmount and before the next mount.
I didn't know about that. I don't need to check manually.
Quoted 6 lines
tests/btrfs/262 line 49
Quoted 3 lines
tests/btrfs/262 new file +3
-0+} + +qgroup_deadlock_testThere's no point in putting all the test code in a function, as the function is only called once.
Of course!
Quoted 6 lines
Otherwise it looks good, and the test works as advertised, it triggers a deadlock on 5.17-rc3+ kernel and passes on a patched kernel.
Thanks for converting the reproducer into a test case.
Thanks for detailed review. I'll back soon with v2.
Thanks,
Sidong
Quoted 16 lines
Quoted 16 lines
tests/btrfs/262 new file +5
-0+ +# success, all done +echo "Silence is golden" +status=0 +exittests/btrfs/262.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 262 +Silence is goldenSignature · 1 line
2.25.1
Quoted 10 lines · Sidong Yang
On Wed, Mar 02, 2022 at 12:10:08PM +0000, Filipe Manana wrote:
Hi, Filipe!
Thanks for review.Quoted 5 lines · Sidong Yang
Quoted 2 lines
Test enabling/disable quota and creating/destroying qgroup repeatedly in asynchronous and confirm it does not cause kernel hang. This is a
in asynchronous -> in parallel
Quoted 8 lines
Sure, Thanks!
Quoted 4 lines
Quoted 2 lines
regression test for the problem reported to linux-btrfs list [1].
It's worth mentioning the deadlock only happens starting with kernel 5.17-rc3.
It only happens in 5.17-rc3 version? I didn't know about it. I'll add mention about this.
Well, in the kernel patch we have:
And that commit was introduced in 5.17-rc3. Maybe it deadlocked in a different way before that commit, perhaps in the way that e804861bd4e6 describes. However I haven't checked how it behaves on a kernel without that commit. But at least we know that currently it deadlocks at 5.17-rc3+.
Quoted 160 lines
Quoted 44 lines
Quoted 10 lines
The hang was recreated using the test case and fixed by kernel patch titled
btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
[1] https://lore.kernel.org/linux-btrfs/[email protected]/
Signed-off-by Sidong Yang <[email protected]>In addition to Shinichiro's comments...
Quoted 13 lines
--- tests/btrfs/262 | 54 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/262.out | 2 ++ 2 files changed, 56 insertions(+) create mode 100755 tests/btrfs/262 create mode 100644 tests/btrfs/262.outtests/btrfs/262 new file +0
-0tests/btrfs/262 line 7
Quoted 7 lines
tests/btrfs/262 new file +7
-0@@ -0,0 +1,54 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 YOUR NAME HERE. All Rights Reserved. +# +# FS QA Test 262 +# +# Test the deadlock between qgroup and quota commandsThe test description should be a lot more clear.
"the deadlock" is vague a gives the wrong idea we only ever had a single deadlock related to qgroups. "qgroup and quota commands" is confusing, and "qgroup" and "quota" are pretty much synonyms, and it should mention which commands.
Plus what we want to test is that we can run some qgroup operations in parallel without triggering a deadlock, crash, etc.
Quoted 8 lines
Perhaps something like:
"""
Test that running qgroup enable, create, destroy and disable commands in parallel does not result in a deadlock, a crash or any filesystem inconsistency.
"""Yeah, It was not clear. I found that this test is not only for checking deadlock. But it checks that test runs without any problem.
Quoted 6 lines
tests/btrfs/262 line 10
Quoted 4 lines
tests/btrfs/262 new file +3
-0+# +. ./common/preamble +_begin_fstest auto qgroupCan also be added to the "quick" group. It takes 1 second in my slowest vm.
Okay, Thanks!
Quoted 15 lines
tests/btrfs/262 line 22
Quoted 13 lines
tests/btrfs/262 new file +12
-0+ +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs + +_require_scratch + +# Run command that enable/disable quota and create/destroy qgroup asynchronouslyWith the more clear test description above, this can go away.
Sure!
Quoted 9 lines
tests/btrfs/262 line 27
Quoted 6 lines
tests/btrfs/262 new file +5
-0+qgroup_deadlock_test() +{ + _scratch_mkfs > /dev/null 2>&1 + _scratch_mount + echo "=== qgroup deadlock test ===" >> $seqres.fullThere's no point in echoing this message to the .full file, it provides no value at all, as testing that is all that this testcase does.
I agree. This is pointless because it's simple test.
Quoted 12 lines
Quoted 12 lines
tests/btrfs/262 new file +11
-0+ + pids=() + for ((i = 0; i < 200; i++)); do + $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG qgroup create 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG qgroup destroy 1/0 $SCRATCH_MNT 2>> $seqres.full & + pids+=($!) + $BTRFS_UTIL_PROG quota disable $SCRATCH_MNT 2>> $seqres.full & + pids+=($!)Quoted 8 lines
tests/btrfs/262 line 43
Quoted 5 lines
tests/btrfs/262 new file +5
-0+ done + + for pid in "${pids[@]}"; do + wait $pid + doneAs pointed before by Shinichiro, a simple 'wait' here is enough, no need to keep track of the PIDs.
Yeah, I don't have to go hard way.
Quoted 10 lines
tests/btrfs/262 line 46
Quoted 4 lines
tests/btrfs/262 new file +3
-0+ + _scratch_unmount + _check_scratch_fsNot needed, the fstests framework automatically runs 'btrfs check' when a test finishes. Doing this explicitly is only necessary when we need to do several mount/unmount operations and want to check the fs is fine after each unmount and before the next mount.
I didn't know about that. I don't need to check manually.
Quoted 6 lines
tests/btrfs/262 line 49
Quoted 3 lines
tests/btrfs/262 new file +3
-0+} + +qgroup_deadlock_testThere's no point in putting all the test code in a function, as the function is only called once.
Of course!
Quoted 6 lines
Otherwise it looks good, and the test works as advertised, it triggers a deadlock on 5.17-rc3+ kernel and passes on a patched kernel.
Thanks for converting the reproducer into a test case.
Thanks for detailed review. I'll back soon with v2.
Thanks,
SidongQuoted 16 lines
Quoted 16 lines
tests/btrfs/262 new file +5
-0+ +# success, all done +echo "Silence is golden" +status=0 +exittests/btrfs/262.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 262 +Silence is goldenSignature · 1 line
2.25.1