fstests: btrfs/028: Don't pollute golden output for killing already finished process
PATCH 1/2 fstests: btrfs/028: Don't pollute golden output for killing already finished process
Sometimes on fast enough test vm, btrfs/028 fails like:
btrfs/028 31s ... - output mismatch (see /home/adam/xfstests-dev/results//btrfs/028.out.bad)
--- tests/btrfs/028.out 2019-07-22 14:13:44.646666660 +0800
+++ /home/adam/xfstests-dev/results//btrfs/028.out.bad 2019-09-18 14:14:45.442131411 +0800
@@ -1,2 +1,3 @@
QA output created by 028
+/home/adam/xfstests-dev/tests/btrfs/028: line 64: kill: (2459) - No such process
Silence is golden
...It's caused by killing already finished process.
There is no need for kill command to pollute the golden output, so just redirect all of its stdout and stderr to null.
--- tests/btrfs/028 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/btrfs/028 +1 -1
@@ -61,7 +61,7 @@ balance_pid=$! # 30s is enough to trigger bug sleep $((30*$TIME_FACTOR)) -kill $fsstress_pid $balance_pid +kill $fsstress_pid $balance_pid &> /dev/null wait # kill _btrfs_stress_balance can't end balance, so call btrfs balance cancel
Signature · 1 line
2.22.0
PATCH 2/2 fstests: btrfs/011: Handle finished scrub/replace operation gracefully
[BUG]
When btrfs/011 is executed on a fast enough system (fully memory backed VM, with test device has unsafe cache mode), the test can fail like this:
btrfs/011 43s ... [failed, exit status 1]- output mismatch (see /home/adam/xfstests-dev/results//btrfs/011.out.bad)
--- tests/btrfs/011.out 2019-07-22 14:13:44.643333326 +0800
+++ /home/adam/xfstests-dev/results//btrfs/011.out.bad 2019-09-18 14:49:28.308798022 +0800
@@ -1,3 +1,4 @@
QA output created by 011
*** test btrfs replace
-*** done
+failed: '/usr/bin/btrfs replace cancel /mnt/scratch'
+(see /home/adam/xfstests-dev/results//btrfs/011.full for details)
...[CAUSE]
Looking into the full output, it shows:
...
Replace from /dev/mapper/test-scratch1 to /dev/mapper/test-scratch2
# /usr/bin/btrfs replace start -f /dev/mapper/test-scratch1 /dev/mapper/test-scratch2 /mnt/scratch # /usr/bin/btrfs replace cancel /mnt/scratch INFO: ioctl(DEV_REPLACE_CANCEL)"/mnt/scratch": not started failed: '/usr/bin/btrfs replace cancel /mnt/scratch'
So this means the replace is already finished before we cancel it. For fast system, it's very common.
[FIX]
Instead of using _run_btrfs_util_prog which requires 0 as return value, we just call "$BTRFS_UTIL_PROG replace cancel" and ignore all its stderr/stdout, and completely rely on "$BTRFS_UTIL_PROG replace status" output to verify the work.
Furthermore if we finished replac before cancelling it, we should replace again to switch the device back, or after the test case, btrfs check will fail as there is no valid btrfs on that replaced device.
--- tests/btrfs/011 | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
tests/btrfs/011 +14 -2
@@ -148,13 +148,25 @@ btrfs_replace_test() # background the replace operation (no '-B' option given) _run_btrfs_util_prog replace start -f $replace_options $source_dev $target_dev $SCRATCH_MNT sleep 1 - _run_btrfs_util_prog replace cancel $SCRATCH_MNT + # 1s is enough for fast system to finish replace, so here we + # ignore all the output, completely rely on later status + # output to determine + $BTRFS_UTIL_PROG replace cancel $SCRATCH_MNT &> /dev/null # 'replace status' waits for the replace operation to finish # before the status is printed $BTRFS_UTIL_PROG replace status $SCRATCH_MNT > $tmp.tmp 2>&1 cat $tmp.tmp >> $seqres.full - grep -q canceled $tmp.tmp || _fail "btrfs replace status (canceled) failed" + grep -q -e canceled -e finished $tmp.tmp ||\ + _fail "btrfs replace status (canceled) failed" + + # If replace finished before cancel, replace them back or + # the final fsck after test case will fail as there is no btrfs + # on the $source_dev anymore + if grep -q -e finished $tmp.tmp ; then + $BTRFS_UTIL_PROG replace start -Bf $replace_options \ + $target_dev $source_dev $SCRATCH_MNT + fi else if [ "${quick}Q" = "thoroughQ" ]; then # On current hardware, the thorough test runs
Signature · 1 line
2.22.0
Re: [PATCH 2/2] fstests: btrfs/011: Handle finished scrub/replace operation gracefully
In reply to Qu WenruoQuoted 67 lines · Qu Wenruo
[BUG]
When btrfs/011 is executed on a fast enough system (fully memory backed VM, with test device has unsafe cache mode), the test can fail like this:btrfs/011 43s ... [failed, exit status 1]- output mismatch (see /home/adam/xfstests-dev/results//btrfs/011.out.bad) --- tests/btrfs/011.out 2019-07-22 14:13:44.643333326 +0800 +++ /home/adam/xfstests-dev/results//btrfs/011.out.bad 2019-09-18 14:49:28.308798022 +0800 @@ -1,3 +1,4 @@ QA output created by 011 *** test btrfs replace -*** done +failed: '/usr/bin/btrfs replace cancel /mnt/scratch' +(see /home/adam/xfstests-dev/results//btrfs/011.full for details) ...[CAUSE]
Looking into the full output, it shows:
...
Replace from/dev/mapper/test-scratch1to/dev/mapper/test-scratch2# /usr/bin/btrfs replace start -f /dev/mapper/test-scratch1 /dev/mapper/test-scratch2 /mnt/scratch # /usr/bin/btrfs replace cancel /mnt/scratch INFO: ioctl(DEV_REPLACE_CANCEL)"/mnt/scratch": not started failed: '/usr/bin/btrfs replace cancel /mnt/scratch'So this means the replace is already finished before we cancel it. For fast system, it's very common.
[FIX]
Instead of using_run_btrfs_util_progwhich requires 0 as return value, we just call "$BTRFS_UTIL_PROG replace cancel" and ignore all its stderr/stdout, and completely rely on "$BTRFS_UTIL_PROG replace status" output to verify the work.Furthermore if we finished replac before cancelling it, we should replace again to switch the device back, or after the test case, btrfs check will fail as there is no valid btrfs on that replaced device.
Signed-off-by Qu Wenruo <[email protected]>--- tests/btrfs/011 | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)tests/btrfs/011 +7
-2@@ -148,13 +148,25 @@ btrfs_replace_test() # background the replace operation (no '-B' option given) _run_btrfs_util_prog replace start -f $replace_options $source_dev $target_dev $SCRATCH_MNT sleep 1 - _run_btrfs_util_prog replace cancel $SCRATCH_MNT + # 1s is enough for fast system to finish replace, so here we + # ignore all the output, completely rely on later status + # output to determine + $BTRFS_UTIL_PROG replace cancel $SCRATCH_MNT &> /dev/null # 'replace status' waits for the replace operation to finish # before the status is printed $BTRFS_UTIL_PROG replace status $SCRATCH_MNT > $tmp.tmp 2>&1 cat $tmp.tmp >> $seqres.full - grep -q canceled $tmp.tmp || _fail "btrfs replace status (canceled) failed" + grep -q -e canceled -e finished $tmp.tmp ||\ + _fail "btrfs replace status (canceled) failed" +
tests/btrfs/011 btrfs_replace_test() · line 172
Quoted 10 lines
tests/btrfs/011 +7
-0+ # If replace finished before cancel, replace them back or + # the final fsck after test case will fail as there is no btrfs + # on the $source_dev anymore + if grep -q -e finished $tmp.tmp ; then + $BTRFS_UTIL_PROG replace start -Bf $replace_options \ + $target_dev $source_dev $SCRATCH_MNT + fi else if [ "${quick}Q" = "thoroughQ" ]; then # On current hardware, the thorough test runs
Faced the same problem before. But I don't have a good solution to fix. Because the idea of the test case appears to test the replace cancel. This change makes it not to error if replace is not running in the first place.
Thanks, Anand
Re: [PATCH 2/2] fstests: btrfs/011: Handle finished scrub/replace operation gracefully
Quoted 79 lines · Anand Jain
Quoted 78 lines · Qu Wenruo
[BUG]
When btrfs/011 is executed on a fast enough system (fully memory backed VM, with test device has unsafe cache mode), the test can fail like this:btrfs/011 43s ... [failed, exit status 1]- output mismatch (see
/home/adam/xfstests-dev/results//btrfs/011.out.bad)
--- tests/btrfs/011.out 2019-07-22 14:13:44.643333326 +0800 +++/home/adam/xfstests-dev/results//btrfs/011.out.bad2019-09-18 14:49:28.308798022 +0800@@ -1,3 +1,4 @@
QA output created by 011
*** test btrfs replace
-*** done
+failed: '/usr/bin/btrfsreplace cancel/mnt/scratch' +(see/home/adam/xfstests-dev/results//btrfs/011.fullfor details)
...[CAUSE]
Looking into the full output, it shows:
...
Replace from/dev/mapper/test-scratch1to/dev/mapper/test-scratch2# /usr/bin/btrfs replace start -f /dev/mapper/test-scratch1 /dev/mapper/test-scratch2 /mnt/scratch # /usr/bin/btrfs replace cancel /mnt/scratch INFO: ioctl(DEV_REPLACE_CANCEL)"/mnt/scratch": not started failed: '/usr/bin/btrfs replace cancel /mnt/scratch'So this means the replace is already finished before we cancel it. For fast system, it's very common.
[FIX]
Instead of using_run_btrfs_util_progwhich requires 0 as return value, we just call "$BTRFS_UTIL_PROG replace cancel" and ignore all its stderr/stdout, and completely rely on "$BTRFS_UTIL_PROG replace status" output to verify the work.Furthermore if we finished replac before cancelling it, we should replace again to switch the device back, or after the test case, btrfs check will fail as there is no valid btrfs on that replaced device.
Signed-off-by Qu Wenruo <[email protected]>---
tests/btrfs/011 | 16 ++++++++++++++--1 file changed, 14 insertions(+), 2 deletions(-)tests/btrfs/011 +0
-0@@ -148,13 +148,25 @@ btrfs_replace_test() # background the replace operation (no '-B' option given)
_run_btrfs_util_progreplace start -f $replace_options $source_dev $target_dev $SCRATCH_MNT
sleep 1
-_run_btrfs_util_progreplace cancel $SCRATCH_MNT
+ # 1s is enough for fast system to finish replace, so here we + # ignore all the output, completely rely on later status + # output to determine
+ $BTRFS_UTIL_PROG replace cancel $SCRATCH_MNT &>/dev/null# 'replace status' waits for the replace operation to finish# before the status is printed $BTRFS_UTIL_PROG replace status $SCRATCH_MNT > $tmp.tmp 2>&1 cat $tmp.tmp >> $seqres.full - grep -q canceled $tmp.tmp || _fail "btrfs replace status (canceled) failed" + grep -q -e canceled -e finished $tmp.tmp ||\ + _fail "btrfs replace status (canceled) failed" + + # If replace finished before cancel, replace them back or + # the final fsck after test case will fail as there is no btrfs + # on the $source_dev anymore + if grep -q -e finished $tmp.tmp ; then + $BTRFS_UTIL_PROG replace start -Bf $replace_options \ + $target_dev $source_dev $SCRATCH_MNT + fi else
Quoted 8 lines
Quoted 3 lines
if [ "${quick}Q" = "thoroughQ" ]; then
# On current hardware, the thorough test runsFaced the same problem before. But I don't have a good solution to fix. Because the idea of the test case appears to test the replace cancel. This change makes it not to error if replace is not running in the first place.
Then what about _notrun if we find that replace/scrub finished too early?
BTW, if we really want to test all regular scrub/replace with cancel/finish combination, we should split this test into 4 test cases, then we can completely afford to skip 2 while still allow the finish cases to run...
Thanks,
Qu
Quoted 3 lines
Thanks, Anand
Re: [PATCH 2/2] fstests: btrfs/011: Handle finished scrub/replace operation gracefully
In reply to Qu WenruoQuoted 21 lines · Qu Wenruo
[BUG]
When btrfs/011 is executed on a fast enough system (fully memory backed VM, with test device has unsafe cache mode), the test can fail like this:btrfs/011 43s ... [failed, exit status 1]- output mismatch (see
/home/adam/xfstests-dev/results//btrfs/011.out.bad)--- tests/btrfs/011.out 2019-07-22 14:13:44.643333326 +0800 +++ /home/adam/xfstests-dev/results//btrfs/011.out.bad 2019-09-18 14:49:28.308798022 +0800 @@ -1,3 +1,4 @@ QA output created by 011 *** test btrfs replace -*** done +failed: '/usr/bin/btrfs replace cancel /mnt/scratch' +(see /home/adam/xfstests-dev/results//btrfs/011.full for details) ...[CAUSE]
Looking into the full output, it shows:
...
Replace from/dev/mapper/test-scratch1to/dev/mapper/test-scratch2
Quoted 7 lines
# /usr/bin/btrfs replace start -f /dev/mapper/test-scratch1 /dev/mapper/test-scratch2 /mnt/scratch # /usr/bin/btrfs replace cancel /mnt/scratch INFO: ioctl(DEV_REPLACE_CANCEL)"/mnt/scratch": not started failed: '/usr/bin/btrfs replace cancel /mnt/scratch'So this means the replace is already finished before we cancel it. For fast system, it's very common.
Does generate heavier load & more data make replace operation last longer? e.g. make more 'noise' by running fsstress instead of dumping /dev/urandom before starting replace.
And does sleep shorter time (0.5s?) before cancel work?
Thanks,
Eryu
Quoted 50 lines
[FIX]
Instead of using_run_btrfs_util_progwhich requires 0 as return value, we just call "$BTRFS_UTIL_PROG replace cancel" and ignore all its stderr/stdout, and completely rely on "$BTRFS_UTIL_PROG replace status" output to verify the work.Furthermore if we finished replac before cancelling it, we should replace again to switch the device back, or after the test case, btrfs check will fail as there is no valid btrfs on that replaced device.
Signed-off-by Qu Wenruo <[email protected]>--- tests/btrfs/011 | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)tests/btrfs/011 +14
-2@@ -148,13 +148,25 @@ btrfs_replace_test() # background the replace operation (no '-B' option given) _run_btrfs_util_prog replace start -f $replace_options $source_dev $target_dev $SCRATCH_MNT sleep 1 - _run_btrfs_util_prog replace cancel $SCRATCH_MNT + # 1s is enough for fast system to finish replace, so here we + # ignore all the output, completely rely on later status + # output to determine + $BTRFS_UTIL_PROG replace cancel $SCRATCH_MNT &> /dev/null # 'replace status' waits for the replace operation to finish # before the status is printed $BTRFS_UTIL_PROG replace status $SCRATCH_MNT > $tmp.tmp 2>&1 cat $tmp.tmp >> $seqres.full - grep -q canceled $tmp.tmp || _fail "btrfs replace status (canceled) failed" + grep -q -e canceled -e finished $tmp.tmp ||\ + _fail "btrfs replace status (canceled) failed" + + # If replace finished before cancel, replace them back or + # the final fsck after test case will fail as there is no btrfs + # on the $source_dev anymore + if grep -q -e finished $tmp.tmp ; then + $BTRFS_UTIL_PROG replace start -Bf $replace_options \ + $target_dev $source_dev $SCRATCH_MNT + fi else if [ "${quick}Q" = "thoroughQ" ]; then # On current hardware, the thorough test runsSignature · 1 line
2.22.0
Re: [PATCH 2/2] fstests: btrfs/011: Handle finished scrub/replace operation gracefully
Quoted 27 lines · Eryu Guan
Quoted 26 lines · Qu Wenruo
[BUG]
When btrfs/011 is executed on a fast enough system (fully memory backed VM, with test device has unsafe cache mode), the test can fail like this:btrfs/011 43s ... [failed, exit status 1]- output mismatch (see
/home/adam/xfstests-dev/results//btrfs/011.out.bad)--- tests/btrfs/011.out 2019-07-22 14:13:44.643333326 +0800 +++ /home/adam/xfstests-dev/results//btrfs/011.out.bad 2019-09-18 14:49:28.308798022 +0800 @@ -1,3 +1,4 @@ QA output created by 011 *** test btrfs replace -*** done +failed: '/usr/bin/btrfs replace cancel /mnt/scratch' +(see /home/adam/xfstests-dev/results//btrfs/011.full for details) ...[CAUSE]
Looking into the full output, it shows:
...
Replace from/dev/mapper/test-scratch1to/dev/mapper/test-scratch2# /usr/bin/btrfs replace start -f /dev/mapper/test-scratch1 /dev/mapper/test-scratch2 /mnt/scratch # /usr/bin/btrfs replace cancel /mnt/scratch INFO: ioctl(DEV_REPLACE_CANCEL)"/mnt/scratch": not started failed: '/usr/bin/btrfs replace cancel /mnt/scratch'
Quoted 6 lines
Quoted 2 lines
So this means the replace is already finished before we cancel it. For fast system, it's very common.
Does generate heavier load & more data make replace operation last longer? e.g. make more 'noise' by running fsstress instead of dumping
/dev/urandombefore starting replace.
That's a great idea.
In fact we can try to write as much data as possible for 3s (using Direct IO), so that we know we have some data which will take at least 3s to read/write.
I'd take a try on this method.
Thanks,
Qu
Quoted 56 lines
And does sleep shorter time (0.5s?) before cancel work?
Thanks,
EryuQuoted 50 lines
[FIX]
Instead of using_run_btrfs_util_progwhich requires 0 as return value, we just call "$BTRFS_UTIL_PROG replace cancel" and ignore all its stderr/stdout, and completely rely on "$BTRFS_UTIL_PROG replace status" output to verify the work.Furthermore if we finished replac before cancelling it, we should replace again to switch the device back, or after the test case, btrfs check will fail as there is no valid btrfs on that replaced device.
Signed-off-by Qu Wenruo <[email protected]>--- tests/btrfs/011 | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)tests/btrfs/011 +14
-2@@ -148,13 +148,25 @@ btrfs_replace_test() # background the replace operation (no '-B' option given) _run_btrfs_util_prog replace start -f $replace_options $source_dev $target_dev $SCRATCH_MNT sleep 1 - _run_btrfs_util_prog replace cancel $SCRATCH_MNT + # 1s is enough for fast system to finish replace, so here we + # ignore all the output, completely rely on later status + # output to determine + $BTRFS_UTIL_PROG replace cancel $SCRATCH_MNT &> /dev/null # 'replace status' waits for the replace operation to finish # before the status is printed $BTRFS_UTIL_PROG replace status $SCRATCH_MNT > $tmp.tmp 2>&1 cat $tmp.tmp >> $seqres.full - grep -q canceled $tmp.tmp || _fail "btrfs replace status (canceled) failed" + grep -q -e canceled -e finished $tmp.tmp ||\ + _fail "btrfs replace status (canceled) failed" + + # If replace finished before cancel, replace them back or + # the final fsck after test case will fail as there is no btrfs + # on the $source_dev anymore + if grep -q -e finished $tmp.tmp ; then + $BTRFS_UTIL_PROG replace start -Bf $replace_options \ + $target_dev $source_dev $SCRATCH_MNT + fi else if [ "${quick}Q" = "thoroughQ" ]; then # On current hardware, the thorough test runsSignature · 1 line
2.22.0