fstests: Always dump dmesg for failed test cases
PATCH 1/2 fstests: Always dump dmesg for failed test cases
When hard-to-hit bugs happened, we really want every piece of info to help us debugging.
Although we already have KEEP_DMESG config, not everyone is utilizing it, thus when hard-to-hit bugs happened, one could only set it and retry until next hit.
This patch will change the behavior by always dumping the dmesg for failed tests, so that developers can always get extra info from any failure.
--- check | 3 +++ 1 file changed, 3 insertions(+)
check +3 -0
@@ -840,6 +840,9 @@ for section in $HOST_OPTIONS_SECTIONS; do # make sure we record the status of the last test we ran. if $err ; then + if [ ! -f $seqres.dmesg ]; then + _dmesg_since_test_start >$seqres.dmesg + fi bad="$bad $seqnum" n_bad=`expr $n_bad + 1` tc_status="fail"
Signature · 1 line
2.24.1
PATCH 2/2 fstests: btrfs: Fix a bug where test case can't grab the 2nd device when glob is used
If SCRATCH_DEV_POOL is definted using glob, e.g. SCRATCH_DEV_POOL="/dev/mapper/test-scratch[123456]", then btrfs/175 will fail like this:
btrfs/175 15s ... - output mismatch (see results//btrfs/175.out.bad)
--- tests/btrfs/175.out 2019-10-22 15:18:14.085632007 +0800
+++ results//btrfs/175.out.bad 2020-01-20 14:53:56.518567916 +0800
@@ -6,3 +6,4 @@
Single on multiple devices
swapon: SCRATCH_MNT/swap: swapon failed: Invalid argument
Single on one device
+ERROR: checking status of : No such file or directory
...
(Run 'diff -u tests/btrfs/175.out results//btrfs/175.out.bad' to see the entire diff)
This is caused by the extra quotation mark (and the complexity nature of bash glob).
# SCRATCH_DEV_POOL="/dev/mapper/test-scratch[123]"
# echo ${SCRATCH_DEV_POOL}
/dev/mapper/test-scratch1 /dev/mapper/test-scratch2 /dev/mapper/test-scratch3
# echo "${SCRATCH_DEV_POOL}"
/dev/mapper/test-scratch[123]To fix the problem, remove the quotation mark out of ${SCRATCH_DEV_POOL} or $SCRATCH_DEV_POOL for all related test cases.
---
The weirdest thing is, only btrfs/17[56], all other related test cases pass without any problem.
Maybe it's time to provide a proper wrapper to do such thing?
--- tests/btrfs/140 | 2 +- tests/btrfs/141 | 2 +- tests/btrfs/142 | 2 +- tests/btrfs/143 | 2 +- tests/btrfs/157 | 2 +- tests/btrfs/158 | 2 +- tests/btrfs/175 | 2 +- tests/btrfs/176 | 6 +++--- 8 files changed, 10 insertions(+), 10 deletions(-)
tests/btrfs/140 +1 -1
@@ -65,7 +65,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 2
tests/btrfs/141 +1 -1
@@ -65,7 +65,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 2
tests/btrfs/142 +1 -1
@@ -66,7 +66,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } start_fail()
tests/btrfs/143 +1 -1
@@ -73,7 +73,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } SYSFS_BDEV=`_sysfs_dev $SCRATCH_DEV`
tests/btrfs/157 +1 -1
@@ -70,7 +70,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 4
tests/btrfs/158 +1 -1
@@ -62,7 +62,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 4
tests/btrfs/175 +1 -1
@@ -63,7 +63,7 @@ _scratch_mount # Create the swap file, then add the device. That way we know it's all on one # device. _format_swapfile "$SCRATCH_MNT/swap" $(($(get_page_size) * 10)) -scratch_dev2="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $2 }')" +scratch_dev2="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $2 }')" $BTRFS_UTIL_PROG device add -f "$scratch_dev2" "$SCRATCH_MNT" swapon "$SCRATCH_MNT/swap" 2>&1 | _filter_scratch swapoff "$SCRATCH_MNT/swap" > /dev/null 2>&1
tests/btrfs/176 +3 -3
@@ -39,9 +39,9 @@ _require_scratch_swapfile # We check the filesystem manually because we move devices around. rm -f "${RESULT_DIR}/require_scratch" -scratch_dev1="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $1 }')" -scratch_dev2="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $2 }')" -scratch_dev3="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $3 }')" +scratch_dev1="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $1 }')" +scratch_dev2="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $2 }')" +scratch_dev3="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $3 }')" echo "Remove device" _scratch_mkfs >> $seqres.full 2>&1
Signature · 1 line
2.24.1
Re: [PATCH 2/2] fstests: btrfs: Fix a bug where test case can't grab the 2nd device when glob is used
Quoted 2 lines · Qu Wenruo
If
SCRATCH_DEV_POOLis definted using glob, e.g. SCRATCH_DEV_POOL="/dev/mapper/test-scratch[123456]", then btrfs/175 will
I don't think this is necessary, and I think it's not a problem that fstests should resolve.
Usually Config file is not always changing, setting SCRATCH_DEV_POOL correctly for once should be fine. A simple command could expand the device list for you:
[root@fedoravm xfstests]# ls /dev/mapper/testvg-lv[1234567]
/dev/mapper/testvg-lv1 /dev/mapper/testvg-lv2 /dev/mapper/testvg-lv3 /dev/mapper/testvg-lv4 /dev/mapper/testvg-lv5 /dev/mapper/testvg-lv6 /dev/mapper/testvg-lv7
Then just copy/paste the device list.
Thanks,
Eryu
Quoted 152 lines
fail like this:
btrfs/175 15s ... - output mismatch (see results//btrfs/175.out.bad)
--- tests/btrfs/175.out 2019-10-22 15:18:14.085632007 +0800
+++ results//btrfs/175.out.bad 2020-01-20 14:53:56.518567916 +0800
@@ -6,3 +6,4 @@
Single on multiple devices
swapon: SCRATCH_MNT/swap: swapon failed: Invalid argument
Single on one device
+ERROR: checking status of : No such file or directory
...
(Run 'diff -u tests/btrfs/175.out results//btrfs/175.out.bad' to see the entire diff)This is caused by the extra quotation mark (and the complexity nature of bash glob).
# SCRATCH_DEV_POOL="/dev/mapper/test-scratch[123]"
# echo ${SCRATCH_DEV_POOL} /dev/mapper/test-scratch1 /dev/mapper/test-scratch2 /dev/mapper/test-scratch3 # echo "${SCRATCH_DEV_POOL}" /dev/mapper/test-scratch[123]To fix the problem, remove the quotation mark out of ${SCRATCH_DEV_POOL} or $SCRATCH_DEV_POOL for all related test cases.
Signed-off-by Qu Wenruo <[email protected]>---
The weirdest thing is, only btrfs/17[56], all other related test cases pass without any problem.Maybe it's time to provide a proper wrapper to do such thing?
--- tests/btrfs/140 | 2 +- tests/btrfs/141 | 2 +- tests/btrfs/142 | 2 +- tests/btrfs/143 | 2 +- tests/btrfs/157 | 2 +- tests/btrfs/158 | 2 +- tests/btrfs/175 | 2 +- tests/btrfs/176 | 6 +++--- 8 files changed, 10 insertions(+), 10 deletions(-)tests/btrfs/140 +1
-1@@ -65,7 +65,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 2tests/btrfs/141 +1
-1@@ -65,7 +65,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 2tests/btrfs/142 +1
-1@@ -66,7 +66,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } start_fail()tests/btrfs/143 +1
-1@@ -73,7 +73,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } SYSFS_BDEV=`_sysfs_dev $SCRATCH_DEV`tests/btrfs/157 +1
-1@@ -70,7 +70,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 4tests/btrfs/158 +1
-1@@ -62,7 +62,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 4tests/btrfs/175 +1
-1@@ -63,7 +63,7 @@ _scratch_mount # Create the swap file, then add the device. That way we know it's all on one # device. _format_swapfile "$SCRATCH_MNT/swap" $(($(get_page_size) * 10)) -scratch_dev2="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $2 }')" +scratch_dev2="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $2 }')" $BTRFS_UTIL_PROG device add -f "$scratch_dev2" "$SCRATCH_MNT" swapon "$SCRATCH_MNT/swap" 2>&1 | _filter_scratch swapoff "$SCRATCH_MNT/swap" > /dev/null 2>&1tests/btrfs/176 +3
-3@@ -39,9 +39,9 @@ _require_scratch_swapfile # We check the filesystem manually because we move devices around. rm -f "${RESULT_DIR}/require_scratch" -scratch_dev1="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $1 }')" -scratch_dev2="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $2 }')" -scratch_dev3="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $3 }')" +scratch_dev1="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $1 }')" +scratch_dev2="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $2 }')" +scratch_dev3="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $3 }')" echo "Remove device" _scratch_mkfs >> $seqres.full 2>&1Signature · 1 line
2.24.1
Re: [PATCH 2/2] fstests: btrfs: Fix a bug where test case can't grab the 2nd device when glob is used
Quoted 7 lines · Eryu Guan
Quoted 2 lines · Qu Wenruo
If
SCRATCH_DEV_POOLis definted using glob, e.g. SCRATCH_DEV_POOL="/dev/mapper/test-scratch[123456]", then btrfs/175 willI don't think this is necessary, and I think it's not a problem that fstests should resolve.
Quoted 8 lines
Usually Config file is not always changing, setting
SCRATCH_DEV_POOLcorrectly for once should be fine. A simple command could expand the device list for you:[root@fedoravm xfstests]# ls /dev/mapper/testvg-lv[1234567]
/dev/mapper/testvg-lv1/dev/mapper/testvg-lv2/dev/mapper/testvg-lv3/dev/mapper/testvg-lv4/dev/mapper/testvg-lv5/dev/mapper/testvg-lv6/dev/mapper/testvg-lv7Then just copy/paste the device list.
Then at least adding some comment for how to set SCRATCH_DEV_POOL would help.
Thanks,
Qu
Quoted 156 lines
Thanks,
EryuQuoted 152 lines
fail like this:
btrfs/175 15s ... - output mismatch (see results//btrfs/175.out.bad)
--- tests/btrfs/175.out 2019-10-22 15:18:14.085632007 +0800
+++ results//btrfs/175.out.bad 2020-01-20 14:53:56.518567916 +0800
@@ -6,3 +6,4 @@
Single on multiple devices
swapon: SCRATCH_MNT/swap: swapon failed: Invalid argument
Single on one device
+ERROR: checking status of : No such file or directory
...
(Run 'diff -u tests/btrfs/175.out results//btrfs/175.out.bad' to see the entire diff)This is caused by the extra quotation mark (and the complexity nature of bash glob).
# SCRATCH_DEV_POOL="/dev/mapper/test-scratch[123]"
# echo ${SCRATCH_DEV_POOL} /dev/mapper/test-scratch1 /dev/mapper/test-scratch2 /dev/mapper/test-scratch3 # echo "${SCRATCH_DEV_POOL}" /dev/mapper/test-scratch[123]To fix the problem, remove the quotation mark out of ${SCRATCH_DEV_POOL} or $SCRATCH_DEV_POOL for all related test cases.
Signed-off-by Qu Wenruo <[email protected]>---
The weirdest thing is, only btrfs/17[56], all other related test cases pass without any problem.Maybe it's time to provide a proper wrapper to do such thing?
--- tests/btrfs/140 | 2 +- tests/btrfs/141 | 2 +- tests/btrfs/142 | 2 +- tests/btrfs/143 | 2 +- tests/btrfs/157 | 2 +- tests/btrfs/158 | 2 +- tests/btrfs/175 | 2 +- tests/btrfs/176 | 6 +++--- 8 files changed, 10 insertions(+), 10 deletions(-)tests/btrfs/140 +1
-1@@ -65,7 +65,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 2tests/btrfs/141 +1
-1@@ -65,7 +65,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 2tests/btrfs/142 +1
-1@@ -66,7 +66,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } start_fail()tests/btrfs/143 +1
-1@@ -73,7 +73,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } SYSFS_BDEV=`_sysfs_dev $SCRATCH_DEV`tests/btrfs/157 +1
-1@@ -70,7 +70,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 4tests/btrfs/158 +1
-1@@ -62,7 +62,7 @@ get_devid() get_device_path() { local devid=$1 - echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}" + echo $SCRATCH_DEV_POOL | $AWK_PROG "{print \$$devid}" } _scratch_dev_pool_get 4tests/btrfs/175 +1
-1@@ -63,7 +63,7 @@ _scratch_mount # Create the swap file, then add the device. That way we know it's all on one # device. _format_swapfile "$SCRATCH_MNT/swap" $(($(get_page_size) * 10)) -scratch_dev2="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $2 }')" +scratch_dev2="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $2 }')" $BTRFS_UTIL_PROG device add -f "$scratch_dev2" "$SCRATCH_MNT" swapon "$SCRATCH_MNT/swap" 2>&1 | _filter_scratch swapoff "$SCRATCH_MNT/swap" > /dev/null 2>&1tests/btrfs/176 +3
-3@@ -39,9 +39,9 @@ _require_scratch_swapfile # We check the filesystem manually because we move devices around. rm -f "${RESULT_DIR}/require_scratch" -scratch_dev1="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $1 }')" -scratch_dev2="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $2 }')" -scratch_dev3="$(echo "${SCRATCH_DEV_POOL}" | awk '{ print $3 }')" +scratch_dev1="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $1 }')" +scratch_dev2="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $2 }')" +scratch_dev3="$(echo ${SCRATCH_DEV_POOL} | awk '{ print $3 }')" echo "Remove device" _scratch_mkfs >> $seqres.full 2>&1Signature · 1 line
2.24.1
Quoted 21 lines · Qu Wenruo
When hard-to-hit bugs happened, we really want every piece of info to help us debugging.
Although we already have
KEEP_DMESGconfig, not everyone is utilizing it, thus when hard-to-hit bugs happened, one could only set it and retry until next hit.This patch will change the behavior by always dumping the dmesg for failed tests, so that developers can always get extra info from any failure.
Signed-off-by Qu Wenruo <[email protected]>--- check | 3 +++ 1 file changed, 3 insertions(+)check +0
-0
check for section in $HOST_OPTIONS_SECTIONS; do · line 845
Quoted 6 lines
check +3
-0@@ -840,6 +840,9 @@ for section in $HOST_OPTIONS_SECTIONS; do # make sure we record the status of the last test we ran. if $err ; then + if [ ! -f $seqres.dmesg ]; then + _dmesg_since_test_start >$seqres.dmesg + fi
So this only saves the dmesg of the last test?
And I don't think this is necessary, even if it saves the dmesgs of all failed tests, this behavior change requires some more diskspace and may fulfill / more easily.
I think if one knows he/she's debugging a hard-to-hit bug, set KEEP_DMESG. Or again, make "save dmesg of every failed test" a tunable behavior.
Thanks,
Eryu
Quoted 6 lines
check +0
-0bad="$bad $seqnum" n_bad=`expr $n_bad + 1` tc_status="fail"Signature · 1 line
2.24.1
Quoted 31 lines · Eryu Guan
Quoted 21 lines · Qu Wenruo
When hard-to-hit bugs happened, we really want every piece of info to help us debugging.
Although we already have
KEEP_DMESGconfig, not everyone is utilizing it, thus when hard-to-hit bugs happened, one could only set it and retry until next hit.This patch will change the behavior by always dumping the dmesg for failed tests, so that developers can always get extra info from any failure.
Signed-off-by Qu Wenruo <[email protected]>--- check | 3 +++ 1 file changed, 3 insertions(+)check +0
-0check for section in $HOST_OPTIONS_SECTIONS; do · line 845
Quoted 6 lines
check +3
-0@@ -840,6 +840,9 @@ for section in $HOST_OPTIONS_SECTIONS; do # make sure we record the status of the last test we ran. if $err ; then + if [ ! -f $seqres.dmesg ]; then + _dmesg_since_test_start >$seqres.dmesg + fiSo this only saves the dmesg of the last test?
Quoted 7 lines
And I don't think this is necessary, even if it saves the dmesgs of all failed tests, this behavior change requires some more diskspace and may fulfill / more easily.
I think if one knows he/she's debugging a hard-to-hit bug, set
KEEP_DMESG. Or again, make "save dmesg of every failed test" a tunable behavior.
Makes sense.
As I just went the same solution.
Thanks,
Qu
Quoted 10 lines
Thanks,
EryuQuoted 6 lines
check +0
-0bad="$bad $seqnum" n_bad=`expr $n_bad + 1` tc_status="fail"Signature · 1 line
2.24.1