fstests: reduce runtime of check -n
PATCH 1/1 fstests: reduce runtime of check -n
kvm-xfstests invokes check -n twice to pre-process and generate the tests-to-run list, which is then being passed as a long list of tests for invkoing check in the command line.
check invokes dirname, basename and sed several times per test just for doing basic string prefix/suffix trimming.
Use bash string pattern matching instead which is much faster.
Note that the following pattern matching expression change:
< test_dir=${test_dir#$SRC_DIR/*}
Quoted 1 line
t=${t#$SRC_DIR/}
does not change the meaning of the expression, because the shortest match of "$SRC_DIR/*" that is being trimmed is "$SRC_DIR/" and removing the tests/ prefix is what this code intended to do.
With check -n, there is no need to cleanup the results dir, but check -n is doing that for every single listed test. Move the cleanup of results dir to before actually running the test.
These improvements to check pre-test code cut down several minutes from the time until tests actually start to run with kvm-xfstests.
---
Zorro,
Just to clarify, this change is not expected to change behavior - only to improve runtime of check -n and the things that happen before the first test is invoked.
Thanks,
Amir.
check | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-)
check +9 -13
@@ -399,9 +399,9 @@ if $have_test_arg; then *) # Expand test pattern (e.g. xfs/???, *fs/001) list=$(cd $SRC_DIR; echo $1) for t in $list; do - test_dir=`dirname $t` - test_dir=${test_dir#$SRC_DIR/*} - test_name=`basename $t` + t=${t#$SRC_DIR/} + test_dir=${t%/*} + test_name=${t#*/} group_file=$SRC_DIR/$test_dir/group.list if grep -Eq "^$test_name" $group_file; then
@@ -849,18 +849,14 @@ function run_section() # the filename for the test and the name output are different. # we don't include the tests/ directory in the name output. - export seqnum=`echo $seq | sed -e "s;$SRC_DIR/;;"` - - # Similarly, the result directory needs to replace the tests/ - # part of the test location. - group=`dirname $seq` + export seqnum=${seq#$SRC_DIR/} + group=${seqnum%/*} if $OPTIONS_HAVE_SECTIONS; then - export RESULT_DIR=`echo $group | sed -e "s;$SRC_DIR;${RESULT_BASE}/$section;"` REPORT_DIR="$RESULT_BASE/$section" else - export RESULT_DIR=`echo $group | sed -e "s;$SRC_DIR;$RESULT_BASE;"` REPORT_DIR="$RESULT_BASE" fi + export RESULT_DIR="$REPORT_DIR/$group" seqres="$REPORT_DIR/$seqnum" # Generate the entire section report with whatever test results
@@ -872,9 +868,6 @@ function run_section() "" &> /dev/null fi - mkdir -p $RESULT_DIR - rm -f ${RESULT_DIR}/require_scratch* - rm -f ${RESULT_DIR}/require_test* echo -n "$seqnum" if $showme; then
@@ -899,6 +892,9 @@ function run_section() fi # really going to try and run this one + mkdir -p $RESULT_DIR + rm -f ${RESULT_DIR}/require_scratch* + rm -f ${RESULT_DIR}/require_test* rm -f $seqres.out.bad $seqres.hints # check if we really should run it
Signature · 1 line
2.34.1
Quoted 22 lines · Amir Goldstein
kvm-xfstests invokes check -n twice to pre-process and generate the tests-to-run list, which is then being passed as a long list of tests for invkoing check in the command line.
check invokes dirname, basename and sed several times per test just for doing basic string prefix/suffix trimming.
Use bash string pattern matching instead which is much faster.Note that the following pattern matching expression change:
< test_dir=${test_dir#$SRC_DIR/*}Quoted 1 line
t=${t#$SRC_DIR/}
does not change the meaning of the expression, because the shortest match of "$SRC_DIR/*" that is being trimmed is "$SRC_DIR/" and removing the tests/ prefix is what this code intended to do.
With check -n, there is no need to cleanup the results dir, but check -n is doing that for every single listed test. Move the cleanup of results dir to before actually running the test.
These improvements to check pre-test code cut down several minutes from the time until tests actually start to run with kvm-xfstests.
Quoted 8 lines
Signed-off-by Amir Goldstein <[email protected]>---
Zorro,
Just to clarify, this change is not expected to change behavior - only to improve runtime of check -n and the things that happen before the first test is invoked.
Hi Amir,
Sure, it's good to me to improve the performace of 'check -n'. But as this patch changes the common logic of check file, I need to make sure it won't bring in regression at first by some testing. If it works fine, I'll ack and merge it.
Thanks,
Zorro
Quoted 69 lines
Thanks,
Amir.check | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-)check +9
-13@@ -399,9 +399,9 @@ if $have_test_arg; then *) # Expand test pattern (e.g. xfs/???, *fs/001) list=$(cd $SRC_DIR; echo $1) for t in $list; do - test_dir=`dirname $t` - test_dir=${test_dir#$SRC_DIR/*} - test_name=`basename $t` + t=${t#$SRC_DIR/} + test_dir=${t%/*} + test_name=${t#*/} group_file=$SRC_DIR/$test_dir/group.list if grep -Eq "^$test_name" $group_file; then@@ -849,18 +849,14 @@ function run_section() # the filename for the test and the name output are different. # we don't include the tests/ directory in the name output. - export seqnum=`echo $seq | sed -e "s;$SRC_DIR/;;"` - - # Similarly, the result directory needs to replace the tests/ - # part of the test location. - group=`dirname $seq` + export seqnum=${seq#$SRC_DIR/} + group=${seqnum%/*} if $OPTIONS_HAVE_SECTIONS; then - export RESULT_DIR=`echo $group | sed -e "s;$SRC_DIR;${RESULT_BASE}/$section;"` REPORT_DIR="$RESULT_BASE/$section" else - export RESULT_DIR=`echo $group | sed -e "s;$SRC_DIR;$RESULT_BASE;"` REPORT_DIR="$RESULT_BASE" fi + export RESULT_DIR="$REPORT_DIR/$group" seqres="$REPORT_DIR/$seqnum" # Generate the entire section report with whatever test results@@ -872,9 +868,6 @@ function run_section() "" &> /dev/null fi - mkdir -p $RESULT_DIR - rm -f ${RESULT_DIR}/require_scratch* - rm -f ${RESULT_DIR}/require_test* echo -n "$seqnum" if $showme; then@@ -899,6 +892,9 @@ function run_section() fi # really going to try and run this one + mkdir -p $RESULT_DIR + rm -f ${RESULT_DIR}/require_scratch* + rm -f ${RESULT_DIR}/require_test* rm -f $seqres.out.bad $seqres.hints # check if we really should run itSignature · 1 line
2.34.1
Quoted 42 lines · Amir Goldstein
kvm-xfstests invokes check -n twice to pre-process and generate the tests-to-run list, which is then being passed as a long list of tests for invkoing check in the command line.
check invokes dirname, basename and sed several times per test just for doing basic string prefix/suffix trimming.
Use bash string pattern matching instead which is much faster.Note that the following pattern matching expression change:
< test_dir=${test_dir#$SRC_DIR/*}Quoted 1 line
t=${t#$SRC_DIR/}
does not change the meaning of the expression, because the shortest match of "$SRC_DIR/*" that is being trimmed is "$SRC_DIR/" and removing the tests/ prefix is what this code intended to do.
With check -n, there is no need to cleanup the results dir, but check -n is doing that for every single listed test. Move the cleanup of results dir to before actually running the test.
These improvements to check pre-test code cut down several minutes from the time until tests actually start to run with kvm-xfstests.
Signed-off-by Amir Goldstein <[email protected]>---
Zorro,
Just to clarify, this change is not expected to change behavior - only to improve runtime of check -n and the things that happen before the first test is invoked.
Thanks,
Amir.check | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-)check +0
-0
check if $have_test_arg; then · line 402
Quoted 7 lines
check +1
-3@@ -399,9 +399,9 @@ if $have_test_arg; then *) # Expand test pattern (e.g. xfs/???, *fs/001) list=$(cd $SRC_DIR; echo $1) for t in $list; do - test_dir=`dirname $t` - test_dir=${test_dir#$SRC_DIR/*} - test_name=`basename $t` + t=${t#$SRC_DIR/}
Actually I'm not sure what's this line for. We get $list by "cd $SRC_DIR; echo $1", is there any chance to have "$SRC_DIR" in $t (or $test_dir) ?
check if $have_test_arg; then · line 404
Quoted 2 lines
check +2
-0+ test_dir=${t%/*} + test_name=${t#*/}
This change looks isn't related with the "check -n". Any reason to change that? Does the 'dirname' and 'basename' have poor performance, or some systems miss them?
I prefer the 'dirname' and 'basename' ways, if there's not a specific reason to change that. Due to they can deal with more complicated path name, e.g. "xfs//001". But the "$(cd $SRC_DIR; echo $1)" looks won't generate a path name likes that, so if you have a proper reason to make this change, that's fine.
Quoted 12 lines
check +0
-5group_file=$SRC_DIR/$test_dir/group.list if grep -Eq "^$test_name" $group_file; then@@ -849,18 +849,14 @@ function run_section() # the filename for the test and the name output are different. # we don't include the tests/ directory in the name output. - export seqnum=`echo $seq | sed -e "s;$SRC_DIR/;;"` - - # Similarly, the result directory needs to replace the tests/ - # part of the test location. - group=`dirname $seq`
check function run_section() · line 859
Quoted 10 lines
check +3
-2+ export seqnum=${seq#$SRC_DIR/} + group=${seqnum%/*} if $OPTIONS_HAVE_SECTIONS; then - export RESULT_DIR=`echo $group | sed -e "s;$SRC_DIR;${RESULT_BASE}/$section;"` REPORT_DIR="$RESULT_BASE/$section" else - export RESULT_DIR=`echo $group | sed -e "s;$SRC_DIR;$RESULT_BASE;"` REPORT_DIR="$RESULT_BASE" fi + export RESULT_DIR="$REPORT_DIR/$group"
Hmm, this code logic looks more concise.
Quoted 14 lines
check +0
-3seqres="$REPORT_DIR/$seqnum" # Generate the entire section report with whatever test results@@ -872,9 +868,6 @@ function run_section() "" &> /dev/null fi - mkdir -p $RESULT_DIR - rm -f ${RESULT_DIR}/require_scratch* - rm -f ${RESULT_DIR}/require_test* echo -n "$seqnum" if $showme; then
check function run_section() · line 897
Quoted 6 lines
check +3
-0@@ -899,6 +892,9 @@ function run_section() fi # really going to try and run this one + mkdir -p $RESULT_DIR + rm -f ${RESULT_DIR}/require_scratch* + rm -f ${RESULT_DIR}/require_test*
Nice catch, loop running these lines really takes much time.
Thanks,
Zorro
Quoted 6 lines
check +0
-0rm -f $seqres.out.bad $seqres.hints # check if we really should run itSignature · 1 line
2.34.1
Hi Zorro and Amir,
Quoted 38 lines · Zorro Lang
Quoted 37 lines · Amir Goldstein
kvm-xfstests invokes check -n twice to pre-process and generate the tests-to-run list, which is then being passed as a long list of tests for invkoing check in the command line.
check invokes dirname, basename and sed several times per test just for doing basic string prefix/suffix trimming.
Use bash string pattern matching instead which is much faster.Note that the following pattern matching expression change:
< test_dir=${test_dir#$SRC_DIR/*}Quoted 1 line
t=${t#$SRC_DIR/}
does not change the meaning of the expression, because the shortest match of "$SRC_DIR/*" that is being trimmed is "$SRC_DIR/" and removing the tests/ prefix is what this code intended to do.
With check -n, there is no need to cleanup the results dir, but check -n is doing that for every single listed test. Move the cleanup of results dir to before actually running the test.
These improvements to check pre-test code cut down several minutes from the time until tests actually start to run with kvm-xfstests.
Signed-off-by Amir Goldstein <[email protected]>---
Zorro,
Just to clarify, this change is not expected to change behavior - only to improve runtime of check -n and the things that happen before the first test is invoked.
Thanks,
Amir.check | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-)
Quoted 15 lines
check if $have_test_arg; then · line 402
Quoted 12 lines
check +1
-3@@ -399,9 +399,9 @@ if $have_test_arg; then *) # Expand test pattern (e.g. xfs/???, *fs/001) list=$(cd $SRC_DIR; echo $1) for t in $list; do - test_dir=`dirname $t` - test_dir=${test_dir#$SRC_DIR/*} - test_name=`basename $t` + t=${t#$SRC_DIR/}Actually I'm not sure what's this line for. We get $list by "cd $SRC_DIR; echo $1", is there any chance to have "$SRC_DIR" in $t (or $test_dir) ?
Indeed, this line can be dropped. One minor change in behaviour here is that a test pattern with a wacky path prefix e.g. ../tests/xfs/00? will no longer work. I think breaking such patterns is okay, but it could also be resolved by more carefully extracting the last two path components.
Quoted 11 lines
check if $have_test_arg; then · line 404
Quoted 2 lines
check +2
-0+ test_dir=${t%/*} + test_name=${t#*/}This change looks isn't related with the "check -n". Any reason to change that? Does the 'dirname' and 'basename' have poor performance, or some systems miss them?
I prefer the 'dirname' and 'basename' ways, if there's not a specific reason to change that. Due to they can deal with more complicated path name, e.g. "xfs//001". But the "$(cd $SRC_DIR; echo $1)" looks won't generate a path name likes that, so if you have a proper reason to make this change, that's fine.
The per-iteration dirname and basename subshells add multiple seconds of latency to ./check -n *fs/* in my test env.
Cheers, David
Quoted 59 lines · David Disseldorp
Hi Zorro and Amir,
Quoted 38 lines · Zorro Lang
Quoted 37 lines · Amir Goldstein
kvm-xfstests invokes check -n twice to pre-process and generate the tests-to-run list, which is then being passed as a long list of tests for invkoing check in the command line.
check invokes dirname, basename and sed several times per test just for doing basic string prefix/suffix trimming.
Use bash string pattern matching instead which is much faster.Note that the following pattern matching expression change:
< test_dir=${test_dir#$SRC_DIR/*}Quoted 1 line
t=${t#$SRC_DIR/}
does not change the meaning of the expression, because the shortest match of "$SRC_DIR/*" that is being trimmed is "$SRC_DIR/" and removing the tests/ prefix is what this code intended to do.
With check -n, there is no need to cleanup the results dir, but check -n is doing that for every single listed test. Move the cleanup of results dir to before actually running the test.
These improvements to check pre-test code cut down several minutes from the time until tests actually start to run with kvm-xfstests.
Signed-off-by Amir Goldstein <[email protected]>---
Zorro,
Just to clarify, this change is not expected to change behavior - only to improve runtime of check -n and the things that happen before the first test is invoked.
Thanks,
Amir.check | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-)Quoted 15 lines
check if $have_test_arg; then · line 402
Quoted 12 lines
check +1
-3@@ -399,9 +399,9 @@ if $have_test_arg; then *) # Expand test pattern (e.g. xfs/???, *fs/001) list=$(cd $SRC_DIR; echo $1) for t in $list; do - test_dir=`dirname $t` - test_dir=${test_dir#$SRC_DIR/*} - test_name=`basename $t` + t=${t#$SRC_DIR/}Actually I'm not sure what's this line for. We get $list by "cd $SRC_DIR; echo $1", is there any chance to have "$SRC_DIR" in $t (or $test_dir) ?
Quoted 6 lines
Indeed, this line can be dropped. One minor change in behaviour here is that a test pattern with a wacky path prefix e.g. ../tests/xfs/00? will no longer work. I think breaking such patterns is okay, but it could also be resolved by more carefully extracting the last two path components.
I didn't want to drop it because I didn't want to change legacy behavior. When user's cwd is fstests root directory, it may be natural for some users to use bash auto completion to run the command
./check tests/generic/001
so there may well be users out these that pass test names including the tests/ prefix and check has always trimmed this prefix.
There is something a bit confusing about expanding of bash patterns:
When the user uses this syntax:
./check tests/generic/00?
the shell expands that pattern to a list of 9 arguments passed to ./check tests/generic/001 tests/generic/002 ...
But when the user uses the documented syntax:
./check generic/00?
check gets a single argument and $(cd $SRC_DIR; echo $1) expands this single argument to a list
All the above is existing behavior which my patch should not have changed.
Quoted 3 lines
Quoted 3 lines
check if $have_test_arg; then · line 404
Quoted 2 lines
check +2
-0+ test_dir=${t%/*} + test_name=${t#*/}
You may change the above to:
+ test_dir=${t%%/*}
+ test_name=${t##*/}
I tested and this deals with "xfs//001" and "xfs//00?" correctly (it trimms the longest match to */ prefix or /* suffix instead of the shortest match.
The line below group= could be changed to %%/* as well but it does not matter because $group is used as part of the results path which can live with //.
Unfortunately, trimming tests// from tests//xfs/001 without also trimming xfs/ is less trivial and while I gave a reason why tests/ prefix may be used in the wild, but I can think of no good reason why tests// prefix would be used in the wild.
Quoted 4 lines
Quoted 4 lines
This change looks isn't related with the "check -n". Any reason to change that? Does the 'dirname' and 'basename' have poor performance, or some systems miss them?
Quoted 8 lines
Quoted 4 lines
I prefer the 'dirname' and 'basename' ways, if there's not a specific reason to change that. Due to they can deal with more complicated path name, e.g. "xfs//001". But the "$(cd $SRC_DIR; echo $1)" looks won't generate a path name likes that, so if you have a proper reason to make this change, that's fine.
The per-iteration
dirnameandbasenamesubshells add multiple seconds of latency to./check -n *fs/*in my test env.
Exactly.
Whoever looks at strace of check will observe that it is very noisy with execve of dirname and basename.
The next step would be to use bash dictionary instead of multiple invocations of grep to filter tests by match to -g <group> or -g xfs/<group>, but that is left for another day...
Thanks,
Amir.
Quoted 1 line · Amir Goldstein
On Wed, Jun 21, 2023 at 12:24 PM David Disseldorp <[email protected]> wrote:
...
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
Actually I'm not sure what's this line for. We get $list by "cd $SRC_DIR; echo $1", is there any chance to have "$SRC_DIR" in $t (or $test_dir) ?
Indeed, this line can be dropped. One minor change in behaviour here is that a test pattern with a wacky path prefix e.g. ../tests/xfs/00? will no longer work. I think breaking such patterns is okay, but it could also be resolved by more carefully extracting the last two path components.
Quoted 7 lines
I didn't want to drop it because I didn't want to change legacy behavior. When user's cwd is fstests root directory, it may be natural for some users to use bash auto completion to run the command
./check tests/generic/001so there may well be users out these that pass test names including the tests/ prefix and check has always trimmed this prefix.
Ah, yes, understood.
Quoted 9 lines
There is something a bit confusing about expanding of bash patterns:
When the user uses this syntax:
./check tests/generic/00?the shell expands that pattern to a list of 9 arguments passed to ./check tests/generic/001 tests/generic/002 ...
Quoted 7 lines
But when the user uses the documented syntax:
./check generic/00?
check gets a single argument and $(cd $SRC_DIR; echo $1) expands this single argument to a listAll the above is existing behavior which my patch should not have changed.
The example I gave was "../tests/xfs/00?", which does get broken by this change. It's relative to SRC_DIR and independent of check parameter expansion. As mentioned, I think it's obscure enough usage that this breakage shouldn't be worth bothering about.
Cheers, David
Quoted 39 lines · David Disseldorp
Quoted 1 line · Amir Goldstein
On Wed, Jun 21, 2023 at 12:24 PM David Disseldorp <[email protected]> wrote:
...
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
Actually I'm not sure what's this line for. We get $list by "cd $SRC_DIR; echo $1", is there any chance to have "$SRC_DIR" in $t (or $test_dir) ?
Indeed, this line can be dropped. One minor change in behaviour here is that a test pattern with a wacky path prefix e.g. ../tests/xfs/00? will no longer work. I think breaking such patterns is okay, but it could also be resolved by more carefully extracting the last two path components.
Quoted 7 lines
I didn't want to drop it because I didn't want to change legacy behavior. When user's cwd is fstests root directory, it may be natural for some users to use bash auto completion to run the command
./check tests/generic/001so there may well be users out these that pass test names including the tests/ prefix and check has always trimmed this prefix.
Ah, yes, understood.
Quoted 14 lines
There is something a bit confusing about expanding of bash patterns:
When the user uses this syntax:
./check tests/generic/00?the shell expands that pattern to a list of 9 arguments passed to ./check tests/generic/001 tests/generic/002 ...
But when the user uses the documented syntax:
./check generic/00?
check gets a single argument and $(cd $SRC_DIR; echo $1) expands this single argument to a list
Quoted 8 lines
Quoted 2 lines
All the above is existing behavior which my patch should not have changed.
The example I gave was "../tests/xfs/00?", which does get broken by this change. It's relative to
SRC_DIRand independent of check parameter expansion. As mentioned, I think it's obscure enough usage that this breakage shouldn't be worth bothering about.
I agree.
Users of tests/xfs/* may be out there (due to bash auto complete) Any other prefix, including ../tests/ is far less likely to exist in the wild (famous last words)
Thanks,
Amir.
Quoted 60 lines · Amir Goldstein
Quoted 59 lines · David Disseldorp
Hi Zorro and Amir,
Quoted 53 lines · Zorro Lang
Quoted 42 lines · Amir Goldstein
kvm-xfstests invokes check -n twice to pre-process and generate the tests-to-run list, which is then being passed as a long list of tests for invkoing check in the command line.
check invokes dirname, basename and sed several times per test just for doing basic string prefix/suffix trimming.
Use bash string pattern matching instead which is much faster.Note that the following pattern matching expression change:
< test_dir=${test_dir#$SRC_DIR/*}Quoted 1 line
t=${t#$SRC_DIR/}
does not change the meaning of the expression, because the shortest match of "$SRC_DIR/*" that is being trimmed is "$SRC_DIR/" and removing the tests/ prefix is what this code intended to do.
With check -n, there is no need to cleanup the results dir, but check -n is doing that for every single listed test. Move the cleanup of results dir to before actually running the test.
These improvements to check pre-test code cut down several minutes from the time until tests actually start to run with kvm-xfstests.
Signed-off-by Amir Goldstein <[email protected]>---
Zorro,
Just to clarify, this change is not expected to change behavior - only to improve runtime of check -n and the things that happen before the first test is invoked.
Thanks,
Amir.check | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-)check +0
-0check if $have_test_arg; then · line 402
Quoted 7 lines
check +1
-3@@ -399,9 +399,9 @@ if $have_test_arg; then *) # Expand test pattern (e.g. xfs/???, *fs/001) list=$(cd $SRC_DIR; echo $1) for t in $list; do - test_dir=`dirname $t` - test_dir=${test_dir#$SRC_DIR/*} - test_name=`basename $t` + t=${t#$SRC_DIR/}Actually I'm not sure what's this line for. We get $list by "cd $SRC_DIR; echo $1", is there any chance to have "$SRC_DIR" in $t (or $test_dir) ?
Quoted 6 lines
Quoted 6 lines
Indeed, this line can be dropped. One minor change in behaviour here is that a test pattern with a wacky path prefix e.g. ../tests/xfs/00? will no longer work. I think breaking such patterns is okay, but it could also be resolved by more carefully extracting the last two path components.
Hi Amir and David,
Sorry for the late reply, I'm on a holiday of Dragon Boat Festival recently, will be back tomorrow :)
Quoted 2 lines
I didn't want to drop it because I didn't want to change legacy behavior.
Sure, I didn't want to remove this line in this patch, due to it's not match the target of this patch. Just speak out to check with you and others :)
As we agree this line is useless, we can deal with it in later patch which improves ./check.
Quoted 4 lines
When user's cwd is fstests root directory, it may be natural for some users to use bash auto completion to run the command ./check tests/generic/001
Quoted 8 lines
so there may well be users out these that pass test names including the tests/ prefix and check has always trimmed this prefix.
There is something a bit confusing about expanding of bash patterns:
When the user uses this syntax:
./check tests/generic/00?
I think the "tests/" prefix is not necessary to support :)
Quoted 17 lines
the shell expands that pattern to a list of 9 arguments passed to ./check tests/generic/001 tests/generic/002 ...
But when the user uses the documented syntax:
./check generic/00?
check gets a single argument and $(cd $SRC_DIR; echo $1) expands this single argument to a listAll the above is existing behavior which my patch should not have changed.
Quoted 3 lines
Quoted 3 lines
check if $have_test_arg; then · line 404
Quoted 2 lines
check +2
-0+ test_dir=${t%/*} + test_name=${t#*/}You may change the above to:
Review
Quoted 2 lines
(quoted diff) +2
-0+ test_dir=${t%%/*} + test_name=${t##*/}
OK, this makes more sense to me. I'll give this a test, and merge this change if no regression. Or you'd like to send a V2 to change more.
Quoted 5 lines
I tested and this deals with "xfs//001" and "xfs//00?" correctly (it trimms the longest match to */ prefix or /* suffix instead of the shortest match.
Quoted 20 lines
The line below group= could be changed to %%/* as well but it does not matter because $group is used as part of the results path which can live with //.
Unfortunately, trimming tests// from tests//xfs/001 without also trimming xfs/ is less trivial and while I gave a reason why tests/ prefix may be used in the wild, but I can think of no good reason why tests// prefix would be used in the wild.
Quoted 11 lines
Quoted 8 lines
This change looks isn't related with the "check -n". Any reason to change that? Does the 'dirname' and 'basename' have poor performance, or some systems miss them?
I prefer the 'dirname' and 'basename' ways, if there's not a specific reason to change that. Due to they can deal with more complicated path name, e.g. "xfs//001". But the "$(cd $SRC_DIR; echo $1)" looks won't generate a path name likes that, so if you have a proper reason to make this change, that's fine.
The per-iteration
dirnameandbasenamesubshells add multiple seconds of latency to./check -n *fs/*in my test env.
Thanks for this feedback. I understand now.
Thanks,
Zorro
Quoted 13 lines
Quoted 1 line
Exactly.
Whoever looks at strace of check will observe that it is very noisy with execve of dirname and basename.The next step would be to use bash dictionary instead of multiple invocations of grep to filter tests by match to -g <group> or -g xfs/<group>, but that is left for another day...
Thanks,
Amir.
Quoted 91 lines · Zorro Lang
Quoted 60 lines · Amir Goldstein
Quoted 59 lines · David Disseldorp
Hi Zorro and Amir,
Quoted 53 lines · Zorro Lang
Quoted 42 lines · Amir Goldstein
kvm-xfstests invokes check -n twice to pre-process and generate the tests-to-run list, which is then being passed as a long list of tests for invkoing check in the command line.
check invokes dirname, basename and sed several times per test just for doing basic string prefix/suffix trimming.
Use bash string pattern matching instead which is much faster.Note that the following pattern matching expression change:
< test_dir=${test_dir#$SRC_DIR/*}Quoted 1 line
t=${t#$SRC_DIR/}
does not change the meaning of the expression, because the shortest match of "$SRC_DIR/*" that is being trimmed is "$SRC_DIR/" and removing the tests/ prefix is what this code intended to do.
With check -n, there is no need to cleanup the results dir, but check -n is doing that for every single listed test. Move the cleanup of results dir to before actually running the test.
These improvements to check pre-test code cut down several minutes from the time until tests actually start to run with kvm-xfstests.
Signed-off-by Amir Goldstein <[email protected]>---
Zorro,
Just to clarify, this change is not expected to change behavior - only to improve runtime of check -n and the things that happen before the first test is invoked.
Thanks,
Amir.check | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-)check +0
-0check if $have_test_arg; then · line 402
Quoted 7 lines
check +1
-3@@ -399,9 +399,9 @@ if $have_test_arg; then *) # Expand test pattern (e.g. xfs/???, *fs/001) list=$(cd $SRC_DIR; echo $1) for t in $list; do - test_dir=`dirname $t` - test_dir=${test_dir#$SRC_DIR/*} - test_name=`basename $t` + t=${t#$SRC_DIR/}Actually I'm not sure what's this line for. We get $list by "cd $SRC_DIR; echo $1", is there any chance to have "$SRC_DIR" in $t (or $test_dir) ?
Quoted 6 lines
Quoted 6 lines
Indeed, this line can be dropped. One minor change in behaviour here is that a test pattern with a wacky path prefix e.g. ../tests/xfs/00? will no longer work. I think breaking such patterns is okay, but it could also be resolved by more carefully extracting the last two path components.
Hi Amir and David,
Sorry for the late reply, I'm on a holiday of Dragon Boat Festival recently, will be back tomorrow :)
Quoted 2 lines
I didn't want to drop it because I didn't want to change legacy behavior.
Sure, I didn't want to remove this line in this patch, due to it's not match the target of this patch. Just speak out to check with you and others :)
As we agree this line is useless, we can deal with it in later patch which improves ./check.
Quoted 7 lines
When user's cwd is fstests root directory, it may be natural for some users to use bash auto completion to run the command ./check tests/generic/001
so there may well be users out these that pass test names including the tests/ prefix and check has always trimmed this prefix.
Quoted 8 lines
Quoted 5 lines
There is something a bit confusing about expanding of bash patterns:
When the user uses this syntax:
./check tests/generic/00?I think the "tests/" prefix is not necessary to support :)
Fine by me.
Quoted 16 lines
Quoted 16 lines
the shell expands that pattern to a list of 9 arguments passed to ./check tests/generic/001 tests/generic/002 ...
But when the user uses the documented syntax:
./check generic/00?
check gets a single argument and $(cd $SRC_DIR; echo $1) expands this single argument to a listAll the above is existing behavior which my patch should not have changed.
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
check +2
-0+ test_dir=${t%/*} + test_name=${t#*/}
Quoted 7 lines
Review
Quoted 3 lines
You may change the above to:
(quoted diff) +2
-0+ test_dir=${t%%/*} + test_name=${t##*/}OK, this makes more sense to me. I'll give this a test, and merge this change if no regression. Or you'd like to send a V2 to change more.
I have no plans to send v2 with more changes.
Please make this change on merge.
Thanks,
Amir.