selftests: ftrace: ftracetest improvements
Masami Hiramatsu
maintainerv4 cover selftests: ftrace: ftracetest improvements
Hello,
Here is v4 of ftracetest improvements, including test return code change and immediate logging features.
This version just fixes a bug in [4/5] so that remove unneeded backslash.
Thank you,
---
Masami Hiramatsu (5):
selftests: ftrace: Do not failure if there is unsupported tests
selftests: ftrace: Add --fail-unsupported option
selftests: ftrace: Add more verbosity for immediate log
selftests: ftrace: Output only to console with "--logdir -"
selftests: ftrace: Check given string is not zero-length
tools/testing/selftests/ftrace/ftracetest | 51 +++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 14 deletions(-)
--
Masami Hiramatsu (Linaro) <[email protected]>
Masami Hiramatsu
maintainerPATCH v4 1/5 selftests: ftrace: Do not failure if there is unsupported tests
Do not return failure exit code (1) for unsupported testcases, since it is expected for stable kernels.
Previously, ftracetest is expected to run only on current release for avoiding regressions. However, nowadays we run it on stable kernels. This means some test cases must return unsupported result. In such case, we should NOT exit ftracetest with error status for unsupported results so that kselftest (upper tests wrapper) shows it passed correctly.
Note that we continue to treat unresolved results as failure, if test writers would like to notice user that the test result should be reviewed, they can use exit_unresolved.
--- tools/testing/selftests/ftrace/ftracetest | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/ftrace/ftracetest +1 -1
@@ -187,7 +187,7 @@ eval_result() { # sigval $UNSUPPORTED) prlog " [UNSUPPORTED]" UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO" - return 1 # this is not a bug, but the result should be reported. + return 0 # this is not a bug. ;; $XFAIL) prlog " [XFAIL]"
Masami Hiramatsu
maintainerPATCH v4 2/5 selftests: ftrace: Add --fail-unsupported option
In reply to Masami HiramatsuAdd --fail-unsupported option to fail the test result if ftracetest gets UNSUPPORTED result. UNSUPPORTED usually happens when the kernel is old (e.g. stable tree) or some kernel feature is disabled.
However, if newer kernel has any bug or regression, it can make test results in UNSUPPORTED too. This option can detect such kernel regression.
--- tools/testing/selftests/ftrace/ftracetest | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
tools/testing/selftests/ftrace/ftracetest +7 -1
@@ -15,6 +15,7 @@ echo " -h|--help Show help message" echo " -k|--keep Keep passed test logs" echo " -v|--verbose Increase verbosity of test messages" echo " -vv Alias of -v -v (Show all results in stdout)" +echo " --fail-unsupported Treat UNSUPPORTED as a failure" echo " -d|--debug Debug mode (trace all shell commands)" echo " -l|--logdir <dir> Save logs on the <dir>" exit $1
@@ -65,6 +66,10 @@ parse_opts() { # opts DEBUG=1 shift 1 ;; + --fail-unsupported) + UNSUPPORTED_RESULT=1 + shift 1 + ;; --logdir|-l) LOG_DIR=$2 shift 2
@@ -108,6 +113,7 @@ LOG_DIR=$TOP_DIR/logs/`date +%Y%m%d-%H%M%S`/ KEEP_LOG=0 DEBUG=0 VERBOSE=0 +UNSUPPORTED_RESULT=0 # Parse command-line options parse_opts $*
@@ -187,7 +193,7 @@ eval_result() { # sigval $UNSUPPORTED) prlog " [UNSUPPORTED]" UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO" - return 0 # this is not a bug. + return $UNSUPPORTED_RESULT # depends on use case ;; $XFAIL) prlog " [XFAIL]"
Masami Hiramatsu
maintainerPATCH v4 3/5 selftests: ftrace: Add more verbosity for immediate log
In reply to Masami HiramatsuAdd 3-level verbosity for showing traced command log on console immediately. Since some test cases can cause kernel pacic if there is a probrem (like regression etc.), we can not know which command caused the problem without traced command log. This verbosity (-vvv) solves that because it shows the log on console immediately. User can get continuous command/error log.
Note that this is a kind of kernel debug mode, if you don't see any kernel related issue, you don't need this verbosity.
--- tools/testing/selftests/ftrace/ftracetest | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
tools/testing/selftests/ftrace/ftracetest +7 -3
@@ -15,6 +15,7 @@ echo " -h|--help Show help message" echo " -k|--keep Keep passed test logs" echo " -v|--verbose Increase verbosity of test messages" echo " -vv Alias of -v -v (Show all results in stdout)" +echo " -vvv Alias of -v -v -v (Show all commands immediately)" echo " --fail-unsupported Treat UNSUPPORTED as a failure" echo " -d|--debug Debug mode (trace all shell commands)" echo " -l|--logdir <dir> Save logs on the <dir>"
@@ -57,9 +58,10 @@ parse_opts() { # opts KEEP_LOG=1 shift 1 ;; - --verbose|-v|-vv) + --verbose|-v|-vv|-vvv) VERBOSE=$((VERBOSE + 1)) [ $1 = '-vv' ] && VERBOSE=$((VERBOSE + 1)) + [ $1 = '-vvv' ] && VERBOSE=$((VERBOSE + 2)) shift 1 ;; --debug|-d)
@@ -258,7 +260,9 @@ run_test() { # testfile testcase $1 echo "execute$INSTANCE: "$1 > $testlog SIG_RESULT=0 - if [ $VERBOSE -ge 2 ]; then + if [ $VERBOSE -ge 3 ]; then + __run_test $1 | tee -a $testlog 2>&1 + elif [ $VERBOSE -eq 2 ]; then __run_test $1 2>> $testlog | tee -a $testlog else __run_test $1 >> $testlog 2>&1
@@ -268,7 +272,7 @@ run_test() { # testfile # Remove test log if the test was done as it was expected. [ $KEEP_LOG -eq 0 ] && rm $testlog else - [ $VERBOSE -ge 1 ] && catlog $testlog + [ $VERBOSE -eq 1 -o $VERBOSE -eq 2 ] && catlog $testlog TOTAL_RESULT=1 fi rm -rf $TMPDIR
Masami Hiramatsu
maintainerPATCH v4 4/5 selftests: ftrace: Output only to console with "--logdir -"
In reply to Masami HiramatsuOutput logs only to console if "-" is given to --logdir option. In this case, ftracetest doesn't record any log on the disk, and all logs immediately shown (including all command logs.) Since there is no "tee" in the middle of command and console, it outputs the log really soon.
This option is useful only when the console is logged.
---
Changes in v4:
- Fix to remove unneeded backslash (Thanks Stafford!)
--- tools/testing/selftests/ftrace/ftracetest | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-)
tools/testing/selftests/ftrace/ftracetest +21 -8
@@ -19,6 +19,7 @@ echo " -vvv Alias of -v -v -v (Show all commands immediately)" echo " --fail-unsupported Treat UNSUPPORTED as a failure" echo " -d|--debug Debug mode (trace all shell commands)" echo " -l|--logdir <dir> Save logs on the <dir>" +echo " If <dir> is -, all logs output in console only" exit $1 }
@@ -127,14 +128,20 @@ if [ -z "$TRACING_DIR" -o ! -d "$TRACING_DIR" ]; then fi # Preparing logs -LOG_FILE=$LOG_DIR/ftracetest.log -mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR" -date > $LOG_FILE +if [ "x$LOG_DIR" = "x-" ]; then + LOG_FILE= + date +else + LOG_FILE=$LOG_DIR/ftracetest.log + mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR" + date > $LOG_FILE +fi + prlog() { # messages - echo "$@" | tee -a $LOG_FILE + [ -z "$LOG_FILE" ] && echo "$@" || echo "$@" | tee -a $LOG_FILE } catlog() { #file - cat $1 | tee -a $LOG_FILE + [ -z "$LOG_FILE" ] && cat $1 || cat $1 | tee -a $LOG_FILE } prlog "=== Ftrace unit tests ==="
@@ -255,12 +262,18 @@ __run_test() { # testfile # Run one test case run_test() { # testfile local testname=`basename $1` - local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + if [ ! -z "$LOG_FILE" ] ; then + local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + else + local testlog=/proc/self/fd/1 + fi export TMPDIR=`mktemp -d /tmp/ftracetest-dir.XXXXXX` testcase $1 echo "execute$INSTANCE: "$1 > $testlog SIG_RESULT=0 - if [ $VERBOSE -ge 3 ]; then + if [ -z "$LOG_FILE" ]; then + __run_test $1 2>&1 + elif [ $VERBOSE -ge 3 ]; then __run_test $1 | tee -a $testlog 2>&1 elif [ $VERBOSE -eq 2 ]; then __run_test $1 2>> $testlog | tee -a $testlog
@@ -270,7 +283,7 @@ run_test() { # testfile eval_result $SIG_RESULT if [ $? -eq 0 ]; then # Remove test log if the test was done as it was expected. - [ $KEEP_LOG -eq 0 ] && rm $testlog + [ $KEEP_LOG -eq 0 -a ! -z "$LOG_FILE" ] && rm $testlog else [ $VERBOSE -eq 1 -o $VERBOSE -eq 2 ] && catlog $testlog TOTAL_RESULT=1
Masami Hiramatsu
maintainerPATCH v4 5/5 selftests: ftrace: Check given string is not zero-length
In reply to Masami HiramatsuUse [ ! -z "$VAR" ] instead of [ "$VAR" ] to check whether the given string variable is not zero-length since it obviously shows what it means.
--- tools/testing/selftests/ftrace/ftracetest | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/testing/selftests/ftrace/ftracetest +3 -3
@@ -8,7 +8,7 @@ # Released under the terms of the GPL v2. usage() { # errno [message] -[ "$2" ] && echo $2 +[ ! -z "$2" ] && echo $2 echo "Usage: ftracetest [options] [testcase(s)] [testcase-directory(s)]" echo " Options:" echo " -h|--help Show help message"
@@ -50,7 +50,7 @@ parse_opts() { # opts local OPT_TEST_CASES= local OPT_TEST_DIR= - while [ "$1" ]; do + while [ ! -z "$1" ]; do case "$1" in --help|-h) usage 0
@@ -96,7 +96,7 @@ parse_opts() { # opts ;; esac done - if [ "$OPT_TEST_CASES" ]; then + if [ ! -z "$OPT_TEST_CASES" ]; then TEST_CASES=$OPT_TEST_CASES fi }
Shuah Khan
maintainerQuoted 20 lines · Masami Hiramatsu
Hello,
Here is v4 of ftracetest improvements, including test return code change and immediate logging features.
This version just fixes a bug in [4/5] so that remove unneeded backslash.
Thank you,
---
Masami Hiramatsu (5):
selftests: ftrace: Do not failure if there is unsupported tests
selftests: ftrace: Add --fail-unsupported option
selftests: ftrace: Add more verbosity for immediate log
selftests: ftrace: Output only to console with "--logdir -"
selftests: ftrace: Check given string is not zero-length
Quoted 7 lines
tools/testing/selftests/ftrace/ftracetest | 51 +++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 14 deletions(-)--
Masami Hiramatsu (Linaro) <[email protected]>
Hi Steve,
Okay to pull these in for 4.14-rc1? Sorry for the delay in responding.
thanks,
-- Shuah
Quoted 9 lines · On Mon, 24 Jul 2017 09:39:25 -0600 Shuah Khan <[email protected]> wrote:
Quoted 7 lines
Masami Hiramatsu (5):
selftests: ftrace: Do not failure if there is unsupported tests
selftests: ftrace: Add --fail-unsupported option
selftests: ftrace: Add more verbosity for immediate log
selftests: ftrace: Output only to console with "--logdir -"
selftests: ftrace: Check given string is not zero-length
Quoted 4 lines
Hi Steve,
Okay to pull these in for 4.14-rc1? Sorry for the delay in responding.
I'm currently traveling, and not really able to test it. But a quick look at the changes look fine to me.
for the series. And yes please pull it in (if you haven't done so already).
Thanks!
-- Steve
Shuah Khan
maintainerQuoted 21 lines · Steven Rostedt
Quoted 9 lines · On Mon, 24 Jul 2017 09:39:25 -0600 Shuah Khan <[email protected]> wrote:
Quoted 7 lines
Masami Hiramatsu (5):
selftests: ftrace: Do not failure if there is unsupported tests
selftests: ftrace: Add --fail-unsupported option
selftests: ftrace: Add more verbosity for immediate log
selftests: ftrace: Output only to console with "--logdir -"
selftests: ftrace: Check given string is not zero-lengthQuoted 4 lines
Hi Steve,
Okay to pull these in for 4.14-rc1? Sorry for the delay in responding.
I'm currently traveling, and not really able to test it. But a quick look at the changes look fine to me.
Quoted 10 lines
for the series. And yes please pull it in (if you haven't done so already).
Thanks!
-- Steve
Hi Steve/Masami,
Applied the series to linux-kselftest next for 4.14-rc1
thanks,
-- Shuah
Masami Hiramatsu
maintainerQuoted 29 lines · On Wed, 26 Jul 2017 15:45:52 -0600 Shuah Khan <[email protected]> wrote:
Quoted 28 lines · Steven Rostedt
Quoted 9 lines · On Mon, 24 Jul 2017 09:39:25 -0600 Shuah Khan <[email protected]> wrote:
Quoted 7 lines
Masami Hiramatsu (5):
selftests: ftrace: Do not failure if there is unsupported tests
selftests: ftrace: Add --fail-unsupported option
selftests: ftrace: Add more verbosity for immediate log
selftests: ftrace: Output only to console with "--logdir -"
selftests: ftrace: Check given string is not zero-lengthQuoted 4 lines
Hi Steve,
Okay to pull these in for 4.14-rc1? Sorry for the delay in responding.
I'm currently traveling, and not really able to test it. But a quick look at the changes look fine to me.
for the series. And yes please pull it in (if you haven't done so already).
Thanks!
Quoted 7 lines
Quoted 3 lines
-- Steve
Hi Steve/Masami,
Applied the series to linux-kselftest next for 4.14-rc1
Thank you Shuah!
Signature · 1 line
Masami Hiramatsu <[email protected]>