selftests: ftrace: ftracetest improvements
Masami Hiramatsu
maintainercover selftests: ftrace: ftracetest improvements
Hello,
Here is v2 of ftracetest improvements, including test return code change and immediate logging features.
The first version is here.
https://patchwork.kernel.org/patch/9821943/
https://patchwork.kernel.org/patch/9821945/
This version adds 2 patches according discussions on previous version. [2/4] adds --fail-unsupported option which makes UNSUPPORTED result failure. [4/4] adds "--logdir -" which logs all results in console but no file.
Changes in v2:
- [2/4]: (new) adds --fail-unsupported option
- [3/4]: Fix not to show failure log twice
- [4/4]: (new) adds --logdir "-" option so that all log goes to console directly.
Thank you,
---
Masami Hiramatsu (4):
selftests: ftrace: Do not failure if there is unsupported tests
selftests: ftrace: Add --fail-unsupported option
selftests: ftrace: Add more verbosity for immediate log
selftests: ftrace: Output only to console with "--logdir -"
tools/testing/selftests/ftrace/ftracetest | 45 ++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 11 deletions(-)
--
Masami Hiramatsu (Linaro) <[email protected]>
Masami Hiramatsu
maintainerPATCH 1/4 selftests: ftrace: Do not failure if there is unsupported tests
Do not return failure exit code (1) for unsupported testcases, since it is expected for stable kernels.
Previously, ftracetest is expected to run only on current release for avoiding regressions. However, nowadays we run it on stable kernels. This means some test cases must return unsupported result. In such case, we should NOT exit ftracetest with error status for unsupported results so that kselftest (upper tests wrapper) shows it passed correctly.
Note that we continue to treat unresolved results as failure, if test writers would like to notice user that the test result should be reviewed, they can use exit_unresolved.
--- tools/testing/selftests/ftrace/ftracetest | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/ftrace/ftracetest +1 -1
@@ -187,7 +187,7 @@ eval_result() { # sigval $UNSUPPORTED) prlog " [UNSUPPORTED]" UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO" - return 1 # this is not a bug, but the result should be reported. + return 0 # this is not a bug. ;; $XFAIL) prlog " [XFAIL]"
Masami Hiramatsu
maintainerPATCH 2/4 selftests: ftrace: Add --fail-unsupported option
In reply to Masami HiramatsuAdd --fail-unsupported option to fail the test result if ftracetest gets UNSUPPORTED result. UNSUPPORTED usually happens when the kernel is old (e.g. stable tree) or some kernel feature is disabled.
However, if newer kernel has any bug or regression, it can make test results in UNSUPPORTED too. This option can detect such kernel regression.
--- tools/testing/selftests/ftrace/ftracetest | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
tools/testing/selftests/ftrace/ftracetest +7 -1
@@ -15,6 +15,7 @@ echo " -h|--help Show help message" echo " -k|--keep Keep passed test logs" echo " -v|--verbose Increase verbosity of test messages" echo " -vv Alias of -v -v (Show all results in stdout)" +echo " --fail-unsupported Treat UNSUPPORTED as a failure" echo " -d|--debug Debug mode (trace all shell commands)" echo " -l|--logdir <dir> Save logs on the <dir>" exit $1
@@ -65,6 +66,10 @@ parse_opts() { # opts DEBUG=1 shift 1 ;; + --fail-unsupported) + UNSUPPORTED_RESULT=1 + shift 1 + ;; --logdir|-l) LOG_DIR=$2 shift 2
@@ -108,6 +113,7 @@ LOG_DIR=$TOP_DIR/logs/`date +%Y%m%d-%H%M%S`/ KEEP_LOG=0 DEBUG=0 VERBOSE=0 +UNSUPPORTED_RESULT=0 # Parse command-line options parse_opts $*
@@ -187,7 +193,7 @@ eval_result() { # sigval $UNSUPPORTED) prlog " [UNSUPPORTED]" UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO" - return 0 # this is not a bug. + return $UNSUPPORTED_RESULT # depends on use case ;; $XFAIL) prlog " [XFAIL]"
Masami Hiramatsu
maintainerPATCH 3/4 selftests: ftrace: Add more verbosity for immediate log
In reply to Masami HiramatsuAdd 3-level verbosity for showing traced command log on console immediately. Since some test cases can cause kernel pacic if there is a probrem (like regression etc.), we can not know which command caused the problem without traced command log. This verbosity (-vvv) solves that because it shows the log on console immediately. User can get continuous command/error log.
Note that this is a kind of kernel debug mode, if you don't see any kernel related issue, you don't need this verbosity.
---
Changes in v2:
- Do not show failure log on console again.
--- tools/testing/selftests/ftrace/ftracetest | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
tools/testing/selftests/ftrace/ftracetest +7 -3
@@ -15,6 +15,7 @@ echo " -h|--help Show help message" echo " -k|--keep Keep passed test logs" echo " -v|--verbose Increase verbosity of test messages" echo " -vv Alias of -v -v (Show all results in stdout)" +echo " -vvv Alias of -v -v -v (Show all commands immediately)" echo " --fail-unsupported Treat UNSUPPORTED as a failure" echo " -d|--debug Debug mode (trace all shell commands)" echo " -l|--logdir <dir> Save logs on the <dir>"
@@ -57,9 +58,10 @@ parse_opts() { # opts KEEP_LOG=1 shift 1 ;; - --verbose|-v|-vv) + --verbose|-v|-vv|-vvv) VERBOSE=$((VERBOSE + 1)) [ $1 = '-vv' ] && VERBOSE=$((VERBOSE + 1)) + [ $1 = '-vvv' ] && VERBOSE=$((VERBOSE + 2)) shift 1 ;; --debug|-d)
@@ -258,7 +260,9 @@ run_test() { # testfile testcase $1 echo "execute$INSTANCE: "$1 > $testlog SIG_RESULT=0 - if [ $VERBOSE -ge 2 ]; then + if [ $VERBOSE -ge 3 ]; then + __run_test $1 | tee -a $testlog 2>&1 + elif [ $VERBOSE -eq 2 ]; then __run_test $1 2>> $testlog | tee -a $testlog else __run_test $1 >> $testlog 2>&1
@@ -268,7 +272,7 @@ run_test() { # testfile # Remove test log if the test was done as it was expected. [ $KEEP_LOG -eq 0 ] && rm $testlog else - [ $VERBOSE -ge 1 ] && catlog $testlog + [ $VERBOSE -eq 1 -o $VERBOSE -eq 2 ] && catlog $testlog TOTAL_RESULT=1 fi rm -rf $TMPDIR
Masami Hiramatsu
maintainerPATCH 4/4 selftests: ftrace: Output only to console with "--logdir -"
In reply to Masami HiramatsuOutput logs only to console if "-" is given to --logdir option. In this case, ftracetest doesn't record any log on the disk, and all logs immediately shown (including all command logs.) Since there is no "tee" in the middle of command and console, it outputs the log really soon.
This option is useful only when the console is logged.
--- tools/testing/selftests/ftrace/ftracetest | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-)
tools/testing/selftests/ftrace/ftracetest +21 -8
@@ -19,6 +19,7 @@ echo " -vvv Alias of -v -v -v (Show all commands immediately)" echo " --fail-unsupported Treat UNSUPPORTED as a failure" echo " -d|--debug Debug mode (trace all shell commands)" echo " -l|--logdir <dir> Save logs on the <dir>" +echo " If <dir> is -, all logs output in console only" exit $1 }
@@ -127,14 +128,20 @@ if [ -z "$TRACING_DIR" -o ! -d "$TRACING_DIR" ]; then fi # Preparing logs -LOG_FILE=$LOG_DIR/ftracetest.log -mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR" -date > $LOG_FILE +if [ "x$LOG_DIR" = "x-" ]; then + LOG_FILE= + date +else + LOG_FILE=$LOG_DIR/ftracetest.log + mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR" + date > $LOG_FILE +fi + prlog() { # messages - echo "$@" | tee -a $LOG_FILE + [ -z "$LOG_FILE" ] && echo "$@" || echo "$@" | tee -a $LOG_FILE } catlog() { #file - cat $1 | tee -a $LOG_FILE + [ -z "$LOG_FILE" ] && cat $1 || cat $1 | tee -a $LOG_FILE } prlog "=== Ftrace unit tests ==="
@@ -255,12 +262,18 @@ __run_test() { # testfile # Run one test case run_test() { # testfile local testname=`basename $1` - local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + if [ "$LOG_FILE" ] ; then + local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + else + local testlog=`/proc/self/fd/1` + fi export TMPDIR=`mktemp -d /tmp/ftracetest-dir.XXXXXX` testcase $1 echo "execute$INSTANCE: "$1 > $testlog SIG_RESULT=0 - if [ $VERBOSE -ge 3 ]; then + if [ -z "$LOG_FILE" ]; then + __run_test $1 2>&1 + elif [ $VERBOSE -ge 3 ]; then __run_test $1 | tee -a $testlog 2>&1 elif [ $VERBOSE -eq 2 ]; then __run_test $1 2>> $testlog | tee -a $testlog
@@ -270,7 +283,7 @@ run_test() { # testfile eval_result $SIG_RESULT if [ $? -eq 0 ]; then # Remove test log if the test was done as it was expected. - [ $KEEP_LOG -eq 0 ] && rm $testlog + [ $KEEP_LOG -eq 0 -a "$LOG_FILE" ] && rm $testlog else [ $VERBOSE -eq 1 -o $VERBOSE -eq 2 ] && catlog $testlog TOTAL_RESULT=1
Steven Rostedt
maintainerRe: [PATCH 4/4] selftests: ftrace: Output only to console with "--logdir -"
Quoted 52 lines · On Tue, 4 Jul 2017 15:38:55 +0900 Masami Hiramatsu <[email protected]> wrote:
Output logs only to console if "-" is given to --logdir option. In this case, ftracetest doesn't record any log on the disk, and all logs immediately shown (including all command logs.) Since there is no "tee" in the middle of command and console, it outputs the log really soon.
This option is useful only when the console is logged.
Signed-off-by Masami Hiramatsu <[email protected]>--- tools/testing/selftests/ftrace/ftracetest | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-)tools/testing/selftests/ftrace/ftracetest +12
-5@@ -19,6 +19,7 @@ echo " -vvv Alias of -v -v -v (Show all commands immediately)" echo " --fail-unsupported Treat UNSUPPORTED as a failure" echo " -d|--debug Debug mode (trace all shell commands)" echo " -l|--logdir <dir> Save logs on the <dir>" +echo " If <dir> is -, all logs output in console only" exit $1 }@@ -127,14 +128,20 @@ if [ -z "$TRACING_DIR" -o ! -d "$TRACING_DIR" ]; then fi # Preparing logs -LOG_FILE=$LOG_DIR/ftracetest.log -mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR" -date > $LOG_FILE +if [ "x$LOG_DIR" = "x-" ]; then + LOG_FILE= + date +else + LOG_FILE=$LOG_DIR/ftracetest.log + mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR" + date > $LOG_FILE +fi + prlog() { # messages - echo "$@" | tee -a $LOG_FILE + [ -z "$LOG_FILE" ] && echo "$@" || echo "$@" | tee -a $LOG_FILE } catlog() { #file - cat $1 | tee -a $LOG_FILE + [ -z "$LOG_FILE" ] && cat $1 || cat $1 | tee -a $LOG_FILE } prlog "=== Ftrace unit tests ==="
tools/testing/selftests/ftrace/ftracetest __run_test() { # testfile · line 265
Quoted 5 lines
tools/testing/selftests/ftrace/ftracetest +1
-1@@ -255,12 +262,18 @@ __run_test() { # testfile # Run one test case run_test() { # testfile local testname=`basename $1` - local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + if [ "$LOG_FILE" ] ; then
Shouldn't this be
if [ ! -z "$LOG_FILE" ]; then
?
tools/testing/selftests/ftrace/ftracetest __run_test() { # testfile · line 268
Quoted 3 lines
tools/testing/selftests/ftrace/ftracetest +3
-0+ local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + else + local testlog=`/proc/self/fd/1`
cute trick.
Quoted 13 lines
tools/testing/selftests/ftrace/ftracetest +4
-1+ fi export TMPDIR=`mktemp -d /tmp/ftracetest-dir.XXXXXX` testcase $1 echo "execute$INSTANCE: "$1 > $testlog SIG_RESULT=0 - if [ $VERBOSE -ge 3 ]; then + if [ -z "$LOG_FILE" ]; then + __run_test $1 2>&1 + elif [ $VERBOSE -ge 3 ]; then __run_test $1 | tee -a $testlog 2>&1 elif [ $VERBOSE -eq 2 ]; then __run_test $1 2>> $testlog | tee -a $testlog
tools/testing/selftests/ftrace/ftracetest run_test() { # testfile · line 286
Quoted 5 lines
tools/testing/selftests/ftrace/ftracetest +1
-1@@ -270,7 +283,7 @@ run_test() { # testfile eval_result $SIG_RESULT if [ $? -eq 0 ]; then # Remove test log if the test was done as it was expected. - [ $KEEP_LOG -eq 0 ] && rm $testlog + [ $KEEP_LOG -eq 0 -a "$LOG_FILE" ] && rm $testlog
again, don't we need to test "$LOG_FILE"?
-- Steve
Quoted 3 lines
tools/testing/selftests/ftrace/ftracetest +0
-0else [ $VERBOSE -eq 1 -o $VERBOSE -eq 2 ] && catlog $testlog TOTAL_RESULT=1
Steven Rostedt
maintainerRe: [PATCH 4/4] selftests: ftrace: Output only to console with "--logdir -"
Quoted 9 lines · On Wed, 5 Jul 2017 12:25:06 -0400 Steven Rostedt <[email protected]> wrote:
tools/testing/selftests/ftrace/ftracetest __run_test() { # testfile · line 265
Quoted 6 lines
tools/testing/selftests/ftrace/ftracetest +1
-1@@ -255,12 +262,18 @@ __run_test() { # testfile # Run one test case run_test() { # testfile local testname=`basename $1` - local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + if [ "$LOG_FILE" ] ; thenShouldn't this be
Quoted 4 lines
if [ ! -z "$LOG_FILE" ]; then?
OK, I just checked it out. I guess "" is considered zero and "<anything>" is considered 1.
Hmm, do we do this in other places too. Just makes me unconfortable.
-- Steve
Masami Hiramatsu
maintainerRe: [PATCH 4/4] selftests: ftrace: Output only to console with "--logdir -"
Quoted 15 lines · On Wed, 5 Jul 2017 12:29:17 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 11 lines · On Wed, 5 Jul 2017 12:25:06 -0400 Steven Rostedt <[email protected]> wrote:
tools/testing/selftests/ftrace/ftracetest __run_test() { # testfile · line 265
Quoted 6 lines
tools/testing/selftests/ftrace/ftracetest +1
-1@@ -255,12 +262,18 @@ __run_test() { # testfile # Run one test case run_test() { # testfile local testname=`basename $1` - local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + if [ "$LOG_FILE" ] ; thenShouldn't this be
if [ ! -z "$LOG_FILE" ]; then
Quoted 7 lines
Quoted 2 lines
?
OK, I just checked it out. I guess "" is considered zero and "<anything>" is considered 1.
Hmm, do we do this in other places too. Just makes me unconfortable.
OK, in that case, I can cleanup the code with [ ! -z "$VAR" ]
Thank you,
Quoted 3 lines
-- Steve
Signature · 1 line
Masami Hiramatsu <[email protected]>
Re: [PATCH 4/4] selftests: ftrace: Output only to console with "--logdir -"
In reply to Steven RostedtQuoted 8 lines · Steven Rostedt
tools/testing/selftests/ftrace/ftracetest __run_test() { # testfile · line 268
Quoted 3 lines · On Tue, 4 Jul 2017 15:38:55 +0900 Masami Hiramatsu <[email protected]> wrote:
tools/testing/selftests/ftrace/ftracetest +3
-0+ local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + else + local testlog=`/proc/self/fd/1`cute trick.
Shouldn't this be:
local testlog=/proc/self/fd/1
without the backticks and subshell? We just want to write to the stdout file not execute it.
Or am missing something.
-Stafford
Steven Rostedt
maintainerRe: [PATCH 4/4] selftests: ftrace: Output only to console with "--logdir -"
Quoted 10 lines · On Thu, 6 Jul 2017 22:06:58 +0900 Stafford Horne <[email protected]> wrote:
Quoted 8 lines · Steven Rostedt
tools/testing/selftests/ftrace/ftracetest __run_test() { # testfile · line 268
Quoted 3 lines · On Tue, 4 Jul 2017 15:38:55 +0900 Masami Hiramatsu <[email protected]> wrote:
tools/testing/selftests/ftrace/ftracetest +3
-0+ local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + else + local testlog=`/proc/self/fd/1`cute trick.
Quoted 8 lines
Shouldn't this be:
local testlog=/proc/self/fd/1
without the backticks and subshell? We just want to write to the stdout file not execute it.
Or am missing something.
Good eyes. I mistook those as quotes and not backticks.
-- Steve
Masami Hiramatsu
maintainerRe: [PATCH 4/4] selftests: ftrace: Output only to console with "--logdir -"
In reply to Stafford HorneQuoted 8 lines · On Thu, 6 Jul 2017 22:06:58 +0900 Stafford Horne <[email protected]> wrote:
Quoted 7 lines · Steven Rostedt
Quoted 3 lines · On Tue, 4 Jul 2017 15:38:55 +0900 Masami Hiramatsu <[email protected]> wrote:
tools/testing/selftests/ftrace/ftracetest +3
-0+ local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX` + else + local testlog=`/proc/self/fd/1`
Quoted 8 lines
Quoted 1 line
cute trick.
Shouldn't this be:
local testlog=/proc/self/fd/1
without the backticks and subshell? We just want to write to the stdout file not execute it.
Yeah, it was my mistake...
Thank you!
Quoted 4 lines
Or am missing something.
-Stafford
Signature · 1 line
Masami Hiramatsu <[email protected]>