Skip to contentSkip to conversation

ftracetest: Add a couple of ftrace test cases

RFC PATCH Superseded Steven Rostedt 4 participants 22 messages Sep 23, 2014 – Sep 30, 2014

Subsystems (2)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

RFC PATCH 1/1 ftracetest: Add a couple of ftrace test cases

[
Masami, I took two of my test scripts and added some basic comments to them and copied them pretty much unchanged into a ftrace directory under test.d. Is this fine, or is there more massaging I need to do to them?

I know the echos don't show up, but I kept them anyway. What should happen with them?
]

Added two test cases to get the feel of adding tests to ftracetest. The two cases are:

function profiling test, to make sure function profiling still works
with function tracing (was a regression)

function graph filter test to make sure that the function graph filter
does filter and also continues to filter when another function tracer is running (like the stack tracer)

---
 .../ftrace/test.d/ftrace/fgraph-filter.tc          | 111 +++++++++++++++++++++
 .../ftrace/test.d/ftrace/func_profiler.tc          |  78 +++++++++++++++
 2 files changed, 189 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +111 -0
new file mode 100644
index 000000000000..42f764d1f8d2
@@ -0,0 +1,111 @@
+#!/bin/sh
+# description: ftrace - function graph filters
+
+# Make sure that function graph filtering works, and is not
+# affected by other tracers enabled (like stack tracer)
+
+if ! grep -q function_graph available_tracers; then
+    echo "no function graph tracer configured"
+    exit 0;
+fi
+
+if [ ! -f set_ftrace_filter ]; then
+    echo "set_ftrace_filter not found? Is dynamic ftrace not set?"
+    exit 0
+fi
+
+do_reset() {
+    echo nop > current_tracer
+    echo 0 > /proc/sys/kernel/stack_tracer_enabled
+    echo 1 > tracing_on
+    echo > trace
+    echo > set_ftrace_filter
+}
+
+echo 0 > tracing_on
+# clear trace
+echo > trace
+# filter something, schedule is always good
+if ! echo "schedule" > set_ftrace_filter; then
+    # test for powerpc 64
+    if ! echo ".schedule" > set_ftrace_filter; then
+	echo "can not enable schedule filter"
+	exit -1
+    fi
+fi
+
+echo function_graph > current_tracer
+echo 1 > tracing_on
+sleep 1
+# search for functions (has "()" on the line), and make sure
+# that only the schedule function was found
+count=`cat trace | grep '()' | grep -v schedule | wc -l`
+if [ $count -ne 0 ]; then
+    echo "Graph filtering not working by itself?"
+    exit -1;
+fi
+
+# Make sure we did find something
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
+    exit -1
+fi
+
+echo "Graph filtering works by itself"
+
+if [ ! -f stack_trace ]; then
+    echo "Stack tracer not configured, can't continue test"
+    # stack tracer not configured in this kernel? pass test anyway
+    do_reset
+    exit 0;
+fi
+
+echo "Now testing with stack tracer"
+
+echo 1 > /proc/sys/kernel/stack_tracer_enabled
+
+echo 0 > tracing_on
+echo > trace
+echo 1 > tracing_on
+sleep 1
+
+count=`cat trace | grep '()' | grep -v schedule | wc -l`
+
+if [ $count -ne 0 ]; then
+    echo "Graph filtering not working with stack tracer?"
+    exit -1
+fi
+
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
+    exit -1
+fi
+
+echo "Graph filtering works with stack tracer"
+
+echo "Now testing if filtering still works without stack tracer"
+
+echo 0 > /proc/sys/kernel/stack_tracer_enabled
+echo > trace
+sleep 1
+
+
+count=`cat trace | grep '()' | grep -v schedule | wc -l`
+
+if [ $count -ne 0 ]; then
+    echo "Graph filtering not working after stack tracer disabled?"
+    exit -1
+fi
+
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
+    exit -1
+fi
+
+do_reset
+
+echo "SUCCESS!"
+exit 0
tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc new file +78 -0

78 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#1 lore raw
Quoted 3 lines · Steven Rostedt

[
Masami, I took two of my test scripts and added some basic comments

Thanks!

Quoted 3 lines

to them and copied them pretty much unchanged into a ftrace directory under test.d. Is this fine, or is there more massaging I need to do to them?

Yeah, ftrace has a double meaning :), so test.d/ftrace is fine to me.

Quoted 3 lines

I know the echos don't show up, but I kept them anyway. What should happen with them?

I think you'd better use exit_unsupported/exit_xfail to notify that the test target is not configured, or expected to fail. Then the user can reconfigure that. Maybe we should keep the detailed log of such results. (you can do it with --keep option)

Quoted 26 lines

]

Added two test cases to get the feel of adding tests to ftracetest. The two cases are:

function profiling test, to make sure function profiling still works
with function tracing (was a regression)

function graph filter test to make sure that the function graph filter
does filter and also continues to filter when another function tracer is running (like the stack tracer)

---
 .../ftrace/test.d/ftrace/fgraph-filter.tc          | 111 +++++++++++++++++++++
 .../ftrace/test.d/ftrace/func_profiler.tc          |  78 +++++++++++++++
 2 files changed, 189 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +0 -0
new file mode 100644
index 000000000000..42f764d1f8d2

tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc line 9

Quoted 9 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +9 -0
@@ -0,0 +1,111 @@
+#!/bin/sh
+# description: ftrace - function graph filters
+
+# Make sure that function graph filtering works, and is not
+# affected by other tracers enabled (like stack tracer)
+
+if ! grep -q function_graph available_tracers; then
+    echo "no function graph tracer configured"
+    exit 0;

This should call exit_unsupported, because the test is not passed.

tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc line 14

Quoted 5 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +5 -0
+fi
+
+if [ ! -f set_ftrace_filter ]; then
+    echo "set_ftrace_filter not found? Is dynamic ftrace not set?"
+    exit 0

Ditto.

Quoted 32 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +32 -0
+fi
+
+do_reset() {
+    echo nop > current_tracer
+    echo 0 > /proc/sys/kernel/stack_tracer_enabled
+    echo 1 > tracing_on
+    echo > trace
+    echo > set_ftrace_filter
+}
+
+echo 0 > tracing_on
+# clear trace
+echo > trace
+# filter something, schedule is always good
+if ! echo "schedule" > set_ftrace_filter; then
+    # test for powerpc 64
+    if ! echo ".schedule" > set_ftrace_filter; then
+	echo "can not enable schedule filter"
+	exit -1
+    fi
+fi
+
+echo function_graph > current_tracer
+echo 1 > tracing_on
+sleep 1
+# search for functions (has "()" on the line), and make sure
+# that only the schedule function was found
+count=`cat trace | grep '()' | grep -v schedule | wc -l`
+if [ $count -ne 0 ]; then
+    echo "Graph filtering not working by itself?"
+    exit -1;
+fi

tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc line 56

Quoted 10 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +10 -0
+
+# Make sure we did find something
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
+    exit -1
+fi
+
+echo "Graph filtering works by itself"
+

I think the following part should be a separated test for stack trace.

tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc line 61

Quoted 5 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +5 -0
+if [ ! -f stack_trace ]; then
+    echo "Stack tracer not configured, can't continue test"
+    # stack tracer not configured in this kernel? pass test anyway
+    do_reset
+    exit 0;

Here should call exit_unsupported.

Quoted 75 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +50 -0
+fi
+
+echo "Now testing with stack tracer"
+
+echo 1 > /proc/sys/kernel/stack_tracer_enabled
+
+echo 0 > tracing_on
+echo > trace
+echo 1 > tracing_on
+sleep 1
+
+count=`cat trace | grep '()' | grep -v schedule | wc -l`
+
+if [ $count -ne 0 ]; then
+    echo "Graph filtering not working with stack tracer?"
+    exit -1
+fi
+
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
+    exit -1
+fi
+
+echo "Graph filtering works with stack tracer"
+
+echo "Now testing if filtering still works without stack tracer"
+
+echo 0 > /proc/sys/kernel/stack_tracer_enabled
+echo > trace
+sleep 1
+
+
+count=`cat trace | grep '()' | grep -v schedule | wc -l`
+
+if [ $count -ne 0 ]; then
+    echo "Graph filtering not working after stack tracer disabled?"
+    exit -1
+fi
+
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
+    exit -1
+fi
+
+do_reset
+
+echo "SUCCESS!"
+exit 0
tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc new file +19 -0
new file mode 100644
index 000000000000..f35c58968d09
@@ -0,0 +1,78 @@
+#!/bin/sh
+# description: ftrace - function profiler with function tracing
+
+# There was a bug after a rewrite of the ftrace infrastructure that
+# caused the function_profiler not to be able to run with the function
+# tracer, because the function_profiler used the function_graph tracer
+# and it was assumed the two could not run simultaneously.
+#
+# There was another related bug where the solution to the first bug
+# broke the way filtering of the function tracer worked.
+#
+# This test triggers those bugs on those kernels.
+#
+# We need function_graph and profiling to to run this test
+if ! grep -q function_graph available_tracers; then
+    echo "no function graph tracer configured"
+    # allow it to pass, as the kernel doesn't support it
+    exit 0;
+fi

tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc line 29

Quoted 10 lines
tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc new file +10 -0
+
+if [ ! -f set_ftrace_filter ]; then
+    echo "set_ftrace_filter not found? Is dynamic ftrace not set?"
+    exit 0
+fi
+
+if [ ! -f function_profile_enabled ]; then
+    echo "function_profile_enabled not found, function profiling enabled?"
+    exit 0
+fi

These checks should call exit_unsupported if failed.

tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc line 35

Quoted 6 lines
tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc new file +6 -0
+
+echo "Testing function tracer with profiler:"
+echo "enable function tracer"
+echo function > current_tracer || exit 1
+echo "enable profiler"
+echo 1 > function_profile_enabled || exit 1

Now you don't need "|| exit 1", since if the command fails, the shell exits with error code.

tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc line 41

Quoted 6 lines
tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc new file +6 -0
+
+sleep 1
+
+echo "Now filter on just schedule"
+echo '*schedule' > set_ftrace_filter
+> trace

echo > trace?

Thank you,

Quoted 38 lines
tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc new file +37 -0
+
+echo "Now disable function profiler"
+echo 0 > function_profile_enabled || exit 1
+
+sleep 1
+
+# make sure only schedule functions exist
+
+echo "testing if only schedule is being traced"
+if grep -v -e '^#' -e 'schedule' trace; then
+	echo "more than schedule was found"
+	exit 1
+fi
+
+echo "Make sure schedule was traced"
+if ! grep -e 'schedule' trace > /dev/null; then
+	cat trace
+	echo "can not find schedule in trace"
+	exit 1
+fi
+
+echo "clear filter"
+echo > set_ftrace_filter || exit 1
+> trace
+
+sleep 1
+
+echo "make sure something other than scheduler is being traced"
+if ! grep -v -e '^#' -e 'schedule' trace > /dev/null; then
+	cat trace
+	echo "no other functions besides schedule was found"
+	exit 1
+fi
+
+echo nop > current_tracer || exit 1
+
+exit 0
Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
author maintainer
#2 lore raw
Quoted 7 lines · On Wed, 24 Sep 2014 11:58:50 +0900 Masami Hiramatsu…
Quoted 3 lines · Steven Rostedt

to them and copied them pretty much unchanged into a ftrace directory under test.d. Is this fine, or is there more massaging I need to do to them?

Yeah, ftrace has a double meaning :), so test.d/ftrace is fine to me.

I was thinking that more complex tests can go into ftrace, and simple tests can go into basic.

Quoted 7 lines
Quoted 4 lines

I know the echos don't show up, but I kept them anyway. What should happen with them?

I think you'd better use exit_unsupported/exit_xfail to notify that the test target is not configured, or expected to fail.

OK, that answers the exit codes (as you also stated below), but what about the echos themselves?

Quoted 2 lines

Then the user can reconfigure that. Maybe we should keep the detailed log of such results. (you can do it with --keep option)

You mean keep the echos, as they are ignored anyway, but if we add --keep, then the echos will show? Maybe that option should be -v, like other tools use for "verbose".

Quoted 6 lines
Quoted 6 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +3 -0
@@ -0,0 +1,111 @@
+#!/bin/sh
+# description: ftrace - function graph filters
+
Quoted 8 lines

tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc line 9

Quoted 6 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +6 -0
+# Make sure that function graph filtering works, and is not
+# affected by other tracers enabled (like stack tracer)
+
+if ! grep -q function_graph available_tracers; then
+    echo "no function graph tracer configured"
+    exit 0;

This should call exit_unsupported, because the test is not passed.

Will update this and others.

Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +4 -0
+# Make sure we did find something
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
Quoted 8 lines

tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc line 56

Quoted 5 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +5 -0
+    exit -1
+fi
+
+echo "Graph filtering works by itself"
+

I think the following part should be a separated test for stack trace.

Good point. A lot of my test scripts do multiple tests. I think for putting them into tools/testing/selftests/ftrace, I'll break them up and make them separate tests. For example, the simple graph filtering above is an example of something that can go into the basic directory, but the test against stack tracer should be in ftrace. What do you think?

Quoted 6 lines

tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc line 22

Quoted 4 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +4 -0
+
+echo "Now filter on just schedule"
+echo '*schedule' > set_ftrace_filter
+> trace

echo > trace?

OK. Hmm, I wonder if we should make a bunch of functions that the tests can use. Like a "clear_trace" call that does this. Can we export functions that this shell will be able to use? When we see lots of duplicate code we may want to have something like that. Well, for this, it may not be that big of a deal, because "echo > trace" is rather simple. But I do have other operations that are a bit more intrusive.

-- Steve

author maintainer
#3 lore raw
Quoted 3 lines · On Wed, 24 Sep 2014 11:42:35 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 2 lines

I think you'd better use exit_unsupported/exit_xfail to notify that the test target is not configured, or expected to fail.

Are these functions defined anywhere? I don't seem them in your v5 patch set, or in selftests anywhere.

-- Steve

#4 lore raw
Quoted 12 lines · Steven Rostedt
Quoted 3 lines · On Wed, 24 Sep 2014 11:42:35 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 2 lines

I think you'd better use exit_unsupported/exit_xfail to notify that the test target is not configured, or expected to fail.

Are these functions defined anywhere? I don't seem them in your v5 patch set, or in selftests anywhere.

-- Steve

Steve,

I am working on adding the framework for c tests. We will have to get this evolved for shell scripts. Please see the following thread:

https://lkml.org/lkml/2014/9/23/926

thanks,
-- Shuah

Signature · 4 lines
Shuah Khan
Sr. Linux Kernel Developer
Samsung Research America (Silicon Valley)
[email protected] | (970) 217-8978
author maintainer
#5 lore raw
Quoted 6 lines · On Wed, 24 Sep 2014 10:01:38 -0600 Shuah Khan <[email protected]> wrote:

I am working on adding the framework for c tests. We will have to get this evolved for shell scripts. Please see the following thread:

https://lkml.org/lkml/2014/9/23/926

Thanks,

Also, what's the rules for a test that requires other tools. Some of my tests require trace-cmd to be installed as well as perf. My stress test runs perf against trace-cmd running hackbench (another required tool), but I believe that perf has hackbench built in so I could replace hackbench with the perf version.

-- Steve

#6 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Wed, 24 Sep 2014 10:01:38 -0600 Shuah Khan <[email protected]> wrote:

I am working on adding the framework for c tests. We will have to get this evolved for shell scripts. Please see the following thread:

https://lkml.org/lkml/2014/9/23/926

Quoted 8 lines

Thanks,

Also, what's the rules for a test that requires other tools. Some of my tests require trace-cmd to be installed as well as perf. My stress test runs perf against trace-cmd running hackbench (another required tool), but I believe that perf has hackbench built in so I could replace hackbench with the perf version.

One simple approach is to have the user install the required tools for the test. Indicate what is needed and fail the test with appropriate message with xskip. This is what I have in mind for tests that can't be run because of a missing config. The same can be extended to external tools that the test depends on.

thanks,
-- Shuah

Signature · 4 lines
Shuah Khan
Sr. Linux Kernel Developer
Samsung Research America (Silicon Valley)
[email protected] | (970) 217-8978
author maintainer
#7 lore raw
Quoted 5 lines · On Wed, 24 Sep 2014 10:22:12 -0600 Shuah Khan <[email protected]> wrote:

One simple approach is to have the user install the required tools for the test. Indicate what is needed and fail the test with appropriate message with xskip. This is what I have in mind for tests that can't be run because of a missing config. The same can be extended to external tools that the test depends on.

Should I hold off adding my tests until we have all this in place. Or should I just push my tests with simply passing if the config isn't supported for now and add it later?

I wont add any of the tests that require other tools besides echo, grep and cat.

-- Steve

#8 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Wed, 24 Sep 2014 10:22:12 -0600 Shuah Khan <[email protected]> wrote:

One simple approach is to have the user install the required tools for the test. Indicate what is needed and fail the test with appropriate message with xskip. This is what I have in mind for tests that can't be run because of a missing config. The same can be extended to external tools that the test depends on.

Quoted 7 lines

Should I hold off adding my tests until we have all this in place. Or should I just push my tests with simply passing if the config isn't supported for now and add it later?

I wont add any of the tests that require other tools besides echo, grep and cat.

Please go ahead and get them in. The framework might have to wait until 3.19, so there is no need to have it gate test development. We can evolve them as we go. If tests require special, it means that they aren't suitable for certain envs such as qemu boot with busybox. There is a plan to add test targets to cover such cases.

I would recommend at least printing a message saying what you are missing if you find something missing.

thanks,
-- Shuah

Signature · 4 lines
Shuah Khan
Sr. Linux Kernel Developer
Samsung Research America (Silicon Valley)
[email protected] | (970) 217-8978
#9 lore raw
Quoted 11 lines · Shuah Khan
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Wed, 24 Sep 2014 10:22:12 -0600 Shuah Khan <[email protected]> wrote:

One simple approach is to have the user install the required tools for the test. Indicate what is needed and fail the test with appropriate message with xskip. This is what I have in mind for tests that can't be run because of a missing config. The same can be extended to external tools that the test depends on.

Quoted 6 lines
Quoted 6 lines

Should I hold off adding my tests until we have all this in place. Or should I just push my tests with simply passing if the config isn't supported for now and add it later?

I wont add any of the tests that require other tools besides echo, grep and cat.

They are supported by busybox :)

Quoted 5 lines

Please go ahead and get them in. The framework might have to wait until 3.19, so there is no need to have it gate test development. We can evolve them as we go. If tests require special, it means that they aren't suitable for certain envs such as qemu boot with busybox. There is a plan to add test targets to cover such cases.

Agreed, anyway, in some env those tools(commands) may not supported by busybox because of the memory size etc. In that case, we can just mark the test unsupported or xfail.

Thank you,

Quoted 7 lines

I would recommend at least printing a message saying what you are missing if you find something missing.

thanks,
-- Shuah

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
author maintainer
#10 lore raw
Quoted 10 lines · On Fri, 26 Sep 2014 21:04:26 +0900 Masami Hiramatsu…
Quoted 5 lines

Please go ahead and get them in. The framework might have to wait until 3.19, so there is no need to have it gate test development. We can evolve them as we go. If tests require special, it means that they aren't suitable for certain envs such as qemu boot with busybox. There is a plan to add test targets to cover such cases.

Agreed, anyway, in some env those tools(commands) may not supported by busybox because of the memory size etc. In that case, we can just mark the test unsupported or xfail.

Note, if memory size is such a big issue it that other tools may not be there, then most of the ftrace features will probably be disabled due to memory constraints on the kernel itself.

As we have a way to state "unsupported" we can just return that from tests that require tools not available, and say why it is the case.

-- Steve

#11 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Fri, 26 Sep 2014 21:04:26 +0900 Masami Hiramatsu…
Quoted 5 lines

Please go ahead and get them in. The framework might have to wait until 3.19, so there is no need to have it gate test development. We can evolve them as we go. If tests require special, it means that they aren't suitable for certain envs such as qemu boot with busybox. There is a plan to add test targets to cover such cases.

Quoted 8 lines
Quoted 4 lines

Agreed, anyway, in some env those tools(commands) may not supported by busybox because of the memory size etc. In that case, we can just mark the test unsupported or xfail.

Note, if memory size is such a big issue it that other tools may not be there, then most of the ftrace features will probably be disabled due to memory constraints on the kernel itself.

Agreed :)

Quoted 3 lines

As we have a way to state "unsupported" we can just return that from tests that require tools not available, and say why it is the case.

OK, if we can count "unsupported" as a test failure (yes, it is unexpectedly failed), it is easy to fix. I'll update XFAIL and other error code support patch and send it.

Thank you,

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#12 lore raw
In reply to Steven Rostedt

Hi Steve,

Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Wed, 24 Sep 2014 10:01:38 -0600 Shuah Khan <[email protected]> wrote:

I am working on adding the framework for c tests. We will have to get this evolved for shell scripts. Please see the following thread:

https://lkml.org/lkml/2014/9/23/926

Quoted 7 lines

Thanks,

Also, what's the rules for a test that requires other tools. Some of my tests require trace-cmd to be installed as well as perf. My stress test runs perf against trace-cmd running hackbench (another required tool), but I believe that perf has hackbench built in so I could replace hackbench with the perf version.

Yeah, and I think we can at least build perf (if not found) for test as we can always have the source. The trace-cmd is different and it'd be another reason I need to work on 'perf ftrace' command soon. ;-)

Thanks,
Namhyung

#13 lore raw
Quoted 21 lines · Namhyung Kim

Hi Steve,

Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Wed, 24 Sep 2014 10:01:38 -0600 Shuah Khan <[email protected]> wrote:

I am working on adding the framework for c tests. We will have to get this evolved for shell scripts. Please see the following thread:

https://lkml.org/lkml/2014/9/23/926

Quoted 7 lines

Thanks,

Also, what's the rules for a test that requires other tools. Some of my tests require trace-cmd to be installed as well as perf. My stress test runs perf against trace-cmd running hackbench (another required tool), but I believe that perf has hackbench built in so I could replace hackbench with the perf version.

Quoted 3 lines

Yeah, and I think we can at least build perf (if not found) for test as we can always have the source. The trace-cmd is different and it'd be another reason I need to work on 'perf ftrace' command soon. ;-)

+1 :)

For the kselftest, we can only expect to have tools in kernel tree (which should be static-linked) and busybox, so that we can use qemu emulator to run kselftest on other archs. I guess perf still need some shared objects to run. That should be solved too.

Anyway, at first we'd better add simpler tests which is not relay on other tools except busybox shell.

Thank you,

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#14 lore raw

Hi Masami,

2014-09-26 (금), 20:54 +0900, Masami Hiramatsu:

Quoted 26 lines
Quoted 21 lines · Namhyung Kim

Hi Steve,

Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Wed, 24 Sep 2014 10:01:38 -0600 Shuah Khan <[email protected]> wrote:

I am working on adding the framework for c tests. We will have to get this evolved for shell scripts. Please see the following thread:

https://lkml.org/lkml/2014/9/23/926

Quoted 7 lines

Thanks,

Also, what's the rules for a test that requires other tools. Some of my tests require trace-cmd to be installed as well as perf. My stress test runs perf against trace-cmd running hackbench (another required tool), but I believe that perf has hackbench built in so I could replace hackbench with the perf version.

Quoted 3 lines

Yeah, and I think we can at least build perf (if not found) for test as we can always have the source. The trace-cmd is different and it'd be another reason I need to work on 'perf ftrace' command soon. ;-)

Quoted 6 lines

+1 :)

For the kselftest, we can only expect to have tools in kernel tree (which should be static-linked) and busybox, so that we can use qemu emulator to run kselftest on other archs. I guess perf still need some shared objects to run. That should be solved too.

The perf should be built with no extra dependencies (other than libc). It won't provide rich features but basic functionality (record, report, stat) should be supported. And I'm also thinking about implementing a very simple elf parser for symbol loading in such a minimal build.

Thanks,
Namhyung

Quoted 7 lines

Anyway, at first we'd better add simpler tests which is not relay on other tools except busybox shell.

Thank you,

#15 lore raw
Quoted 36 lines · Namhyung Kim

Hi Masami,

2014-09-26 (금), 20:54 +0900, Masami Hiramatsu:

Quoted 26 lines
Quoted 21 lines · Namhyung Kim

Hi Steve,

Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Wed, 24 Sep 2014 10:01:38 -0600 Shuah Khan <[email protected]> wrote:

I am working on adding the framework for c tests. We will have to get this evolved for shell scripts. Please see the following thread:

https://lkml.org/lkml/2014/9/23/926

Quoted 7 lines

Thanks,

Also, what's the rules for a test that requires other tools. Some of my tests require trace-cmd to be installed as well as perf. My stress test runs perf against trace-cmd running hackbench (another required tool), but I believe that perf has hackbench built in so I could replace hackbench with the perf version.

Quoted 3 lines

Yeah, and I think we can at least build perf (if not found) for test as we can always have the source. The trace-cmd is different and it'd be another reason I need to work on 'perf ftrace' command soon. ;-)

Quoted 6 lines

+1 :)

For the kselftest, we can only expect to have tools in kernel tree (which should be static-linked) and busybox, so that we can use qemu emulator to run kselftest on other archs. I guess perf still need some shared objects to run. That should be solved too.

Quoted 4 lines

The perf should be built with no extra dependencies (other than libc). It won't provide rich features but basic functionality (record, report, stat) should be supported. And I'm also thinking about implementing a very simple elf parser for symbol loading in such a minimal build.

Maybe you can install some static libraries (e.g. elfutils-libelf-devel-static on fedora) for enabling it. :)
Some libraries may not work, but I could enable libelf by installing that.

Thank you,

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#16 lore raw

Hi Masami,

Quoted 9 lines · Masami Hiramatsu
Quoted 4 lines · Namhyung Kim

The perf should be built with no extra dependencies (other than libc). It won't provide rich features but basic functionality (record, report, stat) should be supported. And I'm also thinking about implementing a very simple elf parser for symbol loading in such a minimal build.

Maybe you can install some static libraries (e.g. elfutils-libelf-devel-static on fedora) for enabling it. :)
Some libraries may not work, but I could enable libelf by installing that.

Yep, installing static libs should also work. But I'd like to have minimal depencency especially for embedded systems. And AFAIK cross building elfutils/libelf always has been a PITA.

Thanks,
Namhyung

author maintainer
#17 lore raw
In reply to Namhyung Kim
Quoted 3 lines · On Fri, 26 Sep 2014 15:06:51 +0900 Namhyung Kim <[email protected]> wrote:

Yeah, and I think we can at least build perf (if not found) for test as we can always have the source. The trace-cmd is different and it'd be another reason I need to work on 'perf ftrace' command soon. ;-)

Well, if a tool is not found. I wouldn't recommend building it. Either case, I'll hold off test that is required.

Hmm, maybe I'll post all my tests on my website, for others to play with. That way I'm not limited in tests that take too much time or use tools not supported by the kernel selftest infrastructure.

-- Steve

#18 lore raw
In reply to Steven Rostedt
Quoted 9 lines · Steven Rostedt
Quoted 3 lines · On Wed, 24 Sep 2014 11:42:35 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 2 lines

I think you'd better use exit_unsupported/exit_xfail to notify that the test target is not configured, or expected to fail.

Are these functions defined anywhere? I don't seem them in your v5 patch set, or in selftests anywhere.

Hmm, could you check my 4/4 patch in the series??
http://www.mail-archive.com/[email protected]/msg730578.html

Thanks,

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
author maintainer
#19 lore raw
Quoted 8 lines · On Thu, 25 Sep 2014 09:35:05 +0900 Masami Hiramatsu…
Quoted 7 lines · Steven Rostedt
Quoted 3 lines · On Wed, 24 Sep 2014 11:42:35 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 2 lines

I think you'd better use exit_unsupported/exit_xfail to notify that the test target is not configured, or expected to fail.

Quoted 5 lines
Quoted 2 lines

Are these functions defined anywhere? I don't seem them in your v5 patch set, or in selftests anywhere.

Hmm, could you check my 4/4 patch in the series??
http://www.mail-archive.com/[email protected]/msg730578.html

Hmm, I somehow missed this. Can you check to make sure my for-next patches are your latest version. I'll pull in 4/4 tomorrow.

Thanks,

-- Steve

#20 lore raw
Quoted 11 lines · Steven Rostedt
Quoted 8 lines · On Thu, 25 Sep 2014 09:35:05 +0900 Masami Hiramatsu…
Quoted 7 lines · Steven Rostedt
Quoted 3 lines · On Wed, 24 Sep 2014 11:42:35 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 2 lines

I think you'd better use exit_unsupported/exit_xfail to notify that the test target is not configured, or expected to fail.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Are these functions defined anywhere? I don't seem them in your v5 patch set, or in selftests anywhere.

Hmm, could you check my 4/4 patch in the series??
http://www.mail-archive.com/[email protected]/msg730578.html

Hmm, I somehow missed this. Can you check to make sure my for-next patches are your latest version. I'll pull in 4/4 tomorrow.

Yeah, I ensured that your for-next patches are the v5. You will be able to cleanly apply 4/4 on it :)

Thank you,

Quoted 5 lines

Thanks,

-- Steve

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#21 lore raw
In reply to Steven Rostedt
Quoted 5 lines · Steven Rostedt
Quoted 1 line · On Wed, 24 Sep 2014 11:58:50 +0900 Masami Hiramatsu…

(2014/09/24 6:38), Steven Rostedt wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

to them and copied them pretty much unchanged into a ftrace directory under test.d. Is this fine, or is there more massaging I need to do to them?

Yeah, ftrace has a double meaning :), so test.d/ftrace is fine to me.

I was thinking that more complex tests can go into ftrace, and simple tests can go into basic.

Hmm, what kind of "complex" test will be there?
I think unit tests are usually simple, selftests/ftrace/test.d/ftrace/ looks redundant if the first ftrace and the second one has same meaning.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I know the echos don't show up, but I kept them anyway. What should happen with them?

I think you'd better use exit_unsupported/exit_xfail to notify that the test target is not configured, or expected to fail.

OK, that answers the exit codes (as you also stated below), but what about the echos themselves?

Again, ftracetest is for unit tests, exit code will return one concrete result, but we can anything with echo (including typo)

Quoted 6 lines
Quoted 3 lines

Then the user can reconfigure that. Maybe we should keep the detailed log of such results. (you can do it with --keep option)

You mean keep the echos, as they are ignored anyway, but if we add --keep, then the echos will show?

No, --keep keeps all output logs of the test script. Since all scripts run under set -x, all commands and outputs are logged to a logfile.

Quoted 2 lines

Maybe that option should be -v, like
other tools use for "verbose".

OK, it will be easy, just turning "> $LOGFILE" into "| tee $LOGFILE" :)

Quoted 31 lines
Quoted 6 lines
Quoted 6 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +3 -0
@@ -0,0 +1,111 @@
+#!/bin/sh
+# description: ftrace - function graph filters
+
Quoted 8 lines

tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc line 9

Quoted 6 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +6 -0
+# Make sure that function graph filtering works, and is not
+# affected by other tracers enabled (like stack tracer)
+
+if ! grep -q function_graph available_tracers; then
+    echo "no function graph tracer configured"
+    exit 0;

This should call exit_unsupported, because the test is not passed.

Will update this and others.

Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +4 -0
+# Make sure we did find something
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
Quoted 8 lines

tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc line 56

Quoted 5 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +5 -0
+    exit -1
+fi
+
+echo "Graph filtering works by itself"
+

I think the following part should be a separated test for stack trace.

Quoted 6 lines

Good point. A lot of my test scripts do multiple tests. I think for putting them into tools/testing/selftests/ftrace, I'll break them up and make them separate tests. For example, the simple graph filtering above is an example of something that can go into the basic directory, but the test against stack tracer should be in ftrace. What do you think?

Hm, I think only really basic operation tests go into the basic directory, like reading and writing (no format checks). And test.d/ftrace may be for more advanced tests, like checking output, setting options, etc.

Quoted 10 lines
Quoted 6 lines

tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc line 22

Quoted 4 lines
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc new file +4 -0
+
+echo "Now filter on just schedule"
+echo '*schedule' > set_ftrace_filter
+> trace

echo > trace?

OK. Hmm, I wonder if we should make a bunch of functions that the tests can use. Like a "clear_trace" call that does this. Can we export functions that this shell will be able to use?

Yeah, it's a good idea :) We can use "." to include it. I think test.d/functions will be good.

Quoted 4 lines

When we see lots of
duplicate code we may want to have something like that. Well, for this, it may not be that big of a deal, because "echo > trace" is rather simple. But I do have other operations that are a bit more intrusive.

Agreed.

Thank you!

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]