introducing tools/perf/testsuite

Hi all,

in the following patches I introduce a skeleton of a testsuite for perf tool. The idea of this testsuite is to primarily cover all the sub-commands perf has and verify their functionality.

The testsuite is designed to be structured in order to be able to keep an order in some amount of testcases. Each group of testcases is located in a base_<something> dir with its init and cleanup scripts and shell files test_<something>.sh that should carry the testcases. Each testcase can have multiple assertions inside. The base_skeleton and base_stat are an example. New tests are supposed to be derived from them.

Since the testsuite is meant to be usable for both smoke and deeper testing, it has some parametrization in common/parametrization.sh.

My idea is to add testcases for the other existing builtin commands and then continue extending it for the upcoming ones.

How does it seem to you? Thoughts?

Michael

Michael Petlan (3):
tools/perf/testsuite: Init commit tools/perf/testsuite: Adding a skeleton for future testcases tools/perf/testsuite: Adding a set of testcases for perf-stat

 tools/perf/testsuite/base_skeleton/cleanup.sh      |  16 +++
 tools/perf/testsuite/base_skeleton/settings.sh     |  10 ++
 tools/perf/testsuite/base_skeleton/setup.sh        |  16 +++
 tools/perf/testsuite/base_skeleton/test_01.sh      |  40 ++++++++
 tools/perf/testsuite/base_stat/cleanup.sh          |  20 ++++
 tools/perf/testsuite/base_stat/settings.sh         |  10 ++
 tools/perf/testsuite/base_stat/test_basic.sh       |  68 ++++++++++++
 tools/perf/testsuite/base_stat/test_hw.sh          |  50 +++++++++
 tools/perf/testsuite/base_stat/test_hwcache.sh     |  51 +++++++++
 .../testsuite/base_stat/test_powerpc_hv_24x7.sh    |  60 +++++++++++
 .../testsuite/common/check_all_lines_matched.pl    |  28 +++++
 .../testsuite/common/check_all_patterns_found.pl   |  30 ++++++
 .../testsuite/common/check_any_pattern_found.pl    |  14 +++
 tools/perf/testsuite/common/init.sh                |  50 +++++++++
 tools/perf/testsuite/common/parametrization.sh     |  18 ++++
 tools/perf/testsuite/common/patterns.sh            |  74 +++++++++++++
 tools/perf/testsuite/common/settings.sh            |  50 +++++++++
 tools/perf/testsuite/test_driver.sh                | 114 +++++++++++++++++++++
 18 files changed, 719 insertions(+)
 create mode 100755 tools/perf/testsuite/base_skeleton/cleanup.sh
 create mode 100644 tools/perf/testsuite/base_skeleton/settings.sh
 create mode 100755 tools/perf/testsuite/base_skeleton/setup.sh
 create mode 100755 tools/perf/testsuite/base_skeleton/test_01.sh
 create mode 100755 tools/perf/testsuite/base_stat/cleanup.sh
 create mode 100644 tools/perf/testsuite/base_stat/settings.sh
 create mode 100755 tools/perf/testsuite/base_stat/test_basic.sh
 create mode 100755 tools/perf/testsuite/base_stat/test_hw.sh
 create mode 100755 tools/perf/testsuite/base_stat/test_hwcache.sh
 create mode 100755 tools/perf/testsuite/base_stat/test_powerpc_hv_24x7.sh
 create mode 100755 tools/perf/testsuite/common/check_all_lines_matched.pl
 create mode 100755 tools/perf/testsuite/common/check_all_patterns_found.pl
 create mode 100755 tools/perf/testsuite/common/check_any_pattern_found.pl
 create mode 100644 tools/perf/testsuite/common/init.sh
 create mode 100644 tools/perf/testsuite/common/parametrization.sh
 create mode 100644 tools/perf/testsuite/common/patterns.sh
 create mode 100644 tools/perf/testsuite/common/settings.sh
 create mode 100755 tools/perf/testsuite/test_driver.sh

1.8.3.1

Hi Michael,

Nice!
I also have ftracetest based homebrew testscripts (shellscripts) for perf-probe. Mine requires just a posix shell and some core utils (no perl), but only covers perf-probe (since it is what I need :) )

I'll post my runtest script series soon.

I like this kind of outside tests instead of perf-test. Some sort of features can be tested by the perf-test, but more complex features like perf-probe SDT support etc. are hard to be tested (or/and to write testcases) inside the perf-test.

I think just a set of shell script is good to go since we can check the result easily and good to review if the testcase is included in a new feature.

Thank you,

Hi Michael,

Here is a couple of patches to add my testsuite(runtest.sh) to perf.

This runs as ftracetest does. I've added some testcases which cover perf-probe features :)

----

  # ./runtests.sh
  === Run tests ===
  [1] Basinc probe adding [PASS]
  [2] Basic probe adding on user binary   [PASS]
  [3] Skip blacklist function     [PASS]
  [4] Basic probe deleting        [PASS]
  [5] probe adding with --force   [PASS]
  [6] Cut off the function suffix from name       [PASS]
  [7] Basic function listup command       [PASS]
  [8] help messages       [PASS]
  [9] invalid commands    [PASS]
  [10] perf probe --list outputs in stdout        [PASS]
  [11] list command parsing       [PASS]
  [12] Skip out-of-text functions [PASS]
  [13] Quiet option check [PASS]
  [14] --line semantic errors     [PASS]
  [15] readable vars command      [PASS]
  [16] perf-probe wildcard support        [PASS]

  # of passed:  16
  # of failed:  0
  # of unresolved:  0
  # of untested:  0
  # of unsupported:  0
  # of xfailed:  0
  # of undefined(test bug):  0
  ----

Thank you,

---

Masami Hiramatsu (2):
perf/runtests: Add runtests.sh shellscript testsuite perf/runtests: Add testcases for perf probe

 tools/perf/runtests/runtests.conf                  |    5 
 tools/perf/runtests/runtests.sh                    |  264 ++++++++++++++++++++
 tools/perf/runtests/test.d/functions               |   13 +
 tools/perf/runtests/test.d/probe/addprobes.tc      |   15 +
 tools/perf/runtests/test.d/probe/adduprobes.tc     |   13 +
 tools/perf/runtests/test.d/probe/blacklist.tc      |   14 +
 tools/perf/runtests/test.d/probe/delete.tc         |    8 +
 tools/perf/runtests/test.d/probe/forceadd.tc       |    8 +
 .../perf/runtests/test.d/probe/funcname-suffix.tc  |   17 +
 tools/perf/runtests/test.d/probe/funcs.tc          |    8 +
 tools/perf/runtests/test.d/probe/help.tc           |    5 
 .../perf/runtests/test.d/probe/invalid_command.tc  |   27 ++
 tools/perf/runtests/test.d/probe/list-stdout.tc    |    7 +
 tools/perf/runtests/test.d/probe/list.tc           |   10 +
 tools/perf/runtests/test.d/probe/out-of-text.tc    |   10 +
 tools/perf/runtests/test.d/probe/quiet.tc          |    6 
 tools/perf/runtests/test.d/probe/semantic-line.tc  |   20 ++
 tools/perf/runtests/test.d/probe/vars.tc           |    9 +
 tools/perf/runtests/test.d/probe/wildcard.tc       |    6 
 19 files changed, 465 insertions(+)
 create mode 100644 tools/perf/runtests/runtests.conf
 create mode 100755 tools/perf/runtests/runtests.sh
 create mode 100644 tools/perf/runtests/test.d/functions
 create mode 100644 tools/perf/runtests/test.d/probe/addprobes.tc
 create mode 100644 tools/perf/runtests/test.d/probe/adduprobes.tc
 create mode 100644 tools/perf/runtests/test.d/probe/blacklist.tc
 create mode 100644 tools/perf/runtests/test.d/probe/delete.tc
 create mode 100644 tools/perf/runtests/test.d/probe/forceadd.tc
 create mode 100644 tools/perf/runtests/test.d/probe/funcname-suffix.tc
 create mode 100644 tools/perf/runtests/test.d/probe/funcs.tc
 create mode 100644 tools/perf/runtests/test.d/probe/help.tc
 create mode 100644 tools/perf/runtests/test.d/probe/invalid_command.tc
 create mode 100644 tools/perf/runtests/test.d/probe/list-stdout.tc
 create mode 100644 tools/perf/runtests/test.d/probe/list.tc
 create mode 100644 tools/perf/runtests/test.d/probe/out-of-text.tc
 create mode 100644 tools/perf/runtests/test.d/probe/quiet.tc
 create mode 100644 tools/perf/runtests/test.d/probe/semantic-line.tc
 create mode 100644 tools/perf/runtests/test.d/probe/vars.tc
 create mode 100644 tools/perf/runtests/test.d/probe/wildcard.tc

--

Add runtests.sh shellscript testsute which is derived from tools/testing/selftests/ftrace/ftracetest.

---
 tools/perf/runtests/runtests.conf |    5 +
 tools/perf/runtests/runtests.sh   |  264 +++++++++++++++++++++++++++++++++++++
 2 files changed, 269 insertions(+)
 create mode 100644 tools/perf/runtests/runtests.conf
 create mode 100755 tools/perf/runtests/runtests.sh

new file mode 100644
index 0000000..545748c3
@@ -0,0 +1,5 @@
+# Config file for runtest
+
+PERF=`abspath ../perf`
+PERF_MANDIR=`abspath ../Documentation/`
+

Add testcases for perf probe command, including following features;

- Add probes on the kernel
- Add probes on user binary
- Delete probes
- Add probes with --force
- Check help message
- Parse --list options
- Skip blacklist functions
- Skip out-of-text address
- Ignore dot suffixes of function names
- List up functions
- Check invalid commands
- Check --list shown in stdout (instead of stderr)
- Check quiet option
- Semantic error checking for --line
- Check --vars command
- Check wildcard support

---
 tools/perf/runtests/test.d/functions               |   13 ++++++++++
 tools/perf/runtests/test.d/probe/addprobes.tc      |   15 +++++++++++
 tools/perf/runtests/test.d/probe/adduprobes.tc     |   13 ++++++++++
 tools/perf/runtests/test.d/probe/blacklist.tc      |   14 ++++++++++
 tools/perf/runtests/test.d/probe/delete.tc         |    8 ++++++
 tools/perf/runtests/test.d/probe/forceadd.tc       |    8 ++++++
 .../perf/runtests/test.d/probe/funcname-suffix.tc  |   17 +++++++++++++
 tools/perf/runtests/test.d/probe/funcs.tc          |    8 ++++++
 tools/perf/runtests/test.d/probe/help.tc           |    5 ++++
 .../perf/runtests/test.d/probe/invalid_command.tc  |   27 ++++++++++++++++++++
 tools/perf/runtests/test.d/probe/list-stdout.tc    |    7 +++++
 tools/perf/runtests/test.d/probe/list.tc           |   10 +++++++
 tools/perf/runtests/test.d/probe/out-of-text.tc    |   10 +++++++
 tools/perf/runtests/test.d/probe/quiet.tc          |    6 ++++
 tools/perf/runtests/test.d/probe/semantic-line.tc  |   20 +++++++++++++++
 tools/perf/runtests/test.d/probe/vars.tc           |    9 +++++++
 tools/perf/runtests/test.d/probe/wildcard.tc       |    6 ++++
 17 files changed, 196 insertions(+)
 create mode 100644 tools/perf/runtests/test.d/functions
 create mode 100644 tools/perf/runtests/test.d/probe/addprobes.tc
 create mode 100644 tools/perf/runtests/test.d/probe/adduprobes.tc
 create mode 100644 tools/perf/runtests/test.d/probe/blacklist.tc
 create mode 100644 tools/perf/runtests/test.d/probe/delete.tc
 create mode 100644 tools/perf/runtests/test.d/probe/forceadd.tc
 create mode 100644 tools/perf/runtests/test.d/probe/funcname-suffix.tc
 create mode 100644 tools/perf/runtests/test.d/probe/funcs.tc
 create mode 100644 tools/perf/runtests/test.d/probe/help.tc
 create mode 100644 tools/perf/runtests/test.d/probe/invalid_command.tc
 create mode 100644 tools/perf/runtests/test.d/probe/list-stdout.tc
 create mode 100644 tools/perf/runtests/test.d/probe/list.tc
 create mode 100644 tools/perf/runtests/test.d/probe/out-of-text.tc
 create mode 100644 tools/perf/runtests/test.d/probe/quiet.tc
 create mode 100644 tools/perf/runtests/test.d/probe/semantic-line.tc
 create mode 100644 tools/perf/runtests/test.d/probe/vars.tc
 create mode 100644 tools/perf/runtests/test.d/probe/wildcard.tc

new file mode 100644
index 0000000..89ca293
@@ -0,0 +1,13 @@
+clear_all_probes() {
+	echo 0 > /sys/kernel/debug/tracing/events/enable
+	echo > /sys/kernel/debug/tracing/kprobe_events
+	echo > /sys/kernel/debug/tracing/uprobe_events
+}
+
+check_perf_probe_option() { #option
+	$PERF probe -h 2>&1 | egrep '[\t ]+'$1'[\t ]+' > /dev/null
+}
+
+check_kernel_debuginfo() {
+	eu-addr2line -k 0x`grep -m 1 vfs_read /proc/kallsyms | cut -f 1 -d" "` | grep vfs_read
+}

new file mode 100644
index 0000000..69c2bc5
@@ -0,0 +1,15 @@
+#!/bin/sh
+#description: Basinc probe adding
+
+clear_all_probes
+$PERF probe --add vfs_read
+
+clear_all_probes
+$PERF probe -a vfs_read
+
+clear_all_probes
+$PERF probe vfs_read
+
+clear_all_probes
+
+! $PERF probe -d vfs_read vfs_read

new file mode 100644
index 0000000..b0e748d
@@ -0,0 +1,13 @@
+#!/bin/sh
+#description: Basic probe adding on user binary
+
+clear_all_probes
+$PERF probe -x /bin/bash --add array_create
+
+clear_all_probes
+$PERF probe -x /bin/bash --add 'array_*'
+
+clear_all_probes
+$PERF probe /bin/bash 'array_*'
+
+clear_all_probes

new file mode 100644
index 0000000..d399439
@@ -0,0 +1,14 @@
+#!/bin/sh
+#description: Skip blacklist function
+
+echo "Check blacklisted function correctly skipped"
+BLACKFUNC=`head -n 1 /sys/kernel/debug/kprobes/blacklist | cut -f2`
+test "$BLACKFUNC" || exit_unsupported
+clear_all_probes
+
+echo "probing on blacklist must fail"
+(! $PERF probe $BLACKFUNC)
+
+echo "it should be skipped"
+$PERF probe $BLACKFUNC 2>&1 | grep "skip it"
+

new file mode 100644
index 0000000..918d6e1
@@ -0,0 +1,8 @@
+#!/bin/sh
+# description: Basic probe deleting
+clear_all_probes
+
+$PERF probe --add vfs_read
+$PERF probe -x $PERF --add cmd_help
+
+$PERF probe -d \* 2>&1 | grep ^Removed | wc -l | grep '^2$'

new file mode 100644
index 0000000..bcefabb
@@ -0,0 +1,8 @@
+#!/bin/sh
+#description: probe adding with --force
+clear_all_probes
+
+$PERF probe --add vfs_read
+! $PERF probe --add 'vfs_read'
+$PERF probe --force --add 'vfs_read'
+$PERF probe --del \*

new file mode 100644
index 0000000..8c68ad0
@@ -0,0 +1,17 @@
+#!/bin/sh
+#description: Cut off the function suffix from name
+
+clear_all_probes
+echo "get a symbol which has a suffix"
+CANDIDATE=`egrep -m 1 '\.(isra|part|constprep)' /proc/kallsyms | cut -f3 -d' '`
+if [ -z "$CANDIDATE" ]; then
+  echo "No suffixed function found. skip it"
+  exit_unsupported
+fi
+echo "add an event on it"
+$PERF probe $CANDIDATE
+
+echo "add the second event on it (with suffix number)"
+$PERF probe -f $CANDIDATE
+
+clear_all_probes

new file mode 100644
index 0000000..98a412f
@@ -0,0 +1,8 @@
+#!/bin/sh
+# description: Basic function listup command
+
+$PERF probe -F &> /dev/null
+$PERF probe -x $PERF -F &> /dev/null
+$PERF probe -x $PERF -F 'map_*' &> /dev/null
+
+

new file mode 100644
index 0000000..67e2c0a
@@ -0,0 +1,5 @@
+#!/bin/sh
+# description: help messages
+! $PERF probe -h
+$PERF probe 2>&1 | grep usage
+

new file mode 100644
index 0000000..351ec99
@@ -0,0 +1,27 @@
+#!/bin/sh
+# description: invalid commands
+echo "Some commands requires an argument"
+! $PERF probe -a
+! $PERF probe -d
+! $PERF probe -L
+! $PERF probe -V
+echo "Some commands can omit the argument"
+$PERF probe -F > /dev/null
+$PERF probe -l > /dev/null
+echo "Commands are mutually exclusive"
+! $PERF probe -a xxx -d xxx
+! $PERF probe -a xxx -L foo
+! $PERF probe -a xxx -V foo
+! $PERF probe -a xxx -l
+! $PERF probe -a xxx -F
+! $PERF probe -d xxx -L foo
+! $PERF probe -d xxx -V foo
+! $PERF probe -d xxx -l
+! $PERF probe -d xxx -F
+! $PERF probe -L foo -V bar
+! $PERF probe -L foo -l
+! $PERF probe -L foo -F
+! $PERF probe -V foo -l
+! $PERF probe -V foo -F
+! $PERF probe -l -F
+

new file mode 100644
index 0000000..d797abf
@@ -0,0 +1,7 @@
+#!/bin/sh
+# description: perf probe --list outputs in stdout
+
+clear_all_probes
+$PERF probe --add 'vfs_read'
+test `$PERF probe --list | wc -l` -ne 0
+clear_all_probes

new file mode 100644
index 0000000..2cfab75
@@ -0,0 +1,10 @@
+#!/bin/sh
+# description: list command parsing
+$PERF probe -l
+$PERF probe -l \*
+$PERF probe -l 'probe:*'
+
+! $PERF probe -l \*\|
+! $PERF probe -l \*\&
+! $PERF probe -l \*\!
+

new file mode 100644
index 0000000..34163a9
@@ -0,0 +1,10 @@
+#!/bin/sh
+#description: Skip out-of-text functions
+
+echo "Check init text correctly skipped"
+INITTEXT=init_setup
+grep $INITTEXT /proc/kallsyms || exit_unsupported
+clear_all_probes
+! $PERF probe $INITTEXT
+$PERF probe $INITTEXT 2>&1 | grep "skip it"
+

new file mode 100644
index 0000000..ac954aa
@@ -0,0 +1,6 @@
+#!/bin/sh
+#description: Quiet option check
+
+test `$PERF probe --quiet --add vfs_read | wc -l` -eq 0
+test `$PERF probe --quiet --list | wc -l` -ne 0
+test `$PERF probe --quiet --del vfs_read | wc -l` -eq 0

new file mode 100644
index 0000000..a6f5869
@@ -0,0 +1,20 @@
+#!/bin/sh
+# description: --line semantic errors
+
+echo "Acceptable --line descriptions"
+! ($PERF probe --line 'func' 2>&1 | grep "Semantic error")
+! ($PERF probe --line 'func:10' 2>&1 | grep "Semantic error")
+! ($PERF probe --line 'func:0-10' 2>&1 | grep "Semantic error")
+! ($PERF probe --line 'func:2+10' 2>&1 | grep "Semantic error")
+! ($PERF probe --line '[email protected]' 2>&1 | grep "Semantic error")
+! ($PERF probe --line '[email protected]:1' 2>&1 | grep "Semantic error")
+! ($PERF probe --line 'source.c:1' 2>&1 | grep "Semantic error")
+! ($PERF probe --line 'source.c:1+1' 2>&1 | grep "Semantic error")
+! ($PERF probe --line 'source.c:1-10' 2>&1 | grep "Semantic error")
+
+echo "Unacceptable --line descriptions"
+$PERF probe --line 'func:foo' 2>&1 | grep "Semantic error"
+$PERF probe --line 'func:1-foo' 2>&1 | grep "Semantic error"
+$PERF probe --line 'func:1+foo' 2>&1 | grep "Semantic error"
+$PERF probe --line 'func;lazy*pattern' 2>&1 | grep "Semantic error"
+

new file mode 100644
index 0000000..9711c91
@@ -0,0 +1,9 @@
+#!/bin/sh
+# description: readable vars command
+
+#check_kernel_debuginfo || exit_unsupported
+
+! $PERF probe -V
+
+$PERF probe -V vfs_read
+

new file mode 100644
index 0000000..ec3e633
@@ -0,0 +1,6 @@
+#!/bin/sh
+#description: perf-probe wildcard support
+
+$PERF probe -nf -a 'vfs_* $params'
+$PERF probe -nf vfs_\* \$params
+

Hi Masami Hiramatsu,

I am sorry for responding late, I had a lot of other work. Thanks for sharing the testsuite, I like it very much, so I added some of your testcases into my skeleton. There is still some work to do regarding that.

I develop the testsuite using my own github repository[1], if you are interested, you may have look at it, comment...

The commit cc094da02683b0908512b6b8676c0648e73fa2c8 added the first part of your suite there.

Thank you again for your interest.

Michael

[1] https://github.com/rfmvh/perftool-testsuite

----- Mensaje original -----

Hi Michael,

From: Michael Petlan [mailto:[email protected]]

Thanks! that helps me a lot.
Arnaldo, I think we'd better include this type of blackbox tests what would you think?

Thanks,

Em Tue, Nov 03, 2015 at 03:24:43AM +0000, 平松雅巳 / HIRAMATU，MASAMI escreveu:

Sure thing, I'll take a look at it at some point, just letting it brew for a while, i.e. for some conversation to take place about how it is being structured, how it compares to what people like you have in place for 'perf probe', etc.

- Arnaldo

Any feedback for the testsuite is appreciated. I am still not satisfied with the coverage, etc. I am just launching an idea being curious, whether it is able to live and survive or not. If you have any ideas, recommendations, contributions, surely come up with them! Also if anyone has questions why it is as it is, about the design of the structure, checking mechanism, etc...

Michael