perf/core improvements and fixes

From: Arnaldo Carvalho de Melo <[email protected]>

Hi Ingo,

Please consider pulling,

- Arnaldo

Test results at the end of this message, as usual.

The following changes since commit cf31b7dd51b542f9a9b2d0de416b5524dfdb1cf2:

Merge tag 'perf-core-for-mingo-4.14-20170814' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core (2017-08-14 19:38:40 +0200)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git tags/perf-core-for-mingo-4.14-20170816

for you to fetch changes up to 35435cd06081d7db96bc617b65ba556f8e24340e:

perf test shell: Replace '|&' with '2>&1 |' to work with more shells (2017-08-16 16:23:26 -0300)

----------------------------------------------------------------
perf core improvements and fixes:

New features:

- Support exporting Intel PT data to sqlite3 with python perf scripts, this is in addition to the postgresql support that was already there (Adrian Hunter)

Infrastructure:

- Handle perf tool builds with less features in perf shell tests, such as those with NO_LIBDWARF=1 or even without 'perf probe' (Arnaldo Carvalho de Melo)

- Replace '|&' with '2>&1 |' to work with more shells in the just introduced perf test shell harness (Kim Phillips)

Architecture related fixes:

- Fix endianness problem when loading parameters in the BPF prologue
generated by perf, noticed using 'perf test BPF' in s390x systems (Wang Nan, Thomas Richter)

----------------------------------------------------------------
Adrian Hunter (5):
perf scripts python: Fix missing call_path_id in export-to-postgresql script perf scripts python: Fix query in call-graph-from-postgresql.py
perf script python: Add support for exporting to sqlite3
perf script python: Rename call-graph-from-postgresql.py to call-graph-from-sql.py perf script python: Add support for sqlite3 to call-graph-from-sql.py

Arnaldo Carvalho de Melo (3):
perf tests shell: Remove duplicate skip_if_no_debuginfo() function perf test shell: Check if 'perf probe' is available, skip tests if not perf test shell vfs_getname: Skip for tools built with NO_LIBDWARF=1

Kim Phillips (1):
perf test shell: Replace '|&' with '2>&1 |' to work with more shells

Wang Nan (1):
perf bpf: Fix endianness problem when loading parameters in prologue

 tools/perf/Documentation/intel-pt.txt              |   6 +-
 .../scripts/python/bin/export-to-sqlite-record     |   8 +
 .../scripts/python/bin/export-to-sqlite-report     |  29 ++
 ...h-from-postgresql.py => call-graph-from-sql.py} |  70 ++--
 tools/perf/scripts/python/export-to-postgresql.py  |   5 +-
 tools/perf/scripts/python/export-to-sqlite.py      | 451 +++++++++++++++++++++
 tools/perf/tests/bpf-script-test-prologue.c        |   4 +-
 tools/perf/tests/shell/lib/probe.sh                |   6 +
 tools/perf/tests/shell/lib/probe_vfs_getname.sh    |  13 +-
 tools/perf/tests/shell/probe_vfs_getname.sh        |   4 +
 .../tests/shell/record+script_probe_vfs_getname.sh |   4 +
 .../perf/tests/shell/trace+probe_libc_inet_pton.sh |   5 +-
 tools/perf/tests/shell/trace+probe_vfs_getname.sh  |   4 +
 tools/perf/util/bpf-prologue.c                     |  49 ++-
 14 files changed, 611 insertions(+), 47 deletions(-)
 create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-record
 create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-report
 rename tools/perf/scripts/python/{call-graph-from-postgresql.py => call-graph-from-sql.py} (87%)
 create mode 100644 tools/perf/scripts/python/export-to-sqlite.py
 create mode 100644 tools/perf/tests/shell/lib/probe.sh

Test results:

The first ones are container (docker) based builds of tools/perf with and without libelf support, objtool where it is supported and samples/bpf/, ditto. Where clang is available, it is also used to build perf with/without libelf.

Several are cross builds, the ones with -x-ARCH, and the android one, and those may not have all the features built, due to lack of multi-arch devel packages, available and being used so far on just a few, like debian:experimental-x-{arm64,mipsel}.

The 'perf test' one will perform a variety of tests exercising tools/perf/util/, tools/lib/{bpf,traceevent,etc}, as well as run perf commands with a variety of command line event specifications to then intercept the sys_perf_event syscall to check that the perf_event_attr fields are set up as expected, among a variety of other unit tests.

The 'perf test' also runs shell scripts exercising the tools, checking if they affect the system in certain ways, like setting up kprobes and uprobes, request callchains for well known programs and check that they are the expected ones, see if 'perf trace' beautifies system call arguments correctly, etc.

Additionally, a new set of tests, script based, runs the tools in a live system, setting probes in place that then gets used by 'perf trace', with its output compared against expected results.

Then there is the 'make -C tools/perf build-test' ones, that build tools/perf/ with a variety of feature sets, exercising the build with an incomplete set of features as well as with a complete one. It is planned to have it run on each of the containers mentioned above, using some container orchestration infrastructure. Get in contact if interested in helping having this in place.

  # dm
   1 alpine:3.4: Ok
   2 alpine:3.5: Ok
   3 alpine:3.6: Ok
   4 alpine:edge: Ok
   5 android-ndk:r12b-arm: Ok
   6 archlinux:latest: Ok
   7 centos:5: Ok
   8 centos:6: Ok
   9 centos:7: Ok
  10 debian:7: Ok
  11 debian:8: Ok
  12 debian:9: Ok
  13 debian:experimental: Ok
  14 debian:experimental-x-arm64: Ok
  15 debian:experimental-x-mips: Ok
  16 debian:experimental-x-mips64: Ok
  17 debian:experimental-x-mipsel: Ok
  18 fedora:20: Ok
  19 fedora:21: Ok
  20 fedora:22: Ok
  21 fedora:23: Ok
  22 fedora:24: Ok
  23 fedora:24-x-ARC-uClibc: Ok
  24 fedora:25: Ok
  25 fedora:26: Ok
  26 fedora:rawhide: Ok
  27 mageia:5: Ok
  28 opensuse:13.2: Ok
  29 opensuse:42.1: Ok
  30 opensuse:42.2: Ok
  31 opensuse:tumbleweed: Ok
  32 oraclelinux:6: Ok
  33 oraclelinux:7: Ok
  34 ubuntu:12.04.5: Ok
  35 ubuntu:14.04.4: Ok
  36 ubuntu:14.04.4-x-linaro-arm64: Ok
  37 ubuntu:15.10: Ok
  38 ubuntu:16.04: Ok
  39 ubuntu:16.04-x-arm: Ok
  40 ubuntu:16.04-x-arm64: Ok
  41 ubuntu:16.04-x-powerpc: Ok
  42 ubuntu:16.04-x-powerpc64: Ok
  43 ubuntu:16.04-x-powerpc64el: Ok
  44 ubuntu:16.04-x-s390: Ok
  45 ubuntu:16.10: Ok
  46 ubuntu:17.04: Ok
  47 ubuntu:17.10: Ok
  #

  # uname -a
  Linux jouet 4.13.0-rc4+ #2 SMP Fri Aug 11 12:39:09 -03 2017 x86_64 x86_64 x86_64 GNU/Linux
  # perf test
   1: vmlinux symtab matches kallsyms                       : Ok
   2: Detect openat syscall event                           : Ok
   3: Detect openat syscall event on all cpus               : Ok
   4: Read samples using the mmap interface                 : Ok
   5: Parse event definition strings                        : Ok
   6: Simple expression parser                              : Ok
   7: PERF_RECORD_* events & perf_sample fields             : Ok
   8: Parse perf pmu format                                 : Ok
   9: DSO data read                                         : Ok
  10: DSO data cache                                        : Ok
  11: DSO data reopen                                       : Ok
  12: Roundtrip evsel->name                                 : Ok
  13: Parse sched tracepoints fields                        : Ok
  14: syscalls:sys_enter_openat event fields                : Ok
  15: Setup struct perf_event_attr                          : Ok
  16: Match and link multiple hists                         : Ok
  17: 'import perf' in python                               : Ok
  18: Breakpoint overflow signal handler                    : Ok
  19: Breakpoint overflow sampling                          : Ok
  20: Number of exit events of a simple workload            : Ok
  21: Software clock events period values                   : Ok
  22: Object code reading                                   : Ok
  23: Sample parsing                                        : Ok
  24: Use a dummy software event to keep tracking           : Ok
  25: Parse with no sample_id_all bit set                   : Ok
  26: Filter hist entries                                   : Ok
  27: Lookup mmap thread                                    : Ok
  28: Share thread mg                                       : Ok
  29: Sort output of hist entries                           : Ok
  30: Cumulate child hist entries                           : Ok
  31: Track with sched_switch                               : Ok
  32: Filter fds with revents mask in a fdarray             : Ok
  33: Add fd to a fdarray, making it autogrow               : Ok
  34: kmod_path__parse                                      : Ok
  35: Thread map                                            : Ok
  36: LLVM search and compile                               :
  36.1: Basic BPF llvm compile                              : Ok
  36.2: kbuild searching                                    : Ok
  36.3: Compile source for BPF prologue generation          : Ok
  36.4: Compile source for BPF relocation                   : Ok
  37: Session topology                                      : Ok
  38: BPF filter                                            :
  38.1: Basic BPF filtering                                 : Ok
  38.2: BPF pinning                                         : Ok
  38.3: BPF prologue generation                             : Ok
  38.4: BPF relocation checker                              : Ok
  39: Synthesize thread map                                 : Ok
  40: Remove thread map                                     : Ok
  41: Synthesize cpu map                                    : Ok
  42: Synthesize stat config                                : Ok
  43: Synthesize stat                                       : Ok
  44: Synthesize stat round                                 : Ok
  45: Synthesize attr update                                : Ok
  46: Event times                                           : Ok
  47: Read backward ring buffer                             : Ok
  48: Print cpu map                                         : Ok
  49: Probe SDT events                                      : Ok
  50: is_printable_array                                    : Ok
  51: Print bitmap                                          : Ok
  52: perf hooks                                            : Ok
  53: builtin clang support                                 : Skip (not compiled in)
  54: unit_number__scnprintf                                : Ok
  55: x86 rdpmc                                             : Ok
  56: Convert perf time to TSC                              : Ok
  57: DWARF unwind                                          : Ok
  58: x86 instruction decoder - new instructions            : Ok
  59: Intel cqm nmi context read                            : Skip
  60: Use vfs_getname probe to get syscall args filenames   : Ok
  61: probe libc's inet_pton & backtrace it with ping       : Ok
  62: Check open filename arg using perf trace + vfs_getname: Ok
  63: Add vfs_getname probe to get syscall args filenames   : Ok
  # 

  $ make -C tools/perf build-test
  make: Entering directory '/home/acme/git/linux/tools/perf'
  - tarpkg: ./tests/perf-targz-src-pkg .
              make_no_libelf_O: make NO_LIBELF=1
              make_clean_all_O: make clean all
                   make_tags_O: make tags
                 make_perf_o_O: make perf.o
             make_util_map_o_O: make util/map.o
            make_install_bin_O: make install-bin
                    make_doc_O: make doc
            make_no_demangle_O: make NO_DEMANGLE=1
  make_no_libdw_dwarf_unwind_O: make NO_LIBDW_DWARF_UNWIND=1
   make_install_prefix_slash_O: make install prefix=/tmp/krava/
        make_with_babeltrace_O: make LIBBABELTRACE=1
           make_no_backtrace_O: make NO_BACKTRACE=1
             make_no_libperl_O: make NO_LIBPERL=1
            make_no_libaudit_O: make NO_LIBAUDIT=1
         make_install_prefix_O: make install prefix=/tmp/krava
           make_no_libbionic_O: make NO_LIBBIONIC=1
                make_minimal_O: make NO_LIBPERL=1 NO_LIBPYTHON=1 NO_NEWT=1 NO_GTK2=1 NO_DEMANGLE=1 NO_LIBELF=1 NO_LIBUNWIND=1 NO_BACKTRACE=1 NO_LIBNUMA=1 NO_LIBAUDIT=1 NO_LIBBIONIC=1 NO_LIBDW_DWARF_UNWIND=1 NO_AUXTRACE=1 NO_LIBBPF=1 NO_LIBCRYPTO=1 NO_SDT=1 NO_JVMTI=1
            make_no_auxtrace_O: make NO_AUXTRACE=1
           make_no_libpython_O: make NO_LIBPYTHON=1
                  make_debug_O: make DEBUG=1
              make_no_libbpf_O: make NO_LIBBPF=1
                make_no_newt_O: make NO_NEWT=1
         make_with_clangllvm_O: make LIBCLANGLLVM=1
                make_no_gtk2_O: make NO_GTK2=1
             make_no_scripts_O: make NO_LIBPYTHON=1 NO_LIBPERL=1
                   make_pure_O: make
                  make_no_ui_O: make NO_NEWT=1 NO_SLANG=1 NO_GTK2=1
               make_no_slang_O: make NO_SLANG=1
                make_install_O: make install
             make_no_libnuma_O: make NO_LIBNUMA=1
       make_util_pmu_bison_o_O: make util/pmu-bison.o
                   make_help_O: make help
           make_no_libunwind_O: make NO_LIBUNWIND=1
  OK
  make: Leaving directory '/home/acme/git/linux/tools/perf'
  $

From: Arnaldo Carvalho de Melo <[email protected]>

---
 tools/perf/tests/shell/lib/probe_vfs_getname.sh | 5 -----
 1 file changed, 5 deletions(-)

index 42308040f882..3eaddf190a30 100644
@@ -21,8 +21,3 @@ skip_if_no_debuginfo() {
 	add_probe_vfs_getname -v 2>&1 | grep -q "^Failed to find the path for kernel" && return 2
 	return 1
 }
-
-skip_if_no_debuginfo() {
-	add_probe_vfs_getname -v 2>&1 | grep -q "^Failed to find the path for kernel" && return 2
-	return 1
-}

2.13.5

From: Arnaldo Carvalho de Melo <[email protected]>

Add a library function that checks if 'perf probe' is built into the tool being tested, skipping tests that need it.

Testing it on a system after removing the library needed to build 'probe' as a perf subcommand:

  # perf test ping vfs_getname
  59: Use vfs_getname probe to get syscall args filenames   : Skip
  60: probe libc's inet_pton & backtrace it with ping       : Skip
  61: Check open filename arg using perf trace + vfs_getname: Skip
  62: Add vfs_getname probe to get syscall args filenames   : Skip
  # perf probe
  perf: 'probe' is not a perf-command. See 'perf --help'.
  #

Now reinstalling elfutils-libelf-devel on this Fedora 26 system to rebuild perf and then retest this:

  # perf test ping vfs_getname
  60: Use vfs_getname probe to get syscall args filenames   : Ok
  61: probe libc's inet_pton & backtrace it with ping       : Ok
  62: Check open filename arg using perf trace + vfs_getname: Ok
  63: Add vfs_getname probe to get syscall args filenames   : Ok
  #

---
 tools/perf/tests/shell/lib/probe.sh                       | 6 ++++++
 tools/perf/tests/shell/lib/probe_vfs_getname.sh           | 2 +-
 tools/perf/tests/shell/probe_vfs_getname.sh               | 4 ++++
 tools/perf/tests/shell/record+script_probe_vfs_getname.sh | 4 ++++
 tools/perf/tests/shell/trace+probe_libc_inet_pton.sh      | 3 +++
 tools/perf/tests/shell/trace+probe_vfs_getname.sh         | 4 ++++
 6 files changed, 22 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/tests/shell/lib/probe.sh

new file mode 100644
index 000000000000..61da3b2c9bca
@@ -0,0 +1,6 @@
+# Arnaldo Carvalho de Melo <[email protected]>, 2017
+
+skip_if_no_perf_probe() {
+	perf probe |& grep -q 'is not a perf-command' && return 2
+	return 0
+}

index 3eaddf190a30..46c1bb707600 100644
@@ -1,6 +1,6 @@
 # Arnaldo Carvalho de Melo <[email protected]>, 2017
 
-perf probe -l | grep -q probe:vfs_getname
+perf probe -l |& grep -q probe:vfs_getname
 had_vfs_getname=$?
 
 cleanup_probe_vfs_getname() {

index c8380137beef..9b7635184dc2 100755
@@ -2,6 +2,10 @@
 #
 # Arnaldo Carvalho de Melo <[email protected]>, 2017
 
+. $(dirname $0)/lib/probe.sh
+
+skip_if_no_perf_probe || exit 2
+
 . $(dirname $0)/lib/probe_vfs_getname.sh
 
 add_probe_vfs_getname || skip_if_no_debuginfo

index 2725c5db699a..ba29535b8580 100755
@@ -7,6 +7,10 @@
 
 # Arnaldo Carvalho de Melo <[email protected]>, 2017
 
+. $(dirname $0)/lib/probe.sh
+
+skip_if_no_perf_probe || exit 2
+
 . $(dirname $0)/lib/probe_vfs_getname.sh
 
 perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)

index 1b9a276e2ace..99dafad61954 100755
@@ -8,6 +8,8 @@
 
 # Arnaldo Carvalho de Melo <[email protected]>, 2017
 
+. $(dirname $0)/lib/probe.sh
+
 trace_libc_inet_pton_backtrace() {
 	idx=0
 	expected[0]="PING.*bytes"

@@ -32,6 +34,7 @@ trace_libc_inet_pton_backtrace() {
 	done
 }
 
+skip_if_no_perf_probe && \
 perf probe -q /lib64/libc-*.so inet_pton && \
 trace_libc_inet_pton_backtrace
 err=$?

index d0fba0c5e354..2e68c5f120da 100755
@@ -8,6 +8,10 @@
 
 # Arnaldo Carvalho de Melo <[email protected]>, 2017
 
+. $(dirname $0)/lib/probe.sh
+
+skip_if_no_perf_probe || exit 2
+
 . $(dirname $0)/lib/probe_vfs_getname.sh
 
 file=$(mktemp /tmp/temporary_file.XXXXX)

2.13.5

From: Arnaldo Carvalho de Melo <[email protected]>

If that is the case, or if the required lib is not present, e.g. elfutils-devel in Fedora systems, then just skip the tests requiring DWARF analysis.

Before:

  # rpm -e elfutils-devel
  # perf test ping vfs_getname
  60: Use vfs_getname probe to get syscall args filenames   : FAILED!
  61: probe libc's inet_pton & backtrace it with ping       : Ok
  62: Check open filename arg using perf trace + vfs_getname: FAILED!
  63: Add vfs_getname probe to get syscall args filenames   : FAILED!
  #

After:

  # perf test vfs_getname
  60: Use vfs_getname probe to get syscall args filenames   : Skip
  62: Check open filename arg using perf trace + vfs_getname: Skip
  63: Add vfs_getname probe to get syscall args filenames   : Skip
  #

Then, reinstalling elfutils-devel, rebuilding the tool and running again:

  # perf test vfs_getname
  60: Use vfs_getname probe to get syscall args filenames   : Ok
  62: Check open filename arg using perf trace + vfs_getname: Ok
  63: Add vfs_getname probe to get syscall args filenames   : Ok
  #

---
 tools/perf/tests/shell/lib/probe_vfs_getname.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

index 46c1bb707600..f832395daad7 100644
@@ -12,12 +12,12 @@ cleanup_probe_vfs_getname() {
 add_probe_vfs_getname() {
 	local verbose=$1
 	if [ $had_vfs_getname -eq 1 ] ; then
-               line=$(perf probe -L getname_flags | egrep 'result.*=.*filename;' | sed -r 's/[[:space:]]+([[:digit:]]+)[[:space:]]+result->uptr.*/\1/')
-               perf probe $verbose "vfs_getname=getname_flags:${line} pathname=result->name:string"
+		line=$(perf probe -L getname_flags |& egrep 'result.*=.*filename;' | sed -r 's/[[:space:]]+([[:digit:]]+)[[:space:]]+result->uptr.*/\1/')
+		perf probe $verbose "vfs_getname=getname_flags:${line} pathname=result->name:string"
 	fi
 }
 
 skip_if_no_debuginfo() {
-	add_probe_vfs_getname -v 2>&1 | grep -q "^Failed to find the path for kernel" && return 2
+	add_probe_vfs_getname -v 2>&1 | egrep -q "^(Failed to find the path for kernel|Debuginfo-analysis is not supported)" && return 2
 	return 1
 }

2.13.5

From: Adrian Hunter <[email protected]>

The export does not work if only branches are exported because of a missing column in the samples table. Fix by adding the missing call_path_id.

---
 tools/perf/scripts/python/export-to-postgresql.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index 7656ff8aa066..f57811443beb 100644
@@ -340,7 +340,8 @@ if branches:
 		'to_sym_offset	bigint,'
 		'to_ip		bigint,'
 		'branch_type	integer,'
-		'in_tx		boolean)')
+		'in_tx		boolean,'
+		'call_path_id	bigint)')
 else:
 	do_query(query, 'CREATE TABLE samples ('
 		'id		bigint		NOT NULL,'

2.13.5

From: Adrian Hunter <[email protected]>

Add a missing space which seemed not to affect PostgreSQL but upsets SQLite.

---
 tools/perf/scripts/python/call-graph-from-postgresql.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index e78fdc2a5a9d..ed9f7f3ccf22 100644
@@ -160,7 +160,7 @@ class TreeItem():
 				  '( SELECT short_name FROM dsos WHERE id = ( SELECT dso_id FROM symbols WHERE id = ( SELECT symbol_id FROM call_paths WHERE id = call_path_id ) ) ), '
 				  '( SELECT ip FROM call_paths where id = call_path_id ) '
 				  'FROM calls WHERE parent_call_path_id = ' + str(self.call_path_id) + ' AND comm_id = ' + str(self.comm_id) + ' AND thread_id = ' + str(self.thread_id) +
-				  'ORDER BY call_path_id')
+				  ' ORDER BY call_path_id')
 		if not ret:
 			raise Exception("Query failed: " + query.lastError().text())
 		last_call_path_id = 0

2.13.5

From: Adrian Hunter <[email protected]>

Add support for exporting to SQLite 3 the same data as the PostgreSQL export.

Committer note:

Tested on RHEL 7.4 using the 1.2.2-4el python-pyside packages from EPEL.

---
 tools/perf/Documentation/intel-pt.txt              |   6 +-
 .../scripts/python/bin/export-to-sqlite-record     |   8 +
 .../scripts/python/bin/export-to-sqlite-report     |  29 ++
 tools/perf/scripts/python/export-to-sqlite.py      | 451 +++++++++++++++++++++
 4 files changed, 491 insertions(+), 3 deletions(-)
 create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-record
 create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-report
 create mode 100644 tools/perf/scripts/python/export-to-sqlite.py

index 4b6cdbf8f935..8e8ae3ad4cbf 100644
@@ -104,9 +104,9 @@ system, asynchronous, interrupt, transaction abort, trace begin, trace end, and
 in transaction, respectively.
 
 While it is possible to create scripts to analyze the data, an alternative
-approach is available to export the data to a postgresql database.  Refer to
-script export-to-postgresql.py for more details, and to script
-call-graph-from-postgresql.py for an example of using the database.
+approach is available to export the data to a sqlite or postgresql database.
+Refer to script export-to-sqlite.py or export-to-postgresql.py for more details,
+and to script call-graph-from-postgresql.py for an example of using the database.
 
 There is also script intel-pt-events.py which provides an example of how to
 unpack the raw data for power events and PTWRITE.

new file mode 100644
index 000000000000..070204fd6d00
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+#
+# export perf data to a sqlite3 database. Can cover
+# perf ip samples (excluding the tracepoints). No special
+# record requirements, just record what you want to export.
+#
+perf record $@

new file mode 100644
index 000000000000..5ff6033e70ba
@@ -0,0 +1,29 @@
+#!/bin/bash
+# description: export perf data to a sqlite3 database
+# args: [database name] [columns] [calls]
+n_args=0
+for i in "$@"
+do
+    if expr match "$i" "-" > /dev/null ; then
+	break
+    fi
+    n_args=$(( $n_args + 1 ))
+done
+if [ "$n_args" -gt 3 ] ; then
+    echo "usage: export-to-sqlite-report [database name] [columns] [calls]"
+    exit
+fi
+if [ "$n_args" -gt 2 ] ; then
+    dbname=$1
+    columns=$2
+    calls=$3
+    shift 3
+elif [ "$n_args" -gt 1 ] ; then
+    dbname=$1
+    columns=$2
+    shift 2
+elif [ "$n_args" -gt 0 ] ; then
+    dbname=$1
+    shift
+fi
+perf script $@ -s "$PERF_EXEC_PATH"/scripts/python/export-to-sqlite.py $dbname $columns $calls

2.13.5

From: Adrian Hunter <[email protected]>

Rename call-graph-from-postgresql.py to call-graph-from-sql.py in preparation for adding support to it for SQLite 3.

---
 tools/perf/Documentation/intel-pt.txt                          |  2 +-
 .../{call-graph-from-postgresql.py => call-graph-from-sql.py}  | 10 +++++-----
 tools/perf/scripts/python/export-to-postgresql.py              |  2 +-
 3 files changed, 7 insertions(+), 7 deletions(-)
 rename tools/perf/scripts/python/{call-graph-from-postgresql.py => call-graph-from-sql.py} (96%)

index 8e8ae3ad4cbf..ab1b0825130a 100644
@@ -106,7 +106,7 @@ in transaction, respectively.
 While it is possible to create scripts to analyze the data, an alternative
 approach is available to export the data to a sqlite or postgresql database.
 Refer to script export-to-sqlite.py or export-to-postgresql.py for more details,
-and to script call-graph-from-postgresql.py for an example of using the database.
+and to script call-graph-from-sql.py for an example of using the database.
 
 There is also script intel-pt-events.py which provides an example of how to
 unpack the raw data for power events and PTWRITE.

similarity index 96%
rename from tools/perf/scripts/python/call-graph-from-postgresql.py
rename to tools/perf/scripts/python/call-graph-from-sql.py
index ed9f7f3ccf22..f18406d3faf7 100644
@@ -1,6 +1,6 @@
 #!/usr/bin/python2
-# call-graph-from-postgresql.py: create call-graph from postgresql database
-# Copyright (c) 2014, Intel Corporation.
+# call-graph-from-sql.py: create call-graph from postgresql database
+# Copyright (c) 2014-2017, Intel Corporation.
 #
 # This program is free software; you can redistribute it and/or modify it
 # under the terms and conditions of the GNU General Public License,

@@ -17,12 +17,12 @@
 # Following on from the example in the export-to-postgresql.py script, a
 # call-graph can be displayed for the pt_example database like this:
 #
-#	python tools/perf/scripts/python/call-graph-from-postgresql.py pt_example
+#	python tools/perf/scripts/python/call-graph-from-sql.py pt_example
 #
 # Note this script supports connecting to remote databases by setting hostname,
 # port, username, password, and dbname e.g.
 #
-#	python tools/perf/scripts/python/call-graph-from-postgresql.py "hostname=myhost username=myuser password=mypassword dbname=pt_example"
+#	python tools/perf/scripts/python/call-graph-from-sql.py "hostname=myhost username=myuser password=mypassword dbname=pt_example"
 #
 # The result is a GUI window with a tree representing a context-sensitive
 # call-graph.  Expanding a couple of levels of the tree and adjusting column

@@ -291,7 +291,7 @@ class MainWindow(QMainWindow):
 
 if __name__ == '__main__':
 	if (len(sys.argv) < 2):
-		print >> sys.stderr, "Usage is: call-graph-from-postgresql.py <database name>"
+		print >> sys.stderr, "Usage is: call-graph-from-sql.py <database name>"
 		raise Exception("Too few arguments")
 
 	dbname = sys.argv[1]

index f57811443beb..efcaf6cac2eb 100644
@@ -59,7 +59,7 @@ import datetime
 #	pt_example=# \q
 #
 # An example of using the database is provided by the script
-# call-graph-from-postgresql.py.  Refer to that script for details.
+# call-graph-from-sql.py.  Refer to that script for details.
 #
 # Tables:
 #

2.13.5

From: Adrian Hunter <[email protected]>

Add support for SQLite 3 to the call-graph-from-sql.py script. The SQL statements work as is, so just detect the database type by checking if the SQLite 3 file exists.

Committer notes:

Tested collecting the PT data on a RHEL7.4, generating the SQLite3 database there and then moving it to a Fedora 26 system where the call-graph-from-sql.py script was run, using python-pyside version 1.2.2-7fc26 to see the callgraphs using Qt4.

---
 tools/perf/scripts/python/call-graph-from-sql.py | 60 ++++++++++++++----------
 1 file changed, 36 insertions(+), 24 deletions(-)

index f18406d3faf7..b494a67a1c67 100644
@@ -1,5 +1,5 @@
 #!/usr/bin/python2
-# call-graph-from-sql.py: create call-graph from postgresql database
+# call-graph-from-sql.py: create call-graph from sql database
 # Copyright (c) 2014-2017, Intel Corporation.
 #
 # This program is free software; you can redistribute it and/or modify it

@@ -11,16 +11,17 @@
 # FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
 # more details.
 
-# To use this script you will need to have exported data using the
-# export-to-postgresql.py script.  Refer to that script for details.
+# To use this script you will need to have exported data using either the
+# export-to-sqlite.py or the export-to-postgresql.py script.  Refer to those
+# scripts for details.
 #
-# Following on from the example in the export-to-postgresql.py script, a
+# Following on from the example in the export scripts, a
 # call-graph can be displayed for the pt_example database like this:
 #
 #	python tools/perf/scripts/python/call-graph-from-sql.py pt_example
 #
-# Note this script supports connecting to remote databases by setting hostname,
-# port, username, password, and dbname e.g.
+# Note that for PostgreSQL, this script supports connecting to remote databases
+# by setting hostname, port, username, password, and dbname e.g.
 #
 #	python tools/perf/scripts/python/call-graph-from-sql.py "hostname=myhost username=myuser password=mypassword dbname=pt_example"
 #

@@ -296,24 +297,35 @@ if __name__ == '__main__':
 
 	dbname = sys.argv[1]
 
-	db = QSqlDatabase.addDatabase('QPSQL')
-
-	opts = dbname.split()
-	for opt in opts:
-		if '=' in opt:
-			opt = opt.split('=')
-			if opt[0] == 'hostname':
-				db.setHostName(opt[1])
-			elif opt[0] == 'port':
-				db.setPort(int(opt[1]))
-			elif opt[0] == 'username':
-				db.setUserName(opt[1])
-			elif opt[0] == 'password':
-				db.setPassword(opt[1])
-			elif opt[0] == 'dbname':
-				dbname = opt[1]
-		else:
-			dbname = opt
+	is_sqlite3 = False
+	try:
+		f = open(dbname)
+		if f.read(15) == "SQLite format 3":
+			is_sqlite3 = True
+		f.close()
+	except:
+		pass
+
+	if is_sqlite3:
+		db = QSqlDatabase.addDatabase('QSQLITE')
+	else:
+		db = QSqlDatabase.addDatabase('QPSQL')
+		opts = dbname.split()
+		for opt in opts:
+			if '=' in opt:
+				opt = opt.split('=')
+				if opt[0] == 'hostname':
+					db.setHostName(opt[1])
+				elif opt[0] == 'port':
+					db.setPort(int(opt[1]))
+				elif opt[0] == 'username':
+					db.setUserName(opt[1])
+				elif opt[0] == 'password':
+					db.setPassword(opt[1])
+				elif opt[0] == 'dbname':
+					dbname = opt[1]
+			else:
+				dbname = opt
 
 	db.setDatabaseName(dbname)
 	if not db.open():

2.13.5

From: Wang Nan <[email protected]>

Perf's BPF prologue generator unconditionally fetches 8 bytes for function parameters, which causes problems on big endian machines. Thomas gives a detailed analysis for this problem:

http://lkml.kernel.org/r/[email protected]

---- 8< ----
I investigated perf test BPF for s390x and have a question regarding the 38.3 subtest (bpf-prologue test) which fails on s390x.

When I turn on trace_printk in tests/bpf-script-test-prologue.c I see this output in /sys/kernel/debug/tracing/trace:

[root@s8360047 perf]# cat /sys/kernel/debug/tracing/trace perf-30229 [000] d..2 170161.535791: : f_mode 2001d00000000 offset:0 orig:0 perf-30229 [000] d..2 170161.535809: : f_mode 6001f00000000 offset:0 orig:0 perf-30229 [000] d..2 170161.535815: : f_mode 6001f00000000 offset:1 orig:0 perf-30229 [000] d..2 170161.535819: : f_mode 2001d00000000 offset:1 orig:0 perf-30229 [000] d..2 170161.535822: : f_mode 2001d00000000 offset:2 orig:1 perf-30229 [000] d..2 170161.535825: : f_mode 6001f00000000 offset:2 orig:1 perf-30229 [000] d..2 170161.535828: : f_mode 6001f00000000 offset:3 orig:1 perf-30229 [000] d..2 170161.535832: : f_mode 2001d00000000 offset:3 orig:1 perf-30229 [000] d..2 170161.535835: : f_mode 2001d00000000 offset:4 orig:0 perf-30229 [000] d..2 170161.535841: : f_mode 6001f00000000 offset:4 orig:0

[...]

There are 3 parameters the eBPF program tests/bpf-script-test-prologue.c
accesses: f_mode (member of struct file at offset 140) offset and orig. They are parameters of the lseek() system call triggered in this test case in function llseek_loop().

What is really strange is the value of f_mode. It is an 8 byte value, whereas in the probe event it is defined as a 4 byte value. The lower 4 bytes are all zero and do not belong to member f_mode. The correct value should be 2001d for read-only and 6001f for read-write open mode.

Here is the output of the 'perf test -vv bpf' trace:
Try to find probe point from debuginfo.
Matched function: null_lseek [2d9310d]
Probe point found: null_lseek+0
Searching 'file' variable in context.
Converting variable file into trace event.
converting f_mode in file
f_mode type is unsigned int.
Opening /sys/kernel/debug/tracing//README write=0
Searching 'offset' variable in context.
Converting variable offset into trace event.
offset type is long long int.
Searching 'orig' variable in context.
Converting variable orig into trace event.
orig type is int.
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing//kprobe_events write=1 Writing event: p:perf_bpf_probe/func _text+8794224 f_mode=+140(%r2):x32
---- 8< ----

This patch parses the type of each argument and converts data from memory to expected type.

Now the test runs successfully on 4.13.0-rc5:

[root@s8360046 perf]# ./perf test bpf

  38: BPF filter                                 :
  38.1: Basic BPF filtering                      : Ok
  38.2: BPF pinning                              : Ok
  38.3: BPF prologue generation                  : Ok
  38.4: BPF relocation checker                   : Ok

[root@s8360046 perf]#

---
 tools/perf/tests/bpf-script-test-prologue.c |  4 ++-
 tools/perf/util/bpf-prologue.c              | 49 +++++++++++++++++++++++++++--
 2 files changed, 50 insertions(+), 3 deletions(-)

index b4ebc75e25ae..43f1e16486f4 100644
@@ -26,9 +26,11 @@ static void (*bpf_trace_printk)(const char *fmt, int fmt_size, ...) =
 	(void *) 6;
 
 SEC("func=null_lseek file->f_mode offset orig")
-int bpf_func__null_lseek(void *ctx, int err, unsigned long f_mode,
+int bpf_func__null_lseek(void *ctx, int err, unsigned long _f_mode,
 			 unsigned long offset, unsigned long orig)
 {
+	fmode_t f_mode = (fmode_t)_f_mode;
+
 	if (err)
 		return 0;
 	if (f_mode & FMODE_WRITE)

2.13.5

From: Kim Phillips <[email protected]>

Since we do not specify bash (and/or zsh) as a requirement, use the standard error redirection that is more widely supported.

---
 tools/perf/tests/shell/lib/probe.sh                  | 2 +-
 tools/perf/tests/shell/lib/probe_vfs_getname.sh      | 4 ++--
 tools/perf/tests/shell/trace+probe_libc_inet_pton.sh | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

index 61da3b2c9bca..6293cc660947 100644
@@ -1,6 +1,6 @@
 # Arnaldo Carvalho de Melo <[email protected]>, 2017
 
 skip_if_no_perf_probe() {
-	perf probe |& grep -q 'is not a perf-command' && return 2
+	perf probe 2>&1 | grep -q 'is not a perf-command' && return 2
 	return 0
 }

index f832395daad7..30a950c9d407 100644
@@ -1,6 +1,6 @@
 # Arnaldo Carvalho de Melo <[email protected]>, 2017
 
-perf probe -l |& grep -q probe:vfs_getname
+perf probe -l 2>&1 | grep -q probe:vfs_getname
 had_vfs_getname=$?
 
 cleanup_probe_vfs_getname() {

@@ -12,7 +12,7 @@ cleanup_probe_vfs_getname() {
 add_probe_vfs_getname() {
 	local verbose=$1
 	if [ $had_vfs_getname -eq 1 ] ; then
-		line=$(perf probe -L getname_flags |& egrep 'result.*=.*filename;' | sed -r 's/[[:space:]]+([[:digit:]]+)[[:space:]]+result->uptr.*/\1/')
+		line=$(perf probe -L getname_flags 2>&1 | egrep 'result.*=.*filename;' | sed -r 's/[[:space:]]+([[:digit:]]+)[[:space:]]+result->uptr.*/\1/')
 		perf probe $verbose "vfs_getname=getname_flags:${line} pathname=result->name:string"
 	fi
 }

index 99dafad61954..462fc755092e 100755
@@ -22,7 +22,7 @@ trace_libc_inet_pton_backtrace() {
 	expected[7]="getaddrinfo[[:space:]]\(/usr/lib.*/libc-[0-9]+\.[0-9]+\.so\)$"
 	expected[8]=".*\(.*/bin/ping.*\)$"
 
-	perf trace --no-syscalls -e probe_libc:inet_pton/max-stack=3/ ping -6 -c 1 ::1 |& grep -v ^$ | while read line ; do
+	perf trace --no-syscalls -e probe_libc:inet_pton/max-stack=3/ ping -6 -c 1 ::1 2>&1 | grep -v ^$ | while read line ; do
 		echo $line
 		echo "$line" | egrep -q "${expected[$idx]}"
 		if [ $? -ne 0 ] ; then

2.13.5

Pulled, thanks a lot Arnaldo!

Ingo