perf/core improvements and fixes

Hi Ingo,

Please consider pulling,

- Arnaldo

Test results at the end of this message, as usual.

* The following description will move to the end in the next pull requests *

The first ones are container (docker) based builds of tools/perf with and without libelf support, objtool where it is supported and samples/bpf/, ditto.

Several are cross builds, the ones with -x-ARCH, and the android one, and those may not have all the features built, due to lack of multi-arch devel packages, available and being used so far on just a few, like debian:experimental-x-{arm64,mipsel}.

The 'perf test' one will perform a variety of tests exercising tools/perf/util/, tools/lib/{bpf,traceevent,etc}, as well as run perf commands with a variety of command line event specifications to then intercept the sys_perf_event syscall to check that the perf_event_attr fields are set up as expected, among a variety of other unit tests.

Then there is the 'make -C tools/perf build-test' ones, that build tools/perf/ with a variety of feature sets, exercising the build with an incomplete set of features as well as with a complete one. It is planned to have it run on each of the containers mentioned above, using some container orchestration infrastructure. Get in contact if interested in helping having this in place.

The following changes since commit ad5013d5699d30ded0cdbbc68b93b2aa28222c6e:

perf/x86/intel: Use ULL constant to prevent undefined shift behaviour (2017-01-11 16:43:30 +0100)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git tags/perf-core-for-mingo-4.11-20170111

for you to fetch changes up to 675f52b23743f396c585fc9d135435be37f320d8:

tools: Sync x86's vmx.h with the kernel (2017-01-11 16:48:02 -0300)

----------------------------------------------------------------
perf/core improvements and fixes:

New features:

- Add more triggers to switch the output file (perf.data.TIMESTAMP).

Now, in addition to switching to a different output file when receiving a SIGUSR2, one can also specify file size and time based triggers:

       perf record -a --switch-output=signal

is equivalent to what we had before:

       perf record -a --switch-output

While we can also ask for the file to be "sliced" by size, taking into account that that will happen only when we get woken up by the kernel, i.e. one has to take into account the --mmap-pages (the size of the perf mmap ring buffer):

       perf record -a --switch-output=2G

will break the perf.data output into multiple files limited to 2GB of samples, right when generating the output.

For time based samples, alert() will be used, so to have 1 minute limited perf.data output files:

      perf record -a --switch-output=1m

(Jiri Olsa)

- Remove the need to use -e only for syscalls and --event only for tracepoints/HW/SW/etc events, i.e. now one can use:

perf trace -e nanosleep,futex,sched:sched_switch ./workload

or:

      perf trace --event nanosleep,futex,sched:sched_switch ./workload

And have it tracing raw_syscalls:sys_{enter,exit} for the nanosleep and futex syscalls, formatting those as strace does while also tracing sched:sched_switch, ordering it all into one strace like output.

Using '!' as the first character in the -e/--event argument remains a way to negate the list of syscalls, i.e. all syscalls except for the ones specified, doesn't affect the other kinds of events.

E.g:

[root@jouet ~] # perf trace -e sched:sched_switch,nanosleep usleep 1

     0.000 ( 0.028 ms): usleep/28150 nanosleep(rqtp: 0x7ffe4201b9f0) ...
     0.028 (         ): sched:sched_switch:usleep:28150 [120] S ==> swapper/0:0 [120])
     0.000 ( 0.065 ms): usleep/28150  ... [continued]: nanosleep()) = 0

[root@jouet ~]#

(Arnaldo Carvalho de Melo)

- 'perf kallsyms' toy tool to look for extended symbol information on the running kernel and demonstrate the machine/thread/symbol APIs for use in other tools, such as 'perf probe' (Arnaldo Carvalho de Melo)

Infrastructure:

- Add missing linux/kernel.h include to subcmd.h (Arnaldo Carvalho de Melo)
tools: Sync x86's vmx.h with the kernel

- Create libdir directory before installing libperf-jvmti.so (Laura Abbott)

- Fix typo in perf_evlist__start_workload() (Soramichi Akiyama)

----------------------------------------------------------------
Arnaldo Carvalho de Melo (5):
tools lib subcmd: Add missing linux/kernel.h include to subcmd.h
perf machine: Add a kallsyms loading constructor
perf kallsyms: Introduce tool to look for extended symbol information on the running kernel perf trace: Allow specifying list of syscalls and events in -e/--expr/--event
tools: Sync x86's vmx.h with the kernel

Jiri Olsa (6):
perf tools: Add unit_number__scnprintf function
perf record: Add struct switch_output
perf record: Change switch-output option to take optional argument perf record: Add switch-output size option argument perf record: Add switch-output size warning
perf record: Add switch-output time option argument

Laura Abbott (1):
perf jvmti: Create libdir directory before installing libperf-jvmti.so

Soramichi Akiyama (1):
perf evlist: Fix typo in perf_evlist__start_workload()

 tools/arch/x86/include/uapi/asm/vmx.h      |   5 +
 tools/lib/subcmd/parse-options.h           |   1 +
 tools/perf/Build                           |   1 +
 tools/perf/Documentation/perf-kallsyms.txt |  24 +++++
 tools/perf/Documentation/perf-record.txt   |  14 ++-
 tools/perf/Documentation/perf-trace.txt    |   8 +-
 tools/perf/Makefile.perf                   |   1 +
 tools/perf/builtin-help.c                  |   2 +-
 tools/perf/builtin-kallsyms.c              |  67 +++++++++++++
 tools/perf/builtin-record.c                | 154 ++++++++++++++++++++++++++---
 tools/perf/builtin-trace.c                 | 120 ++++++++++++++++------
 tools/perf/builtin.h                       |   1 +
 tools/perf/command-list.txt                |   1 +
 tools/perf/perf.c                          |   1 +
 tools/perf/tests/Build                     |   1 +
 tools/perf/tests/builtin-test.c            |   4 +
 tools/perf/tests/tests.h                   |   1 +
 tools/perf/tests/unit_number__scnprintf.c  |  37 +++++++
 tools/perf/util/evlist.c                   |  12 ++-
 tools/perf/util/evlist.h                   |   2 +
 tools/perf/util/machine.c                  |  19 ++++
 tools/perf/util/machine.h                  |   1 +
 tools/perf/util/util.c                     |  13 +++
 tools/perf/util/util.h                     |   1 +
 24 files changed, 439 insertions(+), 52 deletions(-)
 create mode 100644 tools/perf/Documentation/perf-kallsyms.txt
 create mode 100644 tools/perf/builtin-kallsyms.c
 create mode 100644 tools/perf/tests/unit_number__scnprintf.c

  # uname -a
  Linux jouet 4.9.0+ #2 SMP Wed Dec 21 11:54:44 BRT 2016 x86_64 x86_64 x86_64 GNU/Linux
  # perf test
   1: vmlinux symtab matches kallsyms            : Ok
   2: Detect openat syscall event                : Ok
   3: Detect openat syscall event on all cpus    : Ok
   4: Read samples using the mmap interface      : Ok
   5: Parse event definition strings             : Ok
   6: PERF_RECORD_* events & perf_sample fields  : Ok
   7: Parse perf pmu format                      : Ok
   8: DSO data read                              : Ok
   9: DSO data cache                             : Ok
  10: DSO data reopen                            : Ok
  11: Roundtrip evsel->name                      : Ok
  12: Parse sched tracepoints fields             : Ok
  13: syscalls:sys_enter_openat event fields     : Ok
  14: Setup struct perf_event_attr               : Ok
  15: Match and link multiple hists              : Ok
  16: 'import perf' in python                    : Ok
  17: Breakpoint overflow signal handler         : Ok
  18: Breakpoint overflow sampling               : Ok
  19: Number of exit events of a simple workload : Ok
  20: Software clock events period values        : Ok
  21: Object code reading                        : Ok
  22: Sample parsing                             : Ok
  23: Use a dummy software event to keep tracking: Ok
  24: Parse with no sample_id_all bit set        : Ok
  25: Filter hist entries                        : Ok
  26: Lookup mmap thread                         : Ok
  27: Share thread mg                            : Ok
  28: Sort output of hist entries                : Ok
  29: Cumulate child hist entries                : Ok
  30: Track with sched_switch                    : Ok
  31: Filter fds with revents mask in a fdarray  : Ok
  32: Add fd to a fdarray, making it autogrow    : Ok
  33: kmod_path__parse                           : Ok
  34: Thread map                                 : Ok
  35: LLVM search and compile                    :
  35.1: Basic BPF llvm compile                    : Ok
  35.2: kbuild searching                          : Ok
  35.3: Compile source for BPF prologue generation: Ok
  35.4: Compile source for BPF relocation         : Ok
  36: Session topology                           : Ok
  37: BPF filter                                 :
  37.1: Basic BPF filtering                      : Ok
  37.2: BPF prologue generation                  : Ok
  37.3: BPF relocation checker                   : Ok
  38: Synthesize thread map                      : Ok
  39: Remove thread map                          : Ok
  40: Synthesize cpu map                         : Ok
  41: Synthesize stat config                     : Ok
  42: Synthesize stat                            : Ok
  43: Synthesize stat round                      : Ok
  44: Synthesize attr update                     : Ok
  45: Event times                                : Ok
  46: Read backward ring buffer                  : Ok
  47: Print cpu map                              : Ok
  48: Probe SDT events                           : Ok
  49: is_printable_array                         : Ok
  50: Print bitmap                               : Ok
  51: perf hooks                                 : Ok
  52: builtin clang support                      : Skip (not compiled in)
  53: unit_number__scnprintf                     : Ok
  54: x86 rdpmc                                  : Ok
  55: Convert perf time to TSC                   : Ok
  56: DWARF unwind                               : Ok
  57: x86 instruction decoder - new instructions : Ok
  58: Intel cqm nmi context read                 : Skip
  #

[root@jouet ~]# dm
1 alpine:3.4: Ok
2 android-ndk:r12b-arm: Ok
3 archlinux:latest: Ok
4 centos:5: Ok
5 centos:6: Ok
6 centos:7: Ok
7 debian:7: Ok
8 debian:8: Ok
9 debian:experimental: Ok
10 debian:experimental-x-arm64: Ok
11 debian:experimental-x-mips64: Ok
12 debian:experimental-x-mipsel: Ok
13 fedora:20: Ok
14 fedora:21: Ok
15 fedora:22: Ok
16 fedora:23: Ok
17 fedora:24: Ok
18 fedora:24-x-ARC-uClibc: Ok
19 fedora:25: Ok
20 fedora:rawhide: Ok
21 mageia:5: Ok
22 opensuse:13.2: Ok
23 opensuse:42.1: Ok
24 opensuse:tumbleweed: Ok
25 ubuntu:12.04.5: Ok
26 ubuntu:14.04.4-x-linaro-arm64: Ok
27 ubuntu:15.10: Ok
28 ubuntu:16.04: Ok
29 ubuntu:16.04-x-arm: Ok
30 ubuntu:16.04-x-arm64: Ok
31 ubuntu:16.04-x-powerpc: Ok
32 ubuntu:16.04-x-powerpc64: Ok
33 ubuntu:16.04-x-powerpc64el: Ok
34 ubuntu:16.04-x-s390: Ok
35 ubuntu:16.10: Ok
#

  $ make -C tools/perf build-test
  make: Entering directory '/home/acme/git/linux/tools/perf'
  - tarpkg: ./tests/perf-targz-src-pkg .
            make_no_libaudit_O: make NO_LIBAUDIT=1
                 make_static_O: make LDFLAGS=-static
       make_util_pmu_bison_o_O: make util/pmu-bison.o
        make_with_babeltrace_O: make LIBBABELTRACE=1
         make_install_prefix_O: make install prefix=/tmp/krava
                  make_debug_O: make DEBUG=1
             make_no_scripts_O: make NO_LIBPYTHON=1 NO_LIBPERL=1
           make_no_libunwind_O: make NO_LIBUNWIND=1
                   make_help_O: make help
                    make_doc_O: make doc
                make_no_newt_O: make NO_NEWT=1
            make_no_demangle_O: make NO_DEMANGLE=1
            make_install_bin_O: make install-bin
              make_no_libbpf_O: make NO_LIBBPF=1
            make_no_auxtrace_O: make NO_AUXTRACE=1
                 make_perf_o_O: make perf.o
                   make_pure_O: make
              make_clean_all_O: make clean all
           make_no_libbionic_O: make NO_LIBBIONIC=1
                make_install_O: make install
   make_install_prefix_slash_O: make install prefix=/tmp/krava/
             make_util_map_o_O: make util/map.o
             make_no_libnuma_O: make NO_LIBNUMA=1
                  make_no_ui_O: make NO_NEWT=1 NO_SLANG=1 NO_GTK2=1
         make_with_clangllvm_O: make LIBCLANGLLVM=1
                make_minimal_O: make NO_LIBPERL=1 NO_LIBPYTHON=1 NO_NEWT=1 NO_GTK2=1 NO_DEMANGLE=1 NO_LIBELF=1 NO_LIBUNWIND=1 NO_BACKTRACE=1 NO_LIBNUMA=1 NO_LIBAUDIT=1 NO_LIBBIONIC=1 NO_LIBDW_DWARF_UNWIND=1 NO_AUXTRACE=1 NO_LIBBPF=1 NO_LIBCRYPTO=1 NO_SDT=1 NO_JVMTI=1
                make_no_gtk2_O: make NO_GTK2=1
           make_no_libpython_O: make NO_LIBPYTHON=1
           make_no_backtrace_O: make NO_BACKTRACE=1
             make_no_libperl_O: make NO_LIBPERL=1
               make_no_slang_O: make NO_SLANG=1
                   make_tags_O: make tags
              make_no_libelf_O: make NO_LIBELF=1
  make_no_libdw_dwarf_unwind_O: make NO_LIBDW_DWARF_UNWIND=1
  OK
  make: Leaving directory '/home/acme/git/linux/tools/perf'
  $

From: Laura Abbott <[email protected]>

The install command for libperf-jvmti.so does not check if libdir exists before installing. This means that when the install command is run:

install libperf-jvmti.so '/tmp/test_root/usr/lib64';

libperf-jvmti.so will get installed to /usr/lib64 as a file and break further installation. Fix this by ensuring the directory gets created first.

See https://bugzilla.redhat.com/show_bug.cgi?id=1410296

---
 tools/perf/Makefile.perf | 1 +
 1 file changed, 1 insertion(+)

index 8bb16aa9d661..4da19b6ba94a 100644
@@ -661,6 +661,7 @@ ifndef NO_PERF_READ_VDSOX32
 endif
 ifndef NO_JVMTI
 	$(call QUIET_INSTALL, $(LIBJVMTI)) \
+		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(libdir_SQ)'; \
 		$(INSTALL) $(OUTPUT)$(LIBJVMTI) '$(DESTDIR_SQ)$(libdir_SQ)';
 endif
 	$(call QUIET_INSTALL, libexec) \

2.9.3

From: Arnaldo Carvalho de Melo <[email protected]>

As it was getting the BUILD_BUG_ON_ZERO() definition by luck.

---
 tools/lib/subcmd/parse-options.h | 1 +
 1 file changed, 1 insertion(+)

index 11c3be3bcce7..37e2d1a6fc2a 100644
@@ -1,6 +1,7 @@
 #ifndef __SUBCMD_PARSE_OPTIONS_H
 #define __SUBCMD_PARSE_OPTIONS_H
 
+#include <linux/kernel.h>
 #include <stdbool.h>
 #include <stdint.h>
 

2.9.3

From: Arnaldo Carvalho de Melo <[email protected]>

To reduce the boilerplate for searching for functions in the running kernel and modules.

---
 tools/perf/util/machine.c | 19 +++++++++++++++++++
 tools/perf/util/machine.h |  1 +
 2 files changed, 20 insertions(+)

index 9b33bef54581..747a034d1ff3 100644
@@ -87,6 +87,25 @@ struct machine *machine__new_host(void)
 	return NULL;
 }
 
+struct machine *machine__new_kallsyms(void)
+{
+	struct machine *machine = machine__new_host();
+	/*
+	 * FIXME:
+	 * 1) MAP__FUNCTION will go away when we stop loading separate maps for
+	 *    functions and data objects.
+	 * 2) We should switch to machine__load_kallsyms(), i.e. not explicitely
+	 *    ask for not using the kcore parsing code, once this one is fixed
+	 *    to create a map per module.
+	 */
+	if (machine && __machine__load_kallsyms(machine, "/proc/kallsyms", MAP__FUNCTION, true) <= 0) {
+		machine__delete(machine);
+		machine = NULL;
+	}
+
+	return machine;
+}
+
 static void dsos__purge(struct dsos *dsos)
 {
 	struct dso *pos, *n;

index 354de6e56109..a28305029711 100644
@@ -129,6 +129,7 @@ char *machine__mmap_name(struct machine *machine, char *bf, size_t size);
 void machines__set_comm_exec(struct machines *machines, bool comm_exec);
 
 struct machine *machine__new_host(void);
+struct machine *machine__new_kallsyms(void);
 int machine__init(struct machine *machine, const char *root_dir, pid_t pid);
 void machine__exit(struct machine *machine);
 void machine__delete_threads(struct machine *machine);

2.9.3

From: Arnaldo Carvalho de Melo <[email protected]>

Its similar to doing grep on a /proc/kallsyms, but it also shows extra information like the path to the kernel module and the unrelocated addresses in it, to help in diagnosing problems.

It is also helps demonstrate the use of the symbols routines so that tool writers can use them more effectively.

Using it:

  $ perf kallsyms e1000_xmit_frame netif_rx usb_stor_set_xfer_buf
  e1000_xmit_frame: [e1000e] /lib/modules/4.9.0+/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 0xffffffffc046fc10-0xffffffffc0470bb0 (0x19c80-0x1ac20)
  netif_rx: [kernel] [kernel.kallsyms] 0xffffffff916f03a0-0xffffffff916f0410 (0xffffffff916f03a0-0xffffffff916f0410)
  usb_stor_set_xfer_buf: [usb_storage] /lib/modules/4.9.0+/kernel/drivers/usb/storage/usb-storage.ko 0xffffffffc057aea0-0xffffffffc057af19 (0xf10-0xf89)
  $

---
 tools/perf/Build                           |  1 +
 tools/perf/Documentation/perf-kallsyms.txt | 24 +++++++++++
 tools/perf/builtin-help.c                  |  2 +-
 tools/perf/builtin-kallsyms.c              | 67 ++++++++++++++++++++++++++++++
 tools/perf/builtin.h                       |  1 +
 tools/perf/command-list.txt                |  1 +
 tools/perf/perf.c                          |  1 +
 7 files changed, 96 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/Documentation/perf-kallsyms.txt
 create mode 100644 tools/perf/builtin-kallsyms.c

index b12d5d1666e3..0b48806f93c2 100644
@@ -7,6 +7,7 @@ perf-y += builtin-help.o
 perf-y += builtin-sched.o
 perf-y += builtin-buildid-list.o
 perf-y += builtin-buildid-cache.o
+perf-y += builtin-kallsyms.o
 perf-y += builtin-list.o
 perf-y += builtin-record.o
 perf-y += builtin-report.o

new file mode 100644
index 000000000000..954ea9e21236
@@ -0,0 +1,24 @@
+perf-kallsyms(1)
+==============
+
+NAME
+----
+perf-kallsyms - Searches running kernel for symbols
+
+SYNOPSIS
+--------
+[verse]
+'perf kallsyms <options> symbol_name[,symbol_name...]'
+
+DESCRIPTION
+-----------
+This command searches the running kernel kallsyms file for the given symbol(s)
+and prints information about it, including the DSO, the kallsyms begin/end
+addresses and the addresses in the ELF kallsyms symbol table (for symbols in
+modules).
+
+OPTIONS
+-------
+-v::
+--verbose=::
+	Increase verbosity level, showing details about symbol table loading, etc.

index 3bdb2c78a21b..93da24a638be 100644
@@ -434,7 +434,7 @@ int cmd_help(int argc, const char **argv, const char *prefix __maybe_unused)
 	const char * const builtin_help_subcommands[] = {
 		"buildid-cache", "buildid-list", "diff", "evlist", "help", "list",
 		"record", "report", "bench", "stat", "timechart", "top", "annotate",
-		"script", "sched", "kmem", "lock", "kvm", "test", "inject", "mem", "data",
+		"script", "sched", "kallsyms", "kmem", "lock", "kvm", "test", "inject", "mem", "data",
 #ifdef HAVE_LIBELF_SUPPORT
 		"probe",
 #endif

index 0bcf68e98ccc..b55f5be486a1 100644
@@ -23,6 +23,7 @@ int cmd_diff(int argc, const char **argv, const char *prefix);
 int cmd_evlist(int argc, const char **argv, const char *prefix);
 int cmd_help(int argc, const char **argv, const char *prefix);
 int cmd_sched(int argc, const char **argv, const char *prefix);
+int cmd_kallsyms(int argc, const char **argv, const char *prefix);
 int cmd_list(int argc, const char **argv, const char *prefix);
 int cmd_record(int argc, const char **argv, const char *prefix);
 int cmd_report(int argc, const char **argv, const char *prefix);

index ab5cbaa170d0..fb45613dba9e 100644
@@ -12,6 +12,7 @@ perf-diff			mainporcelain common
 perf-config			mainporcelain common
 perf-evlist			mainporcelain common
 perf-inject			mainporcelain common
+perf-kallsyms			mainporcelain common
 perf-kmem			mainporcelain common
 perf-kvm			mainporcelain common
 perf-list			mainporcelain common

index aa23b3347d6b..13c8a7db055e 100644
@@ -47,6 +47,7 @@ static struct cmd_struct commands[] = {
 	{ "diff",	cmd_diff,	0 },
 	{ "evlist",	cmd_evlist,	0 },
 	{ "help",	cmd_help,	0 },
+	{ "kallsyms",	cmd_kallsyms,	0 },
 	{ "list",	cmd_list,	0 },
 	{ "record",	cmd_record,	0 },
 	{ "report",	cmd_report,	0 },

2.9.3

From: Arnaldo Carvalho de Melo <[email protected]>

Makes it easier to specify both events and syscalls (to be formatter strace-like), i.e. previously one would have to do:

  # perf trace -e nanosleep --event sched:sched_switch usleep 1

Now it is possible to do:

  # perf trace -e nanosleep,sched:sched_switch usleep 1
     0.000 ( 0.021 ms): usleep/17962 nanosleep(rqtp: 0x7ffdedd61ec0) ...
     0.021 (         ): sched:sched_switch:usleep:17962 [120] S ==> swapper/1:0 [120])
     0.000 ( 0.066 ms): usleep/17962  ... [continued]: nanosleep()) = 0
  #

The old style --expr and using both -e and --event continues to work.

---
 tools/perf/Documentation/perf-trace.txt |   8 +--
 tools/perf/builtin-trace.c              | 120 ++++++++++++++++++++++++--------
 2 files changed, 96 insertions(+), 32 deletions(-)

index 781b019751a4..afd728672b6f 100644
@@ -35,7 +35,10 @@ OPTIONS
 
 -e::
 --expr::
-	List of syscalls to show, currently only syscall names.
+--event::
+	List of syscalls and other perf events (tracepoints, HW cache events,
+	etc) to show.
+	See 'perf list' for a complete list of events.
 	Prefixing with ! shows all syscalls but the ones specified.  You may
 	need to escape it.
 

@@ -135,9 +138,6 @@ the thread executes on the designated CPUs. Default is to monitor all CPUs.
 --kernel-syscall-graph::
 	 Show the kernel callchains on the syscall exit path.
 
---event::
-	Trace other events, see 'perf list' for a complete list.
-
 --max-stack::
         Set the stack depth limit when parsing the callchain, anything
         beyond the specified depth will be ignored. Note that at this point

2.9.3

From: Soramichi Akiyama <[email protected]>

This patch fixes a typo: s/enable to/unable to/

---
 tools/perf/util/evlist.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index d92e02006fb8..23e6f33edcf2 100644
@@ -1797,7 +1797,7 @@ int perf_evlist__start_workload(struct perf_evlist *evlist)
 		 */
 		ret = write(evlist->workload.cork_fd, &bf, 1);
 		if (ret < 0)
-			perror("enable to write to pipe");
+			perror("unable to write to pipe");
 
 		close(evlist->workload.cork_fd);
 		return ret;

2.9.3

From: Jiri Olsa <[email protected]>

Add unit_number__scnprintf function to display size units and use it in -m option info message.

Before:

  $ perf record -m 10M ls
  rounding mmap pages size to 16777216 bytes (4096 pages)
  ...

After:

  $ perf record -m 10M ls
  rounding mmap pages size to 16M (4096 pages)
  ...

---
 tools/perf/tests/Build                    |  1 +
 tools/perf/tests/builtin-test.c           |  4 ++++
 tools/perf/tests/tests.h                  |  1 +
 tools/perf/tests/unit_number__scnprintf.c | 37 +++++++++++++++++++++++++++++++
 tools/perf/util/evlist.c                  |  8 +++++--
 tools/perf/util/util.c                    | 13 +++++++++++
 tools/perf/util/util.h                    |  1 +
 7 files changed, 63 insertions(+), 2 deletions(-)
 create mode 100644 tools/perf/tests/unit_number__scnprintf.c

index 6676c2dd6dcb..1cb3d9b540e9 100644
@@ -44,6 +44,7 @@ perf-y += is_printable_array.o
 perf-y += bitmap.o
 perf-y += perf-hooks.o
 perf-y += clang.o
+perf-y += unit_number__scnprintf.o
 
 $(OUTPUT)tests/llvm-src-base.c: tests/bpf-script-example.c tests/Build
 	$(call rule_mkdir)

index a77dcc0d24e3..37e326bfd2dc 100644
@@ -247,6 +247,10 @@ static struct test generic_tests[] = {
 		}
 	},
 	{
+		.desc = "unit_number__scnprintf",
+		.func = test__unit_number__scnprint,
+	},
+	{
 		.func = NULL,
 	},
 };

index a512f0c8ff5b..1fa9b9d83aa5 100644
@@ -96,6 +96,7 @@ int test__perf_hooks(int subtest);
 int test__clang(int subtest);
 const char *test__clang_subtest_get_desc(int subtest);
 int test__clang_subtest_get_nr(void);
+int test__unit_number__scnprint(int subtest);
 
 #if defined(__arm__) || defined(__aarch64__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT

new file mode 100644
index 000000000000..623c2aa53c4a
@@ -0,0 +1,37 @@
+#include <linux/compiler.h>
+#include <linux/types.h>
+#include "tests.h"
+#include "util.h"
+#include "debug.h"
+
+int test__unit_number__scnprint(int subtest __maybe_unused)
+{
+	struct {
+		u64		 n;
+		const char	*str;
+	} test[] = {
+		{ 1,			"1B"	},
+		{ 10*1024,		"10K"	},
+		{ 20*1024*1024,		"20M"	},
+		{ 30*1024*1024*1024ULL,	"30G"	},
+		{ 0,			"0B"	},
+		{ 0,			NULL	},
+	};
+	unsigned i = 0;
+
+	while (test[i].str) {
+		char buf[100];
+
+		unit_number__scnprintf(buf, sizeof(buf), test[i].n);
+
+		pr_debug("n %" PRIu64 ", str '%s', buf '%s'\n",
+			 test[i].n, test[i].str, buf);
+
+		if (strcmp(test[i].str, buf))
+			return TEST_FAIL;
+
+		i++;
+	}
+
+	return TEST_OK;
+}

index 23e6f33edcf2..dc4df3d2660e 100644
@@ -1224,12 +1224,16 @@ static long parse_pages_arg(const char *str, unsigned long min,
 	if (pages == 0 && min == 0) {
 		/* leave number of pages at 0 */
 	} else if (!is_power_of_2(pages)) {
+		char buf[100];
+
 		/* round pages up to next power of 2 */
 		pages = roundup_pow_of_two(pages);
 		if (!pages)
 			return -EINVAL;
-		pr_info("rounding mmap pages size to %lu bytes (%lu pages)\n",
-			pages * page_size, pages);
+
+		unit_number__scnprintf(buf, sizeof(buf), pages * page_size);
+		pr_info("rounding mmap pages size to %s (%lu pages)\n",
+			buf, pages);
 	}
 
 	if (pages > max)

index 9ddd98827d12..bf29aed16bd6 100644
@@ -789,3 +789,16 @@ int is_printable_array(char *p, unsigned int len)
 	}
 	return 1;
 }
+
+int unit_number__scnprintf(char *buf, size_t size, u64 n)
+{
+	char unit[4] = "BKMG";
+	int i = 0;
+
+	while (((n / 1024) > 1) && (i < 3)) {
+		n /= 1024;
+		i++;
+	}
+
+	return scnprintf(buf, size, "%" PRIu64 "%c", n, unit[i]);
+}

index 1d639e38aa82..6e8be174ec0b 100644
@@ -363,4 +363,5 @@ int is_printable_array(char *p, unsigned int len);
 
 int timestamp__scnprintf_usec(u64 timestamp, char *buf, size_t sz);
 
+int unit_number__scnprintf(char *buf, size_t size, u64 n);
 #endif /* GIT_COMPAT_UTIL_H */

2.9.3

From: Jiri Olsa <[email protected]>

Next patches will add more --switch-output option arguments, so preparing the data holder.

---
 tools/perf/builtin-record.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

index 4ec10e9427d9..f7e805b30527 100644
@@ -46,6 +46,10 @@
 #include <asm/bug.h>
 #include <linux/time64.h>
 
+struct switch_output {
+	bool		 signal;
+};
+
 struct record {
 	struct perf_tool	tool;
 	struct record_opts	opts;

@@ -62,7 +66,7 @@ struct record {
 	bool			no_buildid_cache_set;
 	bool			buildid_all;
 	bool			timestamp_filename;
-	bool			switch_output;
+	struct switch_output	switch_output;
 	unsigned long long	samples;
 };
 

@@ -842,11 +846,11 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	signal(SIGTERM, sig_handler);
 	signal(SIGSEGV, sigsegv_handler);
 
-	if (rec->opts.auxtrace_snapshot_mode || rec->switch_output) {
+	if (rec->opts.auxtrace_snapshot_mode || rec->switch_output.signal) {
 		signal(SIGUSR2, snapshot_sig_handler);
 		if (rec->opts.auxtrace_snapshot_mode)
 			trigger_on(&auxtrace_snapshot_trigger);
-		if (rec->switch_output)
+		if (rec->switch_output.signal)
 			trigger_on(&switch_output_trigger);
 	} else {
 		signal(SIGUSR2, SIG_IGN);

@@ -1519,7 +1523,7 @@ static struct option __record_options[] = {
 		    "Record build-id of all DSOs regardless of hits"),
 	OPT_BOOLEAN(0, "timestamp-filename", &record.timestamp_filename,
 		    "append timestamp to output filename"),
-	OPT_BOOLEAN(0, "switch-output", &record.switch_output,
+	OPT_BOOLEAN(0, "switch-output", &record.switch_output.signal,
 		    "Switch output when receive SIGUSR2"),
 	OPT_BOOLEAN(0, "dry-run", &dry_run,
 		    "Parse options then exit"),

@@ -1578,7 +1582,7 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 		return -EINVAL;
 	}
 
-	if (rec->switch_output)
+	if (rec->switch_output.signal)
 		rec->timestamp_filename = true;
 
 	if (!rec->itr) {

@@ -1629,7 +1633,7 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 
 	if (rec->no_buildid_cache || rec->no_buildid) {
 		disable_buildid_cache();
-	} else if (rec->switch_output) {
+	} else if (rec->switch_output.signal) {
 		/*
 		 * In 'perf record --switch-output', disable buildid
 		 * generation by default to reduce data file switching

2.9.3

From: Jiri Olsa <[email protected]>

Next patches will add --switch-output option arguments, changing the option to allow that and adding its default value to 'signal'.

---
 tools/perf/builtin-record.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

index f7e805b30527..2bf811acaf8d 100644
@@ -48,6 +48,8 @@
 
 struct switch_output {
 	bool		 signal;
+	const char	*str;
+	bool		 set;
 };
 
 struct record {

@@ -1356,6 +1358,22 @@ static int record__parse_mmap_pages(const struct option *opt,
 	return ret;
 }
 
+static int switch_output_setup(struct record *rec)
+{
+	struct switch_output *s = &rec->switch_output;
+
+	if (!s->set)
+		return 0;
+
+	if (!strcmp(s->str, "signal")) {
+		s->signal = true;
+		pr_debug("switch-output with SIGUSR2 signal\n");
+		return 0;
+	}
+
+	return -1;
+}
+
 static const char * const __record_usage[] = {
 	"perf record [<options>] [<command>]",
 	"perf record [<options>] -- <command> [<options>]",

@@ -1523,8 +1541,9 @@ static struct option __record_options[] = {
 		    "Record build-id of all DSOs regardless of hits"),
 	OPT_BOOLEAN(0, "timestamp-filename", &record.timestamp_filename,
 		    "append timestamp to output filename"),
-	OPT_BOOLEAN(0, "switch-output", &record.switch_output.signal,
-		    "Switch output when receive SIGUSR2"),
+	OPT_STRING_OPTARG_SET(0, "switch-output", &record.switch_output.str,
+			  &record.switch_output.set, "signal",
+			  "Switch output when receive SIGUSR2", "signal"),
 	OPT_BOOLEAN(0, "dry-run", &dry_run,
 		    "Parse options then exit"),
 	OPT_END()

@@ -1582,6 +1601,11 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 		return -EINVAL;
 	}
 
+	if (switch_output_setup(rec)) {
+		parse_options_usage(record_usage, record_options, "switch-output", 0);
+		return -EINVAL;
+	}
+
 	if (rec->switch_output.signal)
 		rec->timestamp_filename = true;
 

2.9.3

From: Jiri Olsa <[email protected]>

It's now possible to specify the threshold size for perf.data like:

  $ perf record --switch-output=2G ...

Once it's reached, the current data are dumped in to the perf.data.<timestamp> file and session does on.

  $ perf record --switch-output=2G ...
  [ perf record: dump data: Woken up 7244 times ]
  [ perf record: Dump perf.data.2017010214093746 ]
  ...

The size is expected to be a number with appended unit character - B/K/M/G.

---
 tools/perf/Documentation/perf-record.txt | 12 +++++-
 tools/perf/builtin-record.c              | 67 +++++++++++++++++++++++++-------
 2 files changed, 63 insertions(+), 16 deletions(-)

index 5054d9147f0f..3d55d2fd48b3 100644
@@ -421,9 +421,17 @@ Configure all used events to run in user space.
 --timestamp-filename
 Append timestamp to output file name.
 
---switch-output::
+--switch-output[=mode]::
 Generate multiple perf.data files, timestamp prefixed, switching to a new one
-when receiving a SIGUSR2.
+based on 'mode' value:
+  "signal" - when receiving a SIGUSR2 (default value) or
+  <size>   - when reaching the size threshold, size is expected to
+             be a number with appended unit character - B/K/M/G
+
+             Note: the precision of  the size  threshold  hugely depends
+             on your configuration  - the number and size of  your  ring
+             buffers (-m). It is generally more precise for higher sizes
+             (like >5M), for lower values expect different sizes.
 
 A possible use case is to, given an external event, slice the perf.data file
 that gets then processed, possibly via a perf script, to decide if that

2.9.3

From: Jiri Olsa <[email protected]>

Adding switch-output size warning if the requested size of lower than the wakeup ring buffer size.

  $ perf record --switch-output=1K ls
  WARNING: switch-output data size lower than wakeup kernel buffer size (258K) expect bigger perf.data sizes
  ...

---
 tools/perf/builtin-record.c | 21 +++++++++++++++++++++
 tools/perf/util/evlist.c    |  2 +-
 tools/perf/util/evlist.h    |  2 ++
 3 files changed, 24 insertions(+), 1 deletion(-)

index 3fa64492ee62..93319e1be3ac 100644
@@ -1377,6 +1377,23 @@ static int record__parse_mmap_pages(const struct option *opt,
 	return ret;
 }
 
+static void switch_output_size_warn(struct record *rec)
+{
+	u64 wakeup_size = perf_evlist__mmap_size(rec->opts.mmap_pages);
+	struct switch_output *s = &rec->switch_output;
+
+	wakeup_size /= 2;
+
+	if (s->size < wakeup_size) {
+		char buf[100];
+
+		unit_number__scnprintf(buf, sizeof(buf), wakeup_size);
+		pr_warning("WARNING: switch-output data size lower than "
+			   "wakeup kernel buffer size (%s) "
+			   "expect bigger perf.data sizes\n", buf);
+	}
+}
+
 static int switch_output_setup(struct record *rec)
 {
 	struct switch_output *s = &rec->switch_output;

@@ -1410,6 +1427,10 @@ static int switch_output_setup(struct record *rec)
 enabled:
 	rec->timestamp_filename = true;
 	s->enabled              = true;
+
+	if (s->size && !rec->opts.no_buffering)
+		switch_output_size_warn(rec);
+
 	return 0;
 }
 

index dc4df3d2660e..b601f2814a30 100644
@@ -1184,7 +1184,7 @@ unsigned long perf_event_mlock_kb_in_pages(void)
 	return pages;
 }
 
-static size_t perf_evlist__mmap_size(unsigned long pages)
+size_t perf_evlist__mmap_size(unsigned long pages)
 {
 	if (pages == UINT_MAX)
 		pages = perf_event_mlock_kb_in_pages();

index 4fd034f22d2f..389b9ccdf8c7 100644
@@ -218,6 +218,8 @@ int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
 		      bool overwrite);
 void perf_evlist__munmap(struct perf_evlist *evlist);
 
+size_t perf_evlist__mmap_size(unsigned long pages);
+
 void perf_evlist__disable(struct perf_evlist *evlist);
 void perf_evlist__enable(struct perf_evlist *evlist);
 void perf_evlist__toggle_enable(struct perf_evlist *evlist);

2.9.3

From: Jiri Olsa <[email protected]>

It's now possible to specify the threshold time for perf.data like:

  $ perf record --switch-output=30s ...

Once it's reached, the current data are dumped in to the perf.data.<timestamp> file and session does on.

  $ perf record --switch-output=30s ...
  [ perf record: dump data: Woken up 44 times ]
  [ perf record: Dump perf.data.2017010213043746 ]
  ...

The time is expected to be a number with appended unit character - s/m/h/d.

---
 tools/perf/Documentation/perf-record.txt |  2 ++
 tools/perf/builtin-record.c              | 44 ++++++++++++++++++++++++++++++--
 2 files changed, 44 insertions(+), 2 deletions(-)

index 3d55d2fd48b3..27256bc68eda 100644
@@ -427,6 +427,8 @@ based on 'mode' value:
   "signal" - when receiving a SIGUSR2 (default value) or
   <size>   - when reaching the size threshold, size is expected to
              be a number with appended unit character - B/K/M/G
+  <time>   - when reaching the time threshold, size is expected to
+             be a number with appended unit character - s/m/h/d
 
              Note: the precision of  the size  threshold  hugely depends
              on your configuration  - the number and size of  your  ring

2.9.3

From: Arnaldo Carvalho de Melo <[email protected]>

To pick the changes from:

1b07304c587d ("KVM: nVMX: support descriptor table exits")

That adds entries to VMX_EXIT_REASONS, that is used by tools/perf/arch/x86/util/kvm-stat.c.

This also picks the changes in:

1dc35dacc16b ("KVM: nVMX: check host CR3 on vmentry and vmexit")

But these are not used in 'perf kvm stat', do it just to silence the kernel/tools file cache coherency detector:

  $ make -C tools/perf
  make: Entering directory '/home/acme/git/linux/tools/perf'
    BUILD:   Doing 'make -j4' parallel build
  Warning: arch/x86/include/uapi/asm/vmx.h differs from kernel

---
 tools/arch/x86/include/uapi/asm/vmx.h | 5 +++++
 1 file changed, 5 insertions(+)

index 37fee272618f..14458658e988 100644
@@ -65,6 +65,8 @@
 #define EXIT_REASON_TPR_BELOW_THRESHOLD 43
 #define EXIT_REASON_APIC_ACCESS         44
 #define EXIT_REASON_EOI_INDUCED         45
+#define EXIT_REASON_GDTR_IDTR           46
+#define EXIT_REASON_LDTR_TR             47
 #define EXIT_REASON_EPT_VIOLATION       48
 #define EXIT_REASON_EPT_MISCONFIG       49
 #define EXIT_REASON_INVEPT              50

@@ -113,6 +115,8 @@
 	{ EXIT_REASON_MCE_DURING_VMENTRY,    "MCE_DURING_VMENTRY" }, \
 	{ EXIT_REASON_TPR_BELOW_THRESHOLD,   "TPR_BELOW_THRESHOLD" }, \
 	{ EXIT_REASON_APIC_ACCESS,           "APIC_ACCESS" }, \
+	{ EXIT_REASON_GDTR_IDTR,	     "GDTR_IDTR" }, \
+	{ EXIT_REASON_LDTR_TR,		     "LDTR_TR" }, \
 	{ EXIT_REASON_EPT_VIOLATION,         "EPT_VIOLATION" }, \
 	{ EXIT_REASON_EPT_MISCONFIG,         "EPT_MISCONFIG" }, \
 	{ EXIT_REASON_INVEPT,                "INVEPT" }, \

@@ -129,6 +133,7 @@
 	{ EXIT_REASON_XRSTORS,               "XRSTORS" }
 
 #define VMX_ABORT_SAVE_GUEST_MSR_FAIL        1
+#define VMX_ABORT_LOAD_HOST_PDPTE_FAIL       2
 #define VMX_ABORT_LOAD_HOST_MSR_FAIL         4
 
 #endif /* _UAPIVMX_H */

2.9.3

Pulled, thanks a lot Arnaldo!

Ingo