perf/pt -> Intel PT/BTS

Hi Ingo,

Please consider pulling, there are several other patches after this,
but I think that this may be acceptable to showcase the capabilities already present, look at the output of 'perf script' and callchains for userspace without using any extra debugging info (no need for DWARF, CFI, nothing), really cool capabilities... :-)

Adrian wrote some docs and I tested it both on a Ivy Bridge machine
where there is only BTS and on a Broadwell machine with the whole shebang, adding the output of the commands to the csets, to further showcase what is there already.

This is on top of my last perf-core-for-mingo tag.

Up to you, please let us know what you think and we'll continue working
on having this in an acceptable form for merging,

Regards,

- Arnaldo

P.S. Kudos for Adrian for the patience with this process, way more is needed to polish this, but the promise is there, cool stuff indeed! :-)

The following changes since commit 36c8bb56a9f718a9a5f35d1834ca9dcec95deb4a:

perf symbols: Check access permission when reading symbol files (2015-06-26 12:11:53 -0300)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git tags/perf-pt-for-mingo

for you to fetch changes up to 04759f172270afb28c8004f5cad62ed55710a499:

perf tools: Add Intel BTS support (2015-06-26 18:36:11 -0300)

----------------------------------------------------------------
Put Intel PT and BTS into initial use (Adrian Hunter)

Signed-off-by Arnaldo Carvalho de Melo <[email protected]>

----------------------------------------------------------------
Adrian Hunter (8):
perf auxtrace: Add Intel PT as an AUX area tracing type perf tools: Add Intel PT packet decoder
perf tools: Add Intel PT instruction decoder perf tools: Add Intel PT log
perf tools: Add Intel PT decoder
perf tools: Add Intel PT support
perf tools: Take Intel PT into use
perf tools: Add Intel BTS support

 tools/build/Makefile.build                         |    2 +
 tools/perf/.gitignore                              |    2 +
 tools/perf/Documentation/intel-bts.txt             |   86 +
 tools/perf/Documentation/intel-pt.txt              |  588 ++++++
 tools/perf/MANIFEST                                |    7 +
 tools/perf/Makefile.perf                           |   12 +-
 tools/perf/arch/x86/util/Build                     |    5 +
 tools/perf/arch/x86/util/auxtrace.c                |   83 +
 tools/perf/arch/x86/util/intel-bts.c               |  458 +++++
 tools/perf/arch/x86/util/intel-pt.c                |  752 ++++++++
 tools/perf/arch/x86/util/pmu.c                     |   18 +
 tools/perf/util/Build                              |    3 +
 tools/perf/util/auxtrace.c                         |    9 +-
 tools/perf/util/auxtrace.h                         |    2 +
 tools/perf/util/intel-bts.c                        |  791 ++++++++
 tools/perf/util/intel-bts.h                        |   43 +
 tools/perf/util/intel-pt-decoder/Build             |   14 +
 .../perf/util/intel-pt-decoder/intel-pt-decoder.c  | 1759 ++++++++++++++++++
 .../perf/util/intel-pt-decoder/intel-pt-decoder.h  |  102 ++
 .../util/intel-pt-decoder/intel-pt-insn-decoder.c  |  246 +++
 .../util/intel-pt-decoder/intel-pt-insn-decoder.h  |   65 +
 tools/perf/util/intel-pt-decoder/intel-pt-log.c    |  155 ++
 tools/perf/util/intel-pt-decoder/intel-pt-log.h    |   52 +
 .../util/intel-pt-decoder/intel-pt-pkt-decoder.c   |  400 +++++
 .../util/intel-pt-decoder/intel-pt-pkt-decoder.h   |   64 +
 tools/perf/util/intel-pt.c                         | 1889 ++++++++++++++++++++
 tools/perf/util/intel-pt.h                         |   51 +
 tools/perf/util/pmu.c                              |    4 -
 28 files changed, 7655 insertions(+), 7 deletions(-)
 create mode 100644 tools/perf/Documentation/intel-bts.txt
 create mode 100644 tools/perf/Documentation/intel-pt.txt
 create mode 100644 tools/perf/arch/x86/util/auxtrace.c
 create mode 100644 tools/perf/arch/x86/util/intel-bts.c
 create mode 100644 tools/perf/arch/x86/util/intel-pt.c
 create mode 100644 tools/perf/arch/x86/util/pmu.c
 create mode 100644 tools/perf/util/intel-bts.c
 create mode 100644 tools/perf/util/intel-bts.h
 create mode 100644 tools/perf/util/intel-pt-decoder/Build
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-decoder.h

 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.h
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-log.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-log.h
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.h
 create mode 100644 tools/perf/util/intel-pt.c
 create mode 100644 tools/perf/util/intel-pt.h

Quoted 93 lines · Arnaldo Carvalho de Melo

Hi Ingo,

Please consider pulling, there are several other patches after this,
but I think that this may be acceptable to showcase the capabilities already present, look at the output of 'perf script' and callchains for userspace without using any extra debugging info (no need for DWARF, CFI, nothing), really cool capabilities... :-)

Adrian wrote some docs and I tested it both on a Ivy Bridge machine
where there is only BTS and on a Broadwell machine with the whole shebang, adding the output of the commands to the csets, to further showcase what is there already.

This is on top of my last perf-core-for-mingo tag.

Up to you, please let us know what you think and we'll continue working
on having this in an acceptable form for merging,

Regards,

- Arnaldo

P.S. Kudos for Adrian for the patience with this process, way more is needed to polish this, but the promise is there, cool stuff indeed! :-)

The following changes since commit 36c8bb56a9f718a9a5f35d1834ca9dcec95deb4a:

perf symbols: Check access permission when reading symbol files (2015-06-26 12:11:53 -0300)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git tags/perf-pt-for-mingo

for you to fetch changes up to 04759f172270afb28c8004f5cad62ed55710a499:

perf tools: Add Intel BTS support (2015-06-26 18:36:11 -0300)

----------------------------------------------------------------
Put Intel PT and BTS into initial use (Adrian Hunter)

Signed-off-by Arnaldo Carvalho de Melo <[email protected]>

----------------------------------------------------------------
Adrian Hunter (8):
perf auxtrace: Add Intel PT as an AUX area tracing type perf tools: Add Intel PT packet decoder
perf tools: Add Intel PT instruction decoder perf tools: Add Intel PT log
perf tools: Add Intel PT decoder
perf tools: Add Intel PT support
perf tools: Take Intel PT into use
perf tools: Add Intel BTS support

 tools/build/Makefile.build                         |    2 +
 tools/perf/.gitignore                              |    2 +
 tools/perf/Documentation/intel-bts.txt             |   86 +
 tools/perf/Documentation/intel-pt.txt              |  588 ++++++
 tools/perf/MANIFEST                                |    7 +
 tools/perf/Makefile.perf                           |   12 +-
 tools/perf/arch/x86/util/Build                     |    5 +
 tools/perf/arch/x86/util/auxtrace.c                |   83 +
 tools/perf/arch/x86/util/intel-bts.c               |  458 +++++
 tools/perf/arch/x86/util/intel-pt.c                |  752 ++++++++
 tools/perf/arch/x86/util/pmu.c                     |   18 +
 tools/perf/util/Build                              |    3 +
 tools/perf/util/auxtrace.c                         |    9 +-
 tools/perf/util/auxtrace.h                         |    2 +
 tools/perf/util/intel-bts.c                        |  791 ++++++++
 tools/perf/util/intel-bts.h                        |   43 +
 tools/perf/util/intel-pt-decoder/Build             |   14 +
 .../perf/util/intel-pt-decoder/intel-pt-decoder.c  | 1759 ++++++++++++++++++
 .../perf/util/intel-pt-decoder/intel-pt-decoder.h  |  102 ++
 .../util/intel-pt-decoder/intel-pt-insn-decoder.c  |  246 +++
 .../util/intel-pt-decoder/intel-pt-insn-decoder.h  |   65 +
 tools/perf/util/intel-pt-decoder/intel-pt-log.c    |  155 ++
 tools/perf/util/intel-pt-decoder/intel-pt-log.h    |   52 +
 .../util/intel-pt-decoder/intel-pt-pkt-decoder.c   |  400 +++++
 .../util/intel-pt-decoder/intel-pt-pkt-decoder.h   |   64 +
 tools/perf/util/intel-pt.c                         | 1889 ++++++++++++++++++++
 tools/perf/util/intel-pt.h                         |   51 +
 tools/perf/util/pmu.c                              |    4 -
 28 files changed, 7655 insertions(+), 7 deletions(-)
 create mode 100644 tools/perf/Documentation/intel-bts.txt
 create mode 100644 tools/perf/Documentation/intel-pt.txt
 create mode 100644 tools/perf/arch/x86/util/auxtrace.c
 create mode 100644 tools/perf/arch/x86/util/intel-bts.c
 create mode 100644 tools/perf/arch/x86/util/intel-pt.c
 create mode 100644 tools/perf/arch/x86/util/pmu.c
 create mode 100644 tools/perf/util/intel-bts.c
 create mode 100644 tools/perf/util/intel-bts.h
 create mode 100644 tools/perf/util/intel-pt-decoder/Build
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-decoder.h

Quoted 8 lines

 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.h
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-log.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-log.h
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.h
 create mode 100644 tools/perf/util/intel-pt.c
 create mode 100644 tools/perf/util/intel-pt.h

Yeah, so I did a 'newbie test':

I pulled the tree and saw that it has a tools/perf/Documentation/intel-bts.txt file and started reading it.

Based on its text:

The Intel BTS kernel driver creates a new PMU for Intel BTS. The perf record option is:

        -e intel_bts//

Currently Intel BTS is limited to per-thread tracing so the --per-thread option is also needed.

I tried the following command which failed:

triton:~/tip> perf record -e intel_bts// --per-thread sleep 1 invalid or unsupported event: 'intel_bts//'
Run 'perf list' for a list of valid events

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-e, --event <event> event selector. use 'perf list' to list available events

That's a really ... unhelpful message. If I typoed something I want to know that. If the kernel does not support something, I want to know about that too. Tooling telling me: "maybe you typoed something, maybe it's not supported, I really don't care" is not very productive.

So this was with a distro kernel, and in the hope that I'm missing some magic new kernel feature, I tried it the latest -tip kernel, but it still gives me the same failure.

So the test newbie user got stuck after wasting some time.

Me as a kernel developer could probably figure it out, but that's not the point:
if newbies cannot discover and use our new features then it's as if they didn't exist, and I'm not pulling non-existent features! ;-)

Could we please improve all this?

Quoted 26 lines

Yeah, so I did a 'newbie test':

I pulled the tree and saw that it has a tools/perf/Documentation/intel-bts.txt file and started reading it.

Based on its text:

The Intel BTS kernel driver creates a new PMU for Intel BTS. The perf record option is:

        -e intel_bts//

Currently Intel BTS is limited to per-thread tracing so the --per-thread option is also needed.

I tried the following command which failed:

triton:~/tip> perf record -e intel_bts// --per-thread sleep 1 invalid or unsupported event: 'intel_bts//'
Run 'perf list' for a list of valid events

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-e, --event <event> event selector. use 'perf list' to list available events

Quoted 4 lines

That's a really ... unhelpful message. If I typoed something I want to know that. If the kernel does not support something, I want to know about that too. Tooling telling me: "maybe you typoed something, maybe it's not supported, I really don't care" is not very productive.

That is not entirely true. The message says "Run 'perf list' for a list of valid events" which will tell you if the event is valid. So you can tell the difference between a typo and unsupported event.

Quoted 6 lines

So this was with a distro kernel, and in the hope that I'm missing some magic new kernel feature, I tried it the latest -tip kernel, but it still gives me the same failure.

So the test newbie user got stuck after wasting some time.

Quoted 5 lines

Me as a kernel developer could probably figure it out, but that's not the
point: if newbies cannot discover and use our new features then it's as if they didn't exist, and I'm not pulling non-existent features! ;-)

Could we please improve all this?

'perf list' shows the event wasn't supported, so I am not sure what more the "newbie" could expect. Do you have any suggestions?

Quoted 31 lines · Adrian Hunter

Quoted 30 lines

Yeah, so I did a 'newbie test':

I pulled the tree and saw that it has a tools/perf/Documentation/intel-bts.txt file and started reading it.

Based on its text:

The Intel BTS kernel driver creates a new PMU for Intel BTS. The perf record option is:

        -e intel_bts//

Currently Intel BTS is limited to per-thread tracing so the --per-thread option is also needed.

I tried the following command which failed:

triton:~/tip> perf record -e intel_bts// --per-thread sleep 1 invalid or unsupported event: 'intel_bts//'
Run 'perf list' for a list of valid events

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-e, --event <event> event selector. use 'perf list' to list available events

That's a really ... unhelpful message. If I typoed something I want to know that. If the kernel does not support something, I want to know about that too. Tooling telling me: "maybe you typoed something, maybe it's not supported, I really don't care" is not very productive.

Quoted 3 lines

That is not entirely true. The message says "Run 'perf list' for a list of valid events" which will tell you if the event is valid. So you can tell the difference between a typo and unsupported event.

Yeah, but my point is: why doesn't the tool do this disambiguation for me? Tools are hard enough to use as-is already, no need to put artificial roadblocks in the path of first time users.

Quoted 7 lines

Quoted 7 lines

So this was with a distro kernel, and in the hope that I'm missing some magic new kernel feature, I tried it the latest -tip kernel, but it still gives me the same failure.

So the test newbie user got stuck after wasting some time.

Quoted 8 lines

Quoted 5 lines

Me as a kernel developer could probably figure it out, but that's not the
point: if newbies cannot discover and use our new features then it's as if they didn't exist, and I'm not pulling non-existent features! ;-)

Could we please improve all this?

'perf list' shows the event wasn't supported, so I am not sure what more the "newbie" could expect. Do you have any suggestions?

So I think a first time user would expect a clear message from the computer: what was wrong with what he wrote and what should he do to fix it.

Btw., here's the 'perf list' output from a system running the latest -tip kernel:

vega:~> uname -a
Linux vega 4.1.0-02935-g390ad45394a3-dirty #567 SMP PREEMPT Mon Jun 29 11:44:48 CEST 2015 x86_64 x86_64 x86_64 GNU/Linux vega:~> perf list | grep -i bts
vega:~>

so is there any kernel feature dependency? It's unclear. If yes, it should be mentioned in the document, and in the tooling output as well. If not then we have a bug somewhere.

I.e. you need to smooth the first time user's rocky path to first use as much as technically possible. Every single such helping step will literally double the number of users who will be able to successfully make use of the new feature.

As a positive example take a look at the newbie's road to 'perf trace':

vega:~> trace
Error: No permissions to read /sys/kernel/debug/tracing/events/raw_syscalls/sys_(enter|exit)
Hint: Try 'sudo mount -o remount,mode=755 /sys/kernel/debug'

Aha, useful message, I need to run this as root:

  # trace

0.000 ( 0.000 ms): sleep/28926 ... [continued]: nanosleep()) = 0
0.051 ( 0.007 ms): sleep/28926 close(fd: 1 ) = 0
0.063 ( 0.005 ms): sleep/28926 close(fd: 2 ) = 0
0.072 ( 0.000 ms): sleep/28926 exit_group(

Ok?

Quoted 34 lines · Adrian Hunter

Quoted 32 lines

Quoted 31 lines

Yeah, so I did a 'newbie test':

I pulled the tree and saw that it has a tools/perf/Documentation/intel-bts.txt file and started reading it.

Based on its text:

The Intel BTS kernel driver creates a new PMU for Intel BTS. The perf record option is:

        -e intel_bts//

Currently Intel BTS is limited to per-thread tracing so the --per-thread option is also needed.

I tried the following command which failed:

triton:~/tip> perf record -e intel_bts// --per-thread sleep 1 invalid or unsupported event: 'intel_bts//'
Run 'perf list' for a list of valid events

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-e, --event <event> event selector. use 'perf list' to list available events

That's a really ... unhelpful message. If I typoed something I want to know that. If the kernel does not support something, I want to know about that too. Tooling telling me: "maybe you typoed something, maybe it's not supported, I really don't care" is not very productive.

Quoted 7 lines

Quoted 3 lines

That is not entirely true. The message says "Run 'perf list' for a list of valid events" which will tell you if the event is valid. So you can tell the difference between a typo and unsupported event.

Yeah, but my point is: why doesn't the tool do this disambiguation for me? Tools are hard enough to use as-is already, no need to put artificial roadblocks in the path of first time users.

That applies to all events e.g.

# perf record -e sched:sched_swotch sleep 1
invalid or unsupported event: 'sched:sched_swotch'                                                                                       
Run 'perf list' for a list of valid events                                                                                               

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-e, --event <event> event selector. use 'perf list' to list available events

So it is a general problem.

Quoted 34 lines · Ingo Molnar

Quoted 31 lines · Adrian Hunter

Quoted 30 lines

Yeah, so I did a 'newbie test':

I pulled the tree and saw that it has a tools/perf/Documentation/intel-bts.txt file and started reading it.

Based on its text:

The Intel BTS kernel driver creates a new PMU for Intel BTS. The perf record option is:

        -e intel_bts//

Currently Intel BTS is limited to per-thread tracing so the --per-thread option is also needed.

I tried the following command which failed:

triton:~/tip> perf record -e intel_bts// --per-thread sleep 1 invalid or unsupported event: 'intel_bts//'
Run 'perf list' for a list of valid events

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-e, --event <event> event selector. use 'perf list' to list available events

That's a really ... unhelpful message. If I typoed something I want to know that. If the kernel does not support something, I want to know about that too. Tooling telling me: "maybe you typoed something, maybe it's not supported, I really don't care" is not very productive.

Quoted 7 lines

Quoted 3 lines

That is not entirely true. The message says "Run 'perf list' for a list of valid events" which will tell you if the event is valid. So you can tell the difference between a typo and unsupported event.

Yeah, but my point is: why doesn't the tool do this disambiguation for me? Tools are hard enough to use as-is already, no need to put artificial roadblocks in the path of first time users.

That applies to all events e.g.

# perf record -e sched:sched_swotch sleep 1
invalid or unsupported event: 'sched:sched_swotch'                                                                                       
Run 'perf list' for a list of valid events                                                                                               

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-e, --event <event> event selector. use 'perf list' to list available events

So it is a general problem.

Quoted 21 lines

Quoted 7 lines

Quoted 7 lines

So this was with a distro kernel, and in the hope that I'm missing some magic new kernel feature, I tried it the latest -tip kernel, but it still gives me the same failure.

So the test newbie user got stuck after wasting some time.

Quoted 8 lines

Quoted 5 lines

Me as a kernel developer could probably figure it out, but that's not the
point: if newbies cannot discover and use our new features then it's as if they didn't exist, and I'm not pulling non-existent features! ;-)

Could we please improve all this?

'perf list' shows the event wasn't supported, so I am not sure what more the "newbie" could expect. Do you have any suggestions?

So I think a first time user would expect a clear message from the computer: what was wrong with what he wrote and what should he do to fix it.

Btw., here's the 'perf list' output from a system running the latest -tip kernel:

Quoted 8 lines

vega:~> uname -a
Linux vega 4.1.0-02935-g390ad45394a3-dirty #567 SMP PREEMPT Mon Jun 29 11:44:48 CEST 2015 x86_64 x86_64 x86_64 GNU/Linux vega:~> perf list | grep -i bts
vega:~>

so is there any kernel feature dependency? It's unclear. If yes, it should be mentioned in the document, and in the tooling output as well. If not then we have a bug somewhere.

I am not aware of any dependencies, apart from perf events itself.

Are you sure you compiled perf tools with the new patches ;-) And it is an Intel CPU?

Quoted 13 lines

I.e. you need to smooth the first time user's rocky path to first use as much as technically possible. Every single such helping step will literally double the number of users who will be able to successfully make use of the new feature.

As a positive example take a look at the newbie's road to 'perf trace':

vega:~> trace
Error: No permissions to read /sys/kernel/debug/tracing/events/raw_syscalls/sys_(enter|exit)
Hint: Try 'sudo mount -o remount,mode=755 /sys/kernel/debug'

Aha, useful message, I need to run this as root:

Quoted 8 lines

  # trace

0.000 ( 0.000 ms): sleep/28926 ... [continued]: nanosleep()) = 0
0.051 ( 0.007 ms): sleep/28926 close(fd: 1 ) = 0
0.063 ( 0.005 ms): sleep/28926 close(fd: 2 ) = 0
0.072 ( 0.000 ms): sleep/28926 exit_group(

Ok?

Could do something like the following:

tools/perf/util/parse-events.c +6 -1

index 09f8d2357108..5ab8fee89361 100644
@@ -666,8 +666,13 @@ int parse_events_add_pmu(struct parse_events_evlist *data,
 	struct perf_evsel *evsel;
 
 	pmu = perf_pmu__find(name);
-	if (!pmu)
+	if (!pmu) {
+		if ((!strcmp(name, "intel_bts") || !strcmp(name, "intel_pt")) &&
+		    data->error)
+			if (asprintf(&data->error->str, "%s is not supported by the running kernel", name) < 0)
+				return -ENOMEM;
 		return -EINVAL;
+	}
 
 	if (pmu->default_config) {
 		memcpy(&attr, pmu->default_config,

Could then add checks for Intel hardware and bts CPU feature flag.

Quoted 86 lines · Adrian Hunter

Quoted 34 lines · Ingo Molnar

Quoted 31 lines · Adrian Hunter

Quoted 30 lines

Yeah, so I did a 'newbie test':

I pulled the tree and saw that it has a tools/perf/Documentation/intel-bts.txt file and started reading it.

Based on its text:

The Intel BTS kernel driver creates a new PMU for Intel BTS. The perf record option is:

        -e intel_bts//

Currently Intel BTS is limited to per-thread tracing so the --per-thread option is also needed.

I tried the following command which failed:

triton:~/tip> perf record -e intel_bts// --per-thread sleep 1 invalid or unsupported event: 'intel_bts//'
Run 'perf list' for a list of valid events

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-e, --event <event> event selector. use 'perf list' to list available events

That's a really ... unhelpful message. If I typoed something I want to know that. If the kernel does not support something, I want to know about that too. Tooling telling me: "maybe you typoed something, maybe it's not supported, I really don't care" is not very productive.

Quoted 7 lines

Quoted 3 lines

That is not entirely true. The message says "Run 'perf list' for a list of valid events" which will tell you if the event is valid. So you can tell the difference between a typo and unsupported event.

Yeah, but my point is: why doesn't the tool do this disambiguation for me? Tools are hard enough to use as-is already, no need to put artificial roadblocks in the path of first time users.

That applies to all events e.g.

# perf record -e sched:sched_swotch sleep 1
invalid or unsupported event: 'sched:sched_swotch'                                                                                       
Run 'perf list' for a list of valid events                                                                                               

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-e, --event <event> event selector. use 'perf list' to list available events

So it is a general problem.

Quoted 29 lines

Quoted 7 lines

Quoted 7 lines

So this was with a distro kernel, and in the hope that I'm missing some magic new kernel feature, I tried it the latest -tip kernel, but it still gives me the same failure.

So the test newbie user got stuck after wasting some time.

Quoted 8 lines

Quoted 5 lines

Me as a kernel developer could probably figure it out, but that's not the
point: if newbies cannot discover and use our new features then it's as if they didn't exist, and I'm not pulling non-existent features! ;-)

Could we please improve all this?

'perf list' shows the event wasn't supported, so I am not sure what more the "newbie" could expect. Do you have any suggestions?

So I think a first time user would expect a clear message from the computer: what was wrong with what he wrote and what should he do to fix it.

Btw., here's the 'perf list' output from a system running the latest -tip kernel:

vega:~> uname -a
Linux vega 4.1.0-02935-g390ad45394a3-dirty #567 SMP PREEMPT Mon Jun 29 11:44:48 CEST 2015 x86_64 x86_64 x86_64 GNU/Linux vega:~> perf list | grep -i bts
vega:~>

so is there any kernel feature dependency? It's unclear. If yes, it should be mentioned in the document, and in the tooling output as well. If not then we have a bug somewhere.

Quoted 5 lines

I am not aware of any dependencies, apart from perf events itself.

Are you sure you compiled perf tools with the new patches ;-) And it is an Intel CPU?

I am going to need to know what hardware it is and cpu feature flags i.e. /proc/cpuinfo

Also ls /sys/bus/event_source/devices

How is this?

From: Adrian Hunter <[email protected]>
Date: Wed, 1 Jul 2015 11:14:50 +0300
Subject: [PATCH] perf tools: Add error messages for missing intel_bts and
intel_pt support

Add error messages to assist users in determining why there is no intel_bts or intel_pt support.

Signed-off-by Adrian Hunter <[email protected]>

---
 tools/perf/arch/x86/util/header.c | 15 ++++++++++++++
 tools/perf/util/header.h          |  3 ++-
 tools/perf/util/parse-events.c    | 41 ++++++++++++++++++++++++++++++++++++++-
 3 files changed, 57 insertions(+), 2 deletions(-)

tools/perf/arch/x86/util/header.c +15 -0

index 146d12a1cec0..afc5bdfd2d15 100644
@@ -57,3 +57,18 @@ get_cpuid(char *buffer, size_t sz)
 	}
 	return -1;
 }
+
+int have_intel_cpu(void)
+{
+	char buffer[64];
+	int ret;
+
+	ret = get_cpuid(buffer, sizeof(buffer));
+	if (ret)
+		return -1;
+
+	if (!strncmp(buffer, "GenuineIntel,", 13))
+		return 1;
+
+	return 0;
+}

tools/perf/util/header.h +2 -1

index d4d57962c591..f6eab49d13d1 100644
@@ -153,8 +153,9 @@ bool is_perf_magic(u64 magic);
 int write_padded(int fd, const void *bf, size_t count, size_t count_aligned);
 
 /*
- * arch specific callback
+ * arch specific callbacks
  */
 int get_cpuid(char *buffer, size_t sz);
+int have_intel_cpu(void);
 
 #endif /* __PERF_HEADER_H */

tools/perf/util/parse-events.c +39 -0

index 09f8d2357108..23fb777b40fa 100644
@@ -656,6 +656,45 @@ static char *pmu_event_name(struct list_head *head_terms)
 	return NULL;
 }
 
+int __weak have_intel_cpu(void)
+{
+	return 0;
+}
+
+static int pmu_not_found_error(char *name, struct parse_events_error *err)
+{
+	int ret;
+
+	if (!err)
+		goto out;
+
+	if (!strcmp(name, "intel_bts")) {
+		ret = have_intel_cpu();
+		if (ret < 0)
+			goto out;
+		if (!ret) {
+			err->str = strdup("intel_bts requires an Intel CPU");
+			goto out;
+		}
+		err->str = strdup("kernel does not support intel_bts (requires 64-bit v4.1 kernel or later and BTS hardware support)");
+		goto out;
+	}
+
+	if (!strcmp(name, "intel_pt")) {
+		ret = have_intel_cpu();
+		if (ret < 0)
+			goto out;
+		if (!ret) {
+			err->str = strdup("intel_pt requires an Intel CPU (Core 5th generation or later)");
+			goto out;
+		}
+		err->str = strdup("kernel does not support intel_pt (requires v4.1 kernel or later and 5th generation Intel Core processor or later)");
+		goto out;
+	}
+out:
+	return -EINVAL;
+}
+
 int parse_events_add_pmu(struct parse_events_evlist *data,
 			 struct list_head *list, char *name,
 			 struct list_head *head_config)

Quoted 106 lines · Adrian Hunter

How is this?

From: Adrian Hunter <[email protected]>
Date: Wed, 1 Jul 2015 11:14:50 +0300
Subject: [PATCH] perf tools: Add error messages for missing intel_bts and
intel_pt support

Add error messages to assist users in determining why there is no intel_bts or intel_pt support.

Signed-off-by Adrian Hunter <[email protected]>

---
 tools/perf/arch/x86/util/header.c | 15 ++++++++++++++
 tools/perf/util/header.h          |  3 ++-
 tools/perf/util/parse-events.c    | 41 ++++++++++++++++++++++++++++++++++++++-
 3 files changed, 57 insertions(+), 2 deletions(-)

tools/perf/arch/x86/util/header.c +15 -0

index 146d12a1cec0..afc5bdfd2d15 100644
@@ -57,3 +57,18 @@ get_cpuid(char *buffer, size_t sz)
 	}
 	return -1;
 }
+
+int have_intel_cpu(void)
+{
+	char buffer[64];
+	int ret;
+
+	ret = get_cpuid(buffer, sizeof(buffer));
+	if (ret)
+		return -1;
+
+	if (!strncmp(buffer, "GenuineIntel,", 13))
+		return 1;
+
+	return 0;
+}

tools/perf/util/header.h +2 -1

index d4d57962c591..f6eab49d13d1 100644
@@ -153,8 +153,9 @@ bool is_perf_magic(u64 magic);
 int write_padded(int fd, const void *bf, size_t count, size_t count_aligned);
 
 /*
- * arch specific callback
+ * arch specific callbacks
  */
 int get_cpuid(char *buffer, size_t sz);
+int have_intel_cpu(void);
 
 #endif /* __PERF_HEADER_H */

tools/perf/util/parse-events.c +39 -0

index 09f8d2357108..23fb777b40fa 100644
@@ -656,6 +656,45 @@ static char *pmu_event_name(struct list_head *head_terms)
 	return NULL;
 }
 
+int __weak have_intel_cpu(void)
+{
+	return 0;
+}
+
+static int pmu_not_found_error(char *name, struct parse_events_error *err)
+{
+	int ret;
+
+	if (!err)
+		goto out;
+
+	if (!strcmp(name, "intel_bts")) {
+		ret = have_intel_cpu();
+		if (ret < 0)
+			goto out;
+		if (!ret) {
+			err->str = strdup("intel_bts requires an Intel CPU");
+			goto out;
+		}
+		err->str = strdup("kernel does not support intel_bts (requires 64-bit v4.1 kernel or later and BTS hardware support)");
+		goto out;
+	}
+
+	if (!strcmp(name, "intel_pt")) {
+		ret = have_intel_cpu();
+		if (ret < 0)
+			goto out;
+		if (!ret) {
+			err->str = strdup("intel_pt requires an Intel CPU (Core 5th generation or later)");
+			goto out;
+		}
+		err->str = strdup("kernel does not support intel_pt (requires v4.1 kernel or later and 5th generation Intel Core processor or later)");
+		goto out;
+	}
+out:
+	return -EINVAL;
+}
+
 int parse_events_add_pmu(struct parse_events_evlist *data,
 			 struct list_head *list, char *name,
 			 struct list_head *head_config)

tools/perf/util/parse-events.c int parse_events_add_pmu(struct parse_events_evlist *data, · line 712

Quoted 8 lines

tools/perf/util/parse-events.c +1 -1

@@ -667,7 +706,7 @@ int parse_events_add_pmu(struct parse_events_evlist *data,
 
 	pmu = perf_pmu__find(name);
 	if (!pmu)
-		return -EINVAL;
+		return pmu_not_found_error(name, data->error);
 
 	if (pmu->default_config) {
 		memcpy(&attr, pmu->default_config,

So I really think we need an extended error reporting feature on the perf kernel side, so that a 'natural' error (plus a string) is reported back to tooling, instead of the current -EINVAL.

No need to do it for everything, doing it for BTS and related functionality would be a good first step to start this.

If you are interested you could try this, or I can try to write something (after the merge window).

So the idea would be to convert such opaque error returns:

        if (attr->config == PERF_COUNT_HW_BRANCH_INSTRUCTIONS &&
            !attr->freq && hwc->sample_period == 1) {
                /* BTS is not supported by this architecture. */
                if (!x86_pmu.bts_active)
                        return -EOPNOTSUPP;

into:

			return perf_err(event, -EOPNOTSUPP, "The BTS hardware feature is not available on this CPU.");

Where perf_err() is a function that on one hand returns -EOPNOTSUPP - so 'legacy' error handling works as before: the syscall will return -EOPNOTSUPP.

But if a new 'extended error reporting' feature bit is set in the perf_attr, then perf_err() also does the following:

- it copies the error string either back to a user-space pointer which is in the perf_attr (plus a max length field)

- or an alternative implementation would be to puts the string into the event's ring buffer, with a special (new) event marker - where it can be recovered by tooling.

(I think the first approach is better, because it would work fine for events without ring-buffers as well.)

Old tooling won't have the feature flag set and won't have the string pointer in perf_attr, so nothing will happen on that case.

New tooling that supports 'extended error reporting' has the feature flag set and the kernel will copy the error string into the provided user-space string buffer, where tooling could use that string to generate more meaningful error messages:

perf syscall error: The BTS hardware feature is not available on this CPU.

Does this make sense to you?

Now an additional complication is the fact that BTS can now also be a separate PMU, listed under /sys/bus/event_source/devices/intel_bts/.

If it's not listed there, we don't know the exact reason: is it not available because it's an old kernel? Or is it the wrong CPU?

We could solve that by extending the sysfs interface and adding an "error" file to the PMU directory: which would contain the reason why the driver was not created.

I.e. if the BTS driver was not created, we'd still have /sys/bus/event_source/devices/intel_bts/error (and no other file), which gives tooling a good way to discover why a particular PMU is not available. This adds a tiny bit more RAM overhead, but it's for the better I think, because tooling could be a lot more certain about what the capabilities of the kernel are.

Quoted 10 lines

Old tooling won't have the feature flag set and won't have the string pointer in perf_attr, so nothing will happen on that case.

New tooling that supports 'extended error reporting' has the feature flag set and the kernel will copy the error string into the provided user-space string buffer, where tooling could use that string to generate more meaningful error messages:

perf syscall error: The BTS hardware feature is not available on this CPU.

Does this make sense to you?

Normally the warts of syscalls are hidden by libraries. I am not sure a library couldn't do just as well with the advantage that it would work for old kernels too. A library could also have information about multiple architectures, not just the one that is actually running e.g. your BTS example won't work on ARM.

A library could provide a similar API to the one you described. Either it just does the syscall and returns, or if extended error information is requested, it probes the API with various options, checks perf_event_paranoid, and generally uses whatever information it can to best figure out what went wrong.

Quoted 6 lines

Now an additional complication is the fact that BTS can now also be a separate PMU, listed under /sys/bus/event_source/devices/intel_bts/.

If it's not listed there, we don't know the exact reason: is it not available because it's an old kernel? Or is it the wrong CPU?

Quoted 8 lines

We could solve that by extending the sysfs interface and adding an "error" file to the PMU directory: which would contain the reason why the driver was not created.

I.e. if the BTS driver was not created, we'd still have /sys/bus/event_source/devices/intel_bts/error (and no other file), which gives tooling a good way to discover why a particular PMU is not available. This adds a tiny bit more RAM overhead, but it's for the better I think, because tooling could be a lot more certain about what the capabilities of the kernel are.

That presumes the user understands what is or is not available on their architecture, because on a non-x86 architecture they still get nothing x86-specific.

It also runs a bit against the way config works e.g. even if the PMU driver is config'ed out it still has to provide a sysfs interface.

So, isn't the patch I proposed sufficient for now?

Quoted 9 lines

So I really think we need an extended error reporting feature on the perf kernel side, so that a 'natural' error (plus a string) is reported back to tooling, instead of the current -EINVAL.

No need to do it for everything, doing it for BTS and related functionality would be a good first step to start this.

If you are interested you could try this, or I can try to write something (after the merge window).

Don't have time sorry.

Quoted 1 line

Does this make sense to you?

Normally the warts of syscalls are hidden by libraries. I am not sure a library couldn't do just as well with the advantage that it would work for old kernels too. A library could also have information about multiple architectures, not just the one that is actually running e.g. your BTS example won't work on ARM.

A library could provide a similar API to the one you described. Either it just does the syscall and returns, or if extended error information is requested, it probes the API with various options, checks perf_event_paranoid, and generally uses whatever information it can to best figure out what went wrong.

Quoted 6 lines

I.e. if the BTS driver was not created, we'd still have /sys/bus/event_source/devices/intel_bts/error (and no other file), which gives tooling a good way to discover why a particular PMU is not available. This adds a tiny bit more RAM overhead, but it's for the better I think, because tooling could be a lot more certain about what the capabilities of the kernel are.

That presumes the user understands what is or is not available on their architecture, because on a non-x86 architecture they still get nothing x86-specific.

It also runs a bit against the way config works e.g. even if the PMU driver is config'ed out it still has to provide a sysfs interface.

So, isn't the patch I proposed sufficient for now?

Quoted 10 lines · Adrian Hunter

Quoted 9 lines

So I really think we need an extended error reporting feature on the perf kernel side, so that a 'natural' error (plus a string) is reported back to tooling, instead of the current -EINVAL.

No need to do it for everything, doing it for BTS and related functionality would be a good first step to start this.

If you are interested you could try this, or I can try to write something (after the merge window).

Quoted 1 line

Don't have time sorry.

So who on the Intel side has time to finish Intel/PT support properly?

As a first step I think we need to disable the Intel/PT kernel side - it was merged with the express promise that proper tooling would come along promptly, but if that's not happening due to lack of resources, then the kernel side is obviously in limbo as well.

Quoted 7 lines

Quoted 2 lines

Does this make sense to you?

Normally the warts of syscalls are hidden by libraries. I am not sure a library couldn't do just as well with the advantage that it would work for old kernels too. A library could also have information about multiple architectures, not just the one that is actually running e.g. your BTS example won't work on ARM.

The method I suggested works _both_ with old and new kernels, with the new kernel simply giving better error feedback.

And these are not warts really, it's simply a threshold issue: above a certain complexity of features it makes sense to introduce a qualitatively better error reporting interface. When I tested Intel PT support I happened to go past that threshold and it became obvious that we need it.

Quoted 4 lines

A library could provide a similar API to the one you described. Either it just does the syscall and returns, or if extended error information is requested, it probes the API with various options, checks perf_event_paranoid, and generally uses whatever information it can to best figure out what went wrong.

So I think eventually a proper libperf will crystalize out of perf's syscall wrappers (I suggested it for a long time), but this has very little impact on the end user who doesn't care whether it's done in perf, or in some intermediate library.

So it can be done in the current perf code just as much - when libperf gets introduced it will be factored out into tools/lib/perf/ without much problem and then other tools can use it too.

Quoted 7 lines

Quoted 6 lines

I.e. if the BTS driver was not created, we'd still have /sys/bus/event_source/devices/intel_bts/error (and no other file), which gives tooling a good way to discover why a particular PMU is not available. This adds a tiny bit more RAM overhead, but it's for the better I think, because tooling could be a lot more certain about what the capabilities of the kernel are.

Quoted 6 lines

That presumes the user understands what is or is not available on their architecture, because on a non-x86 architecture they still get nothing x86-specific.

It also runs a bit against the way config works e.g. even if the PMU driver is config'ed out it still has to provide a sysfs interface.

I partially agree - but it's really a problem that we currently only get two states:

- the PMU driver is there in sysfs
- the PMU driver is not there in sysfs

while we really want to a lot more about why it's not there - and it's the kernel that knows this best, not tooling.

So maybe instead of having a separate directory, we could have a sysfs file that listed all the PMU drivers that the kernel knows about, with a status line that tells us whether they are:

- not configured
- configured but runtime disabled due to reason X or Y or Z
- configured and enabled

OTOH having the directories with a single file in them is a close substitute, and better meets the sysfs 'one file, one value' principle.

Quoted 2 lines

So, isn't the patch I proposed sufficient for now?

Well, no, because it really does the check in the wrong place and introduces possible friction between what tooling thinks is the supported environment and what the kernel thinks - and thus postpones the problem.

I could live with this current solution as the initial version if a subsequent effort fixes it up properly by adding much better error reporting infrastructure, but the 'no time' comment makes me doubt the whole Intel/PT feature set.

So I really think we need an extended error reporting feature on the perf kernel side, so that a 'natural' error (plus a string) is reported back to tooling, instead of the current -EINVAL.

No need to do it for everything, doing it for BTS and related functionality would be a good first step to start this.

If you are interested you could try this, or I can try to write something (after the merge window).

So the idea would be to convert such opaque error returns:

        if (attr->config == PERF_COUNT_HW_BRANCH_INSTRUCTIONS &&
            !attr->freq && hwc->sample_period == 1) {
                /* BTS is not supported by this architecture. */
                if (!x86_pmu.bts_active)
                        return -EOPNOTSUPP;

into:

			return perf_err(event, -EOPNOTSUPP, "The BTS hardware feature is not available on this CPU.");

From a7de169ef2ca5425cd57286bfb61bfd5f8d15738 Mon Sep 17 00:00:00 2001