Skip to contentSkip to conversation

perf/x86/amd: Add AMD Fam19h Branch Sampling support

PATCH Superseded Stephane Eranian 9 participants 41 messages Sep 9, 2021 – Oct 28, 2021

Subsystems (3)

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

cover perf/x86/amd: Add AMD Fam19h Branch Sampling support

This patch series adds support for the AMD Fam19h 16-deep branch sampling feature as described in the AMD PPR Fam19h Model 01h Revision B1 section 2.1.13. This is a model specific extension. It is not an architected AMD feature.

The Branch Sampling Feature (BRS) provides the statistical taken branch information necessary to enable autoFDO-style optimization by compilers, i.e., basic block execution counts.

BRS operates with a 16-deep saturating buffer in MSR registers. There is no hardware branch type filtering. All control flow changes are captured. BRS relies on specific programming of the core PMU of Fam19h. In particular, the following requirements must be met:
- the sampling period be greater than 16 (BRS depth)
- the sampling period must use fixed and not frequency mode

BRS interacts with the NMI interrupt as well. Because enabling BRS is expensive, it is only activated after P event occurrences, where P is the desired sampling period. At P occurrences of the event, the counter overflows, the CPU catches the NMI interrupt, activates BRS for 16 branches until it saturates, and then delivers the NMI to the kernel. Between the overflow and the time BRS activates more branches may be executed skewing the period. All along, the sampling event keeps counting. The skid may be attenuated by reducing the sampling period by 16.

BRS is integrated into perf_events seamlessly via the same PERF_RECORD_BRANCH_STACK sample format. BRS generates branch perf_branch_entry records in the sampling buffer. There is no prediction or latency information supported. The branches are stored in reverse order of execution. The most recent branch is the first entry in each record.

Because BRS must be stopped when a CPU goes into low power mode, the series includes patches to add callbacks on low power entry and exit (mwait and halt).

Given that there is no privilege filterting with BRS, the kernel implements filtering on privlege level.

This version adds a few simple modifications to perf record and report.
1. add the branch-brs event as a builtin such as it can used directly: perf record -e branch-brs ...
2. improve error handling for AMD IBS and is contributed by Kim Phillips.
3. use the better error handling to improve error handling for branch sampling
4. add two new sort dimensions to help display the branch sampling information. Because there is no latency information associated with the branch sampling feature perf report would collapse all samples within a function into a single histogram entry. This is expected because the default sort mode for PERF_SAMPLE_BRANCH_STACK is symbol_from/symbol_to. This propagates to the annotation.

For more detailed view of the branch samples, the new sort dimensions addr_from,addr_to can be used instead as follows:

$ perf report --sort=overhead,comm,dso,addr_from,addr_to 
# Overhead  Command          Shared Object     Source Address  Target Address
# ........  ...............  ................  ..............  ..............
#
     4.21%  test_prg        test_prg         [.] test_threa+0x3c  [.] test_threa+0x4
     4.14%  test_prg        test_prg         [.] test_threa+0x3e  [.] test_threa+0x2
     4.10%  test_prg        test_prg         [.] test_threa+0x4  [.] test_threa+0x3a
     4.07%  test_prg        test_prg         [.] test_threa+0x2  [.] test_threa+0x3c

Versus the default output:

$ perf report 
# Overhead  Command          Source Shared Object  Source Symbol                        Target Symbol                        Basic Block Cycles
# ........  ...............  ....................  ...................................  ...................................  ..................
#
    99.52%  test_prg        test_prg             [.] test_thread                      [.] test_thread                      -                 

BRS can be used with any sampling event. However, it is recommended to use the RETIRED_BRANCH event because it matches what the BRS captures. For convenience, a pseudo event matching the branches captured by BRS is exported by the kernel (branch-brs):

$ perf record -b -e cpu/branch-brs/ -c 1000037 test
$ perf report -D
56531696056126 0x193c000 [0x1a8]: PERF_RECORD_SAMPLE(IP, 0x2): 18122/18230: 0x401d24 period: 1000037 addr: 0
... branch stack: nr:16
.....  0: 0000000000401d24 -> 0000000000401d5a 0 cycles      0
.....  1: 0000000000401d5c -> 0000000000401d24 0 cycles      0
.....  2: 0000000000401d22 -> 0000000000401d5c 0 cycles      0
.....  3: 0000000000401d5e -> 0000000000401d22 0 cycles      0
.....  4: 0000000000401d20 -> 0000000000401d5e 0 cycles      0
.....  5: 0000000000401d3e -> 0000000000401d20 0 cycles      0
.....  6: 0000000000401d42 -> 0000000000401d3e 0 cycles      0
.....  7: 0000000000401d3c -> 0000000000401d42 0 cycles      0
.....  8: 0000000000401d44 -> 0000000000401d3c 0 cycles      0
.....  9: 0000000000401d3a -> 0000000000401d44 0 cycles      0
..... 10: 0000000000401d46 -> 0000000000401d3a 0 cycles      0
..... 11: 0000000000401d38 -> 0000000000401d46 0 cycles      0
..... 12: 0000000000401d48 -> 0000000000401d38 0 cycles      0
..... 13: 0000000000401d36 -> 0000000000401d48 0 cycles      0
..... 14: 0000000000401d4a -> 0000000000401d36 0 cycles      0
..... 15: 0000000000401d34 -> 0000000000401d4a 0 cycles      0
 ... thread: test:18230
 ...... dso: test

Special thanks to Kim Phillips @ AMD for the testing, reviews and contributions.

Kim Phillips (1):
perf tools: improve IBS error handling

Stephane Eranian (12):
perf/core: add union to struct perf_branch_entry x86/cpufeatures: add AMD Fam19h Branch Sampling feature perf/x86/amd: add AMD Fam19h Branch Sampling support perf/x86/amd: add branch-brs helper event for Fam19h BRS perf/x86/amd: enable branch sampling priv level filtering perf/x86/amd: add AMD branch sampling period adjustment perf/core: add idle hooks
perf/x86/core: add idle hooks
perf/x86/amd: add idle hooks for branch sampling perf tools: add branch-brs as a new event
perf tools: improve error handling of AMD Branch Sampling perf report: add addr_from/addr_to sort dimensions

 arch/x86/events/amd/Makefile       |   2 +-
 arch/x86/events/amd/brs.c          | 342 +++++++++++++++++++++++++++++
 arch/x86/events/amd/core.c         | 222 ++++++++++++++++++-
 arch/x86/events/core.c             |  22 +-
 arch/x86/events/intel/lbr.c        |  13 +-
 arch/x86/events/perf_event.h       | 106 +++++++--
 arch/x86/include/asm/cpufeatures.h |   1 +
 arch/x86/include/asm/msr-index.h   |   4 +
 arch/x86/include/asm/mwait.h       |   6 +-
 include/linux/perf_event.h         |   8 +
 include/uapi/linux/perf_event.h    |  19 +-
 kernel/events/core.c               |  58 +++++
 kernel/sched/idle.c                |  19 +-
 tools/perf/util/evsel.c            |  50 +++++
 tools/perf/util/hist.c             |   2 +
 tools/perf/util/hist.h             |   2 +
 tools/perf/util/parse-events.l     |   1 +
 tools/perf/util/sort.c             | 128 +++++++++++
 tools/perf/util/sort.h             |   2 +
 19 files changed, 964 insertions(+), 43 deletions(-)
 create mode 100644 arch/x86/events/amd/brs.c
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
#1 lore raw

PATCH 1/13 perf/core: add union to struct perf_branch_entry

To make it simpler to reset all the info fields on the perf_branch_entry in a single store, we use a union. This avoids missing some of the bitfields and improves generated code by minimizing the number of stores.

Using an anonymous struct around the bitfields to guarantee field ordering.

A single perf_branch_entry.val = 0 guarantees all fields are all zeroes on any arch.

---
 arch/x86/events/intel/lbr.c     | 13 +++----------
 include/uapi/linux/perf_event.h | 19 ++++++++++++-------
 2 files changed, 15 insertions(+), 17 deletions(-)
arch/x86/events/intel/lbr.c +3 -10
index 9e6d6eaeb4cb..27aa48cce3c6 100644
@@ -801,15 +801,9 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
 
 		rdmsrl(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
 
+		cpuc->lbr_entries[i].val	= 0;
 		cpuc->lbr_entries[i].from	= msr_lastbranch.from;
 		cpuc->lbr_entries[i].to		= msr_lastbranch.to;
-		cpuc->lbr_entries[i].mispred	= 0;
-		cpuc->lbr_entries[i].predicted	= 0;
-		cpuc->lbr_entries[i].in_tx	= 0;
-		cpuc->lbr_entries[i].abort	= 0;
-		cpuc->lbr_entries[i].cycles	= 0;
-		cpuc->lbr_entries[i].type	= 0;
-		cpuc->lbr_entries[i].reserved	= 0;
 	}
 	cpuc->lbr_stack.nr = i;
 	cpuc->lbr_stack.hw_idx = tos;
@@ -896,6 +890,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 		if (abort && x86_pmu.lbr_double_abort && out > 0)
 			out--;
 
+		cpuc->lbr_entries[out].val	 = 0;
 		cpuc->lbr_entries[out].from	 = from;
 		cpuc->lbr_entries[out].to	 = to;
 		cpuc->lbr_entries[out].mispred	 = mis;
@@ -903,8 +898,6 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 		cpuc->lbr_entries[out].in_tx	 = in_tx;
 		cpuc->lbr_entries[out].abort	 = abort;
 		cpuc->lbr_entries[out].cycles	 = cycles;
-		cpuc->lbr_entries[out].type	 = 0;
-		cpuc->lbr_entries[out].reserved	 = 0;
 		out++;
 	}
 	cpuc->lbr_stack.nr = out;
@@ -966,6 +959,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
 		to = rdlbr_to(i, lbr);
 		info = rdlbr_info(i, lbr);
 
+		e->val		= 0;
 		e->from		= from;
 		e->to		= to;
 		e->mispred	= get_lbr_mispred(info);
@@ -974,7 +968,6 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
 		e->abort	= !!(info & LBR_INFO_ABORT);
 		e->cycles	= get_lbr_cycles(info);
 		e->type		= get_lbr_br_type(info);
-		e->reserved	= 0;
 	}
 
 	cpuc->lbr_stack.nr = i;
include/uapi/linux/perf_event.h +12 -7
index f92880a15645..eb11f383f4be 100644
@@ -1329,13 +1329,18 @@ union perf_mem_data_src {
 struct perf_branch_entry {
 	__u64	from;
 	__u64	to;
-	__u64	mispred:1,  /* target mispredicted */
-		predicted:1,/* target predicted */
-		in_tx:1,    /* in transaction */
-		abort:1,    /* transaction abort */
-		cycles:16,  /* cycle count to last branch */
-		type:4,     /* branch type */
-		reserved:40;
+	union {
+		__u64	val;	    /* to make it easier to clear all fields */
+		struct {
+			__u64	mispred:1,  /* target mispredicted */
+				predicted:1,/* target predicted */
+				in_tx:1,    /* in transaction */
+				abort:1,    /* transaction abort */
+				cycles:16,  /* cycle count to last branch */
+				type:4,     /* branch type */
+				reserved:40;
+		};
+	};
 };
 
 union perf_sample_weight {
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
maintainer
#2 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Quoted 18 lines · Stephane Eranian
include/uapi/linux/perf_event.h +3 -7
index f92880a15645..eb11f383f4be 100644
@@ -1329,13 +1329,18 @@ union perf_mem_data_src {
 struct perf_branch_entry {
 	__u64	from;
 	__u64	to;
-	__u64	mispred:1,  /* target mispredicted */
-		predicted:1,/* target predicted */
-		in_tx:1,    /* in transaction */
-		abort:1,    /* transaction abort */
-		cycles:16,  /* cycle count to last branch */
-		type:4,     /* branch type */
-		reserved:40;
+	union {
+		__u64	val;	    /* to make it easier to clear all fields */
+		struct {

include/uapi/linux/perf_event.h union perf_mem_data_src { · line 1344

Quoted 10 lines
include/uapi/linux/perf_event.h +9 -0
+			__u64	mispred:1,  /* target mispredicted */
+				predicted:1,/* target predicted */
+				in_tx:1,    /* in transaction */
+				abort:1,    /* transaction abort */
+				cycles:16,  /* cycle count to last branch */
+				type:4,     /* branch type */
+				reserved:40;
+		};
+	};
 };

Hurpmh... all other bitfields have ENDIAN_BITFIELD things except this one. Power folks, could you please have a look?

#3 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Stephane Eranian
include/uapi/linux/perf_event.h +9 -7
index f92880a15645..eb11f383f4be 100644
@@ -1329,13 +1329,18 @@ union perf_mem_data_src {
 struct perf_branch_entry {
 	__u64	from;
 	__u64	to;
-	__u64	mispred:1,  /* target mispredicted */
-		predicted:1,/* target predicted */
-		in_tx:1,    /* in transaction */
-		abort:1,    /* transaction abort */
-		cycles:16,  /* cycle count to last branch */
-		type:4,     /* branch type */
-		reserved:40;
+	union {
+		__u64	val;	    /* to make it easier to clear all fields */
+		struct {
+			__u64	mispred:1,  /* target mispredicted */
+				predicted:1,/* target predicted */
+				in_tx:1,    /* in transaction */
+				abort:1,    /* transaction abort */
+				cycles:16,  /* cycle count to last branch */
+				type:4,     /* branch type */
Quoted 8 lines

include/uapi/linux/perf_event.h union perf_mem_data_src { · line 1344

Quoted 4 lines
include/uapi/linux/perf_event.h +3 -0
+				reserved:40;
+		};
+	};
 };

Hurpmh... all other bitfields have ENDIAN_BITFIELD things except this one. Power folks, could you please have a look?

The bit number of each field changes between big and little endian, but as long as kernel and userspace are the same endian, and both only access values via the bitfields then it works.

It's preferable to have the ENDIAN_BITFIELD thing, so that the bit numbers are stable, but we can't change it now without breaking ABI :/

Adding the union risks having code that accesses val and expects to see mispred in bit 0 for example, which it's not on big endian.

If it's just for clearing easily we could do that with a static inline that sets all the bitfields. With my compiler here (GCC 10) it's smart enough to just turn it into a single unsigned long store of 0.

eg:

static inline void clear_perf_branch_entry_flags(struct perf_branch_entry *e)

{
        e->mispred = 0;
        e->predicted = 0;
        e->in_tx = 0;
        e->abort = 0;
        e->cycles = 0;
        e->type = 0;
        e->reserved = 0;
}

It does look like we have a bug in perf tool though, if I take a perf.data from a big endian system to a little endian one I don't see any of the branch flags decoded. eg:

BE:

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles P 0

LE:

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles 0

                                                         ^
                                                         missing P

I guess we're missing a byte swap somewhere.

cheers

#4 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Quoted 38 lines · Michael Ellerman
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Stephane Eranian
include/uapi/linux/perf_event.h +9 -7
index f92880a15645..eb11f383f4be 100644
@@ -1329,13 +1329,18 @@ union perf_mem_data_src {
 struct perf_branch_entry {
 	__u64	from;
 	__u64	to;
-	__u64	mispred:1,  /* target mispredicted */
-		predicted:1,/* target predicted */
-		in_tx:1,    /* in transaction */
-		abort:1,    /* transaction abort */
-		cycles:16,  /* cycle count to last branch */
-		type:4,     /* branch type */
-		reserved:40;
+	union {
+		__u64	val;	    /* to make it easier to clear all fields */
+		struct {
+			__u64	mispred:1,  /* target mispredicted */
+				predicted:1,/* target predicted */
+				in_tx:1,    /* in transaction */
+				abort:1,    /* transaction abort */
+				cycles:16,  /* cycle count to last branch */
+				type:4,     /* branch type */
Quoted 8 lines

include/uapi/linux/perf_event.h union perf_mem_data_src { · line 1344

Quoted 4 lines
include/uapi/linux/perf_event.h +3 -0
+				reserved:40;
+		};
+	};
 };

Hurpmh... all other bitfields have ENDIAN_BITFIELD things except this one. Power folks, could you please have a look?

The bit number of each field changes between big and little endian, but as long as kernel and userspace are the same endian, and both only access values via the bitfields then it works.

...

Quoted 13 lines

It does look like we have a bug in perf tool though, if I take a perf.data from a big endian system to a little endian one I don't see any of the branch flags decoded. eg:

BE:

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles P 0

LE:

Quoted 7 lines

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles 0

                                                         ^
                                                         missing P

I guess we're missing a byte swap somewhere.

Ugh. We _do_ have a byte swap, but we also need a bit swap.

That works for the single bit fields, not sure if it will for the multi-bit fields.

So that's a bit of a mess :/

cheers

#5 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Michael,

Quoted 60 lines · Michael Ellerman
Quoted 38 lines · Michael Ellerman
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Stephane Eranian
include/uapi/linux/perf_event.h +9 -7
index f92880a15645..eb11f383f4be 100644
@@ -1329,13 +1329,18 @@ union perf_mem_data_src {
 struct perf_branch_entry {
    __u64   from;
    __u64   to;
-   __u64   mispred:1,  /* target mispredicted */
-           predicted:1,/* target predicted */
-           in_tx:1,    /* in transaction */
-           abort:1,    /* transaction abort */
-           cycles:16,  /* cycle count to last branch */
-           type:4,     /* branch type */
-           reserved:40;
+   union {
+           __u64   val;        /* to make it easier to clear all fields */
+           struct {
+                   __u64   mispred:1,  /* target mispredicted */
+                           predicted:1,/* target predicted */
+                           in_tx:1,    /* in transaction */
+                           abort:1,    /* transaction abort */
+                           cycles:16,  /* cycle count to last branch */
+                           type:4,     /* branch type */
Quoted 8 lines

include/uapi/linux/perf_event.h union perf_mem_data_src { · line 1344

Quoted 4 lines
include/uapi/linux/perf_event.h +3 -0
+                           reserved:40;
+           };
+   };
 };

Hurpmh... all other bitfields have ENDIAN_BITFIELD things except this one. Power folks, could you please have a look?

The bit number of each field changes between big and little endian, but as long as kernel and userspace are the same endian, and both only access values via the bitfields then it works.

...

Quoted 19 lines

It does look like we have a bug in perf tool though, if I take a perf.data from a big endian system to a little endian one I don't see any of the branch flags decoded. eg:

BE:

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles P 0

LE:

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles 0

                                                         ^
                                                         missing P
Quoted 9 lines
Quoted 1 line

I guess we're missing a byte swap somewhere.

Ugh. We _do_ have a byte swap, but we also need a bit swap.

That works for the single bit fields, not sure if it will for the multi-bit fields.

So that's a bit of a mess :/

Based on what I see in perf_event.h for other structures, I think I can make up what you would need for struct branch_entry. But Iit would be easier if you could send me a patch that you would have verified on your systems.
Thanks.

#6 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Quoted 66 lines · Stephane Eranian

Michael,

Quoted 62 lines · Michael Ellerman
Quoted 36 lines · Michael Ellerman
Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Stephane Eranian
include/uapi/linux/perf_event.h +8 -7
index f92880a15645..eb11f383f4be 100644
@@ -1329,13 +1329,18 @@ union perf_mem_data_src {
  struct perf_branch_entry {
     __u64   from;
     __u64   to;
-   __u64   mispred:1,  /* target mispredicted */
-           predicted:1,/* target predicted */
-           in_tx:1,    /* in transaction */
-           abort:1,    /* transaction abort */
-           cycles:16,  /* cycle count to last branch */
-           type:4,     /* branch type */
-           reserved:40;
+   union {
+           __u64   val;        /* to make it easier to clear all fields */
+           struct {
+                   __u64   mispred:1,  /* target mispredicted */
+                           predicted:1,/* target predicted */
+                           in_tx:1,    /* in transaction */
+                           abort:1,    /* transaction abort */
+                           cycles:16,  /* cycle count to last branch */
Quoted 8 lines

include/uapi/linux/perf_event.h union perf_mem_data_src { · line 1344

Quoted 5 lines
include/uapi/linux/perf_event.h +4 -0
+                           type:4,     /* branch type */
+                           reserved:40;
+           };
+   };
  };

Hurpmh... all other bitfields have ENDIAN_BITFIELD things except this one. Power folks, could you please have a look?

The bit number of each field changes between big and little endian, but as long as kernel and userspace are the same endian, and both only access values via the bitfields then it works.

...

Quoted 12 lines

It does look like we have a bug in perf tool though, if I take a perf.data from a big endian system to a little endian one I don't see any of the branch flags decoded. eg:

BE:

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles P 0

LE:

Quoted 7 lines

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles 0

                                                          ^
                                                          missing P

I guess we're missing a byte swap somewhere.

Ugh. We _do_ have a byte swap, but we also need a bit swap.

That works for the single bit fields, not sure if it will for the multi-bit fields.

Quoted 7 lines
Quoted 2 lines

So that's a bit of a mess :/

Based on what I see in perf_event.h for other structures, I think I can make up what you would need for struct branch_entry. But Iit would be easier if you could send me a patch that you would have verified on your systems.
Thanks.

Attached patch fixes the issue. Have tested both in both in BE and LE case.

Maddy

From f816ba2e6ef8d5975f78442d7ecb50d66c3c4326 Mon Sep 17 00:00:00 2001
From: Madhavan Srinivasan <[email protected]>
Date: Wed, 15 Sep 2021 22:29:09 +0530
Subject: [RFC PATCH] tools/perf: Add reverse_64b macro

branch_stack struct has bit field definition producing different bit ordering for big/little endian. Because of this, when branch_stack sample collected in a BE system viewed/reported in a LE system, bit fields of the branch stack are not presented properly. To address this issue, a reverse_64b macro is defined and introduced in evsel__parse_sample.

---
 tools/perf/util/evsel.c | 35 +++++++++++++++++++++++++++++++++--

 1 file changed, 33 insertions(+), 2 deletions(-)
tools/perf/util/evsel.c +0 -0
index dbfeceb2546c..3151606e516e 100644
@@ -2221,6 +2221,9 @@ void __weak arch_perf_parse_sample_weight(struct perf_sample *data,

     data->weight = *array;
 }

+#define reverse_64b(src, pos, size)    \
+    (((src >> pos) & (( 1ull <<size) - 1)) << (63 - (pos + size - 1)))
+
 int evsel__parse_sample(struct evsel *evsel, union perf_event *event, struct perf_sample *data)
 {
@@ -2408,6 +2411,8 @@ int evsel__parse_sample(struct evsel *evsel, union perf_event *event, if (type & PERF_SAMPLE_BRANCH_STACK) {
         const u64 max_branch_nr = UINT64_MAX /
                       sizeof(struct branch_entry);
+        struct branch_entry *e;
+        unsigned i;

         OVERFLOW_CHECK_u64(array);
         data->branch_stack = (struct branch_stack *)array++;
@@ -2416,10 +2421,36 @@ int evsel__parse_sample(struct evsel *evsel, union perf_event *event, return -EFAULT;

         sz = data->branch_stack->nr * sizeof(struct branch_entry); -        if (evsel__has_branch_hw_idx(evsel))
+        if (evsel__has_branch_hw_idx(evsel)) {
             sz += sizeof(u64);
-        else
+            e = &data->branch_stack->entries[0];
+        } else {
             data->no_hw_idx = true;
+            e = (struct branch_entry *)&data->branch_stack->hw_idx; +        }
+
+        if (swapped) {
+            for (i = 0; i < data->branch_stack->nr; i++, e++) { +                u64 new_val = 0;
+
+                /* mispred:1  target mispredicted */
+                new_val = reverse_64b(e->flags.value, 0, 1); +                /* predicted:1  target predicted */
+                new_val |= reverse_64b(e->flags.value, 1, 1); +                /* in_tx:1  in transaction */
+                new_val |= reverse_64b(e->flags.value, 2, 1); +                /* abort:1  transaction abort */
+                new_val |= reverse_64b(e->flags.value, 3, 1); +                /* cycles:16  cycle count to last branch */ +                new_val |= reverse_64b(e->flags.value, 4, 16); +                /* type:4  branch type */
+                new_val |= reverse_64b(e->flags.value, 20, 4); +                /* reserved:40 */
+                new_val |= reverse_64b(e->flags.value, 24, 40); +                e->flags.value = new_val;
+            }
+        }
+
         OVERFLOW_CHECK(array, sz, max_size);
         array = (void *)array + sz;
     }

Signature · 1 line
2.31.1
#7 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Hi,

Thanks for fixing this in the perf tool. But what about the struct branch_entry in the header?

Quoted 167 lines · Madhavan Srinivasan
Quoted 66 lines · Stephane Eranian

Michael,

Quoted 62 lines · Michael Ellerman
Quoted 36 lines · Michael Ellerman
Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Stephane Eranian
include/uapi/linux/perf_event.h +8 -7
index f92880a15645..eb11f383f4be 100644
@@ -1329,13 +1329,18 @@ union perf_mem_data_src {
  struct perf_branch_entry {
     __u64   from;
     __u64   to;
-   __u64   mispred:1,  /* target mispredicted */
-           predicted:1,/* target predicted */
-           in_tx:1,    /* in transaction */
-           abort:1,    /* transaction abort */
-           cycles:16,  /* cycle count to last branch */
-           type:4,     /* branch type */
-           reserved:40;
+   union {
+           __u64   val;        /* to make it easier to clear all fields */
+           struct {
+                   __u64   mispred:1,  /* target mispredicted */
+                           predicted:1,/* target predicted */
+                           in_tx:1,    /* in transaction */
+                           abort:1,    /* transaction abort */
+                           cycles:16,  /* cycle count to last branch */
Quoted 8 lines

include/uapi/linux/perf_event.h union perf_mem_data_src { · line 1344

Quoted 5 lines
include/uapi/linux/perf_event.h +4 -0
+                           type:4,     /* branch type */
+                           reserved:40;
+           };
+   };
  };

Hurpmh... all other bitfields have ENDIAN_BITFIELD things except this one. Power folks, could you please have a look?

The bit number of each field changes between big and little endian, but as long as kernel and userspace are the same endian, and both only access values via the bitfields then it works.

...

Quoted 12 lines

It does look like we have a bug in perf tool though, if I take a perf.data from a big endian system to a little endian one I don't see any of the branch flags decoded. eg:

BE:

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles P 0

LE:

Quoted 7 lines

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles 0

                                                          ^
                                                          missing P

I guess we're missing a byte swap somewhere.

Ugh. We _do_ have a byte swap, but we also need a bit swap.

That works for the single bit fields, not sure if it will for the multi-bit fields.

Quoted 7 lines
Quoted 2 lines

So that's a bit of a mess :/

Based on what I see in perf_event.h for other structures, I think I can make up what you would need for struct branch_entry. But Iit would be easier if you could send me a patch that you would have verified on your systems.
Thanks.

Attached patch fixes the issue. Have tested both in both in BE and LE case.

Maddy

From f816ba2e6ef8d5975f78442d7ecb50d66c3c4326 Mon Sep 17 00:00:00 2001
From: Madhavan Srinivasan <[email protected]>
Date: Wed, 15 Sep 2021 22:29:09 +0530
Subject: [RFC PATCH] tools/perf: Add reverse_64b macro

branch_stack struct has bit field definition producing different bit ordering for big/little endian. Because of this, when branch_stack sample collected in a BE system viewed/reported in a LE system, bit fields of the branch stack are not presented properly. To address this issue, a reverse_64b macro is defined and introduced in evsel__parse_sample.

---
  tools/perf/util/evsel.c | 35 +++++++++++++++++++++++++++++++++--
  1 file changed, 33 insertions(+), 2 deletions(-)
tools/perf/util/evsel.c +3 -0
index dbfeceb2546c..3151606e516e 100644
@@ -2221,6 +2221,9 @@ void __weak arch_perf_parse_sample_weight(struct
 perf_sample *data,
      data->weight = *array;
  }
 
+#define reverse_64b(src, pos, size)    \
+    (((src >> pos) & (( 1ull <<size) - 1)) << (63 - (pos + size - 1)))
+
  int evsel__parse_sample(struct evsel *evsel, union perf_event *event,
              struct perf_sample *data)
  {
tools/perf/util/evsel.c +2 -0
@@ -2408,6 +2411,8 @@ int evsel__parse_sample(struct evsel *evsel, union
 perf_event *event,
      if (type & PERF_SAMPLE_BRANCH_STACK) {
          const u64 max_branch_nr = UINT64_MAX /
                        sizeof(struct branch_entry);
+        struct branch_entry *e;
+        unsigned i;
 
          OVERFLOW_CHECK_u64(array);
          data->branch_stack = (struct branch_stack *)array++;
tools/perf/util/evsel.c +28 -2
@@ -2416,10 +2421,36 @@ int evsel__parse_sample(struct evsel *evsel,
 union perf_event *event,
              return -EFAULT;
 
          sz = data->branch_stack->nr * sizeof(struct branch_entry);
-        if (evsel__has_branch_hw_idx(evsel))
+        if (evsel__has_branch_hw_idx(evsel)) {
              sz += sizeof(u64);
-        else
+            e = &data->branch_stack->entries[0];
+        } else {
              data->no_hw_idx = true;
+            e = (struct branch_entry *)&data->branch_stack->hw_idx;
+        }
+
+        if (swapped) {
+            for (i = 0; i < data->branch_stack->nr; i++, e++) {
+                u64 new_val = 0;
+
+                /* mispred:1  target mispredicted */
+                new_val = reverse_64b(e->flags.value, 0, 1);
+                /* predicted:1  target predicted */
+                new_val |= reverse_64b(e->flags.value, 1, 1);
+                /* in_tx:1  in transaction */
+                new_val |= reverse_64b(e->flags.value, 2, 1);
+                /* abort:1  transaction abort */
+                new_val |= reverse_64b(e->flags.value, 3, 1);
+                /* cycles:16  cycle count to last branch */
+                new_val |= reverse_64b(e->flags.value, 4, 16);
+                /* type:4  branch type */
+                new_val |= reverse_64b(e->flags.value, 20, 4);
+                /* reserved:40 */
+                new_val |= reverse_64b(e->flags.value, 24, 40);
+                e->flags.value = new_val;
+            }
+        }
+
          OVERFLOW_CHECK(array, sz, max_size);
          array = (void *)array + sz;
      }

--
2.31.1

#8 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Quoted 4 lines · Stephane Eranian

Hi,

Thanks for fixing this in the perf tool. But what about the struct branch_entry in the header?

I'm not sure what you mean.

We can't change the order of the fields in the header, without breaking existing userspace on BE systems.

It's annoying that the bit numbers are different between LE & BE, but I think it's too late to change that.

So nothing should change in the branch_entry definition in the header.

My comment on your patch was that adding the union with val, makes it easier to misuse the bitfields, because now the values can be accessed via the bitfields and also via val, but when using val you have to know that the bit numbers differ between BE/LE.

Maybe that's over-paranoid on my part, but if we just want val for clearing the values easily then I think the static inline I suggested is preferable.

cheers

Quoted 169 lines
Quoted 167 lines · Madhavan Srinivasan
Quoted 66 lines · Stephane Eranian

Michael,

Quoted 62 lines · Michael Ellerman
Quoted 36 lines · Michael Ellerman
Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Stephane Eranian
include/uapi/linux/perf_event.h +8 -7
index f92880a15645..eb11f383f4be 100644
@@ -1329,13 +1329,18 @@ union perf_mem_data_src {
  struct perf_branch_entry {
     __u64   from;
     __u64   to;
-   __u64   mispred:1,  /* target mispredicted */
-           predicted:1,/* target predicted */
-           in_tx:1,    /* in transaction */
-           abort:1,    /* transaction abort */
-           cycles:16,  /* cycle count to last branch */
-           type:4,     /* branch type */
-           reserved:40;
+   union {
+           __u64   val;        /* to make it easier to clear all fields */
+           struct {
+                   __u64   mispred:1,  /* target mispredicted */
+                           predicted:1,/* target predicted */
+                           in_tx:1,    /* in transaction */
+                           abort:1,    /* transaction abort */
+                           cycles:16,  /* cycle count to last branch */
Quoted 8 lines

include/uapi/linux/perf_event.h union perf_mem_data_src { · line 1344

Quoted 5 lines
include/uapi/linux/perf_event.h +4 -0
+                           type:4,     /* branch type */
+                           reserved:40;
+           };
+   };
  };

Hurpmh... all other bitfields have ENDIAN_BITFIELD things except this one. Power folks, could you please have a look?

The bit number of each field changes between big and little endian, but as long as kernel and userspace are the same endian, and both only access values via the bitfields then it works.

...

Quoted 12 lines

It does look like we have a bug in perf tool though, if I take a perf.data from a big endian system to a little endian one I don't see any of the branch flags decoded. eg:

BE:

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles P 0

LE:

Quoted 7 lines

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles 0

                                                          ^
                                                          missing P

I guess we're missing a byte swap somewhere.

Ugh. We _do_ have a byte swap, but we also need a bit swap.

That works for the single bit fields, not sure if it will for the multi-bit fields.

Quoted 7 lines
Quoted 2 lines

So that's a bit of a mess :/

Based on what I see in perf_event.h for other structures, I think I can make up what you would need for struct branch_entry. But Iit would be easier if you could send me a patch that you would have verified on your systems.
Thanks.

Attached patch fixes the issue. Have tested both in both in BE and LE case.

Maddy

From f816ba2e6ef8d5975f78442d7ecb50d66c3c4326 Mon Sep 17 00:00:00 2001
From: Madhavan Srinivasan <[email protected]>
Date: Wed, 15 Sep 2021 22:29:09 +0530
Subject: [RFC PATCH] tools/perf: Add reverse_64b macro

branch_stack struct has bit field definition producing different bit ordering for big/little endian. Because of this, when branch_stack sample collected in a BE system viewed/reported in a LE system, bit fields of the branch stack are not presented properly. To address this issue, a reverse_64b macro is defined and introduced in evsel__parse_sample.

---
  tools/perf/util/evsel.c | 35 +++++++++++++++++++++++++++++++++--
  1 file changed, 33 insertions(+), 2 deletions(-)
tools/perf/util/evsel.c +3 -0
index dbfeceb2546c..3151606e516e 100644
@@ -2221,6 +2221,9 @@ void __weak arch_perf_parse_sample_weight(struct
 perf_sample *data,
      data->weight = *array;
  }
 
+#define reverse_64b(src, pos, size)    \
+    (((src >> pos) & (( 1ull <<size) - 1)) << (63 - (pos + size - 1)))
+
  int evsel__parse_sample(struct evsel *evsel, union perf_event *event,
              struct perf_sample *data)
  {
tools/perf/util/evsel.c +2 -0
@@ -2408,6 +2411,8 @@ int evsel__parse_sample(struct evsel *evsel, union
 perf_event *event,
      if (type & PERF_SAMPLE_BRANCH_STACK) {
          const u64 max_branch_nr = UINT64_MAX /
                        sizeof(struct branch_entry);
+        struct branch_entry *e;
+        unsigned i;
 
          OVERFLOW_CHECK_u64(array);
          data->branch_stack = (struct branch_stack *)array++;
tools/perf/util/evsel.c +28 -2
@@ -2416,10 +2421,36 @@ int evsel__parse_sample(struct evsel *evsel,
 union perf_event *event,
              return -EFAULT;
 
          sz = data->branch_stack->nr * sizeof(struct branch_entry);
-        if (evsel__has_branch_hw_idx(evsel))
+        if (evsel__has_branch_hw_idx(evsel)) {
              sz += sizeof(u64);
-        else
+            e = &data->branch_stack->entries[0];
+        } else {
              data->no_hw_idx = true;
+            e = (struct branch_entry *)&data->branch_stack->hw_idx;
+        }
+
+        if (swapped) {
+            for (i = 0; i < data->branch_stack->nr; i++, e++) {
+                u64 new_val = 0;
+
+                /* mispred:1  target mispredicted */
+                new_val = reverse_64b(e->flags.value, 0, 1);
+                /* predicted:1  target predicted */
+                new_val |= reverse_64b(e->flags.value, 1, 1);
+                /* in_tx:1  in transaction */
+                new_val |= reverse_64b(e->flags.value, 2, 1);
+                /* abort:1  transaction abort */
+                new_val |= reverse_64b(e->flags.value, 3, 1);
+                /* cycles:16  cycle count to last branch */
+                new_val |= reverse_64b(e->flags.value, 4, 16);
+                /* type:4  branch type */
+                new_val |= reverse_64b(e->flags.value, 20, 4);
+                /* reserved:40 */
+                new_val |= reverse_64b(e->flags.value, 24, 40);
+                e->flags.value = new_val;
+            }
+        }
+
          OVERFLOW_CHECK(array, sz, max_size);
          array = (void *)array + sz;
      }

--
2.31.1

#9 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Hi,

Quoted 4 lines · Michael Ellerman
Quoted 2 lines · Stephane Eranian

Hi,

Quoted 8 lines
Quoted 2 lines

Thanks for fixing this in the perf tool. But what about the struct branch_entry in the header?

I'm not sure what you mean.

We can't change the order of the fields in the header, without breaking existing userspace on BE systems.

Ok, I think I had missed that. You are saying that the #ifdef (__BIG_ENDIAN_BITFIELD) vs __LITTLE_ENDIAN_BITFIELD

is only added to kernel-only data structures?

Quoted 3 lines

It's annoying that the bit numbers are different between LE & BE, but I think it's too late to change that.

I agree.

Quoted 7 lines

So nothing should change in the branch_entry definition in the header.

My comment on your patch was that adding the union with val, makes it easier to misuse the bitfields, because now the values can be accessed via the bitfields and also via val, but when using val you have to know that the bit numbers differ between BE/LE.

Ok, I get it now. We do not need to expose val to user. This is added for kernel code
convenience only. But if we keep it in kernel, that may break some other rules about
uapi headers.

Quoted 175 lines

Maybe that's over-paranoid on my part, but if we just want val for clearing the values easily then I think the static inline I suggested is preferable.

cheers

Quoted 169 lines
Quoted 167 lines · Madhavan Srinivasan
Quoted 66 lines · Stephane Eranian

Michael,

Quoted 62 lines · Michael Ellerman
Quoted 36 lines · Michael Ellerman
Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Stephane Eranian
include/uapi/linux/perf_event.h +8 -7
index f92880a15645..eb11f383f4be 100644
@@ -1329,13 +1329,18 @@ union perf_mem_data_src {
  struct perf_branch_entry {
     __u64   from;
     __u64   to;
-   __u64   mispred:1,  /* target mispredicted */
-           predicted:1,/* target predicted */
-           in_tx:1,    /* in transaction */
-           abort:1,    /* transaction abort */
-           cycles:16,  /* cycle count to last branch */
-           type:4,     /* branch type */
-           reserved:40;
+   union {
+           __u64   val;        /* to make it easier to clear all fields */
+           struct {
+                   __u64   mispred:1,  /* target mispredicted */
+                           predicted:1,/* target predicted */
+                           in_tx:1,    /* in transaction */
+                           abort:1,    /* transaction abort */
+                           cycles:16,  /* cycle count to last branch */
Quoted 8 lines

include/uapi/linux/perf_event.h union perf_mem_data_src { · line 1344

Quoted 5 lines
include/uapi/linux/perf_event.h +4 -0
+                           type:4,     /* branch type */
+                           reserved:40;
+           };
+   };
  };

Hurpmh... all other bitfields have ENDIAN_BITFIELD things except this one. Power folks, could you please have a look?

The bit number of each field changes between big and little endian, but as long as kernel and userspace are the same endian, and both only access values via the bitfields then it works.

...

Quoted 12 lines

It does look like we have a bug in perf tool though, if I take a perf.data from a big endian system to a little endian one I don't see any of the branch flags decoded. eg:

BE:

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles P 0

LE:

Quoted 7 lines

2413132652524 0x1db8 [0x2d0]: PERF_RECORD_SAMPLE(IP, 0x1): 5279/5279: 0xc00000000045c028 period: 923003 addr: 0 ... branch stack: nr:28
..... 0: c00000000045c028 -> c00000000dce7604 0 cycles 0

                                                          ^
                                                          missing P

I guess we're missing a byte swap somewhere.

Ugh. We _do_ have a byte swap, but we also need a bit swap.

That works for the single bit fields, not sure if it will for the multi-bit fields.

Quoted 7 lines
Quoted 2 lines

So that's a bit of a mess :/

Based on what I see in perf_event.h for other structures, I think I can make up what you would need for struct branch_entry. But Iit would be easier if you could send me a patch that you would have verified on your systems.
Thanks.

Attached patch fixes the issue. Have tested both in both in BE and LE case.

Maddy

From f816ba2e6ef8d5975f78442d7ecb50d66c3c4326 Mon Sep 17 00:00:00 2001
From: Madhavan Srinivasan <[email protected]>
Date: Wed, 15 Sep 2021 22:29:09 +0530
Subject: [RFC PATCH] tools/perf: Add reverse_64b macro

branch_stack struct has bit field definition producing different bit ordering for big/little endian. Because of this, when branch_stack sample collected in a BE system viewed/reported in a LE system, bit fields of the branch stack are not presented properly. To address this issue, a reverse_64b macro is defined and introduced in evsel__parse_sample.

---
  tools/perf/util/evsel.c | 35 +++++++++++++++++++++++++++++++++--
  1 file changed, 33 insertions(+), 2 deletions(-)
tools/perf/util/evsel.c +3 -0
index dbfeceb2546c..3151606e516e 100644
@@ -2221,6 +2221,9 @@ void __weak arch_perf_parse_sample_weight(struct
 perf_sample *data,
      data->weight = *array;
  }
 
+#define reverse_64b(src, pos, size)    \
+    (((src >> pos) & (( 1ull <<size) - 1)) << (63 - (pos + size - 1)))
+
  int evsel__parse_sample(struct evsel *evsel, union perf_event *event,
              struct perf_sample *data)
  {
tools/perf/util/evsel.c +2 -0
@@ -2408,6 +2411,8 @@ int evsel__parse_sample(struct evsel *evsel, union
 perf_event *event,
      if (type & PERF_SAMPLE_BRANCH_STACK) {
          const u64 max_branch_nr = UINT64_MAX /
                        sizeof(struct branch_entry);
+        struct branch_entry *e;
+        unsigned i;
 
          OVERFLOW_CHECK_u64(array);
          data->branch_stack = (struct branch_stack *)array++;
tools/perf/util/evsel.c +28 -2
@@ -2416,10 +2421,36 @@ int evsel__parse_sample(struct evsel *evsel,
 union perf_event *event,
              return -EFAULT;
 
          sz = data->branch_stack->nr * sizeof(struct branch_entry);
-        if (evsel__has_branch_hw_idx(evsel))
+        if (evsel__has_branch_hw_idx(evsel)) {
              sz += sizeof(u64);
-        else
+            e = &data->branch_stack->entries[0];
+        } else {
              data->no_hw_idx = true;
+            e = (struct branch_entry *)&data->branch_stack->hw_idx;
+        }
+
+        if (swapped) {
+            for (i = 0; i < data->branch_stack->nr; i++, e++) {
+                u64 new_val = 0;
+
+                /* mispred:1  target mispredicted */
+                new_val = reverse_64b(e->flags.value, 0, 1);
+                /* predicted:1  target predicted */
+                new_val |= reverse_64b(e->flags.value, 1, 1);
+                /* in_tx:1  in transaction */
+                new_val |= reverse_64b(e->flags.value, 2, 1);
+                /* abort:1  transaction abort */
+                new_val |= reverse_64b(e->flags.value, 3, 1);
+                /* cycles:16  cycle count to last branch */
+                new_val |= reverse_64b(e->flags.value, 4, 16);
+                /* type:4  branch type */
+                new_val |= reverse_64b(e->flags.value, 20, 4);
+                /* reserved:40 */
+                new_val |= reverse_64b(e->flags.value, 24, 40);
+                e->flags.value = new_val;
+            }
+        }
+
          OVERFLOW_CHECK(array, sz, max_size);
          array = (void *)array + sz;
      }

--
2.31.1

#10 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Quoted 12 lines · Stephane Eranian

Hi,

Quoted 9 lines · Michael Ellerman
Quoted 4 lines · Stephane Eranian

Hi,

Thanks for fixing this in the perf tool. But what about the struct branch_entry in the header?

I'm not sure what you mean.

Quoted 7 lines
Quoted 3 lines

We can't change the order of the fields in the header, without breaking existing userspace on BE systems.

Ok, I think I had missed that. You are saying that the #ifdef (__BIG_ENDIAN_BITFIELD) vs __LITTLE_ENDIAN_BITFIELD

is only added to kernel-only data structures?

No, we *should* have used __BIG/LITTLE_ENDIAN_BITFIELD for the uapi definition, but we forgot.

Quoted 7 lines
Quoted 3 lines

It's annoying that the bit numbers are different between LE & BE, but I think it's too late to change that.

I agree.

Quoted 2 lines

So nothing should change in the branch_entry definition in the header.

Quoted 7 lines
Quoted 5 lines

My comment on your patch was that adding the union with val, makes it easier to misuse the bitfields, because now the values can be accessed via the bitfields and also via val, but when using val you have to know that the bit numbers differ between BE/LE.

Ok, I get it now. We do not need to expose val to user. This is added for kernel code convenience only.

Yeah. Putting the union with val in the uapi encourages userspace to misuse val to bypass the bitfields, and that risks causing endian bugs.

Quoted 2 lines

But if we keep it in kernel, that may break some other rules about uapi headers.

I don't follow what you mean there.

We could use #ifdef __KERNEL__ in the uapi header to make the union kernel-only, see below, but it's pretty gross.

 struct perf_branch_entry {
	__u64	from;
	__u64	to;
 #ifdef __KERNEL__
	union {
		__u64	val;	    /* to make it easier to clear all fields */
		struct {
 #endif
			__u64	mispred:1,  /* target mispredicted */
				predicted:1,/* target predicted */
				in_tx:1,    /* in transaction */
				abort:1,    /* transaction abort */
				cycles:16,  /* cycle count to last branch */
				type:4,     /* branch type */
				reserved:40;
 #ifdef __KERNEL__
		};
	};

#endif
};

If we just do the inline I suggested we can clear the flags in a single source line, and the generated code seems fine too, eg:

static inline void clear_perf_branch_entry_flags(struct perf_branch_entry *e)

{
	e->mispred = 0;
	e->predicted = 0;
	e->in_tx = 0;
	e->abort = 0;
	e->cycles = 0;
	e->type = 0;
	e->reserved = 0;
}

cheers

#11 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Quoted 17 lines · Michael Ellerman
Quoted 15 lines · Stephane Eranian

Hi,

Quoted 12 lines · Michael Ellerman
Quoted 4 lines · Stephane Eranian

Hi,

Thanks for fixing this in the perf tool. But what about the struct branch_entry in the header?

I'm not sure what you mean.

We can't change the order of the fields in the header, without breaking existing userspace on BE systems.

Quoted 8 lines
Quoted 4 lines

Ok, I think I had missed that. You are saying that the #ifdef (__BIG_ENDIAN_BITFIELD) vs __LITTLE_ENDIAN_BITFIELD

is only added to kernel-only data structures?

No, we *should* have used __BIG/LITTLE_ENDIAN_BITFIELD for the uapi definition, but we forgot.

But are you suggesting it cannot be fixed?

Quoted 52 lines
Quoted 7 lines
Quoted 3 lines

It's annoying that the bit numbers are different between LE & BE, but I think it's too late to change that.

I agree.

Quoted 2 lines

So nothing should change in the branch_entry definition in the header.

Quoted 7 lines
Quoted 5 lines

My comment on your patch was that adding the union with val, makes it easier to misuse the bitfields, because now the values can be accessed via the bitfields and also via val, but when using val you have to know that the bit numbers differ between BE/LE.

Ok, I get it now. We do not need to expose val to user. This is added for kernel code convenience only.

Yeah. Putting the union with val in the uapi encourages userspace to misuse val to bypass the bitfields, and that risks causing endian bugs.

Quoted 2 lines

But if we keep it in kernel, that may break some other rules about uapi headers.

I don't follow what you mean there.

We could use #ifdef __KERNEL__ in the uapi header to make the union kernel-only, see below, but it's pretty gross.

 struct perf_branch_entry {
        __u64   from;
        __u64   to;
 #ifdef __KERNEL__
        union {
                __u64   val;        /* to make it easier to clear all fields */
                struct {
 #endif
                        __u64   mispred:1,  /* target mispredicted */
                                predicted:1,/* target predicted */
                                in_tx:1,    /* in transaction */
                                abort:1,    /* transaction abort */
                                cycles:16,  /* cycle count to last branch */
                                type:4,     /* branch type */
                                reserved:40;
 #ifdef __KERNEL__
                };
        };

#endif
};

If we just do the inline I suggested we can clear the flags in a single source line, and the generated code seems fine too, eg:

static inline void clear_perf_branch_entry_flags(struct perf_branch_entry *e)

Quoted 10 lines
{
        e->mispred = 0;
        e->predicted = 0;
        e->in_tx = 0;
        e->abort = 0;
        e->cycles = 0;
        e->type = 0;
        e->reserved = 0;
}

Ok, let's do the inline then. That looks like a cleaner solution to me assuming the compiler does the right thing.

#12 lore raw

Re: [PATCH v1 01/13] perf/core: add union to struct perf_branch_entry

Quoted 21 lines · Stephane Eranian
Quoted 20 lines · Michael Ellerman
Quoted 18 lines · Stephane Eranian

Hi,

Quoted 4 lines · Michael Ellerman
Quoted 2 lines · Stephane Eranian

Hi,

Quoted 8 lines
Quoted 2 lines

Thanks for fixing this in the perf tool. But what about the struct branch_entry in the header?

I'm not sure what you mean.

We can't change the order of the fields in the header, without breaking existing userspace on BE systems.

Ok, I think I had missed that. You are saying that the #ifdef (__BIG_ENDIAN_BITFIELD) vs __LITTLE_ENDIAN_BITFIELD

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

is only added to kernel-only data structures?

No, we *should* have used __BIG/LITTLE_ENDIAN_BITFIELD for the uapi definition, but we forgot.

But are you suggesting it cannot be fixed?

I'm not saying it *cannot* be fixed

But I don't think it's sufficiently broken to warrant fixing.

Just adding the __BIG/LITTLE_ENDIAN_BITFIELD ifdefs would break the ABI for existing users.

So we'd have to continue to support the existing bitfield layout, and then add a flag for userspace to request a new bitfield layout where the bit numbers are stable across endian.

But that's way too much effort IMHO.

The existing definition works fine, *except* when perf.data files are moved between machines of different endianness. That is pretty rare these days, and can be handled in the perf tool easily enough.

Quoted 62 lines
Quoted 52 lines
Quoted 7 lines
Quoted 3 lines

It's annoying that the bit numbers are different between LE & BE, but I think it's too late to change that.

I agree.

Quoted 2 lines

So nothing should change in the branch_entry definition in the header.

Quoted 7 lines
Quoted 5 lines

My comment on your patch was that adding the union with val, makes it easier to misuse the bitfields, because now the values can be accessed via the bitfields and also via val, but when using val you have to know that the bit numbers differ between BE/LE.

Ok, I get it now. We do not need to expose val to user. This is added for kernel code convenience only.

Yeah. Putting the union with val in the uapi encourages userspace to misuse val to bypass the bitfields, and that risks causing endian bugs.

Quoted 2 lines

But if we keep it in kernel, that may break some other rules about uapi headers.

I don't follow what you mean there.

We could use #ifdef __KERNEL__ in the uapi header to make the union kernel-only, see below, but it's pretty gross.

 struct perf_branch_entry {
        __u64   from;
        __u64   to;
 #ifdef __KERNEL__
        union {
                __u64   val;        /* to make it easier to clear all fields */
                struct {
 #endif
                        __u64   mispred:1,  /* target mispredicted */
                                predicted:1,/* target predicted */
                                in_tx:1,    /* in transaction */
                                abort:1,    /* transaction abort */
                                cycles:16,  /* cycle count to last branch */
                                type:4,     /* branch type */
                                reserved:40;
 #ifdef __KERNEL__
                };
        };

#endif
};

If we just do the inline I suggested we can clear the flags in a single source line, and the generated code seems fine too, eg:

static inline void clear_perf_branch_entry_flags(struct perf_branch_entry *e)

Quoted 10 lines
{
        e->mispred = 0;
        e->predicted = 0;
        e->in_tx = 0;
        e->abort = 0;
        e->cycles = 0;
        e->type = 0;
        e->reserved = 0;
}
Quoted 2 lines

Ok, let's do the inline then. That looks like a cleaner solution to me assuming the compiler does the right thing.

It did when I checked with GCC 10.

cheers

#13 lore raw

PATCH 2/13 x86/cpufeatures: add AMD Fam19h Branch Sampling feature

In reply to Stephane Eranian

This patch adds a cpu feature for AMD Fam19h Branch Sampling feature as bit 31 of EBX on CPUID leaf function 0x80000008.

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index d0ce5cfd3ac1..74fd7ab7d74d 100644
@@ -313,6 +313,7 @@
 #define X86_FEATURE_AMD_SSBD		(13*32+24) /* "" Speculative Store Bypass Disable */
 #define X86_FEATURE_VIRT_SSBD		(13*32+25) /* Virtualized Speculative Store Bypass Disable */
 #define X86_FEATURE_AMD_SSB_NO		(13*32+26) /* "" Speculative Store Bypass is fixed in hardware. */
+#define X86_FEATURE_AMD_BRS		(13*32+31) /* Branch Sampling available */
 
 /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */
 #define X86_FEATURE_DTHERM		(14*32+ 0) /* Digital Thermal Sensor */
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
#14 lore raw

PATCH 3/13 perf/x86/amd: add AMD Fam19h Branch Sampling support

In reply to Stephane Eranian

This patch adds support for the AMD Fam19h 16-deep branch sampling feature as described in the AMD PPR Fam19h Model 01h Revision B1. This is a model specific extension. It is not an architected AMD feature.

The Branch Sampling (BRS) operates with a 16-deep saturating buffer in MSR registers. There is no branch type filtering. All control flow changes are captured. BRS relies on specific programming of the core PMU of Fam19h. In particular, the following requirements must be met:
- the sampling period be greater than 16 (BRS depth)
- the sampling period must use a fixed and not frequency mode

BRS interacts with the NMI interrupt as well. Because enabling BRS is expensive, it is only activated after P event occurrences, where P is the desired sampling period. At P occurrences of the event, the counter overflows, the CPU catches the interrupt, activates BRS for 16 branches until it saturates, and then delivers the NMI to the kernel. Between the overflow and the time BRS activates more branches may be executed skewing the period. All along, the sampling event keeps counting. The skid may be attenuated by reducing the sampling period by 16 (subsequent patch).

BRS is integrated into perf_events seamlessly via the same PERF_RECORD_BRANCH_STACK sample format. BRS generates perf_branch_entry records in the sampling buffer. No prediction information is supported. The branches are stored in reverse order of execution. The most recent branch is the first entry in each record.

No modification to the perf tool is necessary.

BRS can be used with any sampling event. However, it is recommended to use the RETIRED_BRANCH_INSTRUCTIONS event because it matches what the BRS captures.

$ perf record -b -c 1000037 -e cpu/event=0xc2,name=retired_branch_instructions/ test
$ perf report -D
56531696056126 0x193c000 [0x1a8]: PERF_RECORD_SAMPLE(IP, 0x2): 18122/18230: 0x401d24 period: 1000037 addr: 0
... branch stack: nr:16
.....  0: 0000000000401d24 -> 0000000000401d5a 0 cycles      0
.....  1: 0000000000401d5c -> 0000000000401d24 0 cycles      0
.....  2: 0000000000401d22 -> 0000000000401d5c 0 cycles      0
.....  3: 0000000000401d5e -> 0000000000401d22 0 cycles      0
.....  4: 0000000000401d20 -> 0000000000401d5e 0 cycles      0
.....  5: 0000000000401d3e -> 0000000000401d20 0 cycles      0
.....  6: 0000000000401d42 -> 0000000000401d3e 0 cycles      0
.....  7: 0000000000401d3c -> 0000000000401d42 0 cycles      0
.....  8: 0000000000401d44 -> 0000000000401d3c 0 cycles      0
.....  9: 0000000000401d3a -> 0000000000401d44 0 cycles      0
..... 10: 0000000000401d46 -> 0000000000401d3a 0 cycles      0
..... 11: 0000000000401d38 -> 0000000000401d46 0 cycles      0
..... 12: 0000000000401d48 -> 0000000000401d38 0 cycles      0
..... 13: 0000000000401d36 -> 0000000000401d48 0 cycles      0
..... 14: 0000000000401d4a -> 0000000000401d36 0 cycles      0
..... 15: 0000000000401d34 -> 0000000000401d4a 0 cycles      0
 ... thread: test:18230
 ...... dso: test
---
 arch/x86/events/amd/Makefile     |   2 +-
 arch/x86/events/amd/brs.c        | 303 +++++++++++++++++++++++++++++++
 arch/x86/events/amd/core.c       | 202 ++++++++++++++++++++-
 arch/x86/events/core.c           |  15 +-
 arch/x86/events/perf_event.h     |  93 ++++++++--
 arch/x86/include/asm/msr-index.h |   4 +
 6 files changed, 595 insertions(+), 24 deletions(-)
 create mode 100644 arch/x86/events/amd/brs.c
arch/x86/events/amd/Makefile +1 -1
index 6cbe38d5fd9d..cf323ffab5cd 100644
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_CPU_SUP_AMD)		+= core.o
+obj-$(CONFIG_CPU_SUP_AMD)		+= core.o brs.o
 obj-$(CONFIG_PERF_EVENTS_AMD_POWER)	+= power.o
 obj-$(CONFIG_X86_LOCAL_APIC)		+= ibs.o
 obj-$(CONFIG_PERF_EVENTS_AMD_UNCORE)	+= amd-uncore.o
arch/x86/events/amd/brs.c new file +303 -0

303 lines not on this page · raw message

arch/x86/events/amd/core.c +196 -6

285 lines not on this page · raw message

arch/x86/events/core.c +14 -1
index 2a57dbed4894..66a073f21bb7 100644
@@ -681,11 +681,16 @@ void x86_pmu_disable_all(void)
 
 		if (!test_bit(idx, cpuc->active_mask))
 			continue;
+
+		if (is_amd_brs(hwc))
+			amd_brs_disable();
+
 		rdmsrl(x86_pmu_config_addr(idx), val);
 		if (!(val & ARCH_PERFMON_EVENTSEL_ENABLE))
 			continue;
 		val &= ~ARCH_PERFMON_EVENTSEL_ENABLE;
 		wrmsrl(x86_pmu_config_addr(idx), val);
+
 		if (is_counter_pair(hwc))
 			wrmsrl(x86_pmu_config_addr(idx + 1), 0);
 	}
@@ -1334,6 +1339,10 @@ static void x86_pmu_enable(struct pmu *pmu)
 			if (hwc->state & PERF_HES_ARCH)
 				continue;
 
+			/*
+			 * if cpuc->enabled = 0, then no wrmsr as
+			 * per x86_pmu_enable_event()
+			 */
 			x86_pmu_start(event, PERF_EF_RELOAD);
 		}
 		cpuc->n_added = 0;
@@ -1700,11 +1709,15 @@ int x86_pmu_handle_irq(struct pt_regs *regs)
 		 * event overflow
 		 */
 		handled++;
-		perf_sample_data_init(&data, 0, event->hw.last_period);
 
 		if (!x86_perf_event_set_period(event))
 			continue;
 
+		perf_sample_data_init(&data, 0, event->hw.last_period);
+
+		if (has_branch_stack(event))
+			data.br_stack = &cpuc->lbr_stack;
+
 		if (perf_event_overflow(event, &data, regs))
 			x86_pmu_stop(event, 0);
 	}
arch/x86/events/perf_event.h +77 -16

129 lines not on this page · raw message

arch/x86/include/asm/msr-index.h +4 -0
index a7c413432b33..ed73325a8499 100644
@@ -648,6 +648,10 @@
 #define MSR_IA32_PERF_CTL		0x00000199
 #define INTEL_PERF_CTL_MASK		0xffff
 
+/* AMD Branch Sampling configuration */
+#define MSR_AMD_DBG_EXTN_CFG		0xc000010f
+#define MSR_AMD_SAMP_BR_FROM		0xc0010300
+
 #define MSR_IA32_MPERF			0x000000e7
 #define MSR_IA32_APERF			0x000000e8
 
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
#15 lore raw

Re: [PATCH v1 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

Thank you for the patch! Perhaps something to improve:

Hi Stephane,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on tip/perf/core]
[also build test WARNING on next-20210909]
[cannot apply to tip/sched/core tip/x86/core v5.14]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Stephane-Eranian/perf-x86-amd-Add-AMD-Fam19h-Branch-Sampling-support/20210909-160050
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 4034fb207e302cc0b1f304084d379640c1fb1436
config: x86_64-allyesconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/7ff5f8448eecfc188c2240b2a5a5ea0b00f55662
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Stephane-Eranian/perf-x86-amd-Add-AMD-Fam19h-Branch-Sampling-support/20210909-160050
        git checkout 7ff5f8448eecfc188c2240b2a5a5ea0b00f55662
        # save the attached .config to linux build tree
        make W=1 ARCH=x86_64 

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

arch/x86/events/amd/core.c: In function 'amd_pmu_enable_all':

>> arch/x86/events/amd/core.c:676:24: warning: variable 'hwc' set but not used [-Wunused-but-set-variable]
     676 |  struct hw_perf_event *hwc;
         |                        ^~~

vim +/hwc +676 arch/x86/events/amd/core.c

672
673 static void amd_pmu_enable_all(int added)
674 {
675 struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);

 > 676		struct hw_perf_event *hwc;

677 int idx;
678
679 for (idx = 0; idx < x86_pmu.num_counters; idx++) {
680 hwc = &cpuc->events[idx]->hw;
681
682 /* only activate events which are marked as active */
683 if (!test_bit(idx, cpuc->active_mask))
684 continue;
685
686 amd_pmu_enable_event(cpuc->events[idx]);
687 }
688 }
689

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#16 lore raw

Re: [PATCH v1 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

In reply to Stephane Eranian
Thank you for the patch! Perhaps something to improve:

Hi Stephane,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on tip/perf/core]
[also build test WARNING on next-20210909]
[cannot apply to tip/sched/core tip/x86/core v5.14]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Stephane-Eranian/perf-x86-amd-Add-AMD-Fam19h-Branch-Sampling-support/20210909-160050
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 4034fb207e302cc0b1f304084d379640c1fb1436
config: i386-randconfig-a012-20210908 (attached as .config)
compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 261cbe98c38f8c1ee1a482fe76511110e790f58a)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/7ff5f8448eecfc188c2240b2a5a5ea0b00f55662
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Stephane-Eranian/perf-x86-amd-Add-AMD-Fam19h-Branch-Sampling-support/20210909-160050 git checkout 7ff5f8448eecfc188c2240b2a5a5ea0b00f55662
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=i386

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

>> arch/x86/events/amd/core.c:676:24: warning: variable 'hwc' set but not used [-Wunused-but-set-variable]
           struct hw_perf_event *hwc;
                                 ^

1 warning generated.

vim +/hwc +676 arch/x86/events/amd/core.c

672
673 static void amd_pmu_enable_all(int added)
674 {
675 struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);

 > 676		struct hw_perf_event *hwc;

677 int idx;
678
679 for (idx = 0; idx < x86_pmu.num_counters; idx++) {
680 hwc = &cpuc->events[idx]->hw;
681
682 /* only activate events which are marked as active */
683 if (!test_bit(idx, cpuc->active_mask))
684 continue;
685
686 amd_pmu_enable_event(cpuc->events[idx]);
687 }
688 }
689

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#17 lore raw

PATCH 4/13 perf/x86/amd: add branch-brs helper event for Fam19h BRS

In reply to Stephane Eranian

This patch adds a pseudo event called branch-brs to help use the FAM Fam19h Branch Sampling feature (BRS). BRS samples taken branches, so it is best used when sampling on a retired taken branch event (0xc4) which is what BRS captures. Instead of trying to remember the event code or actual event name, users can simply do:

$ perf record -b -e cpu/branch-brs/ -c 1000037 .....
---
 arch/x86/events/amd/core.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arch/x86/events/amd/core.c +16 -0
index 86adb0e879c6..d6d5119260a9 100644
@@ -1106,8 +1106,24 @@ static struct attribute_group group_caps_amd_brs = {
 	.is_visible = amd_brs_is_visible,
 };
 
+#define AMD_FAM19H_BRS_EVENT 0xc4 /* Fam19h RETIRED_TAKEN_BRANCH_INSTRUCTIONS */
+EVENT_ATTR_STR(branch-brs, amd_branch_brs,
+	       "event=" __stringify(AMD_FAM19H_BRS_EVENT)"\n");
+
+static struct attribute *amd_brs_events_attrs[] = {
+	EVENT_PTR(amd_branch_brs),
+	NULL,
+};
+
+static struct attribute_group group_events_amd_brs = {
+	.name       = "events",
+	.attrs      = amd_brs_events_attrs,
+	.is_visible = amd_brs_is_visible,
+};
+
 static const struct attribute_group *amd_attr_update[] = {
 	&group_caps_amd_brs,
+	&group_events_amd_brs,
 	NULL,
 };
 
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
#18 lore raw

PATCH 5/13 perf/x86/amd: enable branch sampling priv level filtering

In reply to Stephane Eranian

The AMD Branch Sampling features does not provide hardware filtering by privilege level. The associated PMU counter does but not the branch sampling by itself. Given how BRS operates there is a possibility that BRS captures kernel level branches even though the event is programmed to count only at the user level. This patch implements a workaround in software by removing the branches which belong to the wrong privilege level. The privilege level is evaluated on the target of the branch and not the source so as to be compatible with other architectures. As a consequence of this patch, the number of entries in the PERF_RECORD_BRANCH_STACK buffer may be less than the maximum (16). It could even be zero. Another consequence is that consecutive entries in the branch stack may not reflect actual code path and may have discontinuities, in case kernel branches were suppressed. But this is no different than what happens on other architectures.

---
 arch/x86/events/amd/brs.c | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
arch/x86/events/amd/brs.c +20 -6
index 86dbc6d06815..23b969001857 100644
@@ -92,10 +92,6 @@ int amd_brs_setup_filter(struct perf_event *event)
 	if ((type & ~PERF_SAMPLE_BRANCH_PLM_ALL) != PERF_SAMPLE_BRANCH_ANY)
 		return -EINVAL;
 
-	/* can only capture at all priv levels due to the way BRS works */
-	if ((type & PERF_SAMPLE_BRANCH_PLM_ALL) != PERF_SAMPLE_BRANCH_PLM_ALL)
-		return -EINVAL;
-
 	return 0;
 }
 
@@ -181,6 +177,21 @@ void amd_brs_disable(void)
 	}
 }
 
+static bool amd_brs_match_plm(struct perf_event *event, u64 to)
+{
+	int type = event->attr.branch_sample_type;
+	int plm_k = PERF_SAMPLE_BRANCH_KERNEL | PERF_SAMPLE_BRANCH_HV;
+	int plm_u = PERF_SAMPLE_BRANCH_USER;
+
+	if (!(type & plm_k) && kernel_ip(to))
+		return 0;
+
+	if (!(type & plm_u) && !kernel_ip(to))
+		return 0;
+
+	return 1;
+}
+
 /*
  * Caller must ensure amd_brs_inuse() is true before calling
  * return:
@@ -237,8 +248,6 @@ void amd_brs_drain(void)
 		if (to == BRS_POISON)
 			break;
 
-		rdmsrl(brs_from(brs_idx), from);
-
 		/*
 		 * Sign-extend SAMP_BR_TO to 64 bits, bits 61-63 are reserved.
 		 * Necessary to generate proper virtual addresses suitable for
@@ -246,6 +255,11 @@ void amd_brs_drain(void)
 		 */
 		to = (u64)(((s64)to << shift) >> shift);
 
+		if (!amd_brs_match_plm(event, to))
+			continue;
+
+		rdmsrl(brs_from(brs_idx), from);
+
 		cpuc->lbr_entries[nr].from = from;
 		cpuc->lbr_entries[nr].to   = to;
 
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
#19 lore raw

PATCH 6/13 perf/x86/amd: add AMD branch sampling period adjustment

In reply to Stephane Eranian

This supplemental patch adds the code to adjust the sampling event period when used with the Branch Sampling feature (BRS). Given the depth of the
BRS (16 on Fam19h Zen3), the period is reduced by that depth such that in
the best case scenario, BRS saturates at the desired sampling period.
In practice, though, the processor may execute more branches. Given a desired period P and a depth D, the kernel programs the actual period at P - D. After P occurrences of the sampling event, the counter overflows. It then may take X branches (skid) before the NMI is caught and held by the hardware and BRS activates. Then, after D branches, BRS saturates and the NMI is delivered. With no skid, the effective period would be (P - D) + D = P. In practice, however, it will likely be (P - D) + X + D. There is no way to eliminate X or predict X.

---
 arch/x86/events/core.c       |  7 +++++++
 arch/x86/events/perf_event.h | 12 ++++++++++++
 2 files changed, 19 insertions(+)
arch/x86/events/core.c +7 -0
index 66a073f21bb7..e1fb0e5ceaa3 100644
@@ -1375,6 +1375,13 @@ int x86_perf_event_set_period(struct perf_event *event)
 	    x86_pmu.set_topdown_event_period)
 		return x86_pmu.set_topdown_event_period(event);
 
+	/*
+	 * decrease period by the depth of the BRS feature to get
+	 * the last N taken branches and approximate the desired period
+	 */
+	if (has_branch_stack(event))
+		period = amd_brs_adjust_period(period);
+
 	/*
 	 * If we are way outside a reasonable range then just skip forward:
 	 */
arch/x86/events/perf_event.h +12 -0
index 12fb1e68f188..a275553e78b9 100644
@@ -1254,6 +1254,14 @@ static inline bool amd_brs_active(void)
 	return cpuc->brs_active;
 }
 
+static inline s64 amd_brs_adjust_period(s64 period)
+{
+	if (period > x86_pmu.lbr_nr)
+		return period - x86_pmu.lbr_nr;
+
+	return period;
+}
+
 #else /* CONFIG_CPU_SUP_AMD */
 
 static inline int amd_pmu_init(void)
@@ -1271,6 +1279,10 @@ static inline void amd_brs_drain(void)
 	return 0;
 }
 
+static inline s64 amd_brs_adjust_period(s64 period)
+{
+	return period;
+}
 #endif /* CONFIG_CPU_SUP_AMD */
 
 static inline int is_pebs_pt(struct perf_event *event)
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
#20 lore raw

PATCH 7/13 perf/core: add idle hooks

In reply to Stephane Eranian

This patch adds a new set of hooks to connect perf_events with the idle task. On some PMU models, it may be necessary to flush or stop the PMU when going to low power. Upon return from low power, the opposite action, i.e., re-enable the PMU, may be necessary. The patch adds perf_pmu_register_lopwr_cb() to register a callback on entry or return from low power. The callback is invoked with a boolean arg. If true, then this is an entry. If false, this is a return.

The callback is invoked from the idle code with interrupts already disabled.

---
 include/linux/perf_event.h |  8 ++++++
 kernel/events/core.c       | 58 ++++++++++++++++++++++++++++++++++++++
 kernel/sched/idle.c        | 15 +++++++++-
 3 files changed, 80 insertions(+), 1 deletion(-)
include/linux/perf_event.h +8 -0
index 2d510ad750ed..32ffc009b2ec 100644
@@ -300,6 +300,7 @@ struct pmu {
 	/* number of address filters this PMU can do */
 	unsigned int			nr_addr_filters;
 
+	struct list_head		lopwr_entry;
 	/*
 	 * Fully disable/enable this PMU, can be used to protect from the PMI
 	 * as well as for lazy/batch writing of the MSRs.
@@ -430,6 +431,8 @@ struct pmu {
 	void (*sched_task)		(struct perf_event_context *ctx,
 					bool sched_in);
 
+	void (*lopwr_cb)		(bool lopwr_in);
+
 	/*
 	 * Kmem cache of PMU specific data
 	 */
@@ -1429,6 +1432,11 @@ extern void perf_event_task_tick(void);
 extern int perf_event_account_interrupt(struct perf_event *event);
 extern int perf_event_period(struct perf_event *event, u64 value);
 extern u64 perf_event_pause(struct perf_event *event, bool reset);
+extern void perf_lopwr_cb(bool lopwr_in);
+extern void perf_lopwr_active_inc(void);
+extern void perf_lopwr_active_dec(void);
+extern void perf_register_lopwr_cb(struct pmu *pmu, void (*lowpwr_cb)(bool));
+
 #else /* !CONFIG_PERF_EVENTS: */
 static inline void *
 perf_aux_output_begin(struct perf_output_handle *handle,
kernel/events/core.c +58 -0

64 lines not on this page · raw message

kernel/sched/idle.c +14 -1
index 912b47aa99d8..14ce130aee1b 100644
@@ -179,6 +179,7 @@ static void cpuidle_idle_call(void)
 	 */
 	if (need_resched()) {
 		local_irq_enable();
+		perf_lopwr_cb(false);
 		return;
 	}
 
@@ -191,7 +192,14 @@ static void cpuidle_idle_call(void)
 	if (cpuidle_not_available(drv, dev)) {
 		tick_nohz_idle_stop_tick();
 
+		if (!cpu_idle_force_poll)
+			perf_lopwr_cb(true);
+
 		default_idle_call();
+
+		if (!cpu_idle_force_poll)
+			perf_lopwr_cb(false);
+
 		goto exit_idle;
 	}
 
@@ -249,8 +257,10 @@ static void cpuidle_idle_call(void)
 	/*
 	 * It is up to the idle functions to reenable local interrupts
 	 */
-	if (WARN_ON_ONCE(irqs_disabled()))
+	if (WARN_ON_ONCE(irqs_disabled())) {
 		local_irq_enable();
+		perf_lopwr_cb(false);
+	}
 }
 
 /*
@@ -279,9 +289,12 @@ static void do_idle(void)
 	__current_set_polling();
 	tick_nohz_idle_enter();
 
+
 	while (!need_resched()) {
 		rmb();
 
+		perf_lopwr_cb(true);
+
 		local_irq_disable();
 
 		if (cpu_is_offline(cpu)) {
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
maintainer
#21 lore raw

Re: [PATCH v1 07/13] perf/core: add idle hooks

Quoted 10 lines · Stephane Eranian

This patch adds a new set of hooks to connect perf_events with the idle task. On some PMU models, it may be necessary to flush or stop the PMU when going to low power. Upon return from low power, the opposite action, i.e., re-enable the PMU, may be necessary. The patch adds perf_pmu_register_lopwr_cb() to register a callback on entry or return from low power. The callback is invoked with a boolean arg. If true, then this is an entry. If false, this is a return.

The callback is invoked from the idle code with interrupts already disabled.

Urghh... we've had this before and always managed to refuse doing this (in generic code).

Also, it's broken..

Quoted 8 lines

+/*
+ * The perf_lopwr_cb() is invoked from the idle task. As such it
+ * cannot grab a mutex that may end up sleeping. The idle task cannot
+ * sleep by construction. Therefore we create a spinlock and a new
+ * list of PMUs to invoke on idle. The list is protected by a spinlock
+ * Normally we would use the pmus_lock and iterate over each PMUs. But
+ * mutex is not possible and we need to iterate only over the PMU which
+ * do require a idle callback.

That rationale is broken, pmus is also SRCU protected and SRCU iteration would actually work from the idle context, unlike:

Quoted 2 lines

+ */
+static DEFINE_SPINLOCK(lopwr_cb_lock);

which is not allowed from preempt disable context, if you really need that lock, you need a raw_spinlock_t.

Quoted 14 lines
kernel/events/core.c +14 -0
+static LIST_HEAD(lopwr_cb_pmus);
+static DEFINE_PER_CPU(int, lopwr_nr_active);
+
+void perf_lopwr_active_inc(void)
+{
+	__this_cpu_inc(lopwr_nr_active);
+}
+
+void perf_lopwr_active_dec(void)
+{
+	__this_cpu_dec(lopwr_nr_active);
+}
+
+/*

kernel/events/core.c void perf_sched_cb_inc(struct pmu *pmu) · line 3557

Quoted 10 lines
kernel/events/core.c +10 -0
+ * lopwr_in = true means going to low power state
+ * lopwr_in = false means returning from low power state
+ */
+void perf_lopwr_cb(bool lopwr_in)
+{
+	struct pmu *pmu;
+	unsigned long flags;
+
+	if (!__this_cpu_read(lopwr_nr_active))
+		return;

We have static_branch and static_call to deal with these things.

kernel/events/core.c void perf_sched_cb_inc(struct pmu *pmu) · line 3564

Quoted 7 lines
kernel/events/core.c +7 -0
+
+	spin_lock_irqsave(&lopwr_cb_lock, flags);
+
+	list_for_each_entry(pmu, &lopwr_cb_pmus, lopwr_entry) {
+		if (pmu->lopwr_cb)
+			pmu->lopwr_cb(lopwr_in);
+	}

I *really* do not like unbound iteration in the idle path.

kernel/events/core.c void perf_sched_cb_inc(struct pmu *pmu) · line 3568

Quoted 4 lines
kernel/events/core.c +4 -0
+
+	spin_unlock_irqrestore(&lopwr_cb_lock, flags);
+}
+EXPORT_SYMBOL_GPL(perf_lopwr_cb);

Why?

Quoted 42 lines
kernel/sched/idle.c +11 -1
index 912b47aa99d8..14ce130aee1b 100644
@@ -179,6 +179,7 @@ static void cpuidle_idle_call(void)
 	 */
 	if (need_resched()) {
 		local_irq_enable();
+		perf_lopwr_cb(false);
 		return;
 	}
 
@@ -191,7 +192,14 @@ static void cpuidle_idle_call(void)
 	if (cpuidle_not_available(drv, dev)) {
 		tick_nohz_idle_stop_tick();
 
+		if (!cpu_idle_force_poll)
+			perf_lopwr_cb(true);
+
 		default_idle_call();
+
+		if (!cpu_idle_force_poll)
+			perf_lopwr_cb(false);
+
 		goto exit_idle;
 	}
 
@@ -249,8 +257,10 @@ static void cpuidle_idle_call(void)
 	/*
 	 * It is up to the idle functions to reenable local interrupts
 	 */
-	if (WARN_ON_ONCE(irqs_disabled()))
+	if (WARN_ON_ONCE(irqs_disabled())) {
 		local_irq_enable();
+		perf_lopwr_cb(false);
+	}
 }
 
 /*
@@ -279,9 +289,12 @@ static void do_idle(void)
 	__current_set_polling();
 	tick_nohz_idle_enter();

kernel/sched/idle.c static void do_idle(void) · line 300

Quoted 10 lines
kernel/sched/idle.c +3 -0
 
+
 	while (!need_resched()) {
 		rmb();
 
+		perf_lopwr_cb(true);
+
 		local_irq_disable();
 
 		if (cpu_is_offline(cpu)) {

I so hate all of this... but mostly, this doesn't seem right, you're unconditionally calling this, even for high power idle states.

*IFF* you really have to do this, stuff it in an AMD idle state driver that knows about the relevant idle states. But really, why can't we just take the his while the event is running?

#22 lore raw

Re: [PATCH v1 07/13] perf/core: add idle hooks

In reply to Stephane Eranian
Thank you for the patch! Yet something to improve:

Hi Stephane,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on tip/perf/core]
[also build test ERROR on next-20210909]
[cannot apply to tip/sched/core tip/x86/core v5.14]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Stephane-Eranian/perf-x86-amd-Add-AMD-Fam19h-Branch-Sampling-support/20210909-160050
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 4034fb207e302cc0b1f304084d379640c1fb1436
config: um-x86_64_defconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/35333fae36d4dee3e2225674fca8d745364482fa
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Stephane-Eranian/perf-x86-amd-Add-AMD-Fam19h-Branch-Sampling-support/20210909-160050
        git checkout 35333fae36d4dee3e2225674fca8d745364482fa
        # save the attached .config to linux build tree
        make W=1 ARCH=um SUBARCH=x86_64

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

kernel/sched/idle.c: In function 'cpuidle_idle_call':

>> kernel/sched/idle.c:182:3: error: implicit declaration of function 'perf_lopwr_cb' [-Werror=implicit-function-declaration]
     182 |   perf_lopwr_cb(false);
         |   ^~~~~~~~~~~~~

cc1: some warnings being treated as errors

vim +/perf_lopwr_cb +182 kernel/sched/idle.c

160
161 /**
162 * cpuidle_idle_call - the main idle function
163 *
164 * NOTE: no locks or semaphores should be used here
165 *
166 * On architectures that support TIF_POLLING_NRFLAG, is called with polling
167 * set, and it returns with polling set. If it ever stops polling, it
168 * must clear the polling bit.
169 */
170 static void cpuidle_idle_call(void)
171 {
172 struct cpuidle_device *dev = cpuidle_get_device();
173 struct cpuidle_driver *drv = cpuidle_get_cpu_driver(dev);
174 int next_state, entered_state;
175
176 /*
177 * Check if the idle task must be rescheduled. If it is the
178 * case, exit the function after re-enabling the local irq.
179 */
180 if (need_resched()) {
181 local_irq_enable();

 > 182			perf_lopwr_cb(false);

183 return;
184 }
185
186 /*
187 * The RCU framework needs to be told that we are entering an idle
188 * section, so no more rcu read side critical sections and one more
189 * step to the grace period
190 */
191
192 if (cpuidle_not_available(drv, dev)) {
193 tick_nohz_idle_stop_tick();
194
195 if (!cpu_idle_force_poll)
196 perf_lopwr_cb(true);
197
198 default_idle_call();
199
200 if (!cpu_idle_force_poll)
201 perf_lopwr_cb(false);
202
203 goto exit_idle;
204 }
205
206 /*
207 * Suspend-to-idle ("s2idle") is a system state in which all user space
208 * has been frozen, all I/O devices have been suspended and the only
209 * activity happens here and in interrupts (if any). In that case bypass
210 * the cpuidle governor and go straight for the deepest idle state
211 * available. Possibly also suspend the local tick and the entire
212 * timekeeping to prevent timer interrupts from kicking us out of idle
213 * until a proper wakeup interrupt happens.
214 */
215
216 if (idle_should_enter_s2idle() || dev->forced_idle_latency_limit_ns) {
217 u64 max_latency_ns;
218
219 if (idle_should_enter_s2idle()) {
220
221 entered_state = call_cpuidle_s2idle(drv, dev);
222 if (entered_state > 0)
223 goto exit_idle;
224
225 max_latency_ns = U64_MAX;
226 } else {
227 max_latency_ns = dev->forced_idle_latency_limit_ns;
228 }
229
230 tick_nohz_idle_stop_tick();
231
232 next_state = cpuidle_find_deepest_state(drv, dev, max_latency_ns);
233 call_cpuidle(drv, dev, next_state);
234 } else {
235 bool stop_tick = true;
236
237 /*
238 * Ask the cpuidle framework to choose a convenient idle state.
239 */
240 next_state = cpuidle_select(drv, dev, &stop_tick);
241
242 if (stop_tick || tick_nohz_tick_stopped())
243 tick_nohz_idle_stop_tick();
244 else
245 tick_nohz_idle_retain_tick();
246
247 entered_state = call_cpuidle(drv, dev, next_state);
248 /*
249 * Give the governor an opportunity to reflect on the outcome
250 */
251 cpuidle_reflect(dev, entered_state);
252 }
253
254 exit_idle:
255 __current_set_polling();
256
257 /*
258 * It is up to the idle functions to reenable local interrupts
259 */
260 if (WARN_ON_ONCE(irqs_disabled())) {
261 local_irq_enable();
262 perf_lopwr_cb(false);
263 }
264 }
265

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#23 lore raw

Re: [PATCH v1 07/13] perf/core: add idle hooks

In reply to Stephane Eranian
Thank you for the patch! Yet something to improve:

Hi Stephane,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on tip/perf/core]
[also build test ERROR on next-20210909]
[cannot apply to tip/sched/core tip/x86/core v5.14]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Stephane-Eranian/perf-x86-amd-Add-AMD-Fam19h-Branch-Sampling-support/20210909-160050
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 4034fb207e302cc0b1f304084d379640c1fb1436
config: riscv-randconfig-r042-20210908 (attached as .config)
compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 261cbe98c38f8c1ee1a482fe76511110e790f58a)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install riscv cross compiling tool for clang build
# apt-get install binutils-riscv64-linux-gnu
# https://github.com/0day-ci/linux/commit/35333fae36d4dee3e2225674fca8d745364482fa
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Stephane-Eranian/perf-x86-amd-Add-AMD-Fam19h-Branch-Sampling-support/20210909-160050 git checkout 35333fae36d4dee3e2225674fca8d745364482fa
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=riscv

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

>> kernel/sched/idle.c:182:3: error: implicit declaration of function 'perf_lopwr_cb' [-Werror,-Wimplicit-function-declaration]
                   perf_lopwr_cb(false);
                   ^
   kernel/sched/idle.c:196:4: error: implicit declaration of function 'perf_lopwr_cb' [-Werror,-Wimplicit-function-declaration]
                           perf_lopwr_cb(true);
                           ^
   kernel/sched/idle.c:262:3: error: implicit declaration of function 'perf_lopwr_cb' [-Werror,-Wimplicit-function-declaration]
                   perf_lopwr_cb(false);
                   ^
   kernel/sched/idle.c:296:3: error: implicit declaration of function 'perf_lopwr_cb' [-Werror,-Wimplicit-function-declaration]
                   perf_lopwr_cb(true);
                   ^

4 errors generated.

vim +/perf_lopwr_cb +182 kernel/sched/idle.c

160
161 /**
162 * cpuidle_idle_call - the main idle function
163 *
164 * NOTE: no locks or semaphores should be used here
165 *
166 * On architectures that support TIF_POLLING_NRFLAG, is called with polling
167 * set, and it returns with polling set. If it ever stops polling, it
168 * must clear the polling bit.
169 */
170 static void cpuidle_idle_call(void)
171 {
172 struct cpuidle_device *dev = cpuidle_get_device();
173 struct cpuidle_driver *drv = cpuidle_get_cpu_driver(dev);
174 int next_state, entered_state;
175
176 /*
177 * Check if the idle task must be rescheduled. If it is the
178 * case, exit the function after re-enabling the local irq.
179 */
180 if (need_resched()) {
181 local_irq_enable();

 > 182			perf_lopwr_cb(false);

183 return;
184 }
185
186 /*
187 * The RCU framework needs to be told that we are entering an idle
188 * section, so no more rcu read side critical sections and one more
189 * step to the grace period
190 */
191
192 if (cpuidle_not_available(drv, dev)) {
193 tick_nohz_idle_stop_tick();
194
195 if (!cpu_idle_force_poll)
196 perf_lopwr_cb(true);
197
198 default_idle_call();
199
200 if (!cpu_idle_force_poll)
201 perf_lopwr_cb(false);
202
203 goto exit_idle;
204 }
205
206 /*
207 * Suspend-to-idle ("s2idle") is a system state in which all user space
208 * has been frozen, all I/O devices have been suspended and the only
209 * activity happens here and in interrupts (if any). In that case bypass
210 * the cpuidle governor and go straight for the deepest idle state
211 * available. Possibly also suspend the local tick and the entire
212 * timekeeping to prevent timer interrupts from kicking us out of idle
213 * until a proper wakeup interrupt happens.
214 */
215
216 if (idle_should_enter_s2idle() || dev->forced_idle_latency_limit_ns) {
217 u64 max_latency_ns;
218
219 if (idle_should_enter_s2idle()) {
220
221 entered_state = call_cpuidle_s2idle(drv, dev);
222 if (entered_state > 0)
223 goto exit_idle;
224
225 max_latency_ns = U64_MAX;
226 } else {
227 max_latency_ns = dev->forced_idle_latency_limit_ns;
228 }
229
230 tick_nohz_idle_stop_tick();
231
232 next_state = cpuidle_find_deepest_state(drv, dev, max_latency_ns);
233 call_cpuidle(drv, dev, next_state);
234 } else {
235 bool stop_tick = true;
236
237 /*
238 * Ask the cpuidle framework to choose a convenient idle state.
239 */
240 next_state = cpuidle_select(drv, dev, &stop_tick);
241
242 if (stop_tick || tick_nohz_tick_stopped())
243 tick_nohz_idle_stop_tick();
244 else
245 tick_nohz_idle_retain_tick();
246
247 entered_state = call_cpuidle(drv, dev, next_state);
248 /*
249 * Give the governor an opportunity to reflect on the outcome
250 */
251 cpuidle_reflect(dev, entered_state);
252 }
253
254 exit_idle:
255 __current_set_polling();
256
257 /*
258 * It is up to the idle functions to reenable local interrupts
259 */
260 if (WARN_ON_ONCE(irqs_disabled())) {
261 local_irq_enable();
262 perf_lopwr_cb(false);
263 }
264 }
265

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#24 lore raw

PATCH 8/13 perf/x86/core: add idle hooks

In reply to Stephane Eranian

This patch adds the perf idle hooks x86 idle routines.

---
 arch/x86/include/asm/mwait.h |  6 +++++-
 kernel/sched/idle.c          | 12 ++++++++----
 2 files changed, 13 insertions(+), 5 deletions(-)
arch/x86/include/asm/mwait.h +5 -1
index 29dd27b5a339..92080fdc1d2a 100644
@@ -4,6 +4,7 @@
 
 #include <linux/sched.h>
 #include <linux/sched/idle.h>
+#include <linux/perf_event.h>
 
 #include <asm/cpufeature.h>
 #include <asm/nospec-branch.h>
@@ -114,8 +115,11 @@ static inline void mwait_idle_with_hints(unsigned long eax, unsigned long ecx)
 		}
 
 		__monitor((void *)&current_thread_info()->flags, 0, 0);
-		if (!need_resched())
+		if (!need_resched()) {
+			perf_lopwr_cb(true);
 			__mwait(eax, ecx);
+			perf_lopwr_cb(false);
+		}
 	}
 	current_clr_polling();
 }
kernel/sched/idle.c +8 -4
index 14ce130aee1b..c0ddc3c32a33 100644
@@ -179,7 +179,6 @@ static void cpuidle_idle_call(void)
 	 */
 	if (need_resched()) {
 		local_irq_enable();
-		perf_lopwr_cb(false);
 		return;
 	}
 
@@ -230,6 +229,9 @@ static void cpuidle_idle_call(void)
 		tick_nohz_idle_stop_tick();
 
 		next_state = cpuidle_find_deepest_state(drv, dev, max_latency_ns);
+		if (!cpu_idle_force_poll)
+			perf_lopwr_cb(true);
+
 		call_cpuidle(drv, dev, next_state);
 	} else {
 		bool stop_tick = true;
@@ -244,12 +246,17 @@ static void cpuidle_idle_call(void)
 		else
 			tick_nohz_idle_retain_tick();
 
+		if (!cpu_idle_force_poll)
+			perf_lopwr_cb(true);
+
 		entered_state = call_cpuidle(drv, dev, next_state);
 		/*
 		 * Give the governor an opportunity to reflect on the outcome
 		 */
 		cpuidle_reflect(dev, entered_state);
 	}
+	if (!cpu_idle_force_poll)
+		perf_lopwr_cb(false);
 
 exit_idle:
 	__current_set_polling();
@@ -259,7 +266,6 @@ static void cpuidle_idle_call(void)
 	 */
 	if (WARN_ON_ONCE(irqs_disabled())) {
 		local_irq_enable();
-		perf_lopwr_cb(false);
 	}
 }
 
@@ -293,8 +299,6 @@ static void do_idle(void)
 	while (!need_resched()) {
 		rmb();
 
-		perf_lopwr_cb(true);
-
 		local_irq_disable();
 
 		if (cpu_is_offline(cpu)) {
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
maintainer
#25 lore raw

Re: [PATCH v1 08/13] perf/x86/core: add idle hooks

Quoted 49 lines · Stephane Eranian
kernel/sched/idle.c +8 -2
index 14ce130aee1b..c0ddc3c32a33 100644
@@ -179,7 +179,6 @@ static void cpuidle_idle_call(void)
 	 */
 	if (need_resched()) {
 		local_irq_enable();
-		perf_lopwr_cb(false);
 		return;
 	}
 
@@ -230,6 +229,9 @@ static void cpuidle_idle_call(void)
 		tick_nohz_idle_stop_tick();
 
 		next_state = cpuidle_find_deepest_state(drv, dev, max_latency_ns);
+		if (!cpu_idle_force_poll)
+			perf_lopwr_cb(true);
+
 		call_cpuidle(drv, dev, next_state);
 	} else {
 		bool stop_tick = true;
@@ -244,12 +246,17 @@ static void cpuidle_idle_call(void)
 		else
 			tick_nohz_idle_retain_tick();
 
+		if (!cpu_idle_force_poll)
+			perf_lopwr_cb(true);
+
 		entered_state = call_cpuidle(drv, dev, next_state);
 		/*
 		 * Give the governor an opportunity to reflect on the outcome
 		 */
 		cpuidle_reflect(dev, entered_state);
 	}
+	if (!cpu_idle_force_poll)
+		perf_lopwr_cb(false);
 
 exit_idle:
 	__current_set_polling();
@@ -259,7 +266,6 @@ static void cpuidle_idle_call(void)
 	 */
 	if (WARN_ON_ONCE(irqs_disabled())) {
 		local_irq_enable();
-		perf_lopwr_cb(false);
 	}
 }
 

kernel/sched/idle.c static void do_idle(void) · line 304

Quoted 8 lines
kernel/sched/idle.c +0 -2
@@ -293,8 +299,6 @@ static void do_idle(void)
 	while (!need_resched()) {
 		rmb();
 
-		perf_lopwr_cb(true);
-
 		local_irq_disable();
 
 		if (cpu_is_offline(cpu)) {

Why are you sendimg me a patch that undoes most of what the previous patch does? That's just bad form.

#26 lore raw

PATCH 9/13 perf/x86/amd: add idle hooks for branch sampling

In reply to Stephane Eranian

On AMD Fam19h Zen3, the branch sampling (BRS) feature must be disabled before entering low power and re-enabled (if was active) when returning from low power. Otherwise, the NMI interrupt may be held up for too long and cause problems. Stopping BRS will cause the NMI to be delivered if it was held up.

This patch connects the branch sampling code to the perf_events idle callbacks.

---
 arch/x86/events/amd/brs.c    | 25 +++++++++++++++++++++++++
 arch/x86/events/amd/core.c   |  4 ++++
 arch/x86/events/perf_event.h |  1 +
 3 files changed, 30 insertions(+)
arch/x86/events/amd/brs.c +25 -0
index 23b969001857..7d27591ba537 100644
@@ -146,6 +146,7 @@ void amd_brs_enable(void)
 
 	/* Set enable bit */
 	set_debug_extn_cfg(cfg.val);
+	perf_lopwr_active_inc();
 }
 
 void amd_brs_disable(void)
@@ -175,6 +176,7 @@ void amd_brs_disable(void)
 		cfg.brsmen = 0;
 		set_debug_extn_cfg(cfg.val);
 	}
+	perf_lopwr_active_dec();
 }
 
 static bool amd_brs_match_plm(struct perf_event *event, u64 to)
@@ -292,6 +294,29 @@ static void amd_brs_poison_buffer(void)
 	wrmsrl(brs_to(idx), BRS_POISON);
 }
 
+/*
+ * called indirectly with irqs masked from mwait_idle_*()
+ */
+void amd_pmu_brs_lopwr_cb(bool lopwr_in)
+{
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+	union amd_debug_extn_cfg cfg;
+
+	/*
+	 * on mwait in, we may end up in non C0 state.
+	 * we must disable branch sampling to avoid holding the NMI
+	 * for too long. We disable it in hardware but we
+	 * keep the state in cpuc, so we can re-enable.
+	 *
+	 * The hardware will deliver the NMI if needed when brsmen cleared
+	 */
+	if (cpuc->brs_active) {
+		cfg.val = get_debug_extn_cfg();
+		cfg.brsmen = !lopwr_in;
+		set_debug_extn_cfg(cfg.val);
+	}
+}
+
 /*
  * On context switch in, we need to make sure no samples from previous user
  * are left in the BRS.
arch/x86/events/amd/core.c +4 -0
index d6d5119260a9..3e1985cd414d 100644
@@ -1184,12 +1184,16 @@ static int __init amd_core_pmu_init(void)
 		 * invoked on context-switch in via sched_task_in(), so enable only when necessary
 		 */
 		if (!amd_brs_init()) {
+			struct pmu *pmu = x86_get_pmu(smp_processor_id());
 			x86_pmu.get_event_constraints = amd_get_event_constraints_f19h;
 			x86_pmu.sched_task = amd_pmu_sched_task;
 			/*
 			 * The put_event_constraints callback is shared with
 			 * Fam17h, set above
 			 */
+
+			/* branch sampling must be stopped when entering low power */
+			perf_register_lopwr_cb(pmu, amd_pmu_brs_lopwr_cb);
 		}
 	}
 
arch/x86/events/perf_event.h +1 -0
index a275553e78b9..73eac9d34bd9 100644
@@ -1242,6 +1242,7 @@ static inline void amd_pmu_brs_del(struct perf_event *event)
 }
 
 void amd_pmu_brs_sched_task(struct perf_event_context *ctx, bool sched_in);
+void amd_pmu_brs_lopwr_cb(bool lopwr_in);
 
 /*
  * check if BRS is activated on the CPU
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
maintainer
#27 lore raw

Re: [PATCH v1 09/13] perf/x86/amd: add idle hooks for branch sampling

Quoted 4 lines · Stephane Eranian

On AMD Fam19h Zen3, the branch sampling (BRS) feature must be disabled before entering low power and re-enabled (if was active) when returning from low power. Otherwise, the NMI interrupt may be held up for too long and cause problems. Stopping BRS will cause the NMI to be delivered if it was held up.

What you sent me ^^^, vs reflow at 72 chars vvv:

On AMD Fam19h Zen3, the branch sampling (BRS) feature must be disabled before entering low power and re-enabled (if was active) when returning from low power. Otherwise, the NMI interrupt may be held up for too long and cause problems. Stopping BRS will cause the NMI to be delivered if it was held up.

(because git show has that stupid extra indentation for Changelogs)

#28 lore raw

PATCH 10/13 perf tools: add branch-brs as a new event

In reply to Stephane Eranian

Although branch-brs is specific to AMD Zen3, we provide it in the list of events recognized by perf as a convenience. It allows:

 $ perf stat -e branch-brs
 $ perf list

and enable some test suite to pass.

---
 tools/perf/util/parse-events.l | 1 +
 1 file changed, 1 insertion(+)
tools/perf/util/parse-events.l +1 -0
index 923849024b15..ae229f5c4758 100644
@@ -361,6 +361,7 @@ mem-loads-aux				|
 mem-stores				|
 topdown-[a-z-]+				|
 tx-capacity-[a-z-]+			|
+branch-brs				|
 el-capacity-[a-z-]+			{ return str(yyscanner, PE_KERNEL_PMU_EVENT); }
 
 L1-dcache|l1-d|l1d|L1-data		|
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
#29 lore raw

PATCH 11/13 perf tools: improve IBS error handling

In reply to Stephane Eranian

From: Kim Phillips <[email protected]>

This patch improves the error message returned on failed perf_event_open() on AMD when using IBS.

---
 tools/perf/util/evsel.c | 42 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
tools/perf/util/evsel.c +42 -0
index f61e5dd53f5d..f203f178fdb9 100644
@@ -2684,12 +2684,52 @@ static bool find_process(const char *name)
 	return ret ? false : true;
 }
 
+static bool is_amd;
+
+static char *fgrep(FILE *inf, const char *str)
+{
+	char line[256];
+	int slen = strlen(str);
+
+	while (!feof(inf)) {
+		if (!fgets(line, 256, inf))
+			break;
+		if (strncmp(line, str, slen))
+			continue;
+
+		return strdup(line);
+	}
+
+	return NULL;
+}
+
+static void detect_amd(void)
+{
+	FILE *inf = fopen("/proc/cpuinfo", "r");
+	char *res;
+
+	if (!inf)
+		return;
+
+	res = fgrep(inf, "vendor_id");
+
+	if (res) {
+		char *s = strchr(res, ':');
+
+		is_amd = s && !strcmp(s, ": AuthenticAMD\n");
+		free(res);
+	}
+	fclose(inf);
+}
+
 int evsel__open_strerror(struct evsel *evsel, struct target *target,
 			 int err, char *msg, size_t size)
 {
 	char sbuf[STRERR_BUFSIZE];
 	int printed = 0, enforced = 0;
 
+	detect_amd();
+
 	switch (err) {
 	case EPERM:
 	case EACCES:
@@ -2782,6 +2822,8 @@ int evsel__open_strerror(struct evsel *evsel, struct target *target,
 			return scnprintf(msg, size, "wrong clockid (%d).", clockid);
 		if (perf_missing_features.aux_output)
 			return scnprintf(msg, size, "The 'aux_output' feature is not supported, update the kernel.");
+		if (is_amd && (evsel->core.attr.precise_ip || !strncmp(evsel->pmu_name, "ibs", 3)) && (evsel->core.attr.exclude_kernel))
+			return scnprintf(msg, size, "AMD IBS can't exclude kernel events.  Try running at a higher privilege level.");
 		break;
 	case ENODATA:
 		return scnprintf(msg, size, "Cannot collect data source with the load latency event alone. "
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
#30 lore raw

Re: [PATCH v1 11/13] perf tools: improve IBS error handling

Em Thu, Sep 09, 2021 at 12:56:58AM -0700, Stephane Eranian escreveu:

Quoted 46 lines

From: Kim Phillips <[email protected]>

This patch improves the error message returned on failed perf_event_open() on AMD when using IBS.

---
 tools/perf/util/evsel.c | 42 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
tools/perf/util/evsel.c +28 -0
index f61e5dd53f5d..f203f178fdb9 100644
@@ -2684,12 +2684,52 @@ static bool find_process(const char *name)
 	return ret ? false : true;
 }
 
+static bool is_amd;
+
+static char *fgrep(FILE *inf, const char *str)
+{
+	char line[256];
+	int slen = strlen(str);
+
+	while (!feof(inf)) {
+		if (!fgets(line, 256, inf))
+			break;
+		if (strncmp(line, str, slen))
+			continue;
+
+		return strdup(line);
+	}
+
+	return NULL;
+}
+
+static void detect_amd(void)
+{
+	FILE *inf = fopen("/proc/cpuinfo", "r");
+	char *res;
+
+	if (!inf)
+		return;
+
+	res = fgrep(inf, "vendor_id");

tools/perf/util/evsel.c static bool find_process(const char *name) · line 2724

Quoted 10 lines
tools/perf/util/evsel.c +10 -0
+
+	if (res) {
+		char *s = strchr(res, ':');
+
+		is_amd = s && !strcmp(s, ": AuthenticAMD\n");
+		free(res);
+	}
+	fclose(inf);
+}
+

We have perf_env for such details, for instance in tools/perf/util/sample-raw.c we have:o

        const char *arch_pf = perf_env__arch(evlist->env);
        const char *cpuid = perf_env__cpuid(evlist->env);

	        else if (arch_pf && !strcmp("x86", arch_pf) &&
                 cpuid && strstarts(cpuid, "AuthenticAMD") &&
                 evlist__has_amd_ibs(evlist)) {
Quoted 22 lines
tools/perf/util/evsel.c +4 -0
 int evsel__open_strerror(struct evsel *evsel, struct target *target,
 			 int err, char *msg, size_t size)
 {
 	char sbuf[STRERR_BUFSIZE];
 	int printed = 0, enforced = 0;
 
+	detect_amd();
+
 	switch (err) {
 	case EPERM:
 	case EACCES:
@@ -2782,6 +2822,8 @@ int evsel__open_strerror(struct evsel *evsel, struct target *target,
 			return scnprintf(msg, size, "wrong clockid (%d).", clockid);
 		if (perf_missing_features.aux_output)
 			return scnprintf(msg, size, "The 'aux_output' feature is not supported, update the kernel.");
+		if (is_amd && (evsel->core.attr.precise_ip || !strncmp(evsel->pmu_name, "ibs", 3)) && (evsel->core.attr.exclude_kernel))
+			return scnprintf(msg, size, "AMD IBS can't exclude kernel events.  Try running at a higher privilege level.");
 		break;
 	case ENODATA:
 		return scnprintf(msg, size, "Cannot collect data source with the load latency event alone. "
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
Signature · 1 line
- Arnaldo
#31 lore raw

Re: [PATCH v1 11/13] perf tools: improve IBS error handling

Quoted 25 lines · Arnaldo Carvalho de Melo

Em Thu, Sep 09, 2021 at 12:56:58AM -0700, Stephane Eranian escreveu:

Quoted 10 lines

From: Kim Phillips <[email protected]>

tools/perf/util/evsel.c +9 -0
+static void detect_amd(void)
+{
+	FILE *inf = fopen("/proc/cpuinfo", "r");
+	char *res;
+
+	if (!inf)
+		return;
+
+	res = fgrep(inf, "vendor_id");

tools/perf/util/evsel.c static bool find_process(const char *name) · line 2724

Quoted 10 lines
tools/perf/util/evsel.c +10 -0
+
+	if (res) {
+		char *s = strchr(res, ':');
+
+		is_amd = s && !strcmp(s, ": AuthenticAMD\n");
+		free(res);
+	}
+	fclose(inf);
+}
+

We have perf_env for such details, for instance in tools/perf/util/sample-raw.c we have:o

Quoted 6 lines
         const char *arch_pf = perf_env__arch(evlist->env);
         const char *cpuid = perf_env__cpuid(evlist->env);

	        else if (arch_pf && !strcmp("x86", arch_pf) &&
                  cpuid && strstarts(cpuid, "AuthenticAMD") &&
                  evlist__has_amd_ibs(evlist)) {

OK, I've rebased this 11/13 patch onto the new perf_env__{arch,cpuid} code, and posted it here:

https://lore.kernel.org/lkml/[email protected]/T/#mc4c9c582e3816ab31af6d0187e6803de1a98ac84

The following 12/13 patch in this series changes, too, but since it depends on prior patches in the series, I'll reply-all to 12/13 with its new version.

Arnaldo, would it be ok to apply the two new patches that replace this 11/13? They don't depend on any others in this series, and it would save Stephane from having to carry them.

Thanks,

Kim

#32 lore raw

Re: [PATCH v1 11/13] perf tools: improve IBS error handling

Quoted 38 lines · Kim Phillips
Quoted 25 lines · Arnaldo Carvalho de Melo

Em Thu, Sep 09, 2021 at 12:56:58AM -0700, Stephane Eranian escreveu:

Quoted 10 lines

From: Kim Phillips <[email protected]>

tools/perf/util/evsel.c +9 -0
+static void detect_amd(void)
+{
+	FILE *inf = fopen("/proc/cpuinfo", "r");
+	char *res;
+
+	if (!inf)
+		return;
+
+	res = fgrep(inf, "vendor_id");

tools/perf/util/evsel.c static bool find_process(const char *name) · line 2724

Quoted 10 lines
tools/perf/util/evsel.c +10 -0
+
+	if (res) {
+		char *s = strchr(res, ':');
+
+		is_amd = s && !strcmp(s, ": AuthenticAMD\n");
+		free(res);
+	}
+	fclose(inf);
+}
+

We have perf_env for such details, for instance in tools/perf/util/sample-raw.c we have:o

Quoted 6 lines
         const char *arch_pf = perf_env__arch(evlist->env);
         const char *cpuid = perf_env__cpuid(evlist->env);

	        else if (arch_pf && !strcmp("x86", arch_pf) &&
                  cpuid && strstarts(cpuid, "AuthenticAMD") &&
                  evlist__has_amd_ibs(evlist)) {

OK, I've rebased this 11/13 patch onto the new perf_env__{arch,cpuid} code, and posted it here:

https://lore.kernel.org/lkml/[email protected]/T/#mc4c9c582e3816ab31af6d0187e6803de1a98ac84

Quoted 7 lines

The following 12/13 patch in this series changes, too, but since it depends on prior patches in the series, I'll reply-all to 12/13 with its new version.

Arnaldo, would it be ok to apply the two new patches that replace this 11/13? They don't depend on any others in this series, and it would save Stephane from having to carry them.

I'll work in this tomorrow, thanks

- Arnaldo

Quoted 4 lines

Thanks,

Kim

#33 lore raw

PATCH 12/13 perf tools: improve error handling of AMD Branch Sampling

In reply to Stephane Eranian

This patch improves the error message printed by perf when perf_event_open() fails on AMD Zen3 when using the branch sampling feature. In the case of EINVAL, there are two main reasons: frequency mode or period is smaller than the depth of the branch sampling buffer (16). The patch checks the parameters of the call and tries to print a relevant message to explain the error:

$ perf record -b -e cpu/branch-brs/ -c 10 ls
Error:
AMD Branch Sampling does not support sampling period smaller than what is reported in /sys/devices/cpu/caps/branches.
$ perf record -b -e cpu/branch-brs/ ls
Error:
AMD Branch Sampling does not support frequency mode sampling, must pass a fixed sampling period via -c option or cpu/branch-brs,period=xxxx/.
---
 tools/perf/util/evsel.c | 8 ++++++++
 1 file changed, 8 insertions(+)
tools/perf/util/evsel.c +8 -0
index f203f178fdb9..8945f08f41b2 100644
@@ -2824,6 +2824,14 @@ int evsel__open_strerror(struct evsel *evsel, struct target *target,
 			return scnprintf(msg, size, "The 'aux_output' feature is not supported, update the kernel.");
 		if (is_amd && (evsel->core.attr.precise_ip || !strncmp(evsel->pmu_name, "ibs", 3)) && (evsel->core.attr.exclude_kernel))
 			return scnprintf(msg, size, "AMD IBS can't exclude kernel events.  Try running at a higher privilege level.");
+
+		if (is_amd && ((evsel->core.attr.config & 0xff) == 0xc4) && (evsel->core.attr.sample_type & PERF_SAMPLE_BRANCH_STACK)) {
+			if (evsel->core.attr.freq)
+				return scnprintf(msg, size, "AMD Branch Sampling does not support frequency mode sampling, must pass a fixed sampling period via -c option or cpu/branch-brs,period=xxxx/.");
+			/* another reason is that the period is too small */
+			return scnprintf(msg, size, "AMD Branch Sampling does not support sampling period smaller than what is reported in /sys/devices/cpu/caps/branches.");
+		}
+
 		break;
 	case ENODATA:
 		return scnprintf(msg, size, "Cannot collect data source with the load latency event alone. "
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
#34 lore raw

Re: [PATCH v1 12/13] perf tools: improve error handling of AMD Branch Sampling

Quoted 10 lines · Stephane Eranian

This patch improves the error message printed by perf when perf_event_open() fails on AMD Zen3 when using the branch sampling feature. In the case of EINVAL, there are two main reasons: frequency mode or period is smaller than the depth of the branch sampling buffer (16). The patch checks the parameters of the call and tries to print a relevant message to explain the error:

$ perf record -b -e cpu/branch-brs/ -c 10 ls
Error:
AMD Branch Sampling does not support sampling period smaller than what is reported in /sys/devices/cpu/caps/branches.
Quoted 6 lines
$ perf record -b -e cpu/branch-brs/ ls
Error:
AMD Branch Sampling does not support frequency mode sampling, must pass a fixed sampling period via -c option or cpu/branch-brs,period=xxxx/.

---

Hi Stephane,

I've rewritten this patch based on Arnaldo's comments to the previous (11/13) patch. The new version attached depends on this 2-patch series:

https://lore.kernel.org/lkml/[email protected]/T/#mc4c9c582e3816ab31af6d0187e6803de1a98ac84

Thanks,

Kim

From a4cbab762719b30bddec2e278cf8b8eb82e83865 Mon Sep 17 00:00:00 2001
From: Stephane Eranian <[email protected]>
Date: Thu, 9 Sep 2021 00:56:59 -0700
Subject: [PATCH] perf tools: Improve error handling of AMD Branch Sampling

This patch improves the error message printed by perf when perf_event_open() fails on AMD Zen3 when using the branch sampling feature. In the case of EINVAL, there are two main reasons: frequency mode or period is smaller than the depth of the branch sampling buffer (16). The patch checks the parameters of the call and tries to print a relevant message to explain the error:

$ perf record -b -e cpu/branch-brs/ -c 10 ls
Error:
AMD Branch Sampling does not support sampling period smaller than what is reported in /sys/devices/cpu/caps/branches.
$ perf record -b -e cpu/branch-brs/ ls
Error:
AMD Branch Sampling does not support frequency mode sampling, must pass a fixed sampling period via -c option or cpu/branch-brs,period=xxxx/.
[Rebased on commit 9fe8895a27a84 ("perf env: Add perf_env__cpuid, perf_env__{nr_}pmu_mappings")]
---
 tools/perf/util/evsel.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
tools/perf/util/evsel.c +14 -0
index f8a9cbd99314..e1f5eff07355 100644
@@ -2753,6 +2753,12 @@ static bool is_amd_ibs(struct evsel *evsel)
 	return evsel->core.attr.precise_ip || !strncmp(evsel->pmu_name, "ibs", 3);
 }
 
+static bool is_amd_brs(struct evsel *evsel)
+{
+	return ((evsel->core.attr.config & 0xff) == 0xc4) &&
+	       (evsel->core.attr.sample_type & PERF_SAMPLE_BRANCH_STACK);
+}
+
 int evsel__open_strerror(struct evsel *evsel, struct target *target,
 			 int err, char *msg, size_t size)
 {
@@ -2863,6 +2869,14 @@ int evsel__open_strerror(struct evsel *evsel, struct target *target,
 					return scnprintf(msg, size,
 	"AMD IBS may only be available in system-wide/per-cpu mode.  Try using -a, or -C and workload affinity");
 			}
+			if (is_amd_brs(evsel)) {
+				if (evsel->core.attr.freq)
+					return scnprintf(msg, size,
+	"AMD Branch Sampling does not support frequency mode sampling, must pass a fixed sampling period via -c option or cpu/branch-brs,period=xxxx/.");
+				/* another reason is that the period is too small */
+				return scnprintf(msg, size,
+	"AMD Branch Sampling does not support sampling period smaller than what is reported in /sys/devices/cpu/caps/branches.");
+			}
 		}
 
 		break;
Signature · 1 line
2.31.1
#35 lore raw

PATCH 13/13 perf report: add addr_from/addr_to sort dimensions

In reply to Stephane Eranian

This patch adds a new pair of sorting dimensions used with branch sampling. With the existing symbol_from/symbol_to, branches captured in the same function would be collapsed into a single function if the latency (cycles) was all the same. That is the case on Intel Broadwell for instance. Since Intel Skylake, the latency is captured by hardware and therefore is used to disambiguate branches.

This patch adds addr_from/addr_to sort dimensions that will sort branches based on their addresses and not the function there are in. The output is still the function name but the offset within the function is provided to uniquely identify each branch. These new sort dimensions also help with annotate because they create different entries in the histogram which, in turn, generates proper branch annotation.

Here is an example using AMD's branch sampling:

$ perf record -a -b -c 1000037 -e cpu/branch-brs/ test_prg
$ perf report
Samples: 6M of event 'cpu/branch-brs/', Event count (approx.): 6901276
Overhead  Command          Source Shared Object  Source Symbol                                   Target Symbol                                   Basic Block Cycle
  99.65%  test_prg	   test_prg              [.] test_thread                                 [.] test_thread                                 -
   0.02%  test_prg         [kernel.vmlinux]      [k] asm_sysvec_apic_timer_interrupt             [k] error_entry                                 -
$ perf report -F overhead,comm,dso,addr_from,addr_to
Samples: 6M of event 'cpu/branch-brs/', Event count (approx.): 6901276
Overhead  Command          Shared Object     Source Address          Target Address
   4.22%  test_prg         test_prg          [.] test_thread+0x3c    [.] test_thread+0x4
   4.13%  test_prg         test_prg          [.] test_thread+0x4     [.] test_thread+0x3a
   4.09%  test_prg         test_prg          [.] test_thread+0x3a    [.] test_thread+0x6
   4.08%  test_prg         test_prg          [.] test_thread+0x2     [.] test_thread+0x3c
   4.06%  test_prg         test_prg          [.] test_thread+0x3e    [.] test_thread+0x2
   3.87%  test_prg         test_prg          [.] test_thread+0x6     [.] test_thread+0x38
   3.84%  test_prg         test_prg          [.] test_thread         [.] test_thread+0x3e
   3.76%  test_prg         test_prg          [.] test_thread+0x1e    [.] test_thread
   3.76%  test_prg         test_prg          [.] test_thread+0x38    [.] test_thread+0x8
   3.56%  test_prg         test_prg          [.] test_thread+0x22    [.] test_thread+0x1e
   3.54%  test_prg         test_prg          [.] test_thread+0x8     [.] test_thread+0x36
   3.47%  test_prg         test_prg          [.] test_thread+0x1c    [.] test_thread+0x22
   3.45%  test_prg         test_prg          [.] test_thread+0x36    [.] test_thread+0xa
   3.28%  test_prg         test_prg          [.] test_thread+0x24    [.] test_thread+0x1c
   3.25%  test_prg         test_prg          [.] test_thread+0xa     [.] test_thread+0x34
   3.24%  test_prg         test_prg          [.] test_thread+0x1a    [.] test_thread+0x24
   3.20%  test_prg         test_prg          [.] test_thread+0x34    [.] test_thread+0xc
   3.04%  test_prg         test_prg          [.] test_thread+0x26    [.] test_thread+0x1a
   3.01%  test_prg         test_prg          [.] test_thread+0xc     [.] test_thread+0x32
   2.98%  test_prg         test_prg          [.] test_thread+0x18    [.] test_thread+0x26
   2.94%  test_prg         test_prg          [.] test_thread+0x32    [.] test_thread+0xe
   2.76%  test_prg         test_prg          [.] test_thread+0x28    [.] test_thread+0x18
   2.73%  test_prg         test_prg          [.] test_thread+0xe     [.] test_thread+0x30
   2.67%  test_prg         test_prg          [.] test_thread+0x30    [.] test_thread+0x10
   2.67%  test_prg         test_prg          [.] test_thread+0x16    [.] test_thread+0x28
   2.46%  test_prg         test_prg          [.] test_thread+0x10    [.] test_thread+0x2e
   2.44%  test_prg         test_prg          [.] test_thread+0x2a    [.] test_thread+0x16
   2.38%  test_prg         test_prg          [.] test_thread+0x14    [.] test_thread+0x2a
   2.32%  test_prg         test_prg          [.] test_thread+0x2e    [.] test_thread+0x12
   2.28%  test_prg         test_prg          [.] test_thread+0x12    [.] test_thread+0x2c
   2.16%  test_prg         test_prg          [.] test_thread+0x2c    [.] test_thread+0x14
   0.02%  test_prg         [kernel.vmlinux]  [k] asm_sysvec_apic_ti+0x5  [k] error_entry
---
 tools/perf/util/hist.c |   2 +
 tools/perf/util/hist.h |   2 +
 tools/perf/util/sort.c | 128 +++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/sort.h |   2 +
 4 files changed, 134 insertions(+)
tools/perf/util/hist.c +2 -0
index 65fe65ba03c2..39924202568f 100644
@@ -124,6 +124,7 @@ void hists__calc_col_len(struct hists *hists, struct hist_entry *h)
 		} else {
 			symlen = unresolved_col_width + 4 + 2;
 			hists__new_col_len(hists, HISTC_SYMBOL_FROM, symlen);
+			hists__new_col_len(hists, HISTC_ADDR_FROM, symlen);
 			hists__set_unres_dso_col_len(hists, HISTC_DSO_FROM);
 		}
 
@@ -138,6 +139,7 @@ void hists__calc_col_len(struct hists *hists, struct hist_entry *h)
 		} else {
 			symlen = unresolved_col_width + 4 + 2;
 			hists__new_col_len(hists, HISTC_SYMBOL_TO, symlen);
+			hists__new_col_len(hists, HISTC_ADDR_TO, symlen);
 			hists__set_unres_dso_col_len(hists, HISTC_DSO_TO);
 		}
 
tools/perf/util/hist.h +2 -0
index 5343b62476e6..9592853fab73 100644
@@ -76,6 +76,8 @@ enum hist_column {
 	HISTC_LOCAL_INS_LAT,
 	HISTC_GLOBAL_INS_LAT,
 	HISTC_P_STAGE_CYC,
+	HISTC_ADDR_FROM,
+	HISTC_ADDR_TO,
 	HISTC_NR_COLS, /* Last entry */
 };
 
tools/perf/util/sort.c +128 -0

146 lines not on this page · raw message

tools/perf/util/sort.h +2 -0
index b67c469aba79..571f6745dced 100644
@@ -250,6 +250,8 @@ enum sort_type {
 	SORT_SRCLINE_FROM,
 	SORT_SRCLINE_TO,
 	SORT_SYM_IPC,
+	SORT_ADDR_FROM,
+	SORT_ADDR_TO,
 
 	/* memory mode specific sort keys */
 	__SORT_MEMORY_MODE,
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
maintainer
#36 lore raw
In reply to Stephane Eranian
Quoted 2 lines · Stephane Eranian

This patch series adds support for the AMD Fam19h 16-deep branch sampling feature as described in the AMD PPR Fam19h Model 01h Revision B1 section 2.1.13.

Yay..

Quoted 4 lines

BRS interacts with the NMI interrupt as well. Because enabling BRS is expensive, it is only activated after P event occurrences, where P is the desired sampling period. At P occurrences of the event, the counter overflows, the CPU catches the NMI interrupt, activates BRS for 16 branches until it saturates, and then delivers the NMI to the kernel.

WTF... ?!? Srsly? You're joking right?

Also, can you please fix you MUA to wrap at 78 chars like normal people?

#37 lore raw
Quoted 5 lines · Peter Zijlstra
Quoted 2 lines · Stephane Eranian

This patch series adds support for the AMD Fam19h 16-deep branch sampling feature as described in the AMD PPR Fam19h Model 01h Revision B1 section 2.1.13.

Quoted 9 lines

Yay..

Quoted 4 lines

BRS interacts with the NMI interrupt as well. Because enabling BRS is expensive, it is only activated after P event occurrences, where P is the desired sampling period. At P occurrences of the event, the counter overflows, the CPU catches the NMI interrupt, activates BRS for 16 branches until it saturates, and then delivers the NMI to the kernel.

WTF... ?!? Srsly? You're joking right?

As I said, this is because of the cost of running BRS usually for millions of branches to keep only the last 16.
Running branch sampling in general on any arch is never totally free.

Quoted 2 lines

Also, can you please fix you MUA to wrap at 78 chars like normal people?

Ok, I fixed that now.

maintainer
#38 lore raw
Quoted 13 lines · Stephane Eranian
Quoted 12 lines · Peter Zijlstra
Quoted 2 lines · Stephane Eranian

This patch series adds support for the AMD Fam19h 16-deep branch sampling feature as described in the AMD PPR Fam19h Model 01h Revision B1 section 2.1.13.

Yay..

Quoted 4 lines

BRS interacts with the NMI interrupt as well. Because enabling BRS is expensive, it is only activated after P event occurrences, where P is the desired sampling period. At P occurrences of the event, the counter overflows, the CPU catches the NMI interrupt, activates BRS for 16 branches until it saturates, and then delivers the NMI to the kernel.

Quoted 6 lines
Quoted 2 lines

WTF... ?!? Srsly? You're joking right?

As I said, this is because of the cost of running BRS usually for millions of branches to keep only the last 16.
Running branch sampling in general on any arch is never totally free.

Holding up the NMI will disrupt the sampling of the other events, which is, IMO unacceptible and would require this event to be exclusive on the whole PMU, simply because sharing it doesn't work.

(also, other NMI sources might object)

Also, by only having LBRs post overflow you can't apply LBR based analysis to other events, which seems quite limiting.

This really seems like a very sub-optimal solution. I mean, it's awesome AMD gets branch records, but this seems a very poor solution.

#39 lore raw
Quoted 18 lines · Peter Zijlstra
Quoted 16 lines · Stephane Eranian
Quoted 14 lines · Peter Zijlstra
Quoted 2 lines · Stephane Eranian

This patch series adds support for the AMD Fam19h 16-deep branch sampling feature as described in the AMD PPR Fam19h Model 01h Revision B1 section 2.1.13.

Yay..

Quoted 4 lines

BRS interacts with the NMI interrupt as well. Because enabling BRS is expensive, it is only activated after P event occurrences, where P is the desired sampling period. At P occurrences of the event, the counter overflows, the CPU catches the NMI interrupt, activates BRS for 16 branches until it saturates, and then delivers the NMI to the kernel.

WTF... ?!? Srsly? You're joking right?

Quoted 8 lines
Quoted 3 lines

As I said, this is because of the cost of running BRS usually for millions of branches to keep only the last 16.
Running branch sampling in general on any arch is never totally free.

Holding up the NMI will disrupt the sampling of the other events, which is, IMO unacceptible and would require this event to be exclusive on the whole PMU, simply because sharing it doesn't work.

Sorry for the long delay, I have been very busy.

You are right on this. It would hold the NMI for 16 taken branches. Making the event exclusive creates a problem with the NMI watchdog. We can try to hack something in to allow NMI watchdog + the sampling event and nothing else.

Quoted 2 lines

(also, other NMI sources might object)

On AMD, there is also IBS op, IBS Fetch both firing on NMI. but that is less of a concern because the instruction address is captured by IBS and the interrupted IP is not useful. So the interrupt skid is not important.

Quoted 3 lines

Also, by only having LBRs post overflow you can't apply LBR based analysis to other events, which seems quite limiting.

This is a very limited functionality designed to support basic sampling primarily to support autoFDO where there is only one sampling event.

Quoted 2 lines

This really seems like a very sub-optimal solution. I mean, it's awesome AMD gets branch records, but this seems a very poor solution.

For now, this is what we have. It is important to get some basic form of branch sampling on Zen3 even if it is not perfect because it enables optimizations such as autoFDO for compilers today. We have verified that autoFDO works well with branch sampling on Zen3.

I hope it will improve in the future.

#40 lore raw
In reply to Stephane Eranian

Hi Stephane,

Quoted 14 lines · Stephane Eranian
Quoted 12 lines · Peter Zijlstra
Quoted 2 lines · Stephane Eranian

This patch series adds support for the AMD Fam19h 16-deep branch sampling feature as described in the AMD PPR Fam19h Model 01h Revision B1 section 2.1.13.

Yay..

Quoted 4 lines

BRS interacts with the NMI interrupt as well. Because enabling BRS is expensive, it is only activated after P event occurrences, where P is the desired sampling period. At P occurrences of the event, the counter overflows, the CPU catches the NMI interrupt, activates BRS for 16 branches until it saturates, and then delivers the NMI to the kernel.

Quoted 6 lines
Quoted 2 lines

WTF... ?!? Srsly? You're joking right?

As I said, this is because of the cost of running BRS usually for millions of branches to keep only the last 16.
Running branch sampling in general on any arch is never totally free.

Could you please share some data on how expensive the BRS is? We are hoping to use
BRS/LBR without PMI (bpf_get_branch_snapshot). If it is too expensive, we may need
some heuristic to turn it on/off.

Thanks,
Song