Skip to contentSkip to conversation

perf_events: add support for sampling taken branches (v2)

PATCH In review Stephane Eranian 3 participants 36 messages Oct 14, 2011 – Dec 9, 2011

Subsystems (9)
Arm (32-bit) x86 PowerPC MIPS +5 more subsystems
lore
Share & tools
lore
#0 lore raw

cover perf_events: add support for sampling taken branches (v2)

This patchset adds an important and useful new feature to
perf_events: branch stack sampling. In other words, the ability to capture taken branches into each sample.

Statistical sampling of taken branch should not be confused for branch tracing. Not all branches are necessarily captured

Sampling taken branches is important for basic block profiling, statistical call graph, function call counts. Many of those measurements can help drive a compiler optimizer.

The branch stack is a software abstraction which sits on top of the PMU hardware. As such, it is not available on all processors. For now, the patch provides the generic interface and the Intel X86 implementation where it leverages the Last Branch Record (LBR) feature (from Core2 to SandyBridge).

Branch stack sampling is supported for both per-thread and system-wide modes.

It is possible to filter the type and privilege level of branches to sample. The target of the branch is used to determine the privilege level.

For each branch, the source and destination are captured. On some hardware platforms, it may be possible to also extract the target prediction and, in that case, it is also exposed to end users.

The branch stack can record a variable number of taken branches per sample. Those branches are always consecutive in time. The number of branches captured depends on the filtering and the underlying hardware. On Intel Nehalem and later, up to 16 consecutive branches can be captured per sample.

Branch sampling is always coupled with an event. It can be any PMU event but it can't be a SW or tracepoint event.

Branch sampling is requested by setting a new sample_type flag called: PERF_SAMPLE_BRANCH_STACK.

To support branch filtering, we introduce a new field to the perf_event_attr struct: branch_sample_type. We chose NOT to overload the config1, config2 field because those are related to the event encoding. Branch stack is a separate feature which is combined with the event.

The branch_sample_type is a bitmask of possible filters. The following filters are defined (more can be added):

- PERF_SAMPLE_BRANCH_ANY     : any control flow change
- PERF_SAMPLE_BRANCH_USER    : capture branches when target is at user level
- PERF_SAMPLE_BRANCH_KERNEL  : capture branches when target is at user level

- PERF_SAMPLE_BRANCH_ANY_CALL: capture call branches (incl. syscalls)
- PERF_SAMPLE_BRANCH_ANY_RET : capture return branches (incl. syscall returns)
- PERF_SAMPLE_BRANCH_IND_CALL: capture indirect calls

It is possible to combine filters, e.g., IND_CALL|USER|KERNEL.

When the privilege level is not specified, the branch stack inherits that of the associated event.

Some processors may not offer hardware branch filtering, e.g., Intel Atom. Some may have HW filtering bugs (e.g., Nehalem). The Intel X86 implementation in this patchset also provides a SW branch filter which works on a best effort basis. It can compensate for the lack of LBR filtering. But first and foremost, it helps work around LBR filtering errata. The goal is to only capture the type of branches requested by the user.

It is possible to combine branch stack sampling with PEBS on Intel X86 processors. Depending on the precise_sampling mode, there are certain filterting restrictions. When precise_sampling=1, then there are no filtering restrictions. When precise_sampling > 1, then only ANY|USER|KERNEL filter can be used. This comes from the fact that the kernel uses LBR to compensate for the PEBS off-by-1 skid on the instruction pointer.

To demonstrate how the perf_event branch stack sampling interface works, the patchset also modifies perf record to capture taken branches. Similarly perf report is enhanced to display a histogram of taken branches.

I would like to thank Roberto Vitillo @ LBL for his work on the perf tool for this.

Enough talking, let's take a simple example. Our trivial test program goes like this:

void f2(void)
{}
void f3(void)
{}
void f1(unsigned long n)
{
if (n & 1UL)

    f2();
  else
    f3();
}

int main(void)
{
unsigned long i;

for (i=0; i < N; i++)
f1(i);
return 0;
}

$ perf record -b any branchy
$ perf report -b
# Events: 23K cycles
#
# Overhead  Source Symbol     Target Symbol
# ........  ................  ................
    18.13%  [.] f1            [.] main                          
    18.10%  [.] main          [.] main                          
    18.01%  [.] main          [.] f1                            
    15.69%  [.] f1            [.] f1                            
     9.11%  [.] f3            [.] f1                            
     6.78%  [.] f1            [.] f3                            
     6.74%  [.] f1            [.] f2                            
     6.71%  [.] f2            [.] f1                            

Of the total number of branches captured, 18.13% were from f1() -> main().

Let's make this clearer by filtering the user call branches only:

$ perf record -b any_call -e cycles:u branchy
$ perf report
# Events: 19K cycles
#
# Overhead  Source Symbol              Target Symbol
# ........  .........................  .........................
#
    52.50%  [.] main                   [.] f1                   
    23.99%  [.] f1                     [.] f3                   
    23.48%  [.] f1                     [.] f2                   
     0.03%  [.] _IO_default_xsputn     [.] _IO_new_file_overflow
     0.01%  [k] _start                 [k] __libc_start_main    

Now it is more obvious. %52 of all the captured branches where calls from main() -> f1(). The rest is split 50/50 between f1() -> f2() and f1() -> f3() which is expected given that f1() dispatches based on odd vs. even values of n which is constantly increasing.

In version 2, we update the patch to tip/master (commit 5734857) and we've incoporated the feedback from v1 concerning anynous bitfield struct for branch_stack_entry and the hanlding of i386 ABI binaries on 64-bit host in the instr decoder for the LBR SW filter.

Roberto Agostino Vitillo (2):
perf: add support for sampling taken branch to perf record
perf: add support for taken branch sampling to perf report

Stephane Eranian (10):
perf_events: add generic taken branch sampling support
perf_events: add Intel LBR MSR definitions
perf_events: add Intel X86 LBR sharing logic
perf_events: sync branch stack sampling with X86 precise_sampling
perf_events: add LBR mappings for PERF_SAMPLE_BRANCH filters
perf_events: implement PERF_SAMPLE_BRANCH for Intel X86
perf_events: add LBR software filter support for Intel X86
perf_events: disable PERF_SAMPLE_BRANCH_* when not supported
perf_events: add hook to flush branch_stack on context switch
perf: add code to support PERF_SAMPLE_BRANCH_STACK

 arch/alpha/kernel/perf_event.c             |    4 +
 arch/arm/kernel/perf_event.c               |    4 +
 arch/mips/kernel/perf_event.c              |    4 +
 arch/powerpc/kernel/perf_event.c           |    4 +
 arch/sh/kernel/perf_event.c                |    4 +
 arch/sparc/kernel/perf_event.c             |    4 +
 arch/x86/include/asm/msr-index.h           |    7 +
 arch/x86/kernel/cpu/perf_event.c           |   62 +++-
 arch/x86/kernel/cpu/perf_event_amd.c       |    3 +
 arch/x86/kernel/cpu/perf_event_intel.c     |  126 +++++--
 arch/x86/kernel/cpu/perf_event_intel_ds.c  |   21 +-
 arch/x86/kernel/cpu/perf_event_intel_lbr.c |  529 ++++++++++++++++++++++++++--
 include/linux/perf_event.h                 |   74 ++++-
 kernel/events/core.c                       |  167 +++++++++
 kernel/events/hw_breakpoint.c              |    6 +
 tools/perf/Documentation/perf-record.txt   |   18 +
 tools/perf/Documentation/perf-report.txt   |    7 +
 tools/perf/builtin-record.c                |   75 ++++
 tools/perf/builtin-report.c                |   93 +++++-
 tools/perf/perf.h                          |   17 +
 tools/perf/util/annotate.c                 |    2 +-
 tools/perf/util/event.h                    |    1 +
 tools/perf/util/evsel.c                    |   10 +
 tools/perf/util/hist.c                     |   97 ++++--
 tools/perf/util/hist.h                     |    6 +
 tools/perf/util/session.c                  |   72 ++++
 tools/perf/util/session.h                  |    5 +
 tools/perf/util/sort.c                     |  348 ++++++++++++++-----
 tools/perf/util/sort.h                     |    5 +
 tools/perf/util/symbol.h                   |   13 +
 30 files changed, 1584 insertions(+), 204 deletions(-)
Signature · 1 line
1.7.4.1
#1 lore raw

PATCH 1/12 perf_events: add generic taken branch sampling support (v2)

This patch adds the ability to sample taken branches to the perf_event interface.

The ability to capture taken branches is very useful for all sorts of analysis. For instance, basic block profiling, call counts, statistical call graph.

This new capability requires hardware assist and as such may not be available on all HW platforms. On Intel X86, it is implemented on top of the Last Branch Record (LBR) facility.

To enable taken branches sampling, the PERF_SAMPLE_BRANCH_STACK bit must be set in attr->sample_type.

Sampled taken branches may be filtered by type and/or priv levels.

The patch adds a new field, called branch_sample_type, to the perf_event_attr structure. It contains a bitmask of filters to apply to the sampled taken branches.

Filters may be implemented in HW. If the HW filter does not exist or is not good enough, some arch may also implement a SW filter.

The following generic filters are currently defined:
- PERF_SAMPLE_USER
only branches whose targets are at the user level

- PERF_SAMPLE_KERNEL
only branches whose targets are at the kernel level

- PERF_SAMPLE_ANY
any type of branches (subject to priv levels filters)

- PERF_SAMPLE_ANY_CALL
any call branches (may incl. syscall on some arch)

- PERF_SAMPLE_ANY_RET
any return branches (may incl. syscall returns on some arch)

- PERF_SAMPLE_IND_CALL
indirect call branches

Obviously filter may be combined. The priv level bits are optional. If not provided, the priv level of the associated event are used. It is possible to collect branches at a priv level different from the associated event.

The number of taken branch records present in each sample may vary based on HW, the type of sampled branches, the executed code. Therefore each sample contains the number of taken branches it contains.

---
 arch/x86/kernel/cpu/perf_event_intel_lbr.c |   21 +++++---
 include/linux/perf_event.h                 |   66 ++++++++++++++++++++++++++--
 kernel/events/core.c                       |   58 ++++++++++++++++++++++++
 3 files changed, 133 insertions(+), 12 deletions(-)
arch/x86/kernel/cpu/perf_event_intel_lbr.c +13 -8
index 3fab3de..b07e051 100644
@@ -144,9 +144,11 @@ static void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
 
 		rdmsrl(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
 
-		cpuc->lbr_entries[i].from  = msr_lastbranch.from;
-		cpuc->lbr_entries[i].to    = msr_lastbranch.to;
-		cpuc->lbr_entries[i].flags = 0;
+		cpuc->lbr_entries[i].from	= msr_lastbranch.from;
+		cpuc->lbr_entries[i].to		= msr_lastbranch.to;
+		cpuc->lbr_entries[i].mispred	= 0;
+		cpuc->lbr_entries[i].predicted	= 0;
+		cpuc->lbr_entries[i].reserved	= 0;
 	}
 	cpuc->lbr_stack.nr = i;
 }
@@ -167,19 +169,22 @@ static void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 
 	for (i = 0; i < x86_pmu.lbr_nr; i++) {
 		unsigned long lbr_idx = (tos - i) & mask;
-		u64 from, to, flags = 0;
+		u64 from, to, mis = 0, pred = 0;
 
 		rdmsrl(x86_pmu.lbr_from + lbr_idx, from);
 		rdmsrl(x86_pmu.lbr_to   + lbr_idx, to);
 
 		if (lbr_format == LBR_FORMAT_EIP_FLAGS) {
-			flags = !!(from & LBR_FROM_FLAG_MISPRED);
+			mis = !!(from & LBR_FROM_FLAG_MISPRED);
+			pred= !mis;
 			from = (u64)((((s64)from) << 1) >> 1);
 		}
 
-		cpuc->lbr_entries[i].from  = from;
-		cpuc->lbr_entries[i].to    = to;
-		cpuc->lbr_entries[i].flags = flags;
+		cpuc->lbr_entries[i].from	= from;
+		cpuc->lbr_entries[i].to		= to;
+		cpuc->lbr_entries[i].mispred	= mis;
+		cpuc->lbr_entries[i].predicted	= pred;
+		cpuc->lbr_entries[i].reserved	= 0;
 	}
 	cpuc->lbr_stack.nr = i;
 }
include/linux/perf_event.h +62 -4

122 lines not on this page · raw message

kernel/events/core.c +58 -0

76 lines not on this page · raw message

Signature · 1 line
1.7.1
maintainer
#2 lore raw

Re: [PATCH 01/12] perf_events: add generic taken branch sampling support (v2)

include/linux/perf_event.h enum perf_event_type { · line 487

Quoted 6 lines · Stephane Eranian
include/linux/perf_event.h +2 -0
@@ -455,6 +483,8 @@ enum perf_event_type {
         *
         *      { u32                   size;
         *        char                  data[size];}&& PERF_SAMPLE_RAW
+        *
+        *      { u64 from, to, flags } lbr[nr];} && PERF_SAMPLE_BRANCH_STACK 
		{ u64	nr;
		  { u64 from, to, flags } brstack[nr]; } && PERF_SAMPLE_BRANCH_STACK

Perhaps? It looks like you lost a line somewhere, even you curly braces are unmatched in the correct way.

#3 lore raw

Re: [PATCH 01/12] perf_events: add generic taken branch sampling support (v2)

Quoted 11 lines · Peter Zijlstra

include/linux/perf_event.h enum perf_event_type { · line 487

Quoted 6 lines · Stephane Eranian
include/linux/perf_event.h +2 -0
@@ -455,6 +483,8 @@ enum perf_event_type {
          *
          *      { u32                   size;
          *        char                  data[size];}&& PERF_SAMPLE_RAW
+        *
+        *      { u64 from, to, flags } lbr[nr];} && PERF_SAMPLE_BRANCH_STACK

               { u64   nr;
                 { u64 from, to, flags } brstack[nr]; } && PERF_SAMPLE_BRANCH_STACK

Quoted 2 lines

Perhaps? It looks like you lost a line somewhere, even you curly braces are unmatched in the correct way.

Yes, that the right way of describing the layout. Will fix that.

maintainer
#4 lore raw

Re: [PATCH 01/12] perf_events: add generic taken branch sampling support (v2)

In reply to Stephane Eranian

include/linux/perf_event.h enum perf_event_sample_format { · line 160

Quoted 3 lines · Stephane Eranian
include/linux/perf_event.h +3 -0
+#define PERF_SAMPLE_BRANCH_PLM_ALL \
+       (PERF_SAMPLE_BRANCH_USER|\
+        PERF_SAMPLE_BRANCH_KERNEL) 

This PLM thing keeps popping up all over, I'm sure it stands for something, but for now it just hurts my eyes.

kernel/events/core.c static int perf_copy_attr(struct perf_event_attr __user *uattr, · line 6018

Quoted 4 lines
kernel/events/core.c +3 -0
+               /* at least one branch bit must be set */
+               if (!(mask & ~PERF_SAMPLE_BRANCH_PLM_ALL))
+                       return -EINVAL;

Why? we can create counters with exclude_user && exclude_kernel as well, I mean, they're useless, but its perfectly valid.

#5 lore raw

Re: [PATCH 01/12] perf_events: add generic taken branch sampling support (v2)

Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Stephane Eranian

+#define PERF_SAMPLE_BRANCH_PLM_ALL \
+       (PERF_SAMPLE_BRANCH_USER|\
+        PERF_SAMPLE_BRANCH_KERNEL)

This PLM thing keeps popping up all over, I'm sure it stands for something, but for now it just hurts my eyes.

Quoted 6 lines
Quoted 4 lines

+               /* at least one branch bit must be set */
+               if (!(mask & ~PERF_SAMPLE_BRANCH_PLM_ALL))
+                       return -EINVAL;

Why? we can create counters with exclude_user && exclude_kernel as well, I mean, they're useless, but its perfectly valid.

I am fine with that change. I can drop this check.

#6 lore raw

PATCH 2/12 perf_events: add Intel LBR MSR definitions (v2)

In reply to Stephane Eranian

This patch adds the LBR definitions for NHM/WSM/SNB and Core. It also adds the definitions for the architected LBR MSR:
LBR_SELECT, LBRT_TOS.

---
 arch/x86/include/asm/msr-index.h           |    7 +++++++
 arch/x86/kernel/cpu/perf_event_intel_lbr.c |   18 +++++++++---------
 2 files changed, 16 insertions(+), 9 deletions(-)
arch/x86/include/asm/msr-index.h +7 -0
index d52609a..e8f5fbf 100644
@@ -56,6 +56,13 @@
 #define MSR_OFFCORE_RSP_0		0x000001a6
 #define MSR_OFFCORE_RSP_1		0x000001a7
 
+#define MSR_LBR_SELECT			0x000001c8
+#define MSR_LBR_TOS			0x000001c9
+#define MSR_LBR_NHM_FROM		0x00000680
+#define MSR_LBR_NHM_TO			0x000006c0
+#define MSR_LBR_CORE_FROM		0x00000040
+#define MSR_LBR_CORE_TO			0x00000060
+
 #define MSR_IA32_PEBS_ENABLE		0x000003f1
 #define MSR_IA32_DS_AREA		0x00000600
 #define MSR_IA32_PERF_CAPABILITIES	0x00000345
arch/x86/kernel/cpu/perf_event_intel_lbr.c +9 -9
index b07e051..e9ac6e9 100644
@@ -205,23 +205,23 @@ void intel_pmu_lbr_read(void)
 void intel_pmu_lbr_init_core(void)
 {
 	x86_pmu.lbr_nr     = 4;
-	x86_pmu.lbr_tos    = 0x01c9;
-	x86_pmu.lbr_from   = 0x40;
-	x86_pmu.lbr_to     = 0x60;
+	x86_pmu.lbr_tos    = MSR_LBR_TOS;
+	x86_pmu.lbr_from   = MSR_LBR_CORE_FROM;
+	x86_pmu.lbr_to     = MSR_LBR_CORE_TO;
 }
 
 void intel_pmu_lbr_init_nhm(void)
 {
 	x86_pmu.lbr_nr     = 16;
-	x86_pmu.lbr_tos    = 0x01c9;
-	x86_pmu.lbr_from   = 0x680;
-	x86_pmu.lbr_to     = 0x6c0;
+	x86_pmu.lbr_tos    = MSR_LBR_TOS;
+	x86_pmu.lbr_from   = MSR_LBR_NHM_FROM;
+	x86_pmu.lbr_to     = MSR_LBR_NHM_TO;
 }
 
 void intel_pmu_lbr_init_atom(void)
 {
 	x86_pmu.lbr_nr	   = 8;
-	x86_pmu.lbr_tos    = 0x01c9;
-	x86_pmu.lbr_from   = 0x40;
-	x86_pmu.lbr_to     = 0x60;
+	x86_pmu.lbr_tos    = MSR_LBR_TOS;
+	x86_pmu.lbr_from   = MSR_LBR_CORE_FROM;
+	x86_pmu.lbr_to     = MSR_LBR_CORE_TO;
 }
Signature · 1 line
1.7.1
#7 lore raw

PATCH 3/12 perf_events: add Intel X86 LBR sharing logic (v2)

In reply to Stephane Eranian

The Intel LBR on some recent processor is capable of filtering branches by type. The filter is configurable via the LBR_SELECT MSR register.

There are limitation on how this register can be used.

On Nehalem/Westmere, the LBR_SELECT is shared by the two HT threads when HT is on. It is private to each core when HT is off.

On SandyBridge, the LBR_SELECT register is private to each thread when HT is on. It is private to each core when HT is off.

The kernel must manage the sharing of LBR_SELECT. It allows multiple users on the same logical CPU to use LBR_SELECT as long as they program it with the same value. Across sibling CPUs (HT threads), the same restriction applies on NHM/WSM.

This patch implements this sharing logic by leveraging the mechanism put in place for managing the offcore_response shared MSR.

We modify __intel_shared_reg_get_constraints() to cause x86_get_event_constraint() to be called because LBR may be associated with events that may be counter constrained.

---
 arch/x86/kernel/cpu/perf_event.c       |    4 ++
 arch/x86/kernel/cpu/perf_event.h       |    4 ++
 arch/x86/kernel/cpu/perf_event_intel.c |   70 ++++++++++++++++++++------------
 3 files changed, 52 insertions(+), 26 deletions(-)
arch/x86/kernel/cpu/perf_event.c +4 -0
index 6408910..cfef90e 100644
@@ -428,6 +428,10 @@ static int __x86_pmu_event_init(struct perf_event *event)
 	/* mark unused */
 	event->hw.extra_reg.idx = EXTRA_REG_NONE;
 
+	/* mark not used */
+	event->hw.extra_reg.idx = EXTRA_REG_NONE;
+	event->hw.branch_reg.idx = EXTRA_REG_NONE;
+
 	return x86_pmu.hw_config(event);
 }
 
arch/x86/kernel/cpu/perf_event.h +4 -0
index b9698d4..8a5c21f 100644
@@ -33,6 +33,7 @@ enum extra_reg_type {
 
 	EXTRA_REG_RSP_0 = 0,	/* offcore_response_0 */
 	EXTRA_REG_RSP_1 = 1,	/* offcore_response_1 */
+	EXTRA_REG_LBR   = 2,	/* lbr_select */
 
 	EXTRA_REG_MAX		/* number of entries needed */
 };
@@ -129,6 +130,7 @@ struct cpu_hw_events {
 	void				*lbr_context;
 	struct perf_branch_stack	lbr_stack;
 	struct perf_branch_entry	lbr_entries[MAX_LBR_ENTRIES];
+	struct er_account		*lbr_sel;
 
 	/*
 	 * Intel host/guest exclude bits
@@ -296,6 +298,8 @@ struct x86_pmu {
 	 */
 	unsigned long	lbr_tos, lbr_from, lbr_to; /* MSR base regs       */
 	int		lbr_nr;			   /* hardware stack size */
+	u64		lbr_sel_mask;   	   /* valid bits in LBR_SELECT */
+	const int	*lbr_sel_map;   	   /* lbr_select mappings */
 
 	/*
 	 * Extra registers for events
arch/x86/kernel/cpu/perf_event_intel.c +44 -26

121 lines not on this page · raw message

Signature · 1 line
1.7.1
#8 lore raw

PATCH 4/12 perf_events: sync branch stack sampling with X86 precise_sampling (v2)

In reply to Stephane Eranian

If precise sampling is enabled on Intel X86, then perf_event uses PEBS. To correct for the off-by-one error of PEBS, perf_event uses LBR when precise_sample > 1.

On Intel X86 PERF_SAMPLE_BRANCH_STACK is implemented using LBR, therefore both features must be coordinated as they may not configure LBR the same way.

For PEBS, LBR needs to capture all branches at all priv levels. This patch sets this up.

The configuration of PERF_SAMPLE_BRANCH_STACK may not be compatible in which case an error must be returned.

---
 arch/x86/kernel/cpu/perf_event.c |   22 ++++++++++++++++++++++
 1 files changed, 22 insertions(+), 0 deletions(-)
arch/x86/kernel/cpu/perf_event.c +22 -0
index cfef90e..e2efa90 100644
@@ -358,6 +358,7 @@ int x86_setup_perfctr(struct perf_event *event)
 int x86_pmu_hw_config(struct perf_event *event)
 {
 	if (event->attr.precise_ip) {
+		u64 *br_type, br_sel;
 		int precise = 0;
 
 		/* Support for constant skid */
@@ -371,6 +372,27 @@ int x86_pmu_hw_config(struct perf_event *event)
 
 		if (event->attr.precise_ip > precise)
 			return -EOPNOTSUPP;
+		/*
+		 * check that PEBS LBR correction does not conflict with
+		 * whatever the user is asking with attr->branch_sample_type
+		 */
+		if (event->attr.precise_ip > 1) {
+
+			br_type = &event->attr.branch_sample_type;
+
+			if (has_branch_stack(event)) {
+				br_sel = *br_type & PERF_SAMPLE_BRANCH_ANY;
+				if (br_sel != PERF_SAMPLE_BRANCH_ANY)
+					return -EOPNOTSUPP;
+			} else {
+				/*
+				 * For PEBS fixups, we capture all
+				 * the branches at all priv levels
+				 */
+				*br_type = PERF_SAMPLE_BRANCH_ANY
+					 | PERF_SAMPLE_BRANCH_PLM_ALL;
+			}
+		}
 	}
 
 	/*
Signature · 1 line
1.7.1
#9 lore raw

PATCH 5/12 perf_events: add LBR mappings for PERF_SAMPLE_BRANCH filters (v2)

In reply to Stephane Eranian

This patch adds the mappings from the generic PERF_SAMPLE_BRANCH_* filters to the actual Intel X86 LBR filters, whenever they exist.

The patch also adds a restriction on Intel Atom, whereby only stepping 10 (PineView) and more recent are supported. Older models, do not have a functional LBR (does not freeze on PMU interrupt).

---
 arch/x86/kernel/cpu/perf_event.h           |    2 +
 arch/x86/kernel/cpu/perf_event_intel.c     |    2 +-
 arch/x86/kernel/cpu/perf_event_intel_lbr.c |  110 +++++++++++++++++++++++++++-
 3 files changed, 111 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/perf_event.h +2 -0
index 8a5c21f..750c7af 100644
@@ -482,6 +482,8 @@ void intel_pmu_lbr_init_nhm(void);
 
 void intel_pmu_lbr_init_atom(void);
 
+void intel_pmu_lbr_init_snb(void);
+
 int p4_pmu_init(void);
 
 int p6_pmu_init(void);
arch/x86/kernel/cpu/perf_event_intel.c +1 -1
index 1303732..6f313c0 100644
@@ -1715,7 +1715,7 @@ __init int intel_pmu_init(void)
 		memcpy(hw_cache_event_ids, snb_hw_cache_event_ids,
 		       sizeof(hw_cache_event_ids));
 
-		intel_pmu_lbr_init_nhm();
+		intel_pmu_lbr_init_snb();
 
 		x86_pmu.event_constraints = intel_snb_event_constraints;
 		x86_pmu.pebs_constraints = intel_snb_pebs_event_constraints;
arch/x86/kernel/cpu/perf_event_intel_lbr.c +108 -2

148 lines not on this page · raw message

Signature · 1 line
1.7.1
maintainer
#10 lore raw

Re: [PATCH 05/12] perf_events: add LBR mappings for PERF_SAMPLE_BRANCH filters (v2)

arch/x86/kernel/cpu/perf_event_intel_lbr.c void intel_pmu_lbr_read(void) · line 325

Quoted 11 lines · Stephane Eranian

void intel_pmu_lbr_init_atom(void)
{

arch/x86/kernel/cpu/perf_event_intel_lbr.c +9 -0
+       /*
+        * only models starting at stepping 10 seems
+        * to have an operational LBR which can freeze
+        * on PMU interrupt
+        */
+       if (boot_cpu_data.x86_mask < 10) {
+               pr_cont("LBR disabled due to erratum");
+               return;
+       } 

Shouldn't that be a separate patch?

#11 lore raw

Re: [PATCH 05/12] perf_events: add LBR mappings for PERF_SAMPLE_BRANCH filters (v2)

Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Stephane Eranian

 void intel_pmu_lbr_init_atom(void)
 {
+       /*
+        * only models starting at stepping 10 seems
+        * to have an operational LBR which can freeze
+        * on PMU interrupt
+        */
+       if (boot_cpu_data.x86_mask < 10) {
+               pr_cont("LBR disabled due to erratum");
+               return;
+       }

Quoted 1 line

Shouldn't that be a separate patch?

I'll make it into a separate patch.

#12 lore raw

PATCH 6/12 perf_events: implement PERF_SAMPLE_BRANCH for Intel X86 (v2)

In reply to Stephane Eranian

This patch implements PERF_SAMPLE_BRANCH support for Intel X86 processors. It connects PERF_SAMPLE_BRANCH to the actual LBR.

The patch adds the hooks in the PMU irq handler to save the LBR on counter overflow for both regular and PEBS modes.

---
 arch/x86/kernel/cpu/perf_event_intel.c     |   35 +++++++++++++
 arch/x86/kernel/cpu/perf_event_intel_ds.c  |   10 ++--
 arch/x86/kernel/cpu/perf_event_intel_lbr.c |   73 +++++++++++++++++++++++++++-
 include/linux/perf_event.h                 |    3 +
 4 files changed, 113 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/perf_event_intel.c +35 -0
index 6f313c0..901217d 100644
@@ -730,6 +730,19 @@ static __initconst const u64 atom_hw_cache_event_ids
  },
 };
 
+static inline bool intel_pmu_needs_lbr_smpl(struct perf_event *event)
+{
+	/* user explicitly requested branch sampling */
+	if (has_branch_stack(event))
+		return true;
+
+	/* implicit branch sampling to correct PEBS skid */
+	if (x86_pmu.intel_cap.pebs_trap && event->attr.precise_ip > 1)
+		return true;
+
+	return false;
+}
+
 static void intel_pmu_disable_all(void)
 {
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
@@ -884,6 +897,13 @@ static void intel_pmu_disable_event(struct perf_event *event)
 	cpuc->intel_ctrl_guest_mask &= ~(1ull << hwc->idx);
 	cpuc->intel_ctrl_host_mask &= ~(1ull << hwc->idx);
 
+	/*
+	 * must disable before any actual event
+	 * because any event may be combined with LBR
+	 */
+	if (intel_pmu_needs_lbr_smpl(event))
+		intel_pmu_lbr_disable(event);
+
 	if (unlikely(hwc->config_base == MSR_ARCH_PERFMON_FIXED_CTR_CTRL)) {
 		intel_pmu_disable_fixed(hwc);
 		return;
@@ -938,6 +958,12 @@ static void intel_pmu_enable_event(struct perf_event *event)
 		intel_pmu_enable_bts(hwc->config);
 		return;
 	}
+	/*
+	 * must enabled before any actual event
+	 * because any event may be combined with LBR
+	 */
+	if (intel_pmu_needs_lbr_smpl(event))
+		intel_pmu_lbr_enable(event);
 
 	if (event->attr.exclude_host)
 		cpuc->intel_ctrl_guest_mask |= (1ull << hwc->idx);
@@ -1060,6 +1086,9 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 
 		data.period = event->hw.last_period;
 
+		if (has_branch_stack(event))
+			data.br_stack = &cpuc->lbr_stack;
+
 		if (perf_event_overflow(event, &data, regs))
 			x86_pmu_stop(event, 0);
 	}
@@ -1308,6 +1337,12 @@ static int intel_pmu_hw_config(struct perf_event *event)
 		event->hw.config = alt_config;
 	}
 
+	if (intel_pmu_needs_lbr_smpl(event)) {
+		ret = intel_pmu_setup_lbr_filter(event);
+		if (ret)
+			return ret;
+	}
+
 	if (event->attr.type != PERF_TYPE_RAW)
 		return 0;
 
arch/x86/kernel/cpu/perf_event_intel_ds.c +4 -6
index c0d238f..d0197ba 100644
@@ -440,9 +440,6 @@ void intel_pmu_pebs_enable(struct perf_event *event)
 
 	cpuc->pebs_enabled |= 1ULL << hwc->idx;
 	WARN_ON_ONCE(cpuc->enabled);
-
-	if (x86_pmu.intel_cap.pebs_trap && event->attr.precise_ip > 1)
-		intel_pmu_lbr_enable(event);
 }
 
 void intel_pmu_pebs_disable(struct perf_event *event)
@@ -455,9 +452,6 @@ void intel_pmu_pebs_disable(struct perf_event *event)
 		wrmsrl(MSR_IA32_PEBS_ENABLE, cpuc->pebs_enabled);
 
 	hwc->config |= ARCH_PERFMON_EVENTSEL_INT;
-
-	if (x86_pmu.intel_cap.pebs_trap && event->attr.precise_ip > 1)
-		intel_pmu_lbr_disable(event);
 }
 
 void intel_pmu_pebs_enable_all(void)
@@ -569,6 +563,7 @@ static void __intel_pmu_pebs_event(struct perf_event *event,
 	 * both formats and we don't use the other fields in this
 	 * routine.
 	 */
+	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
 	struct pebs_record_core *pebs = __pebs;
 	struct perf_sample_data data;
 	struct pt_regs regs;
@@ -599,6 +594,9 @@ static void __intel_pmu_pebs_event(struct perf_event *event,
 	else
 		regs.flags &= ~PERF_EFLAGS_EXACT;
 
+	if (has_branch_stack(event))
+		data.br_stack = &cpuc->lbr_stack;
+
 	if (perf_event_overflow(event, &data, &regs))
 		x86_pmu_stop(event, 0);
 }
arch/x86/kernel/cpu/perf_event_intel_lbr.c +71 -2

116 lines not on this page · raw message

include/linux/perf_event.h +3 -0
index d8f0278..c4fbe84 100644
@@ -159,6 +159,9 @@ enum perf_branch_sample_type {
 	(PERF_SAMPLE_BRANCH_USER|\
 	 PERF_SAMPLE_BRANCH_KERNEL)
 
+#define PERF_SAMPLE_BRANCH_ALL \
+	(PERF_SAMPLE_BRANCH_PLM_ALL|PERF_SAMPLE_BRANCH_ANY)
+
 /*
  * The format of the data returned by read() on a perf event fd,
  * as specified by attr.read_format:
Signature · 1 line
1.7.1
#13 lore raw

PATCH 7/12 perf_events: add LBR software filter support for Intel X86 (v2)

In reply to Stephane Eranian

This patch adds an internal sofware filter to complement the (optional) LBR hardware filter.

The software filter is necessary:
- as a substitute when there is no HW LBR filter (e.g., Atom, Core)
- to complement HW LBR filter in case of errata (e.g., Nehalem/Westmere)
- to provide finer grain filtering (e.g., all processors)

Sometimes, the LBR HW filter cannot distinguish between two types of branches. For instance, to capture syscall as CALLS, it is necessary to enable the LBR_FAR filter which will also capture JMP instructions. Thus, a second pass is necessary to filter those out, this is what the SW filter can do.

The SW filter is built on top of the internal x86 disassembler. It is a best effort filter especially for user level code. It is subject to the availability of the text page of the program.

The SW filter is enabled on all Intel X86 processors. It is bypassed when the user is capturing all branches at all priv levels.

---
 arch/x86/kernel/cpu/perf_event.h           |   12 +
 arch/x86/kernel/cpu/perf_event_intel_ds.c  |   12 +-
 arch/x86/kernel/cpu/perf_event_intel_lbr.c |  322 +++++++++++++++++++++++++++-
 3 files changed, 326 insertions(+), 20 deletions(-)
arch/x86/kernel/cpu/perf_event.h +12 -0
index 750c7af..48ed504 100644
@@ -131,6 +131,7 @@ struct cpu_hw_events {
 	struct perf_branch_stack	lbr_stack;
 	struct perf_branch_entry	lbr_entries[MAX_LBR_ENTRIES];
 	struct er_account		*lbr_sel;
+	u64				br_sel;
 
 	/*
 	 * Intel host/guest exclude bits
@@ -402,6 +403,17 @@ extern struct event_constraint emptyconstraint;
 
 extern struct event_constraint unconstrained;
 
+static inline bool kernel_ip(unsigned long ip)
+{
+#ifdef CONFIG_X86_32
+	return ip > PAGE_OFFSET;
+#else
+	return (long)ip < 0;
+#endif
+}
+
+int intel_pmu_setup_lbr_filter(struct perf_event *event);
+
 #ifdef CONFIG_CPU_SUP_AMD
 
 int amd_pmu_init(void);
arch/x86/kernel/cpu/perf_event_intel_ds.c +1 -11
index d0197ba..8c17380 100644
@@ -3,6 +3,7 @@
 #include <linux/slab.h>
 
 #include <asm/perf_event.h>
+#include <asm/insn.h>
 
 #include "perf_event.h"
 
@@ -470,17 +471,6 @@ void intel_pmu_pebs_disable_all(void)
 		wrmsrl(MSR_IA32_PEBS_ENABLE, 0);
 }
 
-#include <asm/insn.h>
-
-static inline bool kernel_ip(unsigned long ip)
-{
-#ifdef CONFIG_X86_32
-	return ip > PAGE_OFFSET;
-#else
-	return (long)ip < 0;
-#endif
-}
-
 static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
 {
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
arch/x86/kernel/cpu/perf_event_intel_lbr.c +313 -9

384 lines not on this page · raw message

Signature · 1 line
1.7.1
maintainer
#14 lore raw

Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86 (v2)

Quoted 15 lines · Stephane Eranian

This patch adds an internal sofware filter to complement the (optional) LBR hardware filter.

The software filter is necessary:
- as a substitute when there is no HW LBR filter (e.g., Atom, Core)
- to complement HW LBR filter in case of errata (e.g., Nehalem/Westmere)
- to provide finer grain filtering (e.g., all processors)

Sometimes, the LBR HW filter cannot distinguish between two types of branches. For instance, to capture syscall as CALLS, it is necessary to enable the LBR_FAR filter which will also capture JMP instructions. Thus, a second pass is necessary to filter those out, this is what the SW filter can do.

Quoted 6 lines

The SW filter is built on top of the internal x86 disassembler. It is a best effort filter especially for user level code. It is subject to the availability of the text page of the program.

The SW filter is enabled on all Intel X86 processors. It is bypassed when the user is capturing all branches at all priv levels.

This patch is very seriously whitespace challenged.

#15 lore raw

PATCH 8/12 perf_events: disable PERF_SAMPLE_BRANCH_* when not supported (v2)

In reply to Stephane Eranian

PERF_SAMPLE_BRANCH_* is disabled for:
- SW events (sw counters, tracepoints)
- HW breakpoints
- ALL but Intel X86 architecture
- AMD64 processors

---
 arch/alpha/kernel/perf_event.c       |    4 ++++
 arch/arm/kernel/perf_event.c         |    4 ++++
 arch/mips/kernel/perf_event.c        |    4 ++++
 arch/powerpc/kernel/perf_event.c     |    4 ++++
 arch/sh/kernel/perf_event.c          |    4 ++++
 arch/sparc/kernel/perf_event.c       |    4 ++++
 arch/x86/kernel/cpu/perf_event_amd.c |    3 +++
 kernel/events/core.c                 |   24 ++++++++++++++++++++++++
 kernel/events/hw_breakpoint.c        |    6 ++++++
 9 files changed, 57 insertions(+), 0 deletions(-)
arch/alpha/kernel/perf_event.c +4 -0
index 8143cd7..0dae252 100644
@@ -685,6 +685,10 @@ static int alpha_pmu_event_init(struct perf_event *event)
 {
 	int err;
 
+	/* does not support taken branch sampling */
+	if (has_branch_stack(event))
+		return -EOPNOTSUPP;
+
 	switch (event->attr.type) {
 	case PERF_TYPE_RAW:
 	case PERF_TYPE_HARDWARE:
arch/arm/kernel/perf_event.c +4 -0
index 53c9c26..bcb0dd1 100644
@@ -544,6 +544,10 @@ static int armpmu_event_init(struct perf_event *event)
 {
 	int err = 0;
 
+	/* does not support taken branch sampling */
+	if (has_branch_smpl(event))
+		return -EOPNOTSUPP;
+
 	switch (event->attr.type) {
 	case PERF_TYPE_RAW:
 	case PERF_TYPE_HARDWARE:
arch/mips/kernel/perf_event.c +4 -0
index 0aee944..425c35a 100644
@@ -370,6 +370,10 @@ static int mipspmu_event_init(struct perf_event *event)
 {
 	int err = 0;
 
+	/* does not support taken branch sampling */
+	if (has_branch_stack(event))
+		return -EOPNOTSUPP;
+
 	switch (event->attr.type) {
 	case PERF_TYPE_RAW:
 	case PERF_TYPE_HARDWARE:
arch/powerpc/kernel/perf_event.c +4 -0
index 10a140f..5701051 100644
@@ -1078,6 +1078,10 @@ static int power_pmu_event_init(struct perf_event *event)
 	if (!ppmu)
 		return -ENOENT;
 
+	/* does not support taken branch sampling */
+	if (has_branch_stack(event))
+		return -EOPNOTSUPP;
+
 	switch (event->attr.type) {
 	case PERF_TYPE_HARDWARE:
 		ev = event->attr.config;
arch/sh/kernel/perf_event.c +4 -0
index 2ee21a4..7cc9066 100644
@@ -309,6 +309,10 @@ static int sh_pmu_event_init(struct perf_event *event)
 {
 	int err;
 
+	/* does not support taken branch sampling */
+	if (has_branch_stack(event))
+		return -EOPNOTSUPP;
+
 	switch (event->attr.type) {
 	case PERF_TYPE_RAW:
 	case PERF_TYPE_HW_CACHE:
arch/sparc/kernel/perf_event.c +4 -0
index 614da62..8e16a4a 100644
@@ -1105,6 +1105,10 @@ static int sparc_pmu_event_init(struct perf_event *event)
 	if (atomic_read(&nmi_active) < 0)
 		return -ENODEV;
 
+	/* does not support taken branch sampling */
+	if (has_branch_stack(event))
+		return -EOPNOTSUPP;
+
 	switch (attr->type) {
 	case PERF_TYPE_HARDWARE:
 		if (attr->config >= sparc_pmu->max_events)
arch/x86/kernel/cpu/perf_event_amd.c +3 -0
index aeefd45..9ef5749 100644
@@ -138,6 +138,9 @@ static int amd_pmu_hw_config(struct perf_event *event)
 	if (ret)
 		return ret;
 
+	if (has_branch_stack(event))
+		return -EOPNOTSUPP;
+
 	if (event->attr.exclude_host && event->attr.exclude_guest)
 		/*
 		 * When HO == GO == 1 the hardware treats that as GO == HO == 0
kernel/events/core.c +24 -0
index a4c3826..6d30498 100644
@@ -5105,6 +5105,12 @@ static int perf_swevent_init(struct perf_event *event)
 	if (event->attr.type != PERF_TYPE_SOFTWARE)
 		return -ENOENT;
 
+	/*
+	 * no branch sampling for software events
+	 */
+	if (has_branch_stack(event))
+		return -EOPNOTSUPP;
+
 	switch (event_id) {
 	case PERF_COUNT_SW_CPU_CLOCK:
 	case PERF_COUNT_SW_TASK_CLOCK:
@@ -5208,6 +5214,12 @@ static int perf_tp_event_init(struct perf_event *event)
 	if (event->attr.type != PERF_TYPE_TRACEPOINT)
 		return -ENOENT;
 
+	/*
+	 * no branch sampling for tracepoint events
+	 */
+	if (has_branch_stack(event))
+		return -EOPNOTSUPP;
+
 	err = perf_trace_init(event);
 	if (err)
 		return err;
@@ -5431,6 +5443,12 @@ static int cpu_clock_event_init(struct perf_event *event)
 	if (event->attr.config != PERF_COUNT_SW_CPU_CLOCK)
 		return -ENOENT;
 
+	/*
+	 * no branch sampling for software events
+	 */
+	if (has_branch_stack(event))
+		return -EOPNOTSUPP;
+
 	perf_swevent_init_hrtimer(event);
 
 	return 0;
@@ -5503,6 +5521,12 @@ static int task_clock_event_init(struct perf_event *event)
 	if (event->attr.config != PERF_COUNT_SW_TASK_CLOCK)
 		return -ENOENT;
 
+	/*
+	 * no branch sampling for software events
+	 */
+	if (has_branch_stack(event))
+		return -EOPNOTSUPP;
+
 	perf_swevent_init_hrtimer(event);
 
 	return 0;
kernel/events/hw_breakpoint.c +6 -0
index b7971d6..e7fb781 100644
@@ -581,6 +581,12 @@ static int hw_breakpoint_event_init(struct perf_event *bp)
 	if (bp->attr.type != PERF_TYPE_BREAKPOINT)
 		return -ENOENT;
 
+	/*
+	 * no branch sampling for breakpoint events
+	 */
+	if (has_branch_stack(bp))
+		return -EOPNOTSUPP;
+
 	err = register_perf_hw_breakpoint(bp);
 	if (err)
 		return err;
Signature · 1 line
1.7.1
#16 lore raw

PATCH 9/12 perf_events: add hook to flush branch_stack on context switch (v2)

In reply to Stephane Eranian

With branch stack sampling, it is possible to filter by priv levels. In system-wide mode, that means it is possible to capture only user level branches. The builtin SW LBR filter needs to disassemble code based on LBR captured addresses. For that, it needs to know the task the addresses are associated with. Because of context switches, the content of the branch stack buffer may contain addresses from different tasks.

We need a hook on context switch to either flush the branch stack or save it. This patch adds a new hook in struct pmu which is called during context switches. The hook is called only when necessary. That is when a system-wide context has, at least, one event which uses PERF_SAMPLE_BRANCH_STACK. The hook is never called for per-thread context.

In this version, the Intel X86 code simply flushes (reset) the LBR on context switches (fill with zeroes). Those zeroed branches are then filtered out by the SW filter.

---
 arch/x86/kernel/cpu/perf_event.c       |   29 +++++++----
 arch/x86/kernel/cpu/perf_event.h       |    1 +
 arch/x86/kernel/cpu/perf_event_intel.c |   13 +++++
 include/linux/perf_event.h             |    7 ++-
 kernel/events/core.c                   |   85 ++++++++++++++++++++++++++++++++
 5 files changed, 123 insertions(+), 12 deletions(-)
arch/x86/kernel/cpu/perf_event.c +18 -11
index e2efa90..b44aba8 100644
@@ -1436,21 +1436,28 @@ static int x86_pmu_event_init(struct perf_event *event)
 	return err;
 }
 
+static void x86_pmu_flush_branch_stack(void)
+{
+	if (x86_pmu.flush_branch_stack)
+		x86_pmu.flush_branch_stack();
+}
+
 static struct pmu pmu = {
-	.pmu_enable	= x86_pmu_enable,
-	.pmu_disable	= x86_pmu_disable,
+	.pmu_enable		= x86_pmu_enable,
+	.pmu_disable		= x86_pmu_disable,
 
-	.event_init	= x86_pmu_event_init,
+	.event_init		= x86_pmu_event_init,
 
-	.add		= x86_pmu_add,
-	.del		= x86_pmu_del,
-	.start		= x86_pmu_start,
-	.stop		= x86_pmu_stop,
-	.read		= x86_pmu_read,
+	.add			= x86_pmu_add,
+	.del			= x86_pmu_del,
+	.start			= x86_pmu_start,
+	.stop			= x86_pmu_stop,
+	.read			= x86_pmu_read,
 
-	.start_txn	= x86_pmu_start_txn,
-	.cancel_txn	= x86_pmu_cancel_txn,
-	.commit_txn	= x86_pmu_commit_txn,
+	.start_txn		= x86_pmu_start_txn,
+	.cancel_txn		= x86_pmu_cancel_txn,
+	.commit_txn		= x86_pmu_commit_txn,
+	.flush_branch_stack	= x86_pmu_flush_branch_stack,
 };
 
 /*
arch/x86/kernel/cpu/perf_event.h +1 -0
index 48ed504..5ba6a7b 100644
@@ -278,6 +278,7 @@ struct x86_pmu {
 	void		(*cpu_starting)(int cpu);
 	void		(*cpu_dying)(int cpu);
 	void		(*cpu_dead)(int cpu);
+	void		(*flush_branch_stack)(void);
 
 	/*
 	 * Intel Arch Perfmon v2+
arch/x86/kernel/cpu/perf_event_intel.c +13 -0
index 901217d..9cc8a17 100644
@@ -1542,6 +1542,18 @@ static void intel_pmu_cpu_dying(int cpu)
 	fini_debug_store_on_cpu(cpu);
 }
 
+static void intel_pmu_flush_branch_stack(void)
+{
+	/*
+	 * Intel LBR does not tag entries with the
+	 * PID of the current task, then we need to
+	 * flush it on ctxsw
+	 * For now, we simply reset it
+	 */
+	if (x86_pmu.lbr_nr)
+		intel_pmu_lbr_reset();
+}
+
 static __initconst const struct x86_pmu intel_pmu = {
 	.name			= "Intel",
 	.handle_irq		= intel_pmu_handle_irq,
@@ -1569,6 +1581,7 @@ static __initconst const struct x86_pmu intel_pmu = {
 	.cpu_starting		= intel_pmu_cpu_starting,
 	.cpu_dying		= intel_pmu_cpu_dying,
 	.guest_get_msrs		= intel_guest_get_msrs,
+	.flush_branch_stack	= intel_pmu_flush_branch_stack,
 };
 
 static void intel_clovertown_quirks(void)
include/linux/perf_event.h +6 -1
index c4fbe84..83ddb52 100644
@@ -733,6 +733,9 @@ struct pmu {
 	 * for each successful ->add() during the transaction.
 	 */
 	void (*cancel_txn)		(struct pmu *pmu); /* optional */
+	/*
+	 * flush branch stack on context-switches (needed in cpu-wide mode) */
+	void (*flush_branch_stack)	(void);
 };
 
 /**
@@ -961,7 +964,8 @@ struct perf_event_context {
 	u64				parent_gen;
 	u64				generation;
 	int				pin_count;
-	int				nr_cgroups; /* cgroup events present */
+	int				nr_cgroups;	 /* cgroup evts */
+	int				nr_branch_stack; /* branch_stack evt */
 	struct rcu_head			rcu_head;
 };
 
@@ -1026,6 +1030,7 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr,
 extern u64 perf_event_read_value(struct perf_event *event,
 				 u64 *enabled, u64 *running);
 
+
 struct perf_sample_data {
 	u64				type;
 
kernel/events/core.c +85 -0

127 lines not on this page · raw message

Signature · 1 line
1.7.1
maintainer
#17 lore raw

Re: [PATCH 09/12] perf_events: add hook to flush branch_stack on context switch (v2)

Quoted 10 lines · Stephane Eranian

+ * When sampling the branck stack in system-wide, it may be necessary
+ * to flush the stack on context switch. This happens when the branch
+ * stack does not tag its entries with the pid of the current task.
+ * Otherwise it becomes impossible to associate a branch entry with a
+ * task. This ambiguity is more likely to appear when the branch stack
+ * supports priv level filtering and the user sets it to monitor only
+ * at the user level (which could be a useful measurement in system-wide
+ * mode). In that case, the risk is high of having a branch stack with
+ * branch from multiple tasks. Flushing may mean dropping the existing
+ * entries or stashing them somewhere in the PMU specific code layer.

It doesn't need to tag stuff with PID to solve that problem, making the TOS a full 64bit wide counter will work equally well, we'd simply record the TOS value at context switch time and discard everything prior to the last switch-in.

But yeah, we need to flush this stuff under the current scheme.

maintainer
#18 lore raw

Re: [PATCH 09/12] perf_events: add hook to flush branch_stack on context switch (v2)

In reply to Stephane Eranian
Quoted 10 lines · Stephane Eranian
kernel/events/core.c +10 -0
+               /*
+                * check if the context has at least one
+                * event using PERF_SAMPLE_BRANCH_STACK
+                */
+               if (cpuctx->ctx.nr_branch_stack > 0
+                   && pmu->flush_branch_stack) {
+
+                       pmu = cpuctx->ctx.pmu;
+
+                       perf_ctx_lock(cpuctx, cpuctx->task_ctx);

kernel/events/core.c static void perf_event_context_sched_in(struct perf_event_context *ctx, · line 2248

Quoted 10 lines
kernel/events/core.c +10 -0
+
+                       perf_pmu_disable(pmu);
+
+                       pmu->flush_branch_stack();
+
+                       perf_pmu_enable(pmu);
+
+                       perf_ctx_unlock(cpuctx, cpuctx->task_ctx);
+               }
+       } 

(what whitespace looks funny)

So all PMUs not supporting this branch stuff will fail to create a has_branch_stack() event, right? Thus all ctx with !0 nr_branch_stack support it. Doesn't this make the test for pmu->flush_branch_stack redundant?

#19 lore raw

Re: [PATCH 09/12] perf_events: add hook to flush branch_stack on context switch (v2)

Quoted 24 lines · Peter Zijlstra
Quoted 10 lines · Stephane Eranian

+               /*
+                * check if the context has at least one
+                * event using PERF_SAMPLE_BRANCH_STACK
+                */
+               if (cpuctx->ctx.nr_branch_stack > 0
+                   && pmu->flush_branch_stack) {

kernel/events/core.c +4 -0
+
+                       pmu = cpuctx->ctx.pmu;
+
+                       perf_ctx_lock(cpuctx, cpuctx->task_ctx);

kernel/events/core.c static void perf_event_context_sched_in(struct perf_event_context *ctx, · line 2248

Quoted 10 lines
kernel/events/core.c +10 -0
+
+                       perf_pmu_disable(pmu);
+
+                       pmu->flush_branch_stack();
+
+                       perf_pmu_enable(pmu);
+
+                       perf_ctx_unlock(cpuctx, cpuctx->task_ctx);
+               }
+       }

(what whitespace looks funny)

Quoted 6 lines

So all PMUs not supporting this branch stuff will fail to create a has_branch_stack() event, right? Thus all ctx with !0 nr_branch_stack support it. Doesn't this make the test for pmu->flush_branch_stack redundant?

No, nr_branch_stack counts the number of active events with branch_stack. It's like the ctx->nr_cgroups. Processors which do not support branch_stack will always have this field to 0. It's not because a processor supports branch_stack that we need to call flush_branch_stack(), i.e., we use a lazy approach.

maintainer
#20 lore raw

Re: [PATCH 09/12] perf_events: add hook to flush branch_stack on context switch (v2)

Quoted 31 lines · Stephane Eranian
Quoted 24 lines · Peter Zijlstra
Quoted 10 lines · Stephane Eranian
kernel/events/core.c +10 -0
+               /*
+                * check if the context has at least one
+                * event using PERF_SAMPLE_BRANCH_STACK
+                */
+               if (cpuctx->ctx.nr_branch_stack > 0
+                   && pmu->flush_branch_stack) {
+
+                       pmu = cpuctx->ctx.pmu;
+
+                       perf_ctx_lock(cpuctx, cpuctx->task_ctx);

kernel/events/core.c static void perf_event_context_sched_in(struct perf_event_context *ctx, · line 2248

Quoted 10 lines
kernel/events/core.c +10 -0
+
+                       perf_pmu_disable(pmu);
+
+                       pmu->flush_branch_stack();
+
+                       perf_pmu_enable(pmu);
+
+                       perf_ctx_unlock(cpuctx, cpuctx->task_ctx);
+               }
+       }

(what whitespace looks funny)

Quoted 6 lines

So all PMUs not supporting this branch stuff will fail to create a has_branch_stack() event, right? Thus all ctx with !0 nr_branch_stack support it. Doesn't this make the test for pmu->flush_branch_stack redundant?

Quoted 5 lines

No, nr_branch_stack counts the number of active events with branch_stack. It's like the ctx->nr_cgroups. Processors which do not support branch_stack will always have this field to 0. It's not because a processor supports branch_stack that we need to call flush_branch_stack(), i.e., we use a lazy approach.

What you're saying is we can support branch stack and not need flush_branch_stack()? Say in the case the x86 LBR TOS field would be a full u64 counter, since then we could sample the TOS on context switch and filter on that, obviating the hard reset we do now.

And the advantage of testing for the operation as opposed to putting in a dummy function (like we do for most other optional methods) is avoiding all that ctx_lock and pmu_disable muck.

Fair enough.

#21 lore raw

Re: [PATCH 09/12] perf_events: add hook to flush branch_stack on context switch (v2)

Quoted 38 lines · Peter Zijlstra
Quoted 31 lines · Stephane Eranian
Quoted 24 lines · Peter Zijlstra
Quoted 10 lines · Stephane Eranian

+               /*
+                * check if the context has at least one
+                * event using PERF_SAMPLE_BRANCH_STACK
+                */
+               if (cpuctx->ctx.nr_branch_stack > 0
+                   && pmu->flush_branch_stack) {

kernel/events/core.c +4 -0
+
+                       pmu = cpuctx->ctx.pmu;
+
+                       perf_ctx_lock(cpuctx, cpuctx->task_ctx);

kernel/events/core.c static void perf_event_context_sched_in(struct perf_event_context *ctx, · line 2248

Quoted 10 lines
kernel/events/core.c +10 -0
+
+                       perf_pmu_disable(pmu);
+
+                       pmu->flush_branch_stack();
+
+                       perf_pmu_enable(pmu);
+
+                       perf_ctx_unlock(cpuctx, cpuctx->task_ctx);
+               }
+       }

(what whitespace looks funny)

Quoted 6 lines

So all PMUs not supporting this branch stuff will fail to create a has_branch_stack() event, right? Thus all ctx with !0 nr_branch_stack support it. Doesn't this make the test for pmu->flush_branch_stack redundant?

Quoted 5 lines

No, nr_branch_stack counts the number of active events with branch_stack. It's like the ctx->nr_cgroups. Processors which do not support branch_stack will always have this field to 0. It's not because a processor supports branch_stack that we need to call flush_branch_stack(), i.e., we use a lazy approach.

Quoted 5 lines

What you're saying is we can support branch stack and not need flush_branch_stack()? Say in the case the x86 LBR TOS field would be a full u64 counter, since then we could sample the TOS on context switch and filter on that, obviating the hard reset we do now.

The whole motivation behind the flush_branch_stack is explained in the Changelog of the patch. In summary, we need to flush the LBR (regardless of TOS) because in system-wide we need to be able to associate the content of the LBR with a specific task. Given that the HW does not capture the PID in the LBR buffer, the kernel has to intervene. Why don't we have this already? Because we are capturing at all priv levels. But with this patchset, it becomes possible to filter taken branches based on priv levels. Thus, if you only sample at the user level and run in system-wide mode, it is more likely you could end up with branches belonging to two different tasks in the LBR buffer. But you'd have no way of determining this just by looking at the content of the buffer. So instead, we need to flush the LBR on context switch to associate a PID with them.

Because this is an expensive operation, we want to do this only when we sample on LBR. That's what the ctx->nr_branch_stack is about. We could refine that some more by checking for system-wide events with only user priv level on the branch stack. But I did not do that yet.

Does this make more sense now?

Quoted 5 lines

And the advantage of testing for the operation as opposed to putting in a dummy function (like we do for most other optional methods) is avoiding all that ctx_lock and pmu_disable muck.

Fair enough.

maintainer
#22 lore raw

Re: [PATCH 09/12] perf_events: add hook to flush branch_stack on context switch (v2)

Quoted 5 lines · Stephane Eranian

The whole motivation behind the flush_branch_stack is explained in the Changelog of the patch. In summary, we need to flush the LBR (regardless of TOS) because in system-wide we need to be able to associate the content of the LBR with a specific task. Given that the HW does not capture the PID in the LBR buffer, the kernel has to intervene.

That's not regardless of the TOS. If the TOS was a full u64 you wouldn't need the TID (which would be good, since the hardware has no such concept).

Quoted 8 lines

Why don't we have this already?
Because we are capturing at all priv levels. But with this patchset, it becomes possible to filter taken branches based on priv levels. Thus, if you only sample at the user level and run in system-wide mode, it is more likely you could end up with branches belonging to two different tasks in the LBR buffer. But you'd have no way of determining this just by looking at the content of the buffer. So instead, we need to flush the LBR on context switch to associate a PID with them.

Yeah, I get that.

Quoted 6 lines

Because this is an expensive operation, we want to do this only when we sample on LBR. That's what the ctx->nr_branch_stack is about. We could refine that some more by checking for system-wide events with only user priv level on the branch stack. But I did not do that yet.

Does this make more sense now?

It already did. The only thing I wanted to do was get rid of that method check. Initially I overlooked the fact that its optional, even if you support the branch stack. My reply from today argued for it, since installing a dummy method would still have the needless ctx_lock && pmu_disable overhead.

#23 lore raw

Re: [PATCH 09/12] perf_events: add hook to flush branch_stack on context switch (v2)

Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Stephane Eranian

The whole motivation behind the flush_branch_stack is explained in the Changelog of the patch. In summary, we need to flush the LBR (regardless of TOS) because in system-wide we need to be able to associate the content of the LBR with a specific task. Given that the HW does not capture the PID in the LBR buffer, the kernel has to intervene.

That's not regardless of the TOS. If the TOS was a full u64 you wouldn't need the TID (which would be good, since the hardware has no such concept).

Maybe I missed the trick but I don't quite see how a 64-bit TOS would solve the TID
problem. It's not about the wraparound issue, i.e., not like the sampling buffer indexes.
Could you describe the trick again?

Quoted 25 lines
Quoted 8 lines

Why don't we have this already?
Because we are capturing at all priv levels. But with this patchset, it becomes possible to filter taken branches based on priv levels. Thus, if you only sample at the user level and run in system-wide mode, it is more likely you could end up with branches belonging to two different tasks in the LBR buffer. But you'd have no way of determining this just by looking at the content of the buffer. So instead, we need to flush the LBR on context switch to associate a PID with them.

Yeah, I get that.

Quoted 6 lines

Because this is an expensive operation, we want to do this only when we sample on LBR. That's what the ctx->nr_branch_stack is about. We could refine that some more by checking for system-wide events with only user priv level on the branch stack. But I did not do that yet.

Does this make more sense now?

It already did. The only thing I wanted to do was get rid of that method check. Initially I overlooked the fact that its optional, even if you support the branch stack. My reply from today argued for it, since installing a dummy method would still have the needless ctx_lock && pmu_disable overhead.

maintainer
#24 lore raw

Re: [PATCH 09/12] perf_events: add hook to flush branch_stack on context switch (v2)

Quoted 9 lines · Stephane Eranian
Quoted 4 lines

That's not regardless of the TOS. If the TOS was a full u64 you wouldn't need the TID (which would be good, since the hardware has no such concept).

Maybe I missed the trick but I don't quite see how a 64-bit TOS would solve the TID
problem. It's not about the wraparound issue, i.e., not like the sampling buffer indexes.
Could you describe the trick again?

LBR 0
.
.
. <-- TOS % n
.
LBR n-1

So the LBR is an array of n entries which is written to in a cyclic fashion. The Top-Of-Stack or TOS indicates the last written entry and we can read n entries backwards from there.

Something like:

tos = rdmsr(lbr_tos);

  for (i = 0; i < n; i++) {
	idx = (tos - i) % n;

	from = rdmsr(lbr_from + idx);
	to   = rdmsr(lbr_to   + idx);

}

Now the hardware keeps (TOS % n) by limiting the bits in the counter (n = 2^m etc..), if it wouldn't do that, we could sample the TOS on ctxsw and modify the read to:

tos = rdmsr(lbr_tos)

  for (i = 0; i < n && (tos - i) > ctxsw_tos; i++) {
	idx = (tos - i) % n;

	...

}

This would ensure we never read back past the context-switch. But we need the extra bits for this to work, since with the current limited (TOS % n) bits we get into trouble as soon as the ctxsw was more than n branches ago (which is something very likely).

[ With 16 bits we'd get into trouble when the ctxsw was 65536 branch ago, which is still quite possible, at 32 bits we'd need 4G branches, which is rather unlikely, with 64 bits the sun will have died or so.. ]

Also note we can make this extra condition conditional on the event being a task event, so that the cpu events always consume all n entries.

#25 lore raw

PATCH 10/12 perf: add code to support PERF_SAMPLE_BRANCH_STACK (v2)

In reply to Stephane Eranian

From: Roberto Agostino Vitillo <[email protected]>

This patch adds:
- ability to parse samples with PERF_SAMPLE_BRANCH_STACK
- sort on branches
- build histograms on branches

---
 tools/perf/perf.h          |   17 ++
 tools/perf/util/annotate.c |    2 +-
 tools/perf/util/event.h    |    1 +
 tools/perf/util/evsel.c    |   10 ++
 tools/perf/util/hist.c     |   92 +++++++++---
 tools/perf/util/hist.h     |    6 +
 tools/perf/util/session.c  |   72 +++++++++
 tools/perf/util/session.h  |    5 +
 tools/perf/util/sort.c     |  348 +++++++++++++++++++++++++++++++++-----------
 tools/perf/util/sort.h     |    5 +
 tools/perf/util/symbol.h   |   13 ++
 11 files changed, 462 insertions(+), 109 deletions(-)
tools/perf/perf.h +17 -0
index 08b0b5e..a3177e7 100644
@@ -180,6 +180,23 @@ struct ip_callchain {
 	u64 ips[0];
 };
 
+struct branch_flags{
+	u64 mispred : 1;
+	u64 predicted : 1;
+	u64 reserved : 62;
+};
+
+struct branch_entry {
+	u64				from;
+	u64				to;
+	struct branch_flags flags;
+};
+
+struct branch_stack {
+	u64				nr;
+	struct branch_entry	entries[0];
+};
+
 extern bool perf_host, perf_guest;
 extern const char perf_version_string[];
 
tools/perf/util/annotate.c +1 -1
index bc8f477..f071d29 100644
@@ -64,7 +64,7 @@ int symbol__inc_addr_samples(struct symbol *sym, struct map *map,
 
 	pr_debug3("%s: addr=%#" PRIx64 "\n", __func__, map->unmap_ip(map, addr));
 
-	if (addr >= sym->end)
+	if (addr >= sym->end || addr < sym->start)
 		return 0;
 
 	offset = addr - sym->start;
tools/perf/util/event.h +1 -0
index 357a85b..026b1f6 100644
@@ -80,6 +80,7 @@ struct perf_sample {
 	u32 raw_size;
 	void *raw_data;
 	struct ip_callchain *callchain;
+	struct branch_stack *branch_stack;
 };
 
 #define BUILD_ID_SIZE 20
tools/perf/util/evsel.c +10 -0
index b46f6e4..73550ec 100644
@@ -473,5 +473,15 @@ int perf_event__parse_sample(const union perf_event *event, u64 type,
 		data->raw_data = (void *) pdata;
 	}
 
+	if (type & PERF_SAMPLE_BRANCH_STACK) {
+		u64 sz;
+
+		data->branch_stack = (struct branch_stack *)array;
+		array++; /* nr */
+
+		sz = data->branch_stack->nr * sizeof (struct branch_entry);
+		sz /= sizeof(uint64_t);
+		array += sz;
+	}
 	return 0;
 }
tools/perf/util/hist.c +72 -20

127 lines not on this page · raw message

tools/perf/util/hist.h +6 -0
index 7ea1e56..395b2e7 100644
@@ -40,6 +40,7 @@ enum hist_column {
 	HISTC_COMM,
 	HISTC_PARENT,
 	HISTC_CPU,
+	HISTC_MISPREDICT,
 	HISTC_NR_COLS, /* Last entry */
 };
 
@@ -62,6 +63,11 @@ void hists__init(struct hists *hists);
 struct hist_entry *__hists__add_entry(struct hists *self,
 				      struct addr_location *al,
 				      struct symbol *parent, u64 period);
+struct hist_entry *__hists__add_branch_entry(struct hists *self,
+					     struct addr_location *al,
+					     struct symbol *sym_parent,
+					     struct branch_info* bi,
+					     u64 period);
 extern int64_t hist_entry__cmp(struct hist_entry *, struct hist_entry *);
 extern int64_t hist_entry__collapse(struct hist_entry *, struct hist_entry *);
 int hist_entry__fprintf(struct hist_entry *he, size_t size, struct hists *hists,
tools/perf/util/session.c +72 -0

90 lines not on this page · raw message

tools/perf/util/session.h +5 -0
index 514b06d..44a957b 100644
@@ -100,6 +100,11 @@ int __perf_session__process_events(struct perf_session *self,
 int perf_session__process_events(struct perf_session *self,
 				 struct perf_event_ops *event_ops);
 
+
+struct branch_info *perf_session__resolve_bstack(struct perf_session *self,
+						 struct thread *thread,
+						 struct branch_stack *bs);
+
 int perf_session__resolve_callchain(struct perf_session *self,
 				    struct thread *thread,
 				    struct ip_callchain *chain,
tools/perf/util/sort.c +260 -88

422 lines not on this page · raw message

tools/perf/util/sort.h +5 -0
index 03851e3..69fc954 100644
@@ -31,11 +31,14 @@ extern const char *parent_pattern;
 extern const char default_sort_order[];
 extern int sort__need_collapse;
 extern int sort__has_parent;
+extern bool sort__branch_mode;
 extern char *field_sep;
 extern struct sort_entry sort_comm;
 extern struct sort_entry sort_dso;
 extern struct sort_entry sort_sym;
 extern struct sort_entry sort_parent;
+extern struct sort_entry sort_lbr_dso;
+extern struct sort_entry sort_lbr_sym;
 extern enum sort_type sort__first_dimension;
 
 /**
@@ -71,6 +74,7 @@ struct hist_entry {
 		struct hist_entry *pair;
 		struct rb_root	  sorted_chain;
 	};
+	struct branch_info	*branch_info;
 	struct callchain_root	callchain[0];
 };
 
@@ -81,6 +85,7 @@ enum sort_type {
 	SORT_SYM,
 	SORT_PARENT,
 	SORT_CPU,
+	SORT_MISPREDICTED,
 };
 
 /*
tools/perf/util/symbol.h +13 -0
index 29f8d74..a5c84d1 100644
@@ -5,6 +5,7 @@
 #include <stdbool.h>
 #include <stdint.h>
 #include "map.h"
+#include "../perf.h"
 #include <linux/list.h>
 #include <linux/rbtree.h>
 #include <stdio.h>
@@ -118,6 +119,18 @@ struct map_symbol {
 	bool	      has_children;
 };
 
+struct addr_map_symbol {
+	struct map    *map;
+	struct symbol *sym;
+	u64	      addr;
+};
+
+struct branch_info {
+	struct addr_map_symbol from;
+	struct addr_map_symbol to;
+	struct branch_flags flags;
+};
+
 struct addr_location {
 	struct thread *thread;
 	struct map    *map;
Signature · 1 line
1.7.1
#26 lore raw

PATCH 11/12 perf: add support for sampling taken branch to perf record (v2)

In reply to Stephane Eranian

From: Roberto Agostino Vitillo <[email protected]>

This patch adds a new option to enable taken branch stack sampling, i.e., leverage the PERF_SAMPLE_BRANCH_STACK feature of perf_events.

There is a new option to active this mode: -b.
It is possible to pass a set of filters to select the type of branches to sample.

The following filters are available:
- any : any type of branches
- any_call : any function call or system call
- any_ret : any function return or system call return
- any_ind : any indirect branch
- u: only when the branch target is at the user level
- k: only when the branch target is in the kernel

Filters can be combined by passing a comma separated list to the option:

$ perf record -b any_call,u -e cycles:u branchy
---
 tools/perf/Documentation/perf-record.txt |   18 +++++++
 tools/perf/builtin-record.c              |   75 ++++++++++++++++++++++++++++++
 2 files changed, 93 insertions(+), 0 deletions(-)
tools/perf/Documentation/perf-record.txt +18 -0
index 5a520f8..ddc1999 100644
@@ -148,6 +148,24 @@ an empty cgroup (monitor all the time) using, e.g., -G foo,,bar. Cgroups must ha
 corresponding events, i.e., they always refer to events defined earlier on the command
 line.
 
+-b::
+--branch-stack::
+Enable taken branch stack sampling. Each sample captures a series of consecutive
+taken branches. The number of branches captured with each sample depends on the
+underlying hardware, the type of branches of interested and the executed code.
+It is possible to filter the types of branches by enabling filters. The
+following filters are defined: any (any type of branches), any_call (any function
+call or system call), any_ret (any function return or system call return), any_ind
+(any indirect branch), u (only when the branch target is at the user level), k (only when
+the branch target is in the kernel). At least one of any, any_call, any_ret, any_ind
+must be provided. The privilege levels may be ommitted, in which case, the privilege
+levels of the associated event is applied to the branch filter. When sampling on multiple
+events, branch stack sampling is enabled for all the sampling events. The sampled branch
+type is the same for all events. The privilege levels are adjusted based on those of
+the associated event unless specified explicitly with this option. Note that taken
+branch sampling may not be available on all processors. The various filters must
+be specified as a comma separated list: -b any_ret,u,k
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/builtin-record.c +75 -0

99 lines not on this page · raw message

Signature · 1 line
1.7.1
#27 lore raw

PATCH 12/12 perf: add support for taken branch sampling to perf report (v2)

In reply to Stephane Eranian

From: Roberto Agostino Vitillo <[email protected]>

This patch adds support for taken branch sampling, i.e, the PERF_SAMPLE_BRANCH_STACK feature to perf report. In other words, to display histograms based on taken branches rather than executed instructions addresses.

The new option is called -b and it takes no argument. To generate meaningful output, the perf.data must have been obtained using perf record -b xxx ... where xxx is a branch filter option.

The output shows symbols, modules, sorted by 'who branches where' the most often. The percentages reported in the first column refer to the total number of branches captured and not the usual number of samples.

Here is a quick example.
Here branchy is simple test program which looks as follows:

void f2(void)
{}
void f3(void)
{}
void f1(unsigned long n)
{
if (n & 1UL)

    f2();
  else
    f3();
}

int main(void)
{
unsigned long i;

for (i=0; i < N; i++)
f1(i);
return 0;
}

Here is the output captured on Nehalem, if we are
only interested in user level function calls.

$ perf record -b any_call,u -e cycles:u branchy
$ perf report -b --sort=symbol
    52.34%  [.] main                   [.] f1
    24.04%  [.] f1                     [.] f3
    23.60%  [.] f1                     [.] f2
     0.01%  [k] _IO_new_file_xsputn    [k] _IO_file_overflow
     0.01%  [k] _IO_vfprintf_internal  [k] _IO_new_file_xsputn
     0.01%  [k] _IO_vfprintf_internal  [k] strchrnul
     0.01%  [k] __printf               [k] _IO_vfprintf_internal
     0.01%  [k] main                   [k] __printf

About half (52%) of the call branches captured are from main() -> f1(). The second half (24%+23%) is split in two equal shares between f1() -> f2(), f1() ->f3(). The output is as expected given the code.

It should be noted, that using -b in perf record does not eliminate information in the perf.data file. Consequently, a typical profile can also be obtained by perf report by simply not using its -b option.

---
 tools/perf/Documentation/perf-report.txt |    7 ++
 tools/perf/builtin-report.c              |   93 +++++++++++++++++++++++++++---
 2 files changed, 91 insertions(+), 9 deletions(-)
tools/perf/Documentation/perf-report.txt +7 -0
index 212f24d..3163be5 100644
@@ -152,6 +152,13 @@ OPTIONS
 	information which may be very large and thus may clutter the display.
 	It currently includes: cpu and numa topology of the host system.
 
+-b::
+--branch-stack::
+	Use the addresses of sampled taken branches instead of the instruction
+	address to build the histograms. To generate meaningful output, the
+	perf.data file must have been obtained using perf record -b xxx where
+	xxx is a branch filter option.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-annotate[1]
tools/perf/builtin-report.c +84 -9

128 lines not on this page · raw message

Signature · 1 line
1.7.1
#28 lore raw
In reply to Stephane Eranian

Any update on this patchset?

Quoted 206 lines · Stephane Eranian

This patchset adds an important and useful new feature to
perf_events: branch stack sampling. In other words, the ability to capture taken branches into each sample.

Statistical sampling of taken branch should not be confused for branch tracing. Not all branches are necessarily captured

Sampling taken branches is important for basic block profiling, statistical call graph, function call counts. Many of those measurements can help drive a compiler optimizer.

The branch stack is a software abstraction which sits on top of the PMU hardware. As such, it is not available on all processors. For now, the patch provides the generic interface and the Intel X86 implementation where it leverages the Last Branch Record (LBR) feature (from Core2 to SandyBridge).

Branch stack sampling is supported for both per-thread and system-wide modes.

It is possible to filter the type and privilege level of branches to sample. The target of the branch is used to determine the privilege level.

For each branch, the source and destination are captured. On some hardware platforms, it may be possible to also extract the target prediction and, in that case, it is also exposed to end users.

The branch stack can record a variable number of taken branches per sample. Those branches are always consecutive in time. The number of branches captured depends on the filtering and the underlying hardware. On Intel Nehalem and later, up to 16 consecutive branches can be captured per sample.

Branch sampling is always coupled with an event. It can be any PMU event but it can't be a SW or tracepoint event.

Branch sampling is requested by setting a new sample_type flag called: PERF_SAMPLE_BRANCH_STACK.

To support branch filtering, we introduce a new field to the perf_event_attr struct: branch_sample_type. We chose NOT to overload the config1, config2 field because those are related to the event encoding. Branch stack is a separate feature which is combined with the event.

The branch_sample_type is a bitmask of possible filters.
The following filters are defined (more can be added):
- PERF_SAMPLE_BRANCH_ANY     : any control flow change
- PERF_SAMPLE_BRANCH_USER    : capture branches when target is at user level
- PERF_SAMPLE_BRANCH_KERNEL  : capture branches when target is at user level
- PERF_SAMPLE_BRANCH_ANY_CALL: capture call branches (incl. syscalls)
- PERF_SAMPLE_BRANCH_ANY_RET : capture return branches (incl. syscall returns)
- PERF_SAMPLE_BRANCH_IND_CALL: capture indirect calls

It is possible to combine filters, e.g., IND_CALL|USER|KERNEL.

When the privilege level is not specified, the branch stack inherits that of the associated event.

Some processors may not offer hardware branch filtering, e.g., Intel Atom. Some may have HW filtering bugs (e.g., Nehalem). The Intel X86 implementation in this patchset also provides a SW branch filter which works on a best effort basis. It can compensate for the lack of LBR filtering. But first and foremost, it helps work around LBR filtering errata. The goal is to only capture the type of branches requested by the user.

It is possible to combine branch stack sampling with PEBS on Intel X86 processors. Depending on the precise_sampling mode, there are certain filterting restrictions. When precise_sampling=1, then there are no filtering restrictions. When precise_sampling > 1, then only ANY|USER|KERNEL filter can be used. This comes from the fact that the kernel uses LBR to compensate for the PEBS off-by-1 skid on the instruction pointer.

To demonstrate how the perf_event branch stack sampling interface works, the patchset also modifies perf record to capture taken branches. Similarly perf report is enhanced to display a histogram of taken branches.

I would like to thank Roberto Vitillo @ LBL for his work on the perf tool for this.

Enough talking, let's take a simple example. Our trivial test program goes like this:

void f2(void)
{}
void f3(void)
{}
void f1(unsigned long n)
{
 if (n & 1UL)
   f2();
 else
   f3();
}
int main(void)
{
 unsigned long i;

 for (i=0; i < N; i++)
  f1(i);
 return 0;
}

$ perf record -b any branchy
$ perf report -b
# Events: 23K cycles
#
# Overhead  Source Symbol     Target Symbol
# ........  ................  ................

   18.13%  [.] f1            [.] main
   18.10%  [.] main          [.] main
   18.01%  [.] main          [.] f1
   15.69%  [.] f1            [.] f1
    9.11%  [.] f3            [.] f1
    6.78%  [.] f1            [.] f3
    6.74%  [.] f1            [.] f2
    6.71%  [.] f2            [.] f1

Of the total number of branches captured, 18.13% were from f1() -> main().

Let's make this clearer by filtering the user call branches only:

$ perf record -b any_call -e cycles:u branchy
$ perf report
# Events: 19K cycles
#
# Overhead  Source Symbol              Target Symbol
# ........  .........................  .........................
#
   52.50%  [.] main                   [.] f1
   23.99%  [.] f1                     [.] f3
   23.48%  [.] f1                     [.] f2
    0.03%  [.] _IO_default_xsputn     [.] _IO_new_file_overflow
    0.01%  [k] _start                 [k] __libc_start_main

Now it is more obvious. %52 of all the captured branches where calls from main() -> f1(). The rest is split 50/50 between f1() -> f2() and f1() -> f3() which is expected given that f1() dispatches based on odd vs. even values of n which is constantly increasing.

In version 2, we update the patch to tip/master (commit 5734857) and we've incoporated the feedback from v1 concerning anynous bitfield struct for branch_stack_entry and the hanlding of i386 ABI binaries on 64-bit host in the instr decoder for the LBR SW filter.

Roberto Agostino Vitillo (2):
 perf: add support for sampling taken branch to perf record
 perf: add support for taken branch sampling to perf report

Stephane Eranian (10):
 perf_events: add generic taken branch sampling support
 perf_events: add Intel LBR MSR definitions
 perf_events: add Intel X86 LBR sharing logic
 perf_events: sync branch stack sampling with X86 precise_sampling
 perf_events: add LBR mappings for PERF_SAMPLE_BRANCH filters
 perf_events: implement PERF_SAMPLE_BRANCH for Intel X86
 perf_events: add LBR software filter support for Intel X86
 perf_events: disable PERF_SAMPLE_BRANCH_* when not supported
 perf_events: add hook to flush branch_stack on context switch
 perf: add code to support PERF_SAMPLE_BRANCH_STACK

 arch/alpha/kernel/perf_event.c             |    4 + arch/arm/kernel/perf_event.c               |    4 + arch/mips/kernel/perf_event.c              |    4 + arch/powerpc/kernel/perf_event.c           |    4 + arch/sh/kernel/perf_event.c                |    4 + arch/sparc/kernel/perf_event.c             |    4 + arch/x86/include/asm/msr-index.h           |    7 + arch/x86/kernel/cpu/perf_event.c           |   62 +++- arch/x86/kernel/cpu/perf_event_amd.c       |    3 + arch/x86/kernel/cpu/perf_event_intel.c     |  126 +++++-- arch/x86/kernel/cpu/perf_event_intel_ds.c  |   21 +- arch/x86/kernel/cpu/perf_event_intel_lbr.c |  529 ++++++++++++++++++++++++++-- include/linux/perf_event.h                 |   74 ++++- kernel/events/core.c                       |  167 +++++++++ kernel/events/hw_breakpoint.c              |    6 + tools/perf/Documentation/perf-record.txt   |   18 + tools/perf/Documentation/perf-report.txt   |    7 + tools/perf/builtin-record.c                |   75 ++++ tools/perf/builtin-report.c                |   93 +++++- tools/perf/perf.h                          |   17 + tools/perf/util/annotate.c                 |    2 +-

 tools/perf/util/event.h                    |    1 +
 tools/perf/util/evsel.c                    |   10 +
 tools/perf/util/hist.c                     |   97 ++++--
 tools/perf/util/hist.h                     |    6 +

 tools/perf/util/session.c                  |   72 ++++ tools/perf/util/session.h                  |    5 + tools/perf/util/sort.c                     |  348 ++++++++++++++----- tools/perf/util/sort.h                     |    5 + tools/perf/util/symbol.h                   |   13 +

 30 files changed, 1584 insertions(+), 204 deletions(-)

--
1.7.4.1

maintainer
#29 lore raw
Quoted 1 line · Stephane Eranian

Any update on this patchset?

Completely slipped through the cracks in my brain :-) Lemme go have a look.

maintainer
#30 lore raw
In reply to Stephane Eranian
Quoted 4 lines · Stephane Eranian

This patchset adds an important and useful new feature to
perf_events: branch stack sampling. In other words, the ability to capture taken branches into each sample.

Other than the few comments given it all looks good. My main worry is the Intel only aspect, I'd really love for there to be another platform that could implement at least part of this.

#31 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian

This patchset adds an important and useful new feature to
perf_events: branch stack sampling. In other words, the ability to capture taken branches into each sample.

Other than the few comments given it all looks good. My main worry is the Intel only aspect, I'd really love for there to be another platform that could implement at least part of this.

I discussed this with Stephane in Prague and, although it would be lovely to have this on ARM, we simply don't have the hardware to do it. So the nature of series does seem to be x86-centric unless there's way to do a watered down version in software.

Will

maintainer
#32 lore raw
Quoted 6 lines · Will Deacon
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian

This patchset adds an important and useful new feature to
perf_events: branch stack sampling. In other words, the ability to capture taken branches into each sample.

Quoted 8 lines
Quoted 3 lines

Other than the few comments given it all looks good. My main worry is the Intel only aspect, I'd really love for there to be another platform that could implement at least part of this.

I discussed this with Stephane in Prague and, although it would be lovely to have this on ARM, we simply don't have the hardware to do it. So the nature of series does seem to be x86-centric unless there's way to do a watered down version in software.

The only way to do this in software would be like CONFIG_PROFILE_ALL_BRANCHES and that's horrid (and kernel only).

But there's more than Intel & ARM, but it looks like PPC doesn't have this either and I suspect MIPS and SPARC don't either, which doesn't leave us with much else.

So I guess we should just go ahead and merge this and hope more hardware grows this feature in a compatible enough manner.

#33 lore raw
Quoted 16 lines · Peter Zijlstra
Quoted 6 lines · Will Deacon
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian

This patchset adds an important and useful new feature to
perf_events: branch stack sampling. In other words, the ability to capture taken branches into each sample.

Quoted 8 lines
Quoted 3 lines

Other than the few comments given it all looks good. My main worry is the Intel only aspect, I'd really love for there to be another platform that could implement at least part of this.

I discussed this with Stephane in Prague and, although it would be lovely to have this on ARM, we simply don't have the hardware to do it. So the nature of series does seem to be x86-centric unless there's way to do a watered down version in software.

Quoted 7 lines

The only way to do this in software would be like CONFIG_PROFILE_ALL_BRANCHES and that's horrid (and kernel only).

But there's more than Intel & ARM, but it looks like PPC doesn't have this either and I suspect MIPS and SPARC don't either, which doesn't leave us with much else.

There is a hardware branch buffer on all Itanium processors. You can find the description for the McKinley implementation
(Itanium2) branch buffer in section 10.3.9 from:
http://download.intel.com/design/Itanium2/manuals/25111003.pdf

Quoted 2 lines

So I guess we should just go ahead and merge this and hope more hardware grows this feature in a compatible enough manner.

maintainer
#34 lore raw
Quoted 8 lines · Stephane Eranian
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Other than the few comments given it all looks good. My main worry is the Intel only aspect, I'd really love for there to be another platform that could implement at least part of this.

There is a hardware branch buffer on all Itanium processors. You can find the description for the McKinley implementation
(Itanium2) branch buffer in section 10.3.9 from:
http://download.intel.com/design/Itanium2/manuals/25111003.pdf

Yeah, I knew Itanic has it, but one its sinking (and doesn't have perf support), and two its still Intel ;-)

#35 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Stephane Eranian
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Other than the few comments given it all looks good. My main worry is the Intel only aspect, I'd really love for there to be another platform that could implement at least part of this.

Quoted 9 lines
Quoted 4 lines

There is a hardware branch buffer on all Itanium processors. You can find the description for the McKinley implementation (Itanium2) branch buffer in section 10.3.9 from:
http://download.intel.com/design/Itanium2/manuals/25111003.pdf

Yeah, I knew Itanic has it, but one its sinking (and doesn't have perf support), and two its still Intel ;-)

It's just an example of a different implementation of a branch buffer. That's why I mentioned it. The question is: could the branch_stack abstraction map to it. I think it could.

Quoted 1 line