perf, x86: Haswell LBR call stack support (kernel)
Subsystems (3)
v8 cover perf, x86: Haswell LBR call stack support (kernel)
This is the kernel patch.
For many profiling tasks we need the callgraph. For example we often need to see the caller of a lock or the caller of a memcpy or other library function to actually tune the program. Frame pointer unwinding is efficient and works well. But frame pointers are off by default on 64bit code (and on modern 32bit gccs), so there are many binaries around that do not use frame pointers. Profiling unchanged production code is very useful in practice. On some CPUs frame pointer also has a high cost. Dwarf2 unwinding also does not always work and is extremely slow (upto 20% overhead).
Haswell has a new feature that utilizes the existing Last Branch Record facility to record call chains. When the feature is enabled, function call will be collected as normal, but as return instructions are executed the last captured branch record is popped from the on-chip LBR registers. The LBR call stack facility provides an alternative to get callgraph. It has some limitations too, but should work in most cases and is significantly faster than dwarf. Frame pointer unwinding is still the best default, but LBR call stack is a good alternative when nothing else works.
A new call chain recording option "lbr" is introduced into perf tool for LBR call stack. The user can use --call-graph lbr to get the call stack information from hardware.
When profiling bc(1) on Fedora 19:
echo 'scale=2000; 4*a(1)' > cmd; perf record --call-graph lbr bc -l < cmd If enabling LBR, perf report output looks like:
50.36% bc bc [.] bc_divide
|
--- bc_divide
execute
run_code
yyparse
main
__libc_start_main
_start
33.66% bc bc [.] _one_mult
|
--- _one_mult
bc_divide
execute
run_code
yyparse
main
__libc_start_main
_start
7.62% bc bc [.] _bc_do_add
|
--- _bc_do_add
|
|--99.89%-- 0x2000186a8
--0.11%-- [...]
6.83% bc bc [.] _bc_do_sub
|
--- _bc_do_sub
|
|--99.94%-- bc_add
| execute
| run_code
| yyparse
| main
| __libc_start_main
| _start
--0.06%-- [...]
0.46% bc libc-2.17.so [.] __memset_sse2
|
--- __memset_sse2
|
|--54.13%-- bc_new_num
| |
| |--51.00%-- bc_divide
| | execute
| | run_code
| | yyparse
| | main
| | __libc_start_main
| | _start
| |
| |--30.46%-- _bc_do_sub
| | bc_add
| | execute
| | run_code
| | yyparse
| | main
| | __libc_start_main
| | _start
| |
| --18.55%-- _bc_do_add
| bc_add
| execute
| run_code
| yyparse
| main
| __libc_start_main
| _start
|
--45.87%-- bc_divide
execute
run_code
yyparse
main
__libc_start_main
_start
If using FP, perf report output looks like:
echo 'scale=2000; 4*a(1)' > cmd; perf record --call-graph fp bc -l < cmd
50.49% bc bc [.] bc_divide
|
--- bc_divide
33.57% bc bc [.] _one_mult
|
--- _one_mult
7.61% bc bc [.] _bc_do_add
|
--- _bc_do_add
0x2000186a8
6.88% bc bc [.] _bc_do_sub
|
--- _bc_do_sub
0.42% bc libc-2.17.so [.] __memcpy_ssse3_back
|
--- __memcpy_ssse3_backThe LBR call stack has following known limitations
- Zero length calls are not filtered out by hardware
- Exception handing such as setjmp/longjmp will have calls/returns not match
- Pushing different return address onto the stack will have calls/returns not match
- If callstack is deeper than the LBR, only the last entries are captured
Changes since v1
- split change into more patches
- introduce context switch callback and use it to flush LBR
- use the context switch callback to save/restore LBR
- dynamic allocate memory area for storing LBR stack, always switch the memory area during context switch
- disable this feature by default
- more description in change logs
Changes since v2
- don't use xchg to switch PMU specific data
- remove nr_branch_stack from struct perf_event_context
- simplify the save/restore LBR stack logical
- remove unnecessary 'has_branch_stack -> needs_branch_stack' conversion
- more description in change logs
Changes since v3
- remove sysfs attribute file that disable this feature
Changes since v4
- re-organize code that save/resotre LBR stack
- allocate pmu specific data when it's needed
- update code comments
Changes since v5
- Expose LBR call stack data to user perf tool
- Add option for perf report to support LBR call stack
- Some minor changes according to comments
Changes since v6
- rebase on tip.git 05066a2a04
- Modify perf test accordingly
Changes since v7
- LBR call stack as the 3rd callchain options
- Only include kernel patch
Yan, Zheng (13):
perf, x86: Reduce lbr_sel_map size
perf, core: introduce pmu context switch callback
perf, x86: use context switch callback to flush LBR stack perf, x86: Basic Haswell LBR call stack support
perf, core: pmu specific data for perf task context
perf, core: always switch pmu specific data during context switch perf, x86: allocate space for storing LBR stack
perf, x86: track number of events that use LBR callstack perf, x86: Save/resotre LBR stack during context switch perf, core: simplify need branch stack check
perf, x86: re-organize code that implicitly enables LBR/PEBS perf, x86: disable FREEZE_LBRS_ON_PMI when LBR operates in callstack
mode
perf, x86: Discard zero length call entries in LBR call stack
Peter Zijlstra (Intel) (1):
perf, x86: expose LBR callstack to user space tool
arch/x86/kernel/cpu/perf_event.c | 71 +++---- arch/x86/kernel/cpu/perf_event.h | 20 +- arch/x86/kernel/cpu/perf_event_intel.c | 36 +--- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 307 ++++++++++++++++++++++------- include/linux/perf_event.h | 21 +- include/uapi/linux/perf_event.h | 47 +++-- kernel/events/core.c | 178 +++++++++-------- 7 files changed, 451 insertions(+), 229 deletions(-)
Signature · 1 line
1.8.3.2
PATCH v8 1/14 perf, x86: Reduce lbr_sel_map size
From: Yan, Zheng <[email protected]>
The index of lbr_sel_map is bit value of perf branch_sample_type. PERF_SAMPLE_BRANCH_MAX is 1024 at present, so each lbr_sel_map uses 4096 bytes. By using bit shift as index, we can reduce lbr_sel_map size to 40 bytes. This patch defines 'bit shift' for branch types, and use 'bit shift' to define lbr_sel_maps.
--- arch/x86/kernel/cpu/perf_event.h | 4 +++ arch/x86/kernel/cpu/perf_event_intel_lbr.c | 54 ++++++++++++++---------------- include/uapi/linux/perf_event.h | 49 +++++++++++++++++++-------- 3 files changed, 64 insertions(+), 43 deletions(-)
arch/x86/kernel/cpu/perf_event.h +4 -0
@@ -509,6 +509,10 @@ struct x86_pmu {struct perf_guest_switch_msr *(*guest_get_msrs)(int *nr); }; +enum { +PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE =PERF_SAMPLE_BRANCH_MAX_SHIFT , +}; +#define x86_add_quirk (func_) \ do { \ static struct x86_pmu_quirk __quirk __initdata = { \
arch/x86/kernel/cpu/perf_event_intel_lbr.c +25 -29
84 lines not on this page · raw message
include/uapi/linux/perf_event.h +35 -14
56 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH v8 2/14 perf, core: introduce pmu context switch callback
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
The callback is invoked when process is scheduled in or out. It provides mechanism for later patches to save/store the LBR stack. For the schedule in case, the callback is invoked at the same place that flush branch stack callback is invoked. So it also can replace the flush branch stack callback. To avoid unnecessary overhead, the callback is enabled only when there are events use the LBR stack.
--- arch/x86/kernel/cpu/perf_event.c | 7 +++++ arch/x86/kernel/cpu/perf_event.h | 2 ++ include/linux/perf_event.h | 9 +++++++ kernel/events/core.c | 57 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 75 insertions(+)
arch/x86/kernel/cpu/perf_event.c +7 -0
@@ -1879,6 +1879,12 @@ static const struct attribute_group *x86_pmu_attr_groups[] = { NULL, }; +static void x86_pmu_sched_task (struct perf_event_context *ctx,bool sched_in) +{ +if (x86_pmu.sched_task) + x86_pmu.sched_task (ctx, sched_in); +} +static void x86_pmu_flush_branch_stack (void ) {if (x86_pmu.flush_branch_stack)
@@ -1912,6 +1918,7 @@ static struct pmu pmu = { .event_idx = x86_pmu_event_idx, .flush_branch_stack = x86_pmu_flush_branch_stack, + .sched_task = x86_pmu_sched_task, };void arch_perf_update_userpage (struct perf_event_mmap_page *userpg,u64 now)
arch/x86/kernel/cpu/perf_event.h +2 -0
@@ -467,6 +467,8 @@ struct x86_pmu {void (*check_microcode)(void );void (*flush_branch_stack)(void ); +void (*sched_task)(struct perf_event_context *ctx, +bool sched_in);/* * Intel Arch Perfmon v2+
include/linux/perf_event.h +9 -0
@@ -263,6 +263,13 @@ struct pmu {* flush branch stack on context-switches (needed in cpu-wide mode) */ void (*flush_branch_stack) (void ); + +/* +* context-switches callback +*/ +void (*sched_task) (struct perf_event_context *ctx, +bool sched_in); + };/**
@@ -562,6 +569,8 @@ extern void perf_event_delayed_put(struct task_struct *task);extern void perf_event_print_debug (void );extern void perf_pmu_disable (struct pmu *pmu);extern void perf_pmu_enable (struct pmu *pmu); +extern void perf_sched_cb_dec (struct pmu *pmu); +extern void perf_sched_cb_inc (struct pmu *pmu);extern int perf_event_task_disable (void );extern int perf_event_task_enable (void );extern int perf_event_refresh (struct perf_event *event,int refresh);
kernel/events/core.c +57 -0
81 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH v8 3/14 perf, x86: use context switch callback to flush LBR stack
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
Previous commit introduces context switch callback, its function overlaps with the flush branch stack callback. So we can use the context switch callback to flush LBR stack.
This patch adds code that uses the flush branch callback to flush the LBR stack when task is being scheduled in. The callback is enabled only when there are events use the LBR hardware. This patch also removes all old flush branch stack code.
--- arch/x86/kernel/cpu/perf_event.c | 7 --- arch/x86/kernel/cpu/perf_event.h | 3 +- arch/x86/kernel/cpu/perf_event_intel.c | 14 +----- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 27 +++++++++++ include/linux/perf_event.h | 1 - kernel/events/core.c | 77 ------------------------------ 6 files changed, 30 insertions(+), 99 deletions(-)
arch/x86/kernel/cpu/perf_event.c +0 -7
@@ -1885,12 +1885,6 @@ static void x86_pmu_sched_task(struct perf_event_context *ctx, bool sched_in) x86_pmu.sched_task (ctx, sched_in); } -static void x86_pmu_flush_branch_stack (void ) -{ -if (x86_pmu.flush_branch_stack) - x86_pmu.flush_branch_stack (); -} -void perf_check_microcode (void ) {if (x86_pmu.check_microcode)
@@ -1917,7 +1911,6 @@ static struct pmu pmu = { .commit_txn = x86_pmu_commit_txn, .event_idx = x86_pmu_event_idx, - .flush_branch_stack = x86_pmu_flush_branch_stack, .sched_task = x86_pmu_sched_task, };
arch/x86/kernel/cpu/perf_event.h +2 -1
@@ -466,7 +466,6 @@ struct x86_pmu {void (*cpu_dead)(int cpu);void (*check_microcode)(void ); -void (*flush_branch_stack)(void );void (*sched_task)(struct perf_event_context *ctx,bool sched_in);
@@ -727,6 +726,8 @@ void intel_pmu_pebs_disable_all(void);void intel_ds_init (void ); +void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in); +void intel_pmu_lbr_reset (void );void intel_pmu_lbr_enable (struct perf_event *event);
arch/x86/kernel/cpu/perf_event_intel.c +1 -13
@@ -2044,18 +2044,6 @@ static void intel_pmu_cpu_dying(int cpu)fini_debug_store_on_cpu (cpu); } -static void intel_pmu_flush_branch_stack (void ) -{ -/* -* Intel LBR does not tag entries with the -* PID of the current task, then we need to -* flush it on ctxsw -* For now, we simply reset it -*/ -if (x86_pmu.lbr_nr) -intel_pmu_lbr_reset (); -} -PMU_FORMAT_ATTR (offcore_rsp,"config1:0-63" );PMU_FORMAT_ATTR (ldlat,"config1:0-15" );
@@ -2107,7 +2095,7 @@ static __initconst const struct x86_pmu intel_pmu = { .cpu_starting = intel_pmu_cpu_starting, .cpu_dying = intel_pmu_cpu_dying, .guest_get_msrs = intel_guest_get_msrs, - .flush_branch_stack = intel_pmu_flush_branch_stack, + .sched_task = intel_pmu_lbr_sched_task, };static __init void intel_clovertown_quirk (void )
arch/x86/kernel/cpu/perf_event_intel_lbr.c +27 -0
@@ -177,6 +177,31 @@ void intel_pmu_lbr_reset(void)intel_pmu_lbr_reset_64 (); } +void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in) +{ +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + +if (!x86_pmu.lbr_nr) +return ; + +/* +* When sampling the branck stack in system-wide, it may be +* necessary to flush the stack on context switch. This happens +* when the branch stack does not tag its entries with the pid +* of the current task. Otherwise it becomes impossible to +* associate a branch entry with a task. This ambiguity is more +* likely to appear when the branch stack supports priv level +* filtering and the user sets it to monitor only at the user +* level (which could be a useful measurement in system-wide +* mode). In that case, the risk is high of having a branch +* stack with branch from multiple tasks. +*/ +if (sched_in) { +intel_pmu_lbr_reset (); + cpuc->lbr_context = ctx; + } +} +void intel_pmu_lbr_enable (struct perf_event *event) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -195,6 +220,7 @@ void intel_pmu_lbr_enable(struct perf_event *event) cpuc->br_sel = event->hw.branch_reg.reg; cpuc->lbr_users++; +perf_sched_cb_inc (event->ctx->pmu); }void intel_pmu_lbr_disable (struct perf_event *event)
@@ -206,6 +232,7 @@ void intel_pmu_lbr_disable(struct perf_event *event) cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 ); +perf_sched_cb_dec (event->ctx->pmu);if (cpuc->enabled && !cpuc->lbr_users) {__intel_pmu_lbr_disable ();
include/linux/perf_event.h +0 -1
@@ -514,7 +514,6 @@ struct perf_event_context {u64 generation;int pin_count;int nr_cgroups;/* cgroup evts */ -int nr_branch_stack;/* branch_stack evt */ struct rcu_head rcu_head;struct delayed_work orphans_remove;
kernel/events/core.c +0 -77
119 lines not on this page · raw message
Signature · 1 line
1.8.3.2
Peter Zijlstra
maintainerRe: [PATCH V8 03/14] perf, x86: use context switch callback to flush LBR stack
Quoted 51 lines · Kan Liang
---
a/kernel/events/core.c(quoted diff) +0
-46@@ -2673,64 +2666,6 @@ static void perf_event_context_sched_in(struct perf_event_context *ctx, } /* - * When sampling the branck stack in system-wide, it may be necessary - * to flush the stack on context switch. This happens when the branch - * stack does not tag its entries with the pid of the current task. - * Otherwise it becomes impossible to associate a branch entry with a - * task. This ambiguity is more likely to appear when the branch stack - * supports priv level filtering and the user sets it to monitor only - * at the user level (which could be a useful measurement in system-wide - * mode). In that case, the risk is high of having a branch stack with - * branch from multiple tasks. Flushing may mean dropping the existing - * entries or stashing them somewhere in the PMU specific code layer. - * - * This function provides the context switch callback to the lower code - * layer. It is invoked ONLY when there is at least one system-wide context - * with at least one active event using taken branch sampling. - */ -static void perf_branch_stack_sched_in(struct task_struct *prev, - struct task_struct *task) -{ - struct perf_cpu_context *cpuctx; - struct pmu *pmu; - unsigned long flags; - - /* no need to flush branch stack if not changing task */ - if (prev == task) - return; - - local_irq_save(flags); - - rcu_read_lock(); - - list_for_each_entry_rcu(pmu, &pmus, entry) { - cpuctx = this_cpu_ptr(pmu->pmu_cpu_context); - - /* - * check if the context has at least one - * event using PERF_SAMPLE_BRANCH_STACK - */ - if (cpuctx->ctx.nr_branch_stack > 0 - && pmu->flush_branch_stack) { - - perf_ctx_lock(cpuctx, cpuctx->task_ctx); - - perf_pmu_disable(pmu); - - pmu->flush_branch_stack(); -
Review static void perf_event_context_sched_in(struct perf_event_context *ctx, · line 2731
Quoted 10 lines
(quoted diff) +0
-10- perf_pmu_enable(pmu); - - perf_ctx_unlock(cpuctx, cpuctx->task_ctx); - } - } - - rcu_read_unlock(); - - local_irq_restore(flags); -}
So while this patch removes almost everything flush_branch_stack related it leaves a few rudiments and a broken architecture.
---
Subject: perf,powerpc: Fix up flush_branch_stack users
From: Peter Zijlstra <[email protected]>
Date: Wed Jan 14 14:15:39 CET 2015
The recent LBR rework for x86 left a stray flush_branch_stack user in the PowerPC code, fix that up.
--- arch/powerpc/perf/core-book3s.c | 13 +++++++++---- include/linux/perf_event.h | 5 ----- 2 files changed, 9 insertions(+), 9 deletions(-)
arch/powerpc/perf/core-book3s.c +9 -4
@@ -124,7 +124,7 @@ static unsigned long ebb_switch_in(boolstatic inline void power_pmu_bhrb_enable (struct perf_event *event) {}static inline void power_pmu_bhrb_disable (struct perf_event *event) {} -static void power_pmu_flush_branch_stack (void ) {} +static void power_pmu_sched_task (struct perf_event_context *ctx,bool sched_in) {}static inline void power_pmu_bhrb_read (struct cpu_hw_events *cpuhw) {}static void pmao_restore_workaround (bool ebb) { }#endif /* CONFIG_PPC32 */
@@ -350,6 +350,7 @@ static void power_pmu_bhrb_enable(struct cpuhw->bhrb_context = event->ctx; } cpuhw->bhrb_users++; +perf_sched_cb_inc (event->ctx->pmu); }static void power_pmu_bhrb_disable (struct perf_event *event)
@@ -361,6 +362,7 @@ static void power_pmu_bhrb_disable(struc cpuhw->bhrb_users--;WARN_ON_ONCE (cpuhw->bhrb_users <0 ); +perf_sched_cb_dec (event->ctx->pmu);if (!cpuhw->disabled && !cpuhw->bhrb_users) {/* BHRB cannot be turned off when other
@@ -375,9 +377,12 @@ static void power_pmu_bhrb_disable(struc/* Called from ctxsw to prevent one process's branch entries to * mingle with the other process's entries during context switch. */ -static void power_pmu_flush_branch_stack (void ) +static void power_pmu_sched_task (struct perf_event_context *ctx,bool sched_in) { -if (ppmu->bhrb_nr) +if (!ppmu->bhrb_nr) +return ; + +if (sched_in)power_pmu_bhrb_reset (); }/* Calculate the to address for a branch */
@@ -1901,7 +1906,7 @@ static struct pmu power_pmu = { .cancel_txn = power_pmu_cancel_txn, .commit_txn = power_pmu_commit_txn, .event_idx = power_pmu_event_idx, - .flush_branch_stack = power_pmu_flush_branch_stack, + .sched_task = power_pmu_sched_task, };/*
include/linux/perf_event.h +0 -5
@@ -255,11 +255,6 @@ struct pmu {int (*event_idx) (struct perf_event *event);/*optional */ /* -* flush branch stack on context-switches (needed in cpu-wide mode) -*/ -void (*flush_branch_stack) (void ); - -/* * context-switches callback */ void (*sched_task) (struct perf_event_context *ctx,
RE: [PATCH V8 03/14] perf, x86: use context switch callback to flush LBR stack
Quoted 66 lines · Kan Liang
Review static void
Quoted 2 lines
---
a/kernel/events/core.c(quoted diff) +0
-0@@ -2673,64 +2666,6 @@ static voidperf_event_context_sched_in(struct
Quoted 60 lines
(quoted diff) +0
-5perf_event_context *ctx, } /* - * When sampling the branck stack in system-wide, it may be necessary - * to flush the stack on context switch. This happens when the branch - * stack does not tag its entries with the pid of the current task. - * Otherwise it becomes impossible to associate a branch entry with a - * task. This ambiguity is more likely to appear when the branchstack
- * supports priv level filtering and the user sets it to monitor only
- * at the user level (which could be a useful measurement in system-wide
- * mode). In that case, the risk is high of having a branch stack with
- * branch from multiple tasks. Flushing may mean dropping the existing
- * entries or stashing them somewhere in the PMU specific code layer.
- *
- * This function provides the context switch callback to the lower code
- * layer. It is invoked ONLY when there is at least one system-wide context
- * with at least one active event using taken branch sampling.
- */(quoted diff) +0
-36-static void perf_branch_stack_sched_in(struct task_struct *prev, - struct task_struct *task) -{ - struct perf_cpu_context *cpuctx; - struct pmu *pmu; - unsigned long flags; - - /* no need to flush branch stack if not changing task */ - if (prev == task) - return; - - local_irq_save(flags); - - rcu_read_lock(); - - list_for_each_entry_rcu(pmu, &pmus, entry) { - cpuctx = this_cpu_ptr(pmu->pmu_cpu_context); - - /* - * check if the context has at least one - * event using PERF_SAMPLE_BRANCH_STACK - */ - if (cpuctx->ctx.nr_branch_stack > 0 - && pmu->flush_branch_stack) { - - perf_ctx_lock(cpuctx, cpuctx->task_ctx); - - perf_pmu_disable(pmu); - - pmu->flush_branch_stack(); - - perf_pmu_enable(pmu); - - perf_ctx_unlock(cpuctx, cpuctx->task_ctx); - } - }
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +0
-5- - rcu_read_unlock(); - - local_irq_restore(flags); -}So while this patch removes almost everything
flush_branch_stackrelated it leaves a few rudiments and a broken architecture.
Thanks for providing this patch to fix the issue.
Kan
Quoted 88 lines
---
Subject: perf,powerpc: Fix upflush_branch_stackusers
From: Peter Zijlstra <[email protected]>
Date: Wed Jan 14 14:15:39 CET 2015The recent LBR rework for x86 left a stray
flush_branch_stackuser in the PowerPC code, fix that up.Cc Michael Neuling <[email protected]>Cc Anton Blanchard <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/powerpc/perf/core-book3s.c | 13 +++++++++---- include/linux/perf_event.h | 5 ----- 2 files changed, 9 insertions(+), 9 deletions(-)arch/powerpc/perf/core-book3s.c +0
-0@@ -124,7 +124,7 @@ static unsigned long ebb_switch_in(boolstatic inline void power_pmu_bhrb_enable (struct perf_event *event) {}static inline void
power_pmu_bhrb_disable(struct perf_event *event){} - static voidpower_pmu_flush_branch_stack(void){}arch/powerpc/perf/core-book3s.c +2
-0+static void power_pmu_sched_task (struct perf_event_context *ctx,bool +sched_in) {}static inline void power_pmu_bhrb_read (struct cpu_hw_events *cpuhw){} static void
pmao_restore_workaround(bool ebb){ } #endif /*CONFIG_PPC32*/ @@ -350,6 +350,7 @@ static void power_pmu_bhrb_enable(structarch/powerpc/perf/core-book3s.c +1
-0cpuhw->bhrb_context = event->ctx; } cpuhw->bhrb_users++; +perf_sched_cb_inc (event->ctx->pmu); } static void power_pmu_bhrb_disable(struct perf_event *event) @@ -361,6 +362,7 @@ static void power_pmu_bhrb_disable(struc
arch/powerpc/perf/core-book3s.c +8
-2cpuhw->bhrb_users--;WARN_ON_ONCE (cpuhw->bhrb_users <0 ); +perf_sched_cb_dec (event->ctx->pmu);if (!cpuhw->disabled && !cpuhw->bhrb_users) {/* BHRB cannot be turned off when other @@ -375,9 +377,12 @@ static void power_pmu_bhrb_disable(struc /* Called from ctxsw to prevent one process's branch entries to * mingle with the other process's entries during context switch. */ -static void power_pmu_flush_branch_stack (void ) +static void power_pmu_sched_task (struct perf_event_context *ctx,bool +sched_in) { -if (ppmu->bhrb_nr) +if (!ppmu->bhrb_nr) +return ; + +if (sched_in)power_pmu_bhrb_reset (); } /* Calculate the to address for a branch */ @@ -1901,7 +1906,7 @@ staticstruct pmu
power_pmu= {arch/powerpc/perf/core-book3s.c +1
-1.cancel_txn = power_pmu_cancel_txn, .commit_txn = power_pmu_commit_txn, .event_idx = power_pmu_event_idx, - .flush_branch_stack = power_pmu_flush_branch_stack, + .sched_task = power_pmu_sched_task, };/* include/linux/perf_event.h +0
-5@@ -255,11 +255,6 @@ struct pmu {int (*event_idx) (struct perf_event *event);/*optional */ /* -* flush branch stack on context-switches (needed in cpu-wide mode) -*/ -void (*flush_branch_stack) (void ); - -/* * context-switches callback include/linux/perf_event.h +0
-0*/ void (*sched_task) (struct perf_event_context *ctx,
PATCH v8 4/14 perf, x86: Basic Haswell LBR call stack support
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
Haswell has a new feature that utilizes the existing LBR facility to record call chains. To enable this feature, bits (JCC, NEAR_IND_JMP, NEAR_REL_JMP, FAR_BRANCH, EN_CALLSTACK) in LBR_SELECT must be set to 1, bits (NEAR_REL_CALL, NEAR-IND_CALL, NEAR_RET) must be cleared. Due to a hardware bug of Haswell, this feature doesn't work well with FREEZE_LBRS_ON_PMI.
When the call stack feature is enabled, the LBR stack will capture unfiltered call data normally, but as return instructions are executed, the last captured branch record is flushed from the on-chip registers in a last-in first-out (LIFO) manner. Thus, branch information relative to leaf functions will not be captured, while preserving the call stack information of the main line execution path.
This patch defines a separate lbr_sel map for Haswell. The map contains a new entry for the call stack feature.
--- arch/x86/kernel/cpu/perf_event.h | 14 ++++- arch/x86/kernel/cpu/perf_event_intel.c | 2 +- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 91 ++++++++++++++++++++++-------- 3 files changed, 83 insertions(+), 24 deletions(-)
arch/x86/kernel/cpu/perf_event.h +13 -1
@@ -511,7 +511,11 @@ struct x86_pmu { };enum { -PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE =PERF_SAMPLE_BRANCH_MAX_SHIFT , +PERF_SAMPLE_BRANCH_CALL_STACK_SHIFT =PERF_SAMPLE_BRANCH_MAX_SHIFT , +PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE , + +PERF_SAMPLE_BRANCH_CALL_STACK = +1U <<PERF_SAMPLE_BRANCH_CALL_STACK_SHIFT , };#define x86_add_quirk (func_) \
@@ -545,6 +549,12 @@ static struct perf_pmu_events_attr event_attr_##v = { \extern struct x86_pmu x86_pmu__read_mostly ; +static inline bool x86_pmu_has_lbr_callstack (void ) +{ +return x86_pmu.lbr_sel_map && + x86_pmu.lbr_sel_map[PERF_SAMPLE_BRANCH_CALL_STACK_SHIFT ] >0 ; +} + DECLARE_PER_CPU(struct cpu_hw_events, cpu_hw_events);int x86_perf_event_set_period (struct perf_event *event);
@@ -748,6 +758,8 @@ void intel_pmu_lbr_init_atom(void);void intel_pmu_lbr_init_snb (void ); +void intel_pmu_lbr_init_hsw (void ); +int intel_pmu_setup_lbr_filter (struct perf_event *event);int p4_pmu_init (void );
arch/x86/kernel/cpu/perf_event_intel.c +1 -1
@@ -2536,7 +2536,7 @@ __init int intel_pmu_init(void)memcpy (hw_cache_event_ids, snb_hw_cache_event_ids,sizeof (hw_cache_event_ids));memcpy (hw_cache_extra_regs, snb_hw_cache_extra_regs,sizeof (hw_cache_extra_regs)); -intel_pmu_lbr_init_snb (); +intel_pmu_lbr_init_hsw (); x86_pmu.event_constraints = intel_hsw_event_constraints; x86_pmu.pebs_constraints = intel_hsw_pebs_event_constraints;
arch/x86/kernel/cpu/perf_event_intel_lbr.c +69 -22
150 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH v8 5/14 perf, core: pmu specific data for perf task context
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
Introduce a new flag PERF_ATTACH_TASK_DATA for perf event's attach stata. The flag is set by PMU's event_init() callback, it indicates that perf event needs PMU specific data.
The PMU specific data are initialized to zeros. Later patches will use PMU specific data to save LBR stack.
--- include/linux/perf_event.h | 6 ++++++ kernel/events/core.c | 40 ++++++++++++++++++++++++++++++++++++---- 2 files changed, 42 insertions(+), 4 deletions(-)
include/linux/perf_event.h +6 -0
@@ -269,6 +269,10 @@ struct pmu {*/ void (*sched_task) (struct perf_event_context *ctx,bool sched_in); +/* +* PMU specific data size +*/ +size_t task_ctx_size; };
@@ -305,6 +309,7 @@ struct swevent_hlist {#define PERF_ATTACH_CONTEXT 0x01#define PERF_ATTACH_GROUP 0x02#define PERF_ATTACH_TASK 0x04 +#define PERF_ATTACH_TASK_DATA 0x08struct perf_cgroup ;struct ring_buffer ;
@@ -514,6 +519,7 @@ struct perf_event_context {u64 generation;int pin_count;int nr_cgroups;/* cgroup evts */ +void *task_ctx_data;/* pmu specific data */ struct rcu_head rcu_head;struct delayed_work orphans_remove;
kernel/events/core.c +36 -4
@@ -895,6 +895,15 @@ static void get_ctx(struct perf_event_context *ctx)WARN_ON (!atomic_inc_not_zero (&ctx->refcount)); } +static void free_ctx (struct rcu_head *head) +{ +struct perf_event_context *ctx; + + ctx = container_of(head, struct perf_event_context, rcu_head); +kfree (ctx->task_ctx_data); +kfree (ctx); +} +static void put_ctx (struct perf_event_context *ctx) {if (atomic_dec_and_test (&ctx->refcount)) {
@@ -902,7 +911,7 @@ static void put_ctx(struct perf_event_context *ctx)put_ctx (ctx->parent_ctx);if (ctx->task)put_task_struct (ctx->task); -kfree_rcu (ctx, rcu_head); +call_rcu (&ctx->rcu_head, free_ctx); } }
@@ -3188,12 +3197,15 @@ errout:* Returns a matching context with refcount and pincount. */ static struct perf_event_context * -find_get_context (struct pmu *pmu,struct task_struct *task,int cpu) +find_get_context (struct pmu *pmu,struct task_struct *task, +struct perf_event *event) {struct perf_event_context *ctx, *clone_ctx =NULL ;struct perf_cpu_context *cpuctx; +void *task_ctx_data =NULL ;unsigned long flags;int ctxn, err; +int cpu = event->cpu;if (!task) {/* Must be root to operate on a CPU event: */
@@ -3221,11 +3233,24 @@ find_get_context(struct pmu *pmu, struct task_struct *task, int cpu)if (ctxn <0 ) goto errout; +if (event->attach_state &PERF_ATTACH_TASK_DATA ) { + task_ctx_data =kzalloc (pmu->task_ctx_size,GFP_KERNEL ); +if (!task_ctx_data) { + err = -ENOMEM ; + goto errout; + } + } + retry: ctx =perf_lock_task_context (task, ctxn, &flags);if (ctx) { clone_ctx =unclone_ctx (ctx); ++ctx->pin_count; + +if (task_ctx_data && !ctx->task_ctx_data) { + ctx->task_ctx_data = task_ctx_data; + task_ctx_data =NULL ; + }raw_spin_unlock_irqrestore (&ctx->lock, flags);if (clone_ctx)
@@ -3236,6 +3261,11 @@ retry:if (!ctx) goto errout; +if (task_ctx_data) { + ctx->task_ctx_data = task_ctx_data; + task_ctx_data =NULL ; + } + err =0 ;mutex_lock (&task->perf_event_mutex);/*
@@ -3262,9 +3292,11 @@ retry: } } +kfree (task_ctx_data);return ctx; errout: +kfree (task_ctx_data);return ERR_PTR (err); }
@@ -7331,7 +7363,7 @@ SYSCALL_DEFINE5(perf_event_open,/* * Get the target context (task or percpu): */ - ctx =find_get_context (pmu, task, event->cpu); + ctx =find_get_context (pmu, task, event);if (IS_ERR (ctx)) { err =PTR_ERR (ctx); goto err_alloc;
@@ -7500,7 +7532,7 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu,account_event (event); - ctx =find_get_context (event->pmu, task, cpu); + ctx =find_get_context (event->pmu, task, event);if (IS_ERR (ctx)) { err =PTR_ERR (ctx); goto err_free;
Signature · 1 line
1.8.3.2
PATCH v8 6/14 perf, core: always switch pmu specific data during context switch
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
If two tasks were both forked from the same parent task, Events in their perf task contexts can be the same. Perf core may leave out switching the perf event contexts.
Previous patch inroduces pmu specific data. The data is for saving the LBR stack, it is task specific. So we need to switch the data even when context switch is optimized out.
--- kernel/events/core.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/events/core.c +3 -0
@@ -2420,6 +2420,9 @@ static void perf_event_context_sched_out(struct task_struct *task, int ctxn, next->perf_event_ctxp[ctxn] = ctx; ctx->task = next; next_ctx->task = task; + +swap (ctx->task_ctx_data, next_ctx->task_ctx_data); + do_switch =0 ;perf_event_sync_stat (ctx, next_ctx);
Signature · 1 line
1.8.3.2
PATCH v8 7/14 perf, x86: allocate space for storing LBR stack
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
When the LBR call stack is enabled, it is necessary to save/restore the LBR stack on context switch. We can use pmu specific data to store LBR stack when task is scheduled out. This patch adds code that allocates the pmu specific data.
--- arch/x86/kernel/cpu/perf_event.c | 4 ++++ arch/x86/kernel/cpu/perf_event.h | 7 +++++++ 2 files changed, 11 insertions(+)
arch/x86/kernel/cpu/perf_event.c +4 -0
@@ -428,6 +428,9 @@ int x86_pmu_hw_config(struct perf_event *event) } } +if (event->attr.branch_sample_type &PERF_SAMPLE_BRANCH_CALL_STACK ) + event->attach_state |=PERF_ATTACH_TASK_DATA ; +/* * Generate PMC IRQs: * (keep 'enabled' bit clear for now)
@@ -1912,6 +1915,7 @@ static struct pmu pmu = { .event_idx = x86_pmu_event_idx, .sched_task = x86_pmu_sched_task, + .task_ctx_size = sizeof(struct x86_perf_task_context), };void arch_perf_update_userpage (struct perf_event_mmap_page *userpg,u64 now)
arch/x86/kernel/cpu/perf_event.h +7 -0
@@ -510,6 +510,13 @@ struct x86_pmu {struct perf_guest_switch_msr *(*guest_get_msrs)(int *nr); }; +struct x86_perf_task_context { +u64 lbr_from[MAX_LBR_ENTRIES ]; +u64 lbr_to[MAX_LBR_ENTRIES ]; +int lbr_callstack_users; +int lbr_stack_state; +}; +enum {PERF_SAMPLE_BRANCH_CALL_STACK_SHIFT =PERF_SAMPLE_BRANCH_MAX_SHIFT ,PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE ,
Signature · 1 line
1.8.3.2
PATCH v8 8/14 perf, x86: track number of events that use LBR callstack
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
When enabling/disabling an event, check if the event uses the LBR callstack feature, adjust the LBR callstack usage count accordingly. Later patch will use the usage count to decide if LBR stack should be saved/restored.
--- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
arch/x86/kernel/cpu/perf_event_intel_lbr.c +19 -0
@@ -205,9 +205,15 @@ void intel_pmu_lbr_sched_task(struct perf_event_context *ctx, bool sched_in) } } +static inline bool branch_user_callstack (unsigned br_sel) +{ +return (br_sel &X86_BR_USER ) && (br_sel &X86_BR_CALL_STACK ); +} +void intel_pmu_lbr_enable (struct perf_event *event) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct x86_perf_task_context *task_ctx;if (!x86_pmu.lbr_nr)return ;
@@ -222,6 +228,12 @@ void intel_pmu_lbr_enable(struct perf_event *event) } cpuc->br_sel = event->hw.branch_reg.reg; +if (branch_user_callstack (cpuc->br_sel) && event->ctx && + event->ctx->task_ctx_data) { + task_ctx = event->ctx->task_ctx_data; + task_ctx->lbr_callstack_users++; + } + cpuc->lbr_users++;perf_sched_cb_inc (event->ctx->pmu); }
@@ -229,10 +241,17 @@ void intel_pmu_lbr_enable(struct perf_event *event)void intel_pmu_lbr_disable (struct perf_event *event) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct x86_perf_task_context *task_ctx;if (!x86_pmu.lbr_nr)return ; +if (branch_user_callstack (cpuc->br_sel) && event->ctx && + event->ctx->task_ctx_data) { + task_ctx = event->ctx->task_ctx_data; + task_ctx->lbr_callstack_users--; + } + cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 );perf_sched_cb_dec (event->ctx->pmu);
Signature · 1 line
1.8.3.2
PATCH v8 9/14 perf, x86: Save/resotre LBR stack during context switch
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
When the LBR call stack is enabled, it is necessary to save/restore the LBR stack on context switch. The solution is saving/restoring the LBR stack to/from task's perf event context.
The LBR stack is saved/restored only when there are events that use the LBR call stack. If no event uses LBR call stack, the LBR stack is reset when task is scheduled in.
--- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 88 ++++++++++++++++++++++++++---- 1 file changed, 76 insertions(+), 12 deletions(-)
arch/x86/kernel/cpu/perf_event_intel_lbr.c +76 -12
@@ -180,14 +180,90 @@ void intel_pmu_lbr_reset(void)intel_pmu_lbr_reset_64 (); } +/* +* TOS = most recently recorded branch +*/ +static inline u64 intel_pmu_lbr_tos (void ) +{ +u64 tos; + +rdmsrl (x86_pmu.lbr_tos, tos); +return tos; +} + +enum { +LBR_NONE , +LBR_VALID , +}; + +static void __intel_pmu_lbr_restore (struct x86_perf_task_context *task_ctx) +{ +int i; +unsigned lbr_idx, mask; +u64 tos; + +if (task_ctx->lbr_callstack_users ==0 || + task_ctx->lbr_stack_state ==LBR_NONE ) { +intel_pmu_lbr_reset (); +return ; + } + + mask = x86_pmu.lbr_nr -1 ; + tos =intel_pmu_lbr_tos (); +for (i =0 ; i < x86_pmu.lbr_nr; i++) { + lbr_idx = (tos - i) & mask; +wrmsrl (x86_pmu.lbr_from + lbr_idx, task_ctx->lbr_from[i]); +wrmsrl (x86_pmu.lbr_to + lbr_idx, task_ctx->lbr_to[i]); + } + task_ctx->lbr_stack_state =LBR_NONE ; +} + +static void __intel_pmu_lbr_save (struct x86_perf_task_context *task_ctx) +{ +int i; +unsigned lbr_idx, mask; +u64 tos; + +if (task_ctx->lbr_callstack_users ==0 ) { + task_ctx->lbr_stack_state =LBR_NONE ; +return ; + } + + mask = x86_pmu.lbr_nr -1 ; + tos =intel_pmu_lbr_tos (); +for (i =0 ; i < x86_pmu.lbr_nr; i++) { + lbr_idx = (tos - i) & mask; +rdmsrl (x86_pmu.lbr_from + lbr_idx, task_ctx->lbr_from[i]); +rdmsrl (x86_pmu.lbr_to + lbr_idx, task_ctx->lbr_to[i]); + } + task_ctx->lbr_stack_state =LBR_VALID ; +} +void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct x86_perf_task_context *task_ctx;if (!x86_pmu.lbr_nr)return ;/* +* If LBR callstack feature is enabled and the stack was saved when +* the task was scheduled out, restore the stack. Otherwise flush +* the LBR stack. +*/ + task_ctx = ctx ? ctx->task_ctx_data :NULL ; +if (task_ctx) { +if (sched_in) { +__intel_pmu_lbr_restore (task_ctx); + cpuc->lbr_context = ctx; + }else { +__intel_pmu_lbr_save (task_ctx); + } +return ; + } + +/* * When sampling the branck stack in system-wide, it may be * necessary to flush the stack on context switch. This happens * when the branch stack does not tag its entries with the pid
@@ -279,18 +355,6 @@ void intel_pmu_lbr_disable_all(void)__intel_pmu_lbr_disable (); } -/* -* TOS = most recently recorded branch -*/ -static inline u64 intel_pmu_lbr_tos (void ) -{ -u64 tos; - -rdmsrl (x86_pmu.lbr_tos, tos); - -return tos; -} -static void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc) {unsigned long mask = x86_pmu.lbr_nr -1 ;
Signature · 1 line
1.8.3.2
PATCH v8 10/14 perf, core: simplify need branch stack check
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
Using event->attr.branch_sample_type to replace intel_pmu_needs_lbr_smpl() for avoiding duplicating code that implicitly enables the LBR.
Currently, branch stack can be enabled by user explicitly requested branch sampling or implicit branch sampling to correct PEBS skid. For user explicitly requested branch sampling, the branch_sample_type is explicitly set by user. For PEBS case, the branch_sample_type is also implicitly set to PERF_SAMPLE_BRANCH_ANY in x86_pmu_hw_config.
--- arch/x86/kernel/cpu/perf_event_intel.c | 20 +++----------------- include/linux/perf_event.h | 5 +++++ kernel/events/core.c | 3 +++ 3 files changed, 11 insertions(+), 17 deletions(-)
arch/x86/kernel/cpu/perf_event_intel.c +3 -17
@@ -1029,20 +1029,6 @@ static __initconst const u64 slm_hw_cache_event_ids }, }; -static inline bool intel_pmu_needs_lbr_smpl (struct perf_event *event) -{ -/* user explicitly requested branch sampling */ -if (has_branch_stack (event)) -return true; - -/* implicit branch sampling to correct PEBS skid */ -if (x86_pmu.intel_cap.pebs_trap && event->attr.precise_ip >1 && - x86_pmu.intel_cap.pebs_format <2 ) -return true; - -return false; -} -static void intel_pmu_disable_all (void ) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -1207,7 +1193,7 @@ static void intel_pmu_disable_event(struct perf_event *event)* must disable before any actual event * because any event may be combined with LBR */ -if (intel_pmu_needs_lbr_smpl (event)) +if (needs_branch_stack (event))intel_pmu_lbr_disable (event);if (unlikely (hwc->config_base ==MSR_ARCH_PERFMON_FIXED_CTR_CTRL )) {
@@ -1268,7 +1254,7 @@ static void intel_pmu_enable_event(struct perf_event *event)* must enabled before any actual event * because any event may be combined with LBR */ -if (intel_pmu_needs_lbr_smpl (event)) +if (needs_branch_stack (event))intel_pmu_lbr_enable (event);if (event->attr.exclude_host)
@@ -1747,7 +1733,7 @@ static int intel_pmu_hw_config(struct perf_event *event)if (event->attr.precise_ip && x86_pmu.pebs_aliases) x86_pmu.pebs_aliases (event); -if (intel_pmu_needs_lbr_smpl (event)) { +if (needs_branch_stack (event)) { ret =intel_pmu_setup_lbr_filter (event);if (ret)return ret;
include/linux/perf_event.h +5 -0
@@ -796,6 +796,11 @@ static inline bool has_branch_stack(struct perf_event *event)return event->attr.sample_type &PERF_SAMPLE_BRANCH_STACK ; } +static inline bool needs_branch_stack (struct perf_event *event) +{ +return event->attr.branch_sample_type !=0 ; +} +extern int perf_output_begin (struct perf_output_handle *handle,struct perf_event *event,unsigned int size);extern void perf_output_end (struct perf_output_handle *handle);
kernel/events/core.c +3 -0
@@ -7015,6 +7015,9 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,if (attr->inherit && (attr->read_format &PERF_FORMAT_GROUP )) goto err_ns; +if (!has_branch_stack (event)) + event->attr.branch_sample_type =0 ; + pmu =perf_init_event (event);if (!pmu) goto err_ns;
Signature · 1 line
1.8.3.2
PATCH v8 11/14 perf, x86: re-organize code that implicitly enables LBR/PEBS
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
make later patch more readable, no logic change.
--- arch/x86/kernel/cpu/perf_event.c | 59 ++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 30 deletions(-)
arch/x86/kernel/cpu/perf_event.c +29 -30
@@ -395,36 +395,35 @@ int x86_pmu_hw_config(struct perf_event *event)if (event->attr.precise_ip > precise)return -EOPNOTSUPP ; -/* -* check that PEBS LBR correction does not conflict with -* whatever the user is asking with attr->branch_sample_type -*/ -if (event->attr.precise_ip >1 && - x86_pmu.intel_cap.pebs_format <2 ) { -u64 *br_type = &event->attr.branch_sample_type; - -if (has_branch_stack (event)) { -if (!precise_br_compat (event)) -return -EOPNOTSUPP ; - -/* branch_sample_type is compatible */ - - }else { -/* -* user did not specify branch_sample_type -* -* For PEBS fixups, we capture all -* the branches at the priv level of the -* event. -*/ - *br_type =PERF_SAMPLE_BRANCH_ANY ; - -if (!event->attr.exclude_user) - *br_type |=PERF_SAMPLE_BRANCH_USER ; - -if (!event->attr.exclude_kernel) - *br_type |=PERF_SAMPLE_BRANCH_KERNEL ; - } + } +/* +* check that PEBS LBR correction does not conflict with +* whatever the user is asking with attr->branch_sample_type +*/ +if (event->attr.precise_ip >1 && x86_pmu.intel_cap.pebs_format <2 ) { +u64 *br_type = &event->attr.branch_sample_type; + +if (has_branch_stack (event)) { +if (!precise_br_compat (event)) +return -EOPNOTSUPP ; + +/* branch_sample_type is compatible */ + + }else { +/* +* user did not specify branch_sample_type +* +* For PEBS fixups, we capture all +* the branches at the priv level of the +* event. +*/ + *br_type =PERF_SAMPLE_BRANCH_ANY ; + +if (!event->attr.exclude_user) + *br_type |=PERF_SAMPLE_BRANCH_USER ; + +if (!event->attr.exclude_kernel) + *br_type |=PERF_SAMPLE_BRANCH_KERNEL ; } }
Signature · 1 line
1.8.3.2
PATCH v8 12/14 perf, x86: disable FREEZE_LBRS_ON_PMI when LBR operates in callstack mode
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
LBR callstack is designed for PEBS, It does not work well with FREEZE_LBRS_ON_PMI for non PEBS event. If FREEZE_LBRS_ON_PMI is set for non PEBS event, PMIs near call/return instructions may cause superfluous increase/decrease of LBR_TOS.
This patch modifies __intel_pmu_lbr_enable() to not enable FREEZE_LBRS_ON_PMI when LBR operates in callstack mode. We currently don't use LBR callstack to capture kernel space callchain, so disabling FREEZE_LBRS_ON_PMI should not be a problem.
--- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/perf_event_intel_lbr.c +13 -4
@@ -131,14 +131,23 @@ static void intel_pmu_lbr_filter(struct cpu_hw_events *cpuc);static void __intel_pmu_lbr_enable (void ) { -u64 debugctl;struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +u64 debugctl, lbr_select =0 ; -if (cpuc->lbr_sel) -wrmsrl (MSR_LBR_SELECT , cpuc->lbr_sel->config); +if (cpuc->lbr_sel) { + lbr_select = cpuc->lbr_sel->config; +wrmsrl (MSR_LBR_SELECT , lbr_select); + }rdmsrl (MSR_IA32_DEBUGCTLMSR , debugctl); - debugctl |= (DEBUGCTLMSR_LBR |DEBUGCTLMSR_FREEZE_LBRS_ON_PMI ); + debugctl |=DEBUGCTLMSR_LBR ; +/* +* LBR callstack does not work well with FREEZE_LBRS_ON_PMI. +* If FREEZE_LBRS_ON_PMI is set, PMI near call/return instructions +* may cause superfluous increase/decrease of LBR_TOS. +*/ +if (!(lbr_select &LBR_CALL_STACK )) + debugctl |=DEBUGCTLMSR_FREEZE_LBRS_ON_PMI ;wrmsrl (MSR_IA32_DEBUGCTLMSR , debugctl); }
Signature · 1 line
1.8.3.2
PATCH v8 13/14 perf, x86: Discard zero length call entries in LBR call stack
In reply to Kan LiangFrom: Yan, Zheng <[email protected]>
"Zero length call" uses the attribute of the call instruction to push the immediate instruction pointer on to the stack and then pops off that address into a register. This is accomplished without any matching return instruction. It confuses the hardware and make the recorded call stack incorrect.
We can partially resolve this issue by: decode call instructions and discard any zero length call entry in the LBR stack.
--- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/perf_event_intel_lbr.c +11 -2
@@ -94,7 +94,8 @@ enum {X86_BR_ABORT =1 <<12 ,/* transaction abort */ X86_BR_IN_TX =1 <<13 ,/* in transaction */ X86_BR_NO_TX =1 <<14 ,/* not in transaction */ - X86_BR_CALL_STACK = 1 << 15,/* call stack */ +X86_BR_ZERO_CALL =1 <<15 ,/* zero length call */ + X86_BR_CALL_STACK = 1 << 16,/* call stack */ };#define X86_BR_PLM (X86_BR_USER | X86_BR_KERNEL)
@@ -111,13 +112,15 @@ enum { X86_BR_JMP |\ X86_BR_IRQ |\ X86_BR_ABORT |\ - X86_BR_IND_CALL) + X86_BR_IND_CALL |\ + X86_BR_ZERO_CALL)#define X86_BR_ALL (X86_BR_PLM | X86_BR_ANY)#define X86_BR_ANY_CALL \ (X86_BR_CALL |\ X86_BR_IND_CALL |\ + X86_BR_ZERO_CALL |\ X86_BR_SYSCALL |\ X86_BR_IRQ |\ X86_BR_INT)
@@ -689,6 +692,12 @@ static int branch_type(unsigned long from, unsigned long to, int abort) ret =X86_BR_INT ;break ;case 0xe8 :/* call near rel */ +insn_get_immediate (&insn); +if (insn.immediate1.value ==0 ) { +/* zero length call */ + ret =X86_BR_ZERO_CALL ; +break ; + }case 0x9a :/* call far absolute */ ret =X86_BR_CALL ;break ;
Signature · 1 line
1.8.3.2
PATCH v8 14/14 perf, x86: expose LBR callstack to user space tool
In reply to Kan LiangFrom: Peter Zijlstra (Intel) <[email protected]>
With LBR call stack feature enable, there are three callchain options. Enable the 3rd callchain options LBR callstack to user space tool.
--- arch/x86/kernel/cpu/perf_event.h | 8 -------- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 8 ++++---- include/uapi/linux/perf_event.h | 14 ++++++-------- 3 files changed, 10 insertions(+), 20 deletions(-)
arch/x86/kernel/cpu/perf_event.h +0 -8
@@ -517,14 +517,6 @@ struct x86_perf_task_context {int lbr_stack_state; }; -enum { -PERF_SAMPLE_BRANCH_CALL_STACK_SHIFT =PERF_SAMPLE_BRANCH_MAX_SHIFT , -PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE , - -PERF_SAMPLE_BRANCH_CALL_STACK = -1U <<PERF_SAMPLE_BRANCH_CALL_STACK_SHIFT , -}; -#define x86_add_quirk (func_) \ do { \ static struct x86_pmu_quirk __quirk __initdata = { \
arch/x86/kernel/cpu/perf_event_intel_lbr.c +4 -4
@@ -537,7 +537,7 @@ static int intel_pmu_setup_hw_lbr_filter(struct perf_event *event)u64 mask =0 , v;int i; -for (i =0 ; i <PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE ; i++) { +for (i =0 ; i <PERF_SAMPLE_BRANCH_MAX_SHIFT ; i++) {if (!(br_type & (1ULL << i)))continue ;
@@ -808,7 +808,7 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)/* * Map interface branch filters onto LBR filters */ -static const int nhm_lbr_sel_map[PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE ] = { +static const int nhm_lbr_sel_map[PERF_SAMPLE_BRANCH_MAX_SHIFT ] = { [PERF_SAMPLE_BRANCH_ANY_SHIFT ] =LBR_ANY , [PERF_SAMPLE_BRANCH_USER_SHIFT ] =LBR_USER , [PERF_SAMPLE_BRANCH_KERNEL_SHIFT ] =LBR_KERNEL ,
@@ -827,7 +827,7 @@ static const int nhm_lbr_sel_map[PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE] = { [PERF_SAMPLE_BRANCH_COND_SHIFT] = LBR_JCC, }; -static const int snb_lbr_sel_map[PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE ] = { +static const int snb_lbr_sel_map[PERF_SAMPLE_BRANCH_MAX_SHIFT ] = { [PERF_SAMPLE_BRANCH_ANY_SHIFT ] =LBR_ANY , [PERF_SAMPLE_BRANCH_USER_SHIFT ] =LBR_USER , [PERF_SAMPLE_BRANCH_KERNEL_SHIFT ] =LBR_KERNEL ,
@@ -839,7 +839,7 @@ static const int snb_lbr_sel_map[PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE] = { [PERF_SAMPLE_BRANCH_COND_SHIFT] = LBR_JCC, }; -static const int hsw_lbr_sel_map[PERF_SAMPLE_BRANCH_SELECT_MAP_SIZE ] = { +static const int hsw_lbr_sel_map[PERF_SAMPLE_BRANCH_MAX_SHIFT ] = { [PERF_SAMPLE_BRANCH_ANY_SHIFT ] =LBR_ANY , [PERF_SAMPLE_BRANCH_USER_SHIFT ] =LBR_USER , [PERF_SAMPLE_BRANCH_KERNEL_SHIFT ] =LBR_KERNEL ,
include/uapi/linux/perf_event.h +6 -8
@@ -164,6 +164,7 @@ enum perf_branch_sample_type_shift {PERF_SAMPLE_BRANCH_IN_TX_SHIFT =8 ,/* in transaction */ PERF_SAMPLE_BRANCH_NO_TX_SHIFT =9 ,/* not in transaction */ PERF_SAMPLE_BRANCH_COND_SHIFT =10 ,/* conditional branches */ +PERF_SAMPLE_BRANCH_CALL_STACK_SHIFT =11 ,/* call/ret stack*/ PERF_SAMPLE_BRANCH_MAX_SHIFT /* non-ABI */ };
@@ -174,18 +175,15 @@ enum perf_branch_sample_type {PERF_SAMPLE_BRANCH_HV =1U <<PERF_SAMPLE_BRANCH_HV_SHIFT ,PERF_SAMPLE_BRANCH_ANY =1U <<PERF_SAMPLE_BRANCH_ANY_SHIFT , -PERF_SAMPLE_BRANCH_ANY_CALL = -1U <<PERF_SAMPLE_BRANCH_ANY_CALL_SHIFT , -PERF_SAMPLE_BRANCH_ANY_RETURN = -1U <<PERF_SAMPLE_BRANCH_ANY_RETURN_SHIFT , -PERF_SAMPLE_BRANCH_IND_CALL = -1U <<PERF_SAMPLE_BRANCH_IND_CALL_SHIFT , -PERF_SAMPLE_BRANCH_ABORT_TX = -1U <<PERF_SAMPLE_BRANCH_ABORT_TX_SHIFT , +PERF_SAMPLE_BRANCH_ANY_CALL =1U <<PERF_SAMPLE_BRANCH_ANY_CALL_SHIFT , +PERF_SAMPLE_BRANCH_ANY_RETURN =1U <<PERF_SAMPLE_BRANCH_ANY_RETURN_SHIFT , +PERF_SAMPLE_BRANCH_IND_CALL =1U <<PERF_SAMPLE_BRANCH_IND_CALL_SHIFT , +PERF_SAMPLE_BRANCH_ABORT_TX =1U <<PERF_SAMPLE_BRANCH_ABORT_TX_SHIFT ,PERF_SAMPLE_BRANCH_IN_TX =1U <<PERF_SAMPLE_BRANCH_IN_TX_SHIFT ,PERF_SAMPLE_BRANCH_NO_TX =1U <<PERF_SAMPLE_BRANCH_NO_TX_SHIFT ,PERF_SAMPLE_BRANCH_COND =1U <<PERF_SAMPLE_BRANCH_COND_SHIFT , +PERF_SAMPLE_BRANCH_CALL_STACK =1U <<PERF_SAMPLE_BRANCH_CALL_STACK_SHIFT , PERF_SAMPLE_BRANCH_MAX = 1U << PERF_SAMPLE_BRANCH_MAX_SHIFT, };
Signature · 1 line
1.8.3.2
Peter Zijlstra
maintainerQuoted 16 lines · Kan Liang
Yan, Zheng (13):
perf, x86: Reducelbr_sel_mapsize
perf, core: introduce pmu context switch callback
perf, x86: use context switch callback to flush LBR stack perf, x86: Basic Haswell LBR call stack support
perf, core: pmu specific data for perf task context
perf, core: always switch pmu specific data during context switch perf, x86: allocate space for storing LBR stack
perf, x86: track number of events that use LBR callstack perf, x86: Save/resotre LBR stack during context switch perf, core: simplify need branch stack check
perf, x86: re-organize code that implicitly enables LBR/PEBS perf, x86: disableFREEZE_LBRS_ON_PMIwhen LBR operates in callstack
mode
perf, x86: Discard zero length call entries in LBR call stack
Quoted 2 lines
Peter Zijlstra (Intel) (1):
perf, x86: expose LBR callstack to user space tool
So are these the exact same patches I already had, or did you change anything else? Its not immediately clear, lemme try and figure out.
Quoted 20 lines · Kan Liang
Quoted 16 lines
Yan, Zheng (13):
perf, x86: Reducelbr_sel_mapsize
perf, core: introduce pmu context switch callback
perf, x86: use context switch callback to flush LBR stack perf, x86: Basic Haswell LBR call stack support
perf, core: pmu specific data for perf task context
perf, core: always switch pmu specific data during context switch perf, x86: allocate space for storing LBR stack
perf, x86: track number of events that use LBR callstack perf, x86: Save/resotre LBR stack during context switch perf, core: simplify need branch stack check
perf, x86: re-organize code that implicitly enables LBR/PEBS perf, x86: disableFREEZE_LBRS_ON_PMIwhen LBR operates in callstack
mode
perf, x86: Discard zero length call entries in LBR call stack
Quoted 5 lines
Quoted 2 lines
Peter Zijlstra (Intel) (1):
perf, x86: expose LBR callstack to user space toolSo are these the exact same patches I already had, or did you change anything else? Its not immediately clear, lemme try and figure out.
Yes, Zheng's patches are exact same patches as you already had. I only remove 11, 13,16, 17 from previous patch set and add your new patch (no change). The significant changes are in user tool part, I sent them out as a separate patch set.
Thanks,
Kan
Peter Zijlstra
maintainerQuoted 4 lines · Kan
Yes, Zheng's patches are exact same patches as you already had. I only remove 11, 13,16, 17 from previous patch set and add your new patch (no change).
OK thanks for confirming, I didn't find any changes, but that could equally be because I didn't look ahrd enough ;-)
Quoted 2 lines
The significant changes are in user tool part, I sent them out as a separate patch set.
Right, saw those, picked them up while asking acme/jolsa for an ack.
Thanks!