: perf/core: fix restoring of Intel LBR call stack on a context switch
Subsystems (2)
v3 cover : perf/core: fix restoring of Intel LBR call stack on a context switch
Restore Intel LBR call stack from cloned inactive task perf context on a context switch. This change inherently addresses inconsistency in LBR call stack data provided on a sample in record profiling mode:
$ perf record -N -B -T -R --call-graph lbr \
-e cpu/period=0xcdfe60,event=0x3c,name=\'CPU_CLK_UNHALTED.THREAD\'/Duk \
--clockid=monotonic_raw -- ./miniFE.x nx 25 ny 25 nz 25Let's assume threads A, B, C belonging to the same process.
B and C are siblings of A and their perf contexts are treated as equivalent. At some point B blocks on a futex (non preempt context switch). B's LBRs are preserved at B's perf context task_ctx_data and B's events are removed from PMU and disabled. B's perf context becomes inactive.
Later C gets on a cpu, runs, gets profiled and eventually switches to the awaken but not yet running B. The optimized context switch path is executed swapping B's and C's task_ctx_data pointers at perf event contexts. So C's task_ctx_data will refer preserved B's LBRs on the following switch-in event.
However, as far B's perf context is inactive there is no enabled events in there and B's task_ctx_data->lbr_callstack_users is equal to 0. When B gets on the cpu B's events reviving is skipped following the optimized context switch path and B's task_ctx_data->lbr_callstack_users remains 0. Thus B's LBR's are not restored by pmu sched_task() code called in the end of perf context switch-in callback for B.
In the report that manifests as having short fragments of B's call stack, still tracked by LBR's HW between adjacent samples, but the whole thread call tree doesn't aggregate.
The fix has been evaluated when profiling miniFE [1] (C++, OpenMP) workload running 64 threads on Intel Skylake EP(64 core, 2 sockets):
$ perf report --call-graph callee,flat
5.3.0-rc6+ (tip perf/core) - fixed
- 92.66% 82.64% miniFE.x libiomp5.so [.] _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
- 69.14% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
__kmp_fork_barrier
__kmp_launch_thread
_INTERNAL_24_______src_z_Linux_util_c_3e0095e6::__kmp_launch_worker
start_thread
__clone
- 21.89% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
__kmp_barrier
__kmpc_reduce_nowait
miniFE::cg_solve<miniFE::CSRMatrix<double, int, int>, miniFE::Vector<double, int, int>, miniFE::matvec_std<miniFE::CSRMatrix<double, int, int>, miniFE::Vector<double, int, in
__kmp_invoke_microtask
__kmp_invoke_task_func
__kmp_launch_thread
_INTERNAL_24_______src_z_Linux_util_c_3e0095e6::__kmp_launch_worker
start_thread
__clone
- 1.63% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
__kmp_barrier
__kmpc_reduce_nowait
main
__kmp_invoke_microtask
__kmp_invoke_task_func
__kmp_launch_thread
_INTERNAL_24_______src_z_Linux_util_c_3e0095e6::__kmp_launch_worker
start_thread
__clone5.0.13-300.fc30.x86_64 - no fix
- 90.29% 81.01% miniFE.x libiomp5.so [.] _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
- 33.45% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
__kmp_fork_barrier
__kmp_launch_thread
_INTERNAL_24_______src_z_Linux_util_c_3e0095e6::__kmp_launch_worker
start_thread
__clone
87.63% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
- 54.79% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
__kmp_fork_barrier
__kmp_launch_thread
- 9.18% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
__kmp_barrier
__kmpc_reduce_nowait
miniFE::cg_solve<miniFE::CSRMatrix<double, int, int>, miniFE::Vector<double, int, int>, miniFE::matvec_std<miniFE::CSRMatrix<double, int, int>, miniFE::Vector<double, int, in
__kmp_invoke_microtask
__kmp_invoke_task_func
__kmp_launch_thread
_INTERNAL_24_______src_z_Linux_util_c_3e0095e6::__kmp_launch_worker
start_thread
__clone
- 41.28% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
__kmp_fork_barrier
__kmp_launch_thread
_INTERNAL_24_______src_z_Linux_util_c_3e0095e6::__kmp_launch_worker
- 15.77% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
__kmp_barrier
__kmpc_reduce_nowait
miniFE::cg_solve<miniFE::CSRMatrix<double, int, int>, miniFE::Vector<double, int, int>, miniFE::matvec_std<miniFE::CSRMatrix<double, int, int>, miniFE::Vector<double, int, in
__kmp_invoke_microtask
__kmp_invoke_task_func
__kmp_launch_thread
- 11.56% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
__kmp_barrier
__kmpc_reduce_nowait
miniFE::cg_solve<miniFE::CSRMatrix<double, int, int>, miniFE::Vector<double, int, int>, miniFE::matvec_std<miniFE::CSRMatrix<double, int, int>, miniFE::Vector<double, int, in
__kmp_invoke_microtask
__kmp_invoke_task_func
__kmp_launch_thread
_INTERNAL_24_______src_z_Linux_util_c_3e0095e6::__kmp_launch_worker
- 2.33% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_release
__kmp_barrier
__kmpc_reduce_nowait
main
__kmp_invoke_microtask
__kmp_invoke_task_func
__kmp_launch_thread
_INTERNAL_24_______src_z_Linux_util_c_3e0095e6::__kmp_launch_worker
start_thread
__clone
0.67% _INTERNAL_25_______src_kmp_barrier_cpp_1d20fae8::__kmp_hyper_barrier_gather
0.57% __kmp_hardware_timestamp[1] https://www.hpcadvisorycouncil.com/pdf/miniFE_Analysis_and_Profiling.pdf
---
Alexey Budankov (4):
perf/core,x86: introduce sync_task_ctx() method at struct pmu perf/x86: install platform specific sync_task_ctx adapter perf/x86/intel: implement LBR callstacks context synchronization perf/core,x86: synchronize PMU task contexts on optimized context
switches
arch/x86/events/core.c | 7 +++++++ arch/x86/events/intel/core.c | 7 +++++++ arch/x86/events/intel/lbr.c | 9 +++++++++ arch/x86/events/perf_event.h | 11 +++++++++++ include/linux/perf_event.h | 7 +++++++ kernel/events/core.c | 9 +++++++++ 6 files changed, 50 insertions(+)
---
Changes in v3:
- replaced assignment with swap at intel_pmu_lbr_sync_task_ctx()
Changes in v2:
- implemented sync_task_ctx() method at perf,x86,intel pmu types;
- employed the method on the optimized context switch path between equivalent perf event contexts;
Signature · 1 line
2.20.1
PATCH v3 1/4 perf/core,x86: introduce sync_task_ctx() method at struct pmu
Declare sync_task_ctx() methods at the generic and x86 specific pmu types to bridge calls to platform specific pmu code on optimized context switch path between equivalent task perf event contexts.
--- arch/x86/events/perf_event.h | 8 ++++++++ include/linux/perf_event.h | 7 +++++++ 2 files changed, 15 insertions(+)
arch/x86/events/perf_event.h +8 -0
@@ -682,6 +682,14 @@ struct x86_pmu {*/ atomic_t lbr_exclusive[x86_lbr_exclusive_max]; +/* +* perf task context (i.e. struct perf_event_context::task_ctx_data) switch helper +* to bridge calls from perf/core to perf/x86. See struct pmu::sync_task_ctx() usage +* for examples; +*/ +void (*sync_task_ctx)(struct x86_perf_task_context *one, +struct x86_perf_task_context *another); +/* * AMD bits */
include/linux/perf_event.h +7 -0
@@ -409,6 +409,13 @@ struct pmu {*/ size_t task_ctx_size; +/* +* PMU specific parts of task perf event context (i.e. ctx->task_ctx_data) +* can be synchronized using this function. See Intel LBR callstack support +* implementation and Perf core context switch handling callbacks for usage +* examples. +*/ +void (*sync_task_ctx) (void *one,void *another);/* * Set up pmu-private data structures for an AUX area
Signature · 1 line
2.20.1
Peter Zijlstra
maintainerRe: [PATCH v3 1/4] perf/core,x86: introduce sync_task_ctx() method at struct pmu
include/linux/perf_event.h struct pmu { · line 416
Quoted 13 lines · Alexey Budankov
include/linux/perf_event.h +5
-0@@ -409,6 +409,13 @@ struct pmu {*/ size_t task_ctx_size; +/* +* PMU specific parts of task perf event context (i.e. ctx->task_ctx_data) +* can be synchronized using this function. See Intel LBR callstack support +* implementation and Perf core context switch handling callbacks for usage +* examples.
You're forgetting to mark this: Optional
include/linux/perf_event.h struct pmu { · line 418
Quoted 3 lines
include/linux/perf_event.h +2
-0+*/ +void (*sync_task_ctx) (void *one,void *another);
The traditional argment names for context switching functions are prev and next.
Re: [PATCH v3 1/4] perf/core,x86: introduce sync_task_ctx() method at struct pmu
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Alexey Budankov
include/linux/perf_event.h +0
-0@@ -409,6 +409,13 @@ struct pmu {*/ size_t task_ctx_size;
Quoted 7 lines
include/linux/perf_event.h struct pmu { · line 416
Quoted 5 lines
include/linux/perf_event.h +5
-0+/* +* PMU specific parts of task perf event context (i.e. ctx->task_ctx_data) +* can be synchronized using this function. See Intel LBR callstack support +* implementation and Perf core context switch handling callbacks for usage +* examples. You're forgetting to mark this: Optional
Fixed in v4.
Quoted 7 lines
include/linux/perf_event.h struct pmu { · line 418
Quoted 4 lines
include/linux/perf_event.h +2
-0+*/ +void (*sync_task_ctx) (void *one,void *another);The traditional argment names for context switching functions are prev and next.
Fixed in v4.
Thanks,
Alexey
PATCH v3 2/4 perf/x86: install platform specific sync_task_ctx adapter
In reply to Alexey BudankovBridge perf core and x86 sync_task_ctx() method calls.
--- arch/x86/events/core.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/x86/events/core.c +7 -0
@@ -2243,6 +2243,12 @@ static void x86_pmu_sched_task(struct perf_event_context *ctx, bool sched_in) x86_pmu.sched_task (ctx, sched_in); } +static void x86_pmu_sync_task_ctx (void *one,void *another) +{ +if (x86_pmu.sync_task_ctx) + x86_pmu.sync_task_ctx (one, another); +} +void perf_check_microcode (void ) {if (x86_pmu.check_microcode)
@@ -2297,6 +2303,7 @@ static struct pmu pmu = { .event_idx = x86_pmu_event_idx, .sched_task = x86_pmu_sched_task, .task_ctx_size = sizeof(struct x86_perf_task_context), + .sync_task_ctx = x86_pmu_sync_task_ctx, .check_period = x86_pmu_check_period, .aux_output_match = x86_pmu_aux_output_match,
Signature · 1 line
2.20.1
PATCH v3 3/4 perf/x86/intel: implement LBR callstacks context synchronization
In reply to Alexey BudankovImplement intel_pmu_lbr_sync_task_ctx() method updating counters of the events that requested LBR callstack data on a sample.
The counter can be zero for the case when task context belongs to a thread that has just come from a block on a futex and the context contains saved (lbr_stack_state == LBR_VALID) LBR register values.
For the values to be restored at LBR registers on the next thread's switch-in event it swaps the counter value with the one that is expected to be non zero at the previous equivalent task perf event context.
Swap operation type ensures the previous task perf event context stays consistent with the amount of events that requested LBR callstack data on a sample.
---
Changes in v3:
- replaced assignment with swap at intel_pmu_lbr_sync_task_ctx()
--- arch/x86/events/intel/lbr.c | 9 +++++++++ arch/x86/events/perf_event.h | 3 +++ 2 files changed, 12 insertions(+)
arch/x86/events/intel/lbr.c +9 -0
@@ -417,6 +417,15 @@ static void __intel_pmu_lbr_save(struct x86_perf_task_context *task_ctx) cpuc->last_log_id = ++task_ctx->log_id; } +void intel_pmu_lbr_sync_task_ctx (struct x86_perf_task_context *one, +struct x86_perf_task_context *another) +{ +if (!one || !another) +return ; + +swap (one->lbr_callstack_users, another->lbr_callstack_users); +} +void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
arch/x86/events/perf_event.h +3 -0
@@ -1024,6 +1024,9 @@ void intel_pmu_store_pebs_lbrs(struct pebs_lbr *lbr);void intel_ds_init (void ); +void intel_pmu_lbr_sync_task_ctx (struct x86_perf_task_context *one, +struct x86_perf_task_context *another); +void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in);u64 lbr_from_signext_quirk_wr (u64 val);
Signature · 1 line
2.20.1
PATCH v3 4/4 perf/core,x86: synchronize PMU task contexts on optimized context switches
In reply to Alexey BudankovInstall Intel specific PMU task context synchronization adapter and extend optimized context switch path with PMU specific task context synchronization to fix LBR callstack virtualization on context switches.
--- arch/x86/events/intel/core.c | 7 +++++++ kernel/events/core.c | 9 +++++++++ 2 files changed, 16 insertions(+)
arch/x86/events/intel/core.c +7 -0
@@ -3819,6 +3819,12 @@ static void intel_pmu_sched_task(struct perf_event_context *ctx,intel_pmu_lbr_sched_task (ctx, sched_in); } +static void intel_pmu_sync_task_ctx (struct x86_perf_task_context *one, +struct x86_perf_task_context *another) +{ +intel_pmu_lbr_sync_task_ctx (one, another); +} +static int intel_pmu_check_period (struct perf_event *event,u64 value) {return intel_pmu_has_bts_period (event, value) ? -EINVAL :0 ;
@@ -3954,6 +3960,7 @@ static __initconst const struct x86_pmu intel_pmu = { .guest_get_msrs = intel_guest_get_msrs, .sched_task = intel_pmu_sched_task, + .sync_task_ctx = intel_pmu_sync_task_ctx, .check_period = intel_pmu_check_period,
kernel/events/core.c +9 -0
@@ -3204,11 +3204,20 @@ static void perf_event_context_sched_out(struct task_struct *task, int ctxn,raw_spin_lock (&ctx->lock);raw_spin_lock_nested (&next_ctx->lock,SINGLE_DEPTH_NESTING );if (context_equiv (ctx, next_ctx)) { +struct pmu *pmu = ctx->pmu; +WRITE_ONCE (ctx->task, next);WRITE_ONCE (next_ctx->task, task);swap (ctx->task_ctx_data, next_ctx->task_ctx_data); +/* +* PMU specific parts of task perf context may require +* additional synchronization, at least for proper Intel +* LBR callstack data profiling; +*/ + pmu->sync_task_ctx (ctx->task_ctx_data, + next_ctx->task_ctx_data);/* * RCU_INIT_POINTER here is safe because we've not * modified the ctx and the above modification of
Signature · 1 line
2.20.1
Re: [PATCH v3 4/4] perf/core,x86: synchronize PMU task contexts on optimized context switches
kernel/events/core.c static void perf_event_context_sched_out(struct task_struct *task, int ctxn, · line 3220
Quoted 7 lines · Alexey Budankov
kernel/events/core.c +7
-0+/* +* PMU specific parts of task perf context may require +* additional synchronization, at least for proper Intel +* LBR callstack data profiling; +*/ + pmu->sync_task_ctx (ctx->task_ctx_data, + next_ctx->task_ctx_data);
Firstly, I'm pretty sure you never run this on a CPU where pmu->sync_task_ctx is NULL, right? ;-)
Secondly, even on Intel CPUs in many cases we'll just call into a ~2 deep function pointer based call hierarchy, just to find that nothing needs to be done, because there's no LBR call stack maintained:
+ if (!one || !another)
+ return;
So while it's technically a layering violation, it might make sense to elevate this check to the generic layer and say that synchronization calls by the core layer will always provide two valid pointers?
Thanks,
Ingo
Re: [PATCH v3 4/4] perf/core,x86: synchronize PMU task contexts on optimized context switches
Quoted 13 lines · Ingo Molnar
kernel/events/core.c static void perf_event_context_sched_out(struct task_struct *task, int ctxn, · line 3220
Quoted 7 lines · Alexey Budankov
kernel/events/core.c +7
-0+/* +* PMU specific parts of task perf context may require +* additional synchronization, at least for proper Intel +* LBR callstack data profiling; +*/ + pmu->sync_task_ctx (ctx->task_ctx_data, + next_ctx->task_ctx_data);Firstly, I'm pretty sure you never run this on a CPU where
pmu->sync_task_ctxis NULL, right? ;-)
Yes, right.
Quoted 11 lines
Secondly, even on Intel CPUs in many cases we'll just call into a ~2 deep function pointer based call hierarchy, just to find that nothing needs to be done, because there's no LBR call stack maintained:
(quoted diff) +2
-0+ if (!one || !another) + return;So while it's technically a layering violation, it might make sense to elevate this check to the generic layer and say that synchronization calls by the core layer will always provide two valid pointers?
This would also keep performance benefit of avoiding double indirect pointer calls for cases when LBR callstack is not requested.
As far it has the comment saying that check is intentionally placed at the core layer it doesn't look like a violation but rather performance optimization.
Let me come up with v4 to see how it would look like.
Thanks for observation,
Alexey
Quoted 5 lines
Thanks,
Ingo
Peter Zijlstra
maintainerRe: [PATCH v3 4/4] perf/core,x86: synchronize PMU task contexts on optimized context switches
In reply to Ingo MolnarQuoted 14 lines · Ingo Molnar
kernel/events/core.c static void perf_event_context_sched_out(struct task_struct *task, int ctxn, · line 3220
Quoted 7 lines · Alexey Budankov
kernel/events/core.c +7
-0+/* +* PMU specific parts of task perf context may require +* additional synchronization, at least for proper Intel +* LBR callstack data profiling; +*/ + pmu->sync_task_ctx (ctx->task_ctx_data, + next_ctx->task_ctx_data);Firstly, I'm pretty sure you never run this on a CPU where
pmu->sync_task_ctxis NULL, right? ;-)
Quoted 2 lines
Secondly, even on Intel CPUs in many cases we'll just call into a ~2 deep function pointer based call hierarchy, just to find that nothing needs to
See prototype here for getting rid of at least one layer of indirect calls:
https://lkml.kernel.org/r/[email protected]
Quoted 8 lines
be done, because there's no LBR call stack maintained:
(quoted diff) +2
-0+ if (!one || !another) + return;So while it's technically a layering violation, it might make sense to elevate this check to the generic layer and say that synchronization calls by the core layer will always provide two valid pointers?
Alternatively we can write the thing like:
if (pmu->swap_task_ctx) pmu->swap_task_ctx(ctx, next_ctx) else swap(ctx->task_ctx_data, next_ctx->task_ctx_data);