perf: Rewrite core context handling
Subsystems (7)
PATCH v2 1/1 perf: Rewrite core context handling
From: Peter Zijlstra <[email protected]>
There have been various issues and limitations with the way perf uses (task) contexts to track events. Most notable is the single hardware PMU task context, which has resulted in a number of yucky things (both proposed and merged).
Notably:
- HW breakpoint PMU
- ARM big.little PMU / Intel ADL PMU
- Intel Branch Monitoring PMU
- AMD IBS PMU
- S390 cpum_cf PMU
- PowerPC trace_imc PMU
*Current design:*
Currently we have a per task and per cpu perf_event_contexts:
task_struct::perf_events_ctxp[] <-> perf_event_context <-> perf_cpu_context
^ | ^ | ^
`---------------------------------' | `--> pmu ---'
v ^
perf_event ------'Each task has an array of pointers to a perf_event_context. Each perf_event_context has a direct relation to a PMU and a group of events for that PMU. The task related perf_event_context's have a pointer back to that task.
Each PMU has a per-cpu pointer to a per-cpu perf_cpu_context, which includes a perf_event_context, which again has a direct relation to that PMU, and a group of events for that PMU.
The perf_cpu_context also tracks which task context is currently associated with that CPU and includes a few other things like the hrtimer for rotation etc.
Each perf_event is then associated with its PMU and one perf_event_context.
*Proposed design:*
New design proposed by this patch reduce to a single task context and a single CPU context but adds some intermediate data-structures:
task_struct::perf_event_ctxp -> perf_event_context <- perf_cpu_context
^ | ^ ^
`---------------------------' | |
| | perf_cpu_pmu_context <--.
| `----. ^ |
| | | |
| v v |
| ,--> perf_event_pmu_context |
| | |
| | | v v |
perf_event ---> pmu ----------------'
With new design, perf_event_context will hold all pmu events in the respective(pinned/flexible) rbtrees. This can be achieved by adding pmu to rbtree key:
{cpu, pmu, cgroup, group_index}
Each perf_event_context carry a list of perf_event_pmu_context which is used to hold per-pmu-per-context state. For ex, it keeps track of currently active events for that pmu, a pmu specific task_ctx_data, a flag to tell whether rotation is required or not etc.
Similarly, perf_cpu_pmu_context is used to hold per-pmu-per-cpu state like hrtimer details to drive the event rotation, a pointer to perf_event_pmu_context of currently running task and some other ancillary information.
Each perf_event is associated to it's pmu, perf_event_context and perf_event_pmu_context.
*Tests:*
With this, perf test and perf_event_test runs fine without any issues on Intel and AMD machines. However, one WARN_ON() was reported on Intel machine[1] with perf fuzzer. Note that, so far I've run these tests to verify stability of the change only, not functional correctness. Also, some of the known issues marked with XXX needs to be resolved.
Further optimizations to current implementation are possible. For ex, ctx_resched() can be optimized to reschedule only single pmu events. ctx_sched_in() can be optimized to use pmu=NULL while scheduling entire context. However, this will lead to significant changes to rbtree and minheap layouts.
[1]: https://lore.kernel.org/lkml/[email protected]
---
v1: https://lore.kernel.org/lkml/[email protected]v1->v2:
- Rebase to tip/perf/core (82aad7ff7ac25)
- Make perf_event_context->refcount as ctx <-> epc instead of ctx <-> event. This change is mostly copy pasted from Peter's add on patch from v1, however, that patch missed to migrate sibling events at cpu offline time which is fixed in this version.
- Use rbtree instead of event list while iterating over tracepoint pmu events in perf_tp_event().
- Bail out and return -EBUSY if flexible event scheduling fails in merge_sched_in().
arch/arm64/kernel/perf_event.c | 24 +- arch/powerpc/perf/core-book3s.c | 8 +- arch/s390/kernel/perf_pai_crypto.c | 2 +- arch/x86/events/amd/brs.c | 2 +- arch/x86/events/amd/lbr.c | 6 +- arch/x86/events/core.c | 44 +- arch/x86/events/intel/core.c | 23 +- arch/x86/events/intel/ds.c | 4 +- arch/x86/events/intel/lbr.c | 30 +- arch/x86/events/perf_event.h | 30 +- drivers/perf/arm_pmu.c | 11 +- include/linux/perf/arm_pmu.h | 2 +- include/linux/perf_event.h | 115 +- include/linux/sched.h | 2 +- kernel/events/core.c | 2071 ++++++++++++++-------------- 15 files changed, 1227 insertions(+), 1147 deletions(-)
arch/arm64/kernel/perf_event.c +17 -7
@@ -806,10 +806,20 @@ static void armv8pmu_disable_event(struct perf_event *event)static void armv8pmu_start (struct arm_pmu *cpu_pmu) { -struct perf_event_context *task_ctx = -this_cpu_ptr (cpu_pmu->pmu.pmu_cpu_context)->task_ctx; +/* XXX: Get this verified by Arm folks */ +struct perf_event_context *ctx; +int nr_user =0 ; + +rcu_read_lock (); + ctx =rcu_dereference (current->perf_event_ctxp); +if (ctx) { +raw_spin_lock (&ctx->lock); + nr_user = ctx->nr_user; +raw_spin_unlock (&ctx->lock); + } +rcu_read_unlock (); -if (sysctl_perf_user_access && task_ctx && task_ctx->nr_user) +if (sysctl_perf_user_access && nr_user)armv8pmu_enable_user_access (cpu_pmu);else armv8pmu_disable_user_access ();
@@ -1019,10 +1029,10 @@ static int armv8pmu_set_event_filter(struct hw_perf_event *event,return 0 ; } -static int armv8pmu_filter_match (struct perf_event *event) +static bool armv8pmu_filter (struct pmu *pmu,int cpu) { -unsigned long evtype = event->hw.config_base &ARMV8_PMU_EVTYPE_EVENT ; -return evtype !=ARMV8_PMUV3_PERFCTR_CHAIN ; +struct arm_pmu *armpmu =to_arm_pmu (pmu); +return !cpumask_test_cpu (smp_processor_id (), &armpmu->supported_cpus); }static void armv8pmu_reset (void *info)
@@ -1253,7 +1263,7 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu, char *name, cpu_pmu->stop = armv8pmu_stop; cpu_pmu->reset = armv8pmu_reset; cpu_pmu->set_event_filter = armv8pmu_set_event_filter; - cpu_pmu->filter_match = armv8pmu_filter_match; + cpu_pmu->filter = armv8pmu_filter; cpu_pmu->pmu.event_idx = armv8pmu_user_event_idx;
arch/powerpc/perf/core-book3s.c +4 -4
@@ -132,7 +132,7 @@ static unsigned long ebb_switch_in(bool ebb, struct cpu_hw_events *cpuhw)static inline void power_pmu_bhrb_enable (struct perf_event *event) {}static inline void power_pmu_bhrb_disable (struct perf_event *event) {} -static void power_pmu_sched_task (struct perf_event_context *ctx,bool sched_in) {} +static void power_pmu_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {}static inline void power_pmu_bhrb_read (struct perf_event *event,struct cpu_hw_events *cpuhw) {}static void pmao_restore_workaround (bool ebb) { }#endif /* CONFIG_PPC32 */
@@ -424,7 +424,7 @@ static void power_pmu_bhrb_enable(struct perf_event *event) cpuhw->bhrb_context = event->ctx; } cpuhw->bhrb_users++; -perf_sched_cb_inc (event->ctx->pmu); +perf_sched_cb_inc (event->pmu); }static void power_pmu_bhrb_disable (struct perf_event *event)
@@ -436,7 +436,7 @@ static void power_pmu_bhrb_disable(struct perf_event *event)WARN_ON_ONCE (!cpuhw->bhrb_users); cpuhw->bhrb_users--; -perf_sched_cb_dec (event->ctx->pmu); +perf_sched_cb_dec (event->pmu);if (!cpuhw->disabled && !cpuhw->bhrb_users) {/* BHRB cannot be turned off when other
@@ -451,7 +451,7 @@ static void power_pmu_bhrb_disable(struct perf_event *event)/* Called from ctxsw to prevent one process's branch entries to * mingle with the other process's entries during context switch. */ -static void power_pmu_sched_task (struct perf_event_context *ctx,bool sched_in) +static void power_pmu_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {if (!ppmu->bhrb_nr)return ;
arch/s390/kernel/perf_pai_crypto.c +1 -1
@@ -379,7 +379,7 @@ static int paicrypt_push_sample(void)/* Called on schedule-in and schedule-out. No access to event structure, * but for sampling only event CRYPTO_ALL is allowed. */ -static void paicrypt_sched_task (struct perf_event_context *ctx,bool sched_in) +static void paicrypt_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {/* We started with a clean page on event installation. So read out * results on schedule_out and if page was dirty, clear values.
arch/x86/events/amd/brs.c +1 -1
@@ -384,7 +384,7 @@ static void amd_brs_poison_buffer(void)* On ctxswin, sched_in = true, called after the PMU has started * On ctxswout, sched_in = false, called before the PMU is stopped */ -void amd_pmu_brs_sched_task (struct perf_event_context *ctx,bool sched_in) +void amd_pmu_brs_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
arch/x86/events/amd/lbr.c +3 -3
@@ -352,7 +352,7 @@ void amd_pmu_lbr_add(struct perf_event *event) cpuc->br_sel = reg->reg; } -perf_sched_cb_inc (event->ctx->pmu); +perf_sched_cb_inc (event->pmu);if (!cpuc->lbr_users++ && !event->total_time_running)amd_pmu_lbr_reset ();
@@ -370,10 +370,10 @@ void amd_pmu_lbr_del(struct perf_event *event) cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 ); -perf_sched_cb_dec (event->ctx->pmu); +perf_sched_cb_dec (event->pmu); } -void amd_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in) +void amd_pmu_lbr_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
arch/x86/events/core.c +14 -30
92 lines not on this page · raw message
arch/x86/events/intel/core.c +10 -13
@@ -4527,8 +4527,6 @@ static bool init_hybrid_pmu(int cpu)cpumask_set_cpu (cpu, &pmu->supported_cpus); cpuc->pmu = &pmu->pmu; -x86_pmu_update_cpu_context (&pmu->pmu, cpu); -return true; }
@@ -4662,17 +4660,17 @@ static void intel_pmu_cpu_dead(int cpu)cpumask_clear_cpu (cpu, &hybrid_pmu (cpuc->pmu)->supported_cpus); } -static void intel_pmu_sched_task (struct perf_event_context *ctx, +static void intel_pmu_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) { -intel_pmu_pebs_sched_task (ctx, sched_in); -intel_pmu_lbr_sched_task (ctx, sched_in); +intel_pmu_pebs_sched_task (pmu_ctx, sched_in); +intel_pmu_lbr_sched_task (pmu_ctx, sched_in); } -static void intel_pmu_swap_task_ctx (struct perf_event_context *prev, -struct perf_event_context *next) +static void intel_pmu_swap_task_ctx (struct perf_event_pmu_context *prev_epc, +struct perf_event_pmu_context *next_epc) { -intel_pmu_lbr_swap_task_ctx (prev, next); +intel_pmu_lbr_swap_task_ctx (prev_epc, next_epc); }static int intel_pmu_check_period (struct perf_event *event,u64 value)
@@ -4696,12 +4694,11 @@ static int intel_pmu_aux_output_match(struct perf_event *event)return is_intel_pt_event (event); } -static int intel_pmu_filter_match (struct perf_event *event) +static void intel_pmu_filter (struct pmu *pmu,int cpu,bool *ret) { -struct x86_hybrid_pmu *pmu =hybrid_pmu (event->pmu); -unsigned int cpu =smp_processor_id (); +struct x86_hybrid_pmu *hpmu =hybrid_pmu (pmu); -return cpumask_test_cpu (cpu, &pmu->supported_cpus); + *ret = !cpumask_test_cpu (cpu, &hpmu->supported_cpus); }PMU_FORMAT_ATTR (offcore_rsp,"config1:0-63" );
@@ -6374,7 +6371,7 @@ __init int intel_pmu_init(void)static_call_update (intel_pmu_set_topdown_event_period, &adl_set_topdown_event_period); - x86_pmu.filter_match = intel_pmu_filter_match; + x86_pmu.filter = intel_pmu_filter; x86_pmu.get_event_constraints = adl_get_event_constraints; x86_pmu.hw_config = adl_hw_config; x86_pmu.limit_period = spr_limit_period;
arch/x86/events/intel/ds.c +2 -2
@@ -1054,7 +1054,7 @@ static inline bool pebs_needs_sched_cb(struct cpu_hw_events *cpuc)return cpuc->n_pebs && (cpuc->n_pebs == cpuc->n_large_pebs); } -void intel_pmu_pebs_sched_task (struct perf_event_context *ctx,bool sched_in) +void intel_pmu_pebs_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -1162,7 +1162,7 @@ static void pebs_update_state(bool needed_cb, struct cpu_hw_events *cpuc, struct perf_event *event, bool add) { - struct pmu *pmu = event->ctx->pmu; + struct pmu *pmu = event->pmu;/* * Make sure we get updated with the first PEBS * event. It will trigger also during removal, but
arch/x86/events/intel/lbr.c +15 -15
@@ -515,21 +515,21 @@ static void __intel_pmu_lbr_save(void *ctx) cpuc->last_log_id = ++task_context_opt (ctx)->log_id; } -void intel_pmu_lbr_swap_task_ctx (struct perf_event_context *prev, -struct perf_event_context *next) +void intel_pmu_lbr_swap_task_ctx (struct perf_event_pmu_context *prev_epc, +struct perf_event_pmu_context *next_epc) {void *prev_ctx_data, *next_ctx_data; -swap (prev->task_ctx_data, next->task_ctx_data); +swap (prev_epc->task_ctx_data, next_epc->task_ctx_data);/* -* Architecture specific synchronization makes sense in -* case both prev->task_ctx_data and next->task_ctx_data +* Architecture specific synchronization makes sense in case +* both prev_epc->task_ctx_data and next_epc->task_ctx_data * pointers are allocated. */ - prev_ctx_data = next->task_ctx_data; - next_ctx_data = prev->task_ctx_data; + prev_ctx_data = next_epc->task_ctx_data; + next_ctx_data = prev_epc->task_ctx_data;if (!prev_ctx_data || !next_ctx_data)return ;
@@ -538,7 +538,7 @@ void intel_pmu_lbr_swap_task_ctx(struct perf_event_context *prev, task_context_opt(next_ctx_data)->lbr_callstack_users); } -void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in) +void intel_pmu_lbr_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);void *task_ctx;
@@ -551,7 +551,7 @@ void intel_pmu_lbr_sched_task(struct perf_event_context *ctx, bool sched_in)* the task was scheduled out, restore the stack. Otherwise flush * the LBR stack. */ - task_ctx = ctx ? ctx->task_ctx_data :NULL ; + task_ctx = pmu_ctx ? pmu_ctx->task_ctx_data :NULL ;if (task_ctx) {if (sched_in)__intel_pmu_lbr_restore (task_ctx);
@@ -587,8 +587,8 @@ void intel_pmu_lbr_add(struct perf_event *event) cpuc->br_sel = event->hw.branch_reg.reg; -if (branch_user_callstack (cpuc->br_sel) && event->ctx->task_ctx_data) -task_context_opt (event->ctx->task_ctx_data)->lbr_callstack_users++; +if (branch_user_callstack (cpuc->br_sel) && event->pmu_ctx->task_ctx_data) +task_context_opt (event->pmu_ctx->task_ctx_data)->lbr_callstack_users++;/* * Request pmu::sched_task() callback, which will fire inside the
@@ -611,7 +611,7 @@ void intel_pmu_lbr_add(struct perf_event *event)*/ if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 ) cpuc->lbr_pebs_users++; -perf_sched_cb_inc (event->ctx->pmu); +perf_sched_cb_inc (event->pmu);if (!cpuc->lbr_users++ && !event->total_time_running)intel_pmu_lbr_reset (); }
@@ -664,8 +664,8 @@ void intel_pmu_lbr_del(struct perf_event *event)return ;if (branch_user_callstack (cpuc->br_sel) && - event->ctx->task_ctx_data) -task_context_opt (event->ctx->task_ctx_data)->lbr_callstack_users--; + event->pmu_ctx->task_ctx_data) +task_context_opt (event->pmu_ctx->task_ctx_data)->lbr_callstack_users--;if (event->hw.flags &PERF_X86_EVENT_LBR_SELECT ) cpuc->lbr_select =0 ;
@@ -675,7 +675,7 @@ void intel_pmu_lbr_del(struct perf_event *event) cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 );WARN_ON_ONCE (cpuc->lbr_pebs_users <0 ); -perf_sched_cb_dec (event->ctx->pmu); +perf_sched_cb_dec (event->pmu); }static inline bool vlbr_exclude_host (void )
arch/x86/events/perf_event.h +14 -16
@@ -811,7 +811,7 @@ struct x86_pmu {void (*cpu_dead)(int cpu);void (*check_microcode)(void ); -void (*sched_task)(struct perf_event_context *ctx, +void (*sched_task)(struct perf_event_pmu_context *pmu_ctx,bool sched_in);/*
@@ -894,12 +894,12 @@ struct x86_pmu {int num_topdown_events;/* -* perf task context (i.e. struct perf_event_context::task_ctx_data) +* perf task context (i.e. struct perf_event_pmu_context::task_ctx_data) * switch helper to bridge calls from perf/core to perf/x86. * See struct pmu::swap_task_ctx() usage for examples; */ -void (*swap_task_ctx)(struct perf_event_context *prev, -struct perf_event_context *next); +void (*swap_task_ctx)(struct perf_event_pmu_context *prev_epc, +struct perf_event_pmu_context *next_epc);/* * AMD bits
@@ -925,7 +925,7 @@ struct x86_pmu {int (*aux_output_match) (struct perf_event *event); -int (*filter_match)(struct perf_event *event); +void (*filter)(struct pmu *pmu,int cpu,bool *ret);/* * Hybrid support *
@@ -1180,8 +1180,6 @@ int x86_pmu_handle_irq(struct pt_regs *regs);void x86_pmu_show_pmu_cap (int num_counters,int num_counters_fixed,u64 intel_ctrl); -void x86_pmu_update_cpu_context (struct pmu *pmu,int cpu); -extern struct event_constraint emptyconstraint;extern struct event_constraint unconstrained;
@@ -1306,7 +1304,7 @@ void amd_pmu_lbr_reset(void);void amd_pmu_lbr_read (void );void amd_pmu_lbr_add (struct perf_event *event);void amd_pmu_lbr_del (struct perf_event *event); -void amd_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in); +void amd_pmu_lbr_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in);void amd_pmu_lbr_enable_all (void );void amd_pmu_lbr_disable_all (void );int amd_pmu_lbr_hw_config (struct perf_event *event);
@@ -1330,7 +1328,7 @@ static inline void amd_pmu_brs_add(struct perf_event *event) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -perf_sched_cb_inc (event->ctx->pmu); +perf_sched_cb_inc (event->pmu); cpuc->lbr_users++;/* * No need to reset BRS because it is reset
@@ -1345,10 +1343,10 @@ static inline void amd_pmu_brs_del(struct perf_event *event) cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 ); - perf_sched_cb_dec(event->ctx->pmu); + perf_sched_cb_dec(event->pmu); } -void amd_pmu_brs_sched_task (struct perf_event_context *ctx,bool sched_in); +void amd_pmu_brs_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in);#else static inline int amd_brs_init (void ) {
@@ -1373,7 +1371,7 @@ static inline void amd_pmu_brs_del(struct perf_event *event) { } -static inline void amd_pmu_brs_sched_task (struct perf_event_context *ctx,bool sched_in) +static inline void amd_pmu_brs_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) { }
@@ -1533,7 +1531,7 @@ void intel_pmu_pebs_enable_all(void);void intel_pmu_pebs_disable_all (void ); -void intel_pmu_pebs_sched_task (struct perf_event_context *ctx,bool sched_in); +void intel_pmu_pebs_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in);void intel_pmu_auto_reload_read (struct perf_event *event);
@@ -1541,10 +1539,10 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);void intel_ds_init (void ); -void intel_pmu_lbr_swap_task_ctx (struct perf_event_context *prev, -struct perf_event_context *next); +void intel_pmu_lbr_swap_task_ctx (struct perf_event_pmu_context *prev_epc, +struct perf_event_pmu_context *next_epc); -void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in); +void intel_pmu_lbr_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in);u64 lbr_from_signext_quirk_wr (u64 val);
drivers/perf/arm_pmu.c +5 -6
@@ -550,15 +550,15 @@ static void armpmu_disable(struct pmu *pmu)* microarchitecture, and aren't suitable for another. Thus, only match CPUs of * the same microarchitecture. */ -static int armpmu_filter_match (struct perf_event *event) +static int armpmu_filter (struct perf_event *event) {struct arm_pmu *armpmu =to_arm_pmu (event->pmu);unsigned int cpu =smp_processor_id ();int ret; ret =cpumask_test_cpu (cpu, &armpmu->supported_cpus); -if (ret && armpmu->filter_match) -return armpmu->filter_match (event); +if (ret && armpmu->filter) +return armpmu->filter (event->pmu, cpu);return ret; }
@@ -885,14 +885,13 @@ static struct arm_pmu *__armpmu_alloc(gfp_t flags) .start = armpmu_start, .stop = armpmu_stop, .read = armpmu_read, - .filter_match = armpmu_filter_match, + .filter = armpmu_filter, .attr_groups = pmu->attr_groups,/* * This is a CPU PMU potentially in a heterogeneous * configuration (e.g. big.LITTLE). This is not an uncore PMU, * and we have taken ctx sharing into account (e.g. with our -* pmu::filter_match callback and pmu::event_init group -* validation). +* pmu::filter callback and pmu::event_init group validation). */ .capabilities = PERF_PMU_CAP_HETEROGENEOUS_CPUS, };
include/linux/perf/arm_pmu.h +1 -1
@@ -100,7 +100,7 @@ struct arm_pmu {void (*stop)(struct arm_pmu *);void (*reset)(void *);int (*map_event)(struct perf_event *event); -int (*filter_match)(struct perf_event *event); +bool (*filter)(struct pmu *pmu,int cpu);int num_events;bool secure_access;/* 32-bit ARM only */ #define ARMV8_PMUV3_MAX_COMMON_EVENTS 0x40
include/linux/perf_event.h +84 -31
221 lines not on this page · raw message
include/linux/sched.h +1 -1
@@ -1228,7 +1228,7 @@ struct task_struct {unsigned int futex_state;#endif #ifdef CONFIG_PERF_EVENTS -struct perf_event_context *perf_event_ctxp[perf_nr_task_contexts]; +struct perf_event_context *perf_event_ctxp;struct mutex perf_event_mutex;struct list_head perf_event_list;#endif
kernel/events/core.c +1055 -1016
2,279 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.37.3
Part of the diff is not shown here. View the raw message.
Peter Zijlstra
maintainerQuoted 20 lines · Ravi Bangoria
kernel/events/core.c +20
-0+static void perf_event_swap_task_ctx_data (struct perf_event_context *prev_ctx, +struct perf_event_context *next_ctx) +{ +struct perf_event_pmu_context *prev_epc, *next_epc; + +if (!prev_ctx->nr_task_data) +return ; + + prev_epc =list_first_entry (&prev_ctx->pmu_ctx_list, + struct perf_event_pmu_context, + pmu_ctx_entry); + next_epc =list_first_entry (&next_ctx->pmu_ctx_list, + struct perf_event_pmu_context, + pmu_ctx_entry); + +while (&prev_epc->pmu_ctx_entry != &prev_ctx->pmu_ctx_list && + &next_epc->pmu_ctx_entry != &next_ctx->pmu_ctx_list) { + +WARN_ON_ONCE (prev_epc->pmu != next_epc->pmu); +
kernel/events/core.c static void perf_event_sync_stat(struct perf_event_context *ctx, · line 3383
Quoted 10 lines
kernel/events/core.c +10
-0+/* +* PMU specific parts of task perf context can require +* additional synchronization. As an example of such +* synchronization see implementation details of Intel +* LBR call stack data profiling; +*/ +if (prev_epc->pmu->swap_task_ctx) + prev_epc->pmu->swap_task_ctx (prev_epc, next_epc); +else +swap (prev_epc->task_ctx_data, next_epc->task_ctx_data);
Did I forget to advance the iterators here?
Quoted 2 lines
(quoted diff) +2
-0+ } +}
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · Ravi Bangoria
kernel/events/core.c +24
-0+static void perf_event_swap_task_ctx_data (struct perf_event_context *prev_ctx, +struct perf_event_context *next_ctx) +{ +struct perf_event_pmu_context *prev_epc, *next_epc; + +if (!prev_ctx->nr_task_data) +return ; + + prev_epc =list_first_entry (&prev_ctx->pmu_ctx_list, + struct perf_event_pmu_context, + pmu_ctx_entry); + next_epc =list_first_entry (&next_ctx->pmu_ctx_list, + struct perf_event_pmu_context, + pmu_ctx_entry); + +while (&prev_epc->pmu_ctx_entry != &prev_ctx->pmu_ctx_list && + &next_epc->pmu_ctx_entry != &next_ctx->pmu_ctx_list) { + +WARN_ON_ONCE (prev_epc->pmu != next_epc->pmu); + +/* +* PMU specific parts of task perf context can require +* additional synchronization. As an example of such +* synchronization see implementation details of Intel
Quoted 8 lines
kernel/events/core.c static void perf_event_sync_stat(struct perf_event_context *ctx, · line 3383
Quoted 6 lines
kernel/events/core.c +6
-0+* LBR call stack data profiling; +*/ +if (prev_epc->pmu->swap_task_ctx) + prev_epc->pmu->swap_task_ctx (prev_epc, next_epc); +else +swap (prev_epc->task_ctx_data, next_epc->task_ctx_data);Did I forget to advance the iterators here?
Yeah. Seems so. I overlooked it too.
Thanks,
Ravi
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Peter Zijlstra
maintainerQuoted 36 lines · Ravi Bangoria
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · Ravi Bangoria
kernel/events/core.c +24
-0+static void perf_event_swap_task_ctx_data (struct perf_event_context *prev_ctx, +struct perf_event_context *next_ctx) +{ +struct perf_event_pmu_context *prev_epc, *next_epc; + +if (!prev_ctx->nr_task_data) +return ; + + prev_epc =list_first_entry (&prev_ctx->pmu_ctx_list, + struct perf_event_pmu_context, + pmu_ctx_entry); + next_epc =list_first_entry (&next_ctx->pmu_ctx_list, + struct perf_event_pmu_context, + pmu_ctx_entry); + +while (&prev_epc->pmu_ctx_entry != &prev_ctx->pmu_ctx_list && + &next_epc->pmu_ctx_entry != &next_ctx->pmu_ctx_list) { + +WARN_ON_ONCE (prev_epc->pmu != next_epc->pmu); + +/* +* PMU specific parts of task perf context can require +* additional synchronization. As an example of such +* synchronization see implementation details of Intel Quoted 8 lines
kernel/events/core.c static void perf_event_sync_stat(struct perf_event_context *ctx, · line 3383
Quoted 6 lines
kernel/events/core.c +6
-0+* LBR call stack data profiling; +*/ +if (prev_epc->pmu->swap_task_ctx) + prev_epc->pmu->swap_task_ctx (prev_epc, next_epc); +else +swap (prev_epc->task_ctx_data, next_epc->task_ctx_data);Did I forget to advance the iterators here?
Quoted 1 line
Yeah. Seems so. I overlooked it too.
OK; so I'm not slowly going crazy staring at this code ;-) Let me go add it now then. :-)
But first I gotta taxi the kids around for a bit, bbl.
Peter Zijlstra
maintainerQuoted 37 lines · Peter Zijlstra
Quoted 36 lines · Ravi Bangoria
Quoted 34 lines · Peter Zijlstra
Quoted 20 lines · Ravi Bangoria
(quoted diff) +20
-0+static void perf_event_swap_task_ctx_data(struct perf_event_context *prev_ctx, + struct perf_event_context *next_ctx) +{ + struct perf_event_pmu_context *prev_epc, *next_epc; + + if (!prev_ctx->nr_task_data) + return; + + prev_epc = list_first_entry(&prev_ctx->pmu_ctx_list, + struct perf_event_pmu_context, + pmu_ctx_entry); + next_epc = list_first_entry(&next_ctx->pmu_ctx_list, + struct perf_event_pmu_context, + pmu_ctx_entry); + + while (&prev_epc->pmu_ctx_entry != &prev_ctx->pmu_ctx_list && + &next_epc->pmu_ctx_entry != &next_ctx->pmu_ctx_list) { + + WARN_ON_ONCE(prev_epc->pmu != next_epc->pmu); +Review
Quoted 10 lines
(quoted diff) +10
-0+ /* + * PMU specific parts of task perf context can require + * additional synchronization. As an example of such + * synchronization see implementation details of Intel + * LBR call stack data profiling; + */ + if (prev_epc->pmu->swap_task_ctx) + prev_epc->pmu->swap_task_ctx(prev_epc, next_epc); + else + swap(prev_epc->task_ctx_data, next_epc->task_ctx_data);Did I forget to advance the iterators here?
Quoted 6 lines
Quoted 1 line
Yeah. Seems so. I overlooked it too.
OK; so I'm not slowly going crazy staring at this code ;-) Let me go add it now then. :-)
But first I gotta taxi the kids around for a bit, bbl.
OK, so I've been going over the perf_event_pmu_context life-time thing as well, there were a bunch of XXXs there and I'm not sure Im happy with things, but I'd also forgotten most of it.
Ideally epc works like it's a regular member of ctx -- locking wise that is, but I'm not sure we can make that stick -- see the ctx->mutex issues we have with put_ctx().
As such, I'm going to have to re-audit all the epc usage to see if pure ctx->lock is sufficient.
I did do make epc RCU freed, because pretty much everything is and that was easy enough to make happen -- it means we don't need to worry about that.
But I'm going cross-eyes from staring at this all day, so more tomorrow. The below is what I currently have.
---
include/linux/perf_event.h +6 -5
@@ -833,13 +833,13 @@ struct perf_event {* `--------[1:n]---------' `-[n:1]-> pmu <-[1:n]-' * * -* XXX destroy epc when empty -* refcount, !rcu +* epc lifetime is refcount based and RCU freed (similar to perf_event_context). +* epc locking is as if it were a member of perf_event_context; specifically: * -* XXX epc locking +* modification, both: ctx->mutex && ctx->lock +* reading, either: ctx->mutex || ctx->lock * -* event->pmu_ctx ctx->mutex && inactive -* ctx->pmu_ctx_list ctx->mutex && ctx->lock +* XXX except this isn't true ... see put_pmu_ctx(). * */ struct perf_event_pmu_context {
@@ -857,6 +857,7 @@ struct perf_event_pmu_context {unsigned int nr_events;atomic_t refcount;/* event <-> epc */ +struct rcu_head rcu_head;void *task_ctx_data;/* pmu specific data */ /*
kernel/events/core.c +55 -31
190 lines not on this page · raw message
Quoted 128 lines · Peter Zijlstra
Quoted 37 lines · Peter Zijlstra
Quoted 36 lines · Ravi Bangoria
Quoted 34 lines · Peter Zijlstra
Quoted 20 lines · Ravi Bangoria
(quoted diff) +20
-0+static void perf_event_swap_task_ctx_data(struct perf_event_context *prev_ctx, + struct perf_event_context *next_ctx) +{ + struct perf_event_pmu_context *prev_epc, *next_epc; + + if (!prev_ctx->nr_task_data) + return; + + prev_epc = list_first_entry(&prev_ctx->pmu_ctx_list, + struct perf_event_pmu_context, + pmu_ctx_entry); + next_epc = list_first_entry(&next_ctx->pmu_ctx_list, + struct perf_event_pmu_context, + pmu_ctx_entry); + + while (&prev_epc->pmu_ctx_entry != &prev_ctx->pmu_ctx_list && + &next_epc->pmu_ctx_entry != &next_ctx->pmu_ctx_list) { + + WARN_ON_ONCE(prev_epc->pmu != next_epc->pmu); +Review
Quoted 10 lines
(quoted diff) +10
-0+ /* + * PMU specific parts of task perf context can require + * additional synchronization. As an example of such + * synchronization see implementation details of Intel + * LBR call stack data profiling; + */ + if (prev_epc->pmu->swap_task_ctx) + prev_epc->pmu->swap_task_ctx(prev_epc, next_epc); + else + swap(prev_epc->task_ctx_data, next_epc->task_ctx_data);Did I forget to advance the iterators here?
Quoted 6 lines
Quoted 1 line
Yeah. Seems so. I overlooked it too.
OK; so I'm not slowly going crazy staring at this code ;-) Let me go add it now then. :-)
But first I gotta taxi the kids around for a bit, bbl.
OK, so I've been going over the
perf_event_pmu_contextlife-time thing as well, there were a bunch of XXXs there and I'm not sure Im happy with things, but I'd also forgotten most of it.Ideally epc works like it's a regular member of ctx -- locking wise that is, but I'm not sure we can make that stick -- see the
ctx->mutexissues we have withput_ctx().As such, I'm going to have to re-audit all the epc usage to see if pure
ctx->lockis sufficient.I did do make epc RCU freed, because pretty much everything is and that was easy enough to make happen -- it means we don't need to worry about that.
But I'm going cross-eyes from staring at this all day, so more tomorrow. The below is what I currently have.
---
include/linux/perf_event.h +6
-5@@ -833,13 +833,13 @@ struct perf_event {* `--------[1:n]---------' `-[n:1]-> pmu <-[1:n]-' * * -* XXX destroy epc when empty -* refcount, !rcu +* epc lifetime is refcount based and RCU freed (similar to perf_event_context). +* epc locking is as if it were a member of perf_event_context; specifically: * -* XXX epc locking +* modification, both: ctx->mutex && ctx->lock +* reading, either: ctx->mutex || ctx->lock * -* event->pmu_ctx ctx->mutex && inactive -* ctx->pmu_ctx_list ctx->mutex && ctx->lock +* XXX except this isn't true ... see put_pmu_ctx(). * */ struct perf_event_pmu_context {@@ -857,6 +857,7 @@ struct perf_event_pmu_context {unsigned int nr_events;atomic_t refcount;/* event <-> epc */ +struct rcu_head rcu_head;void *task_ctx_data;/* pmu specific data */ /* kernel/events/core.c +12
-3@@ -1727,6 +1727,10 @@ perf_event_groups_next(struct perf_eventreturn NULL ; } +#define perf_event_groups_for_cpu_pmu (event, groups, cpu, pmu) \ + for (event = perf_event_groups_first(groups, cpu, pmu, NULL); \ + event; event = perf_event_groups_next(event, pmu)) +/* * Iterate through the whole groups tree. */ @@ -3366,6 +3370,14 @@ static void perf_event_sync_stat(struct } } +#define list_for_each_entry_double (pos1, pos2, head1, head2, member) \ + for (pos1 = list_first_entry(head1, typeof(*pos1), member), \ + pos2 = list_first_entry(head2, typeof(*pos2), member); \ + !list_entry_is_head(pos1, head1, member) && \ + !list_entry_is_head(pos2, head2, member); \ + pos1 = list_next_entry(pos1, member), \ + pos2 = list_next_entry(pos2, member)) +static void perf_event_swap_task_ctx_data (struct perf_event_context *prev_ctx,struct perf_event_context *next_ctx) {@@ -3374,16 +3386,9 @@ static void perf_event_swap_task_ctx_datif (!prev_ctx->nr_task_data)return ; - prev_epc =list_first_entry (&prev_ctx->pmu_ctx_list, - struct perf_event_pmu_context, - pmu_ctx_entry);
kernel/events/core.c static void perf_event_swap_task_ctx_dat · line 3391
Quoted 10 lines
kernel/events/core.c +3
-7- next_epc =list_first_entry (&next_ctx->pmu_ctx_list, - struct perf_event_pmu_context, - pmu_ctx_entry); - -while (&prev_epc->pmu_ctx_entry != &prev_ctx->pmu_ctx_list && - &next_epc->pmu_ctx_entry != &next_ctx->pmu_ctx_list) { - +list_for_each_entry_double (prev_epc, next_epc, + &prev_ctx->pmu_ctx_list, &next_ctx->pmu_ctx_list, + pmu_ctx_entry) {
There are more places which can use list_for_each_entry_double(). I'll fix those.
Quoted 53 lines
kernel/events/core.c +17
-2WARN_ON_ONCE (prev_epc->pmu != next_epc->pmu);/* @@ -3706,7 +3711,6 @@ static noinline int visit_groups_merge(sperf_assert_pmu_disabled ((*evt)->pmu_ctx->pmu); } -min_heapify_all (&event_heap, &perf_min_heap);while (event_heap.nr) {@@ -3845,7 +3849,6 @@ ctx_sched_in(struct perf_event_context */* start ctx time */ __update_context_time (ctx, false);perf_cgroup_set_timestamp (cpuctx); -// XXX ctx->task =? task /* * CPU-release for the below ->is_active store, * see __load_acquire() in perf_event_time_now() @@ -4815,6 +4818,15 @@ find_get_pmu_context(struct pmu *pmu, st__perf_init_event_pmu_context (new, pmu); +/* +* XXX +* +* lockdep_assert_held(&ctx->mutex); +* +* can't because perf_event_init_task() doesn't actually hold the +* child_ctx->mutex. +*/ +raw_spin_lock_irq (&ctx->lock);list_for_each_entry (epc, &ctx->pmu_ctx_list, pmu_ctx_entry) {if (epc->pmu == pmu) {@@ -4849,6 +4861,14 @@ static void get_pmu_ctx(struct perf_evenWARN_ON_ONCE (!atomic_inc_not_zero (&epc->refcount)); } +static void free_epc_rcu (struct rcu_head *head) +{ +struct perf_event_pmu_context *epc =container_of (head,typeof (*epc), rcu_head); + +kfree (epc->task_ctx_data); +kfree (epc); +} +static void put_pmu_ctx (struct perf_event_pmu_context *epc) {unsigned long flags;@@ -4859,7 +4879,14 @@ static void put_pmu_ctx(struct perf_evenif (epc->ctx) {struct perf_event_context *ctx = epc->ctx;
kernel/events/core.c static void put_pmu_ctx(struct perf_even · line 4889
Quoted 10 lines
kernel/events/core.c +8
-1-// XXX ctx->mutex +/* +* XXX +* +* lockdep_assert_held(&ctx->mutex); +* +* can't because of the call-site in _free_event()/put_event() +* which isn't always called under ctx->mutex. +*/
Yes. I came across the same and could not figure out how to solve this. So Just kept XXX as is.
Quoted 38 lines
kernel/events/core.c +2
-11WARN_ON_ONCE (list_empty (&epc->pmu_ctx_entry));raw_spin_lock_irqsave (&ctx->lock, flags);@@ -4874,17 +4901,15 @@ static void put_pmu_ctx(struct perf_evenif (epc->embedded)return ; -kfree (epc->task_ctx_data); -kfree (epc); +call_rcu (&epc->rcu_head, free_epc_rcu); }static void perf_event_free_filter (struct perf_event *event);static void free_event_rcu (struct rcu_head *head) { -struct perf_event *event; +struct perf_event *event =container_of (head,typeof (*event), rcu_head); - event = container_of(head, struct perf_event, rcu_head);if (event->ns)put_pid_ns (event->ns);perf_event_free_filter (event);@@ -12643,13 +12668,6 @@ perf_event_create_kernel_counter(struct goto err_alloc; } - pmu_ctx =find_get_pmu_context (pmu, ctx, event); -if (IS_ERR (pmu_ctx)) { - err =PTR_ERR (pmu_ctx); - goto err_ctx; - } - event->pmu_ctx = pmu_ctx; -WARN_ON_ONCE (ctx->parent_ctx);mutex_lock (&ctx->mutex);if (ctx->task ==TASK_TOMBSTONE ) {
kernel/events/core.c perf_event_create_kernel_counter(struct · line 12683
Quoted 9 lines
kernel/events/core.c +6
-0@@ -12657,6 +12675,13 @@ perf_event_create_kernel_counter(struct goto err_unlock; } + pmu_ctx =find_get_pmu_context (pmu, ctx, event); +if (IS_ERR (pmu_ctx)) { + err =PTR_ERR (pmu_ctx); + goto err_unlock; + } + event->pmu_ctx = pmu_ctx;
We should call find_get_pmu_context() with ctx->mutex held and thus above perf_event_create_kernel_counter() change. Is my understanding correct?
Quoted 44 lines
kernel/events/core.c +6
-6+if (!task) {/* * Check if the @cpu we're creating an event for is online. @@ -12668,13 +12693,13 @@ perf_event_create_kernel_counter(struct container_of(ctx, struct perf_cpu_context, ctx);if (!cpuctx->online) { err = -ENODEV ; - goto err_unlock; + goto err_pmu_ctx; } }if (!exclusive_event_installable (event, ctx)) { err = -EBUSY ; - goto err_unlock; + goto err_pmu_ctx; }perf_install_in_context (ctx, event, event->cpu);@@ -12683,9 +12708,10 @@ perf_event_create_kernel_counter(structreturn event; +err_pmu_ctx: +put_pmu_ctx (pmu_ctx); err_unlock:mutex_unlock (&ctx->mutex); -err_ctx:perf_unpin_context (ctx);put_ctx (ctx); err_alloc:@@ -12702,9 +12728,7 @@ static void __perf_pmu_remove(struct per {struct perf_event *event, *sibling; -for (event =perf_event_groups_first (groups, cpu, pmu,NULL ); - event; event =perf_event_groups_next (event, pmu)) { - + perf_event_groups_for_cpu_pmu(event, groups, cpu, pmu) {perf_remove_from_context (event,0 );unaccount_event_cpu (event, cpu);put_pmu_ctx (event->pmu_ctx);
kernel/events/core.c void perf_event_free_task(struct task_st · line 13025
Quoted 5 lines
kernel/events/core.c +1
-1@@ -12998,7 +13022,7 @@ void perf_event_free_task(struct task_ststruct perf_event_context *ctx;struct perf_event *event, *tmp; - ctx =rcu_dereference (task->perf_event_ctxp); + ctx =rcu_access_pointer (task->perf_event_ctxp);
We dereference ctx pointer but with mutex and lock held. And thus rcu_access_pointer() is sufficient. Is my understanding correct?
Thanks,
Ravi
Peter Zijlstra
maintainerQuoted 19 lines · Ravi Bangoria
Quoted 19 lines
kernel/events/core.c +8
-0@@ -3366,6 +3370,14 @@ static void perf_event_sync_stat(struct } } +#define list_for_each_entry_double (pos1, pos2, head1, head2, member) \ + for (pos1 = list_first_entry(head1, typeof(*pos1), member), \ + pos2 = list_first_entry(head2, typeof(*pos2), member); \ + !list_entry_is_head(pos1, head1, member) && \ + !list_entry_is_head(pos2, head2, member); \ + pos1 = list_next_entry(pos1, member), \ + pos2 = list_next_entry(pos2, member)) +static void perf_event_swap_task_ctx_data (struct perf_event_context *prev_ctx,struct perf_event_context *next_ctx) {@@ -3374,16 +3386,9 @@ static void perf_event_swap_task_ctx_datif (!prev_ctx->nr_task_data)return ;
Quoted 16 lines
kernel/events/core.c static void perf_event_swap_task_ctx_dat · line 3391
Quoted 13 lines
kernel/events/core.c +3
-10- prev_epc =list_first_entry (&prev_ctx->pmu_ctx_list, - struct perf_event_pmu_context, - pmu_ctx_entry); - next_epc =list_first_entry (&next_ctx->pmu_ctx_list, - struct perf_event_pmu_context, - pmu_ctx_entry); - -while (&prev_epc->pmu_ctx_entry != &prev_ctx->pmu_ctx_list && - &next_epc->pmu_ctx_entry != &next_ctx->pmu_ctx_list) { - +list_for_each_entry_double (prev_epc, next_epc, + &prev_ctx->pmu_ctx_list, &next_ctx->pmu_ctx_list, + pmu_ctx_entry) {There are more places which can use
list_for_each_entry_double(). I'll fix those.
I've gone and renamed it: double_list_for_each_entry(), but yeah, didn't look too hard for other users.
Quoted 4 lines
Quoted 4 lines
kernel/events/core.c +0
-0@@ -4859,7 +4879,14 @@ static void put_pmu_ctx(struct perf_evenif (epc->ctx) {struct perf_event_context *ctx = epc->ctx;
Quoted 12 lines
kernel/events/core.c static void put_pmu_ctx(struct perf_even · line 4889
Quoted 9 lines
kernel/events/core.c +8
-1-// XXX ctx->mutex +/* +* XXX +* +* lockdep_assert_held(&ctx->mutex); +* +* can't because of the call-site in _free_event()/put_event() +* which isn't always called under ctx->mutex. +*/ Yes. I came across the same and could not figure out how to solve this. So Just kept XXX as is.
Yeah, I can sorta fix it, but it's ugly so there we are.
Quoted 8 lines
Quoted 8 lines
kernel/events/core.c +0
-0WARN_ON_ONCE (list_empty (&epc->pmu_ctx_entry));raw_spin_lock_irqsave (&ctx->lock, flags);@@ -12657,6 +12675,13 @@ perf_event_create_kernel_counter(struct goto err_unlock; }
Quoted 10 lines
kernel/events/core.c perf_event_create_kernel_counter(struct · line 12683
Quoted 6 lines
kernel/events/core.c +6
-0+ pmu_ctx =find_get_pmu_context (pmu, ctx, event); +if (IS_ERR (pmu_ctx)) { + err =PTR_ERR (pmu_ctx); + goto err_unlock; + } + event->pmu_ctx = pmu_ctx;We should call
find_get_pmu_context()withctx->mutexheld and thus aboveperf_event_create_kernel_counter()change. Is my understanding correct?
That's the intent yeah. But due to not always holding ctx->mutex over put_pmu_ctx() this might be moot. I'm almost through auditing epc usage and I think ctx->lock is sufficient, fingers crossed.
Quoted 9 lines
Quoted 9 lines
kernel/events/core.c +1
-0+if (!task) {/* * Check if the @cpu we're creating an event for is online. @@ -12998,7 +13022,7 @@ void perf_event_free_task(struct task_ststruct perf_event_context *ctx;struct perf_event *event, *tmp;
Quoted 5 lines
kernel/events/core.c void perf_event_free_task(struct task_st · line 13025
Quoted 2 lines
kernel/events/core.c +1
-1- ctx =rcu_dereference (task->perf_event_ctxp); + ctx =rcu_access_pointer (task->perf_event_ctxp);We dereference ctx pointer but with mutex and lock held. And thus
rcu_access_pointer()is sufficient. Is my understanding correct?
We do not in fact hold ctx->lock here IIRC; but this is a NULL test, if it is !NULL we know we have a reference on it and are good.
Peter Zijlstra
maintainerQuoted 3 lines · Peter Zijlstra
That's the intent yeah. But due to not always holding
ctx->mutexoverput_pmu_ctx()this might be moot. I'm almost through auditing epc usage and I thinkctx->lockis sufficient, fingers crossed.
So the very last epc usage threw a spanner into the works and made things complicated.
Specifically sys_perf_event_open()'s group_leader case uses event->pmu_ctx while only holding ctx->mutex. Therefore we can't fully let go of ctx->mutex locking and purely rely on ctx->lock.
Now the good news is that the annoying put_pmu_ctx() without holding ctx->mutex case doesn't actually matter here. Since we hold a reference on the group_leader (per the filedesc) the event can't go away, therefore it must have a pmu_ctx, and then holding ctx->mutex ensures the pmu_ctx is stable -- iow it serializes against sys_perf_event_open()'s move_group and perf_pmu_migrate_context() changing the epc around.
So we're going with the normal mutex+lock for modification rule, but allow the weird put_pmu_ctx() exception.
I have the below delta.
I'm hoping we can call this done -- I'm going to see if I can bribe Mark to take a look at the arm64 thing soon and then hopefully queue the whole thing once -rc1 happens. That should give us a good long soak until the next merge window.
---
include/linux/perf_event.h +18 -10
@@ -826,21 +826,28 @@ struct perf_event { };/* -* ,------------------------[1:n]---------------------. +* ,-----------------------[1:n]----------------------. * V V * perf_event_context <-[1:n]-> perf_event_pmu_context <--- perf_event * ^ ^ | | * `--------[1:n]---------' `-[n:1]-> pmu <-[1:n]-' * * -* XXX destroy epc when empty -* refcount, !rcu -* -* XXX epc locking -* -* event->pmu_ctx ctx->mutex && inactive -* ctx->pmu_ctx_list ctx->mutex && ctx->lock -* +* struct perf_event_pmu_context lifetime is refcount based and RCU freed +* (similar to perf_event_context). Locking is as if it were a member of +* perf_event_context; specifically: +* +* modification, both: ctx->mutex && ctx->lock +* reading, either: ctx->mutex || ctx->lock +* +* There is one exception to this; namely put_pmu_ctx() isn't always called +* with ctx->mutex held; this means that as long as we can guarantee the epc +* has events the above rules hold. +* +* Specificially, sys_perf_event_open()'s group_leader case depends on +* ctx->mutex pinning the configuration. Since we hold a reference on +* group_leader (through the filedesc) it can't fo away, therefore it's +* associated pmu_ctx must exist and cannot change due to ctx->mutex. */ struct perf_event_pmu_context {struct pmu *pmu;
@@ -857,6 +864,7 @@ struct perf_event_pmu_context {unsigned int nr_events;atomic_t refcount;/* event <-> epc */ +struct rcu_head rcu_head;void *task_ctx_data;/* pmu specific data */ /*
@@ -906,7 +914,7 @@ struct perf_event_context {int nr_freq;int rotate_disable; -refcount_t refcount; +refcount_t refcount;/* event <-> ctx */ struct task_struct *task;/*
kernel/events/core.c +60 -31
202 lines not on this page · raw message
Quoted 24 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra
That's the intent yeah. But due to not always holding
ctx->mutexoverput_pmu_ctx()this might be moot. I'm almost through auditing epc usage and I thinkctx->lockis sufficient, fingers crossed.So the very last epc usage threw a spanner into the works and made things complicated.
Specifically sys_perf_event_open()'s
group_leadercase usesevent->pmu_ctxwhile only holdingctx->mutex. Therefore we can't fully let go ofctx->mutexlocking and purely rely onctx->lock.Now the good news is that the annoying
put_pmu_ctx()without holdingctx->mutexcase doesn't actually matter here. Since we hold a reference on thegroup_leader(per the filedesc) the event can't go away, therefore it must have apmu_ctx, and then holdingctx->mutexensures thepmu_ctxis stable -- iow it serializes against sys_perf_event_open()'smove_groupandperf_pmu_migrate_context()changing the epc around.So we're going with the normal mutex+lock for modification rule, but allow the weird
put_pmu_ctx()exception.
Quoted 6 lines
I have the below delta.
I'm hoping we can call this done -- I'm going to see if I can bribe Mark to take a look at the arm64 thing soon and then hopefully queue the whole thing once -rc1 happens. That should give us a good long soak until the next merge window.
Sounds good. Thanks for all the help!
I've glanced through the changes and they looks fine, below are few minor points.
Quoted 3 lines
+ * Specificially, sys_perf_event_open()'s
group_leadercase depends on
+ *ctx->mutexpinning the configuration. Since we hold a reference on
+ *group_leader(through the filedesc) it can't fo away, therefore it's
typo: can't go away
include/linux/perf_event.h struct perf_event_context { · line 917
Quoted 2 lines
include/linux/perf_event.h +1
-1-refcount_t refcount; +refcount_t refcount;/* event <-> ctx */
Ok. We need to remove all those // XXX get/put_ctx() from code which we added to make refcount a pmu_ctx <-> ctx.
kernel/events/core.c static void perf_event_sync_stat(struct · line 3382
Quoted 10 lines
kernel/events/core.c +8
-0+#define double_list_for_each_entry (pos1, pos2, head1, head2, member) \ + for (pos1 = list_first_entry(head1, typeof(*pos1), member), \ + pos2 = list_first_entry(head2, typeof(*pos2), member); \ + !list_entry_is_head(pos1, head1, member) && \ + !list_entry_is_head(pos2, head2, member); \ + pos1 = list_next_entry(pos1, member), \ + pos2 = list_next_entry(pos2, member)) +static void perf_event_swap_task_ctx_data (struct perf_event_context *prev_ctx,struct perf_event_context *next_ctx)
While this is unrelated to this patch, shouldn't we also need to swap event->hw.target? A purely hypothetical scenario: Consider two processes having clone contexts (for example, two children of the same parent). While process switch between these two, the perf event context would get swapped but event->hw.target will point to other sibling's task_struct. If any one process exit just after single context swap, _free_event() will call put_task_context() on sibling process' task_struct.
kernel/events/core.c SYSCALL_DEFINE5(perf_event_open, · line 12468
Quoted 7 lines
kernel/events/core.c +3
-0@@ -12436,6 +12463,9 @@ SYSCALL_DEFINE5(perf_event_open,* Allow the addition of software events to hw * groups, this is safe because software events * never fail to schedule. +* +* Note the comment that goes with struct +* pmu_event_pmu_context.
typo: perf_event_pmu_context
The good (or bad? ;)) news is, perf test and Vince's perf_event_tests are running fine without any regression on my machine.
Thanks,
Ravi
Peter Zijlstra
maintainerQuoted 5 lines · Ravi Bangoria
include/linux/perf_event.h struct perf_event_context { · line 917
Quoted 2 lines
include/linux/perf_event.h +1
-1-refcount_t refcount; +refcount_t refcount;/* event <-> ctx */ Ok. We need to remove all those // XXX get/put_ctx() from code which we added to make refcount a
pmu_ctx<-> ctx.
Them already gone :-) I've not yet fixed up the typoes, but current version should be here:
https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=perf/core
Thanks!
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Ravi Bangoria
include/linux/perf_event.h struct perf_event_context { · line 917
Quoted 2 lines
include/linux/perf_event.h +1
-1-refcount_t refcount; +refcount_t refcount;/* event <-> ctx */ Ok. We need to remove all those // XXX get/put_ctx() from code which we added to make refcount a
pmu_ctx<-> ctx.
Quoted 6 lines
Them already gone :-) I've not yet fixed up the typoes, but current version should be here:
https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=perf/core
Thanks!
I've been running perf-fuzzer on Xeon machine since yesterday and I don't see any issue. Will do the same on my AMD machine as well over the weekend.
Thanks,
Ravi
Peter Zijlstra
maintainerQuoted 12 lines · Ravi Bangoria
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Ravi Bangoria
include/linux/perf_event.h struct perf_event_context { · line 917
Quoted 2 lines
include/linux/perf_event.h +1
-1-refcount_t refcount; +refcount_t refcount;/* event <-> ctx */ Ok. We need to remove all those // XXX get/put_ctx() from code which we added to make refcount a
pmu_ctx<-> ctx.Them already gone :-) I've not yet fixed up the typoes, but current version should be here:
Quoted 6 lines
Quoted 3 lines
https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=perf/core
Thanks!
I've been running perf-fuzzer on Xeon machine since yesterday and I don't see any issue. Will do the same on my AMD machine as well over the weekend.
Awesome -- I've started fuzzing on the ADL (with the big.LITTLE PMU setup) and I've had it run on my very aged IVB-EP machine. Both so far (knock on wood) with no issues.
The most modern AMD machine I have at hand is a 2 socket Interlagos, and I doubt anybody really much cares about that these days -- but I can run it for giggles.
Quoted 12 lines · Ravi Bangoria
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Ravi Bangoria
include/linux/perf_event.h struct perf_event_context { · line 917
Quoted 2 lines
include/linux/perf_event.h +1
-1-refcount_t refcount; +refcount_t refcount;/* event <-> ctx */ Ok. We need to remove all those // XXX get/put_ctx() from code which we added to make refcount a
pmu_ctx<-> ctx.Them already gone :-) I've not yet fixed up the typoes, but current version should be here:
Quoted 6 lines
Quoted 3 lines
https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=perf/core
Thanks!
I've been running perf-fuzzer on Xeon machine since yesterday and I don't see any issue. Will do the same on my AMD machine as well over the weekend.
Only one WARN_ON() hit. Otherwise all good.
https://lore.kernel.org/lkml/[email protected]
Thanks,
Ravi
The following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: bd27568117664b8b3e259721393df420ed51f57b Gitweb: https://git.kernel.org/tip/bd27568117664b8b3e259721393df420ed51f57b Author: Peter Zijlstra <[email protected]> AuthorDate: Sat, 08 Oct 2022 11:54:24 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 27 Oct 2022 20:12:16 +02:00
perf: Rewrite core context handling
There have been various issues and limitations with the way perf uses (task) contexts to track events. Most notable is the single hardware PMU task context, which has resulted in a number of yucky things (both proposed and merged).
Notably:
- HW breakpoint PMU
- ARM big.little PMU / Intel ADL PMU
- Intel Branch Monitoring PMU
- AMD IBS PMU
- S390 cpum_cf PMU
- PowerPC trace_imc PMU
*Current design:*
Currently we have a per task and per cpu perf_event_contexts:
task_struct::perf_events_ctxp[] <-> perf_event_context <-> perf_cpu_context
^ | ^ | ^
`---------------------------------' | `--> pmu ---'
v ^
perf_event ------'Each task has an array of pointers to a perf_event_context. Each perf_event_context has a direct relation to a PMU and a group of events for that PMU. The task related perf_event_context's have a pointer back to that task.
Each PMU has a per-cpu pointer to a per-cpu perf_cpu_context, which includes a perf_event_context, which again has a direct relation to that PMU, and a group of events for that PMU.
The perf_cpu_context also tracks which task context is currently associated with that CPU and includes a few other things like the hrtimer for rotation etc.
Each perf_event is then associated with its PMU and one perf_event_context.
*Proposed design:*
New design proposed by this patch reduce to a single task context and a single CPU context but adds some intermediate data-structures:
task_struct::perf_event_ctxp -> perf_event_context <- perf_cpu_context
^ | ^ ^
`---------------------------' | |
| | perf_cpu_pmu_context <--.
| `----. ^ |
| | | |
| v v |
| ,--> perf_event_pmu_context |
| | |
| | | v v |
perf_event ---> pmu ----------------'
With the new design, perf_event_context will hold all events for all pmus in the (respective pinned/flexible) rbtrees. This can be achieved by adding pmu to rbtree key:
{cpu, pmu, cgroup, group_index}
Each perf_event_context carries a list of perf_event_pmu_context which is used to hold per-pmu-per-context state. For example, it keeps track of currently active events for that pmu, a pmu specific task_ctx_data, a flag to tell whether rotation is required or not etc.
Additionally, perf_cpu_pmu_context is used to hold per-pmu-per-cpu state like hrtimer details to drive the event rotation, a pointer to perf_event_pmu_context of currently running task and some other ancillary information.
Each perf_event is associated to it's pmu, perf_event_context and perf_event_pmu_context.
Further optimizations to current implementation are possible. For example, ctx_resched() can be optimized to reschedule only single pmu events.
Much thanks to Ravi for picking this up and pushing it towards completion.
--- arch/arm64/kernel/perf_event.c | 18 +- arch/powerpc/perf/core-book3s.c | 8 +- arch/s390/kernel/perf_pai_crypto.c | 2 +- arch/s390/kernel/perf_pai_ext.c | 2 +- arch/x86/events/amd/brs.c | 2 +- arch/x86/events/amd/lbr.c | 6 +- arch/x86/events/core.c | 44 +- arch/x86/events/intel/core.c | 23 +- arch/x86/events/intel/ds.c | 4 +- arch/x86/events/intel/lbr.c | 30 +- arch/x86/events/perf_event.h | 30 +- drivers/perf/arm_pmu.c | 16 +- include/linux/perf/arm_pmu.h | 2 +- include/linux/perf_event.h | 125 +- include/linux/sched.h | 2 +- kernel/events/core.c | 1958 +++++++++++++-------------- 16 files changed, 1178 insertions(+), 1094 deletions(-)
arch/arm64/kernel/perf_event.c +11 -7
@@ -806,10 +806,14 @@ static void armv8pmu_disable_event(struct perf_event *event)static void armv8pmu_start (struct arm_pmu *cpu_pmu) { -struct perf_event_context *task_ctx = -this_cpu_ptr (cpu_pmu->pmu.pmu_cpu_context)->task_ctx; +struct perf_event_context *ctx; +int nr_user =0 ; -if (sysctl_perf_user_access && task_ctx && task_ctx->nr_user) + ctx =perf_cpu_task_ctx (); +if (ctx) + nr_user = ctx->nr_user; + +if (sysctl_perf_user_access && nr_user)armv8pmu_enable_user_access (cpu_pmu);else armv8pmu_disable_user_access ();
@@ -1019,10 +1023,10 @@ static int armv8pmu_set_event_filter(struct hw_perf_event *event,return 0 ; } -static int armv8pmu_filter_match (struct perf_event *event) +static bool armv8pmu_filter (struct pmu *pmu,int cpu) { -unsigned long evtype = event->hw.config_base &ARMV8_PMU_EVTYPE_EVENT ; -return evtype !=ARMV8_PMUV3_PERFCTR_CHAIN ; +struct arm_pmu *armpmu =to_arm_pmu (pmu); +return !cpumask_test_cpu (smp_processor_id (), &armpmu->supported_cpus); }static void armv8pmu_reset (void *info)
@@ -1253,7 +1257,7 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu, char *name, cpu_pmu->stop = armv8pmu_stop; cpu_pmu->reset = armv8pmu_reset; cpu_pmu->set_event_filter = armv8pmu_set_event_filter; - cpu_pmu->filter_match = armv8pmu_filter_match; + cpu_pmu->filter = armv8pmu_filter; cpu_pmu->pmu.event_idx = armv8pmu_user_event_idx;
arch/powerpc/perf/core-book3s.c +4 -4
@@ -132,7 +132,7 @@ static unsigned long ebb_switch_in(bool ebb, struct cpu_hw_events *cpuhw)static inline void power_pmu_bhrb_enable (struct perf_event *event) {}static inline void power_pmu_bhrb_disable (struct perf_event *event) {} -static void power_pmu_sched_task (struct perf_event_context *ctx,bool sched_in) {} +static void power_pmu_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {}static inline void power_pmu_bhrb_read (struct perf_event *event,struct cpu_hw_events *cpuhw) {}static void pmao_restore_workaround (bool ebb) { }#endif /* CONFIG_PPC32 */
@@ -424,7 +424,7 @@ static void power_pmu_bhrb_enable(struct perf_event *event) cpuhw->bhrb_context = event->ctx; } cpuhw->bhrb_users++; -perf_sched_cb_inc (event->ctx->pmu); +perf_sched_cb_inc (event->pmu); }static void power_pmu_bhrb_disable (struct perf_event *event)
@@ -436,7 +436,7 @@ static void power_pmu_bhrb_disable(struct perf_event *event)WARN_ON_ONCE (!cpuhw->bhrb_users); cpuhw->bhrb_users--; -perf_sched_cb_dec (event->ctx->pmu); +perf_sched_cb_dec (event->pmu);if (!cpuhw->disabled && !cpuhw->bhrb_users) {/* BHRB cannot be turned off when other
@@ -451,7 +451,7 @@ static void power_pmu_bhrb_disable(struct perf_event *event)/* Called from ctxsw to prevent one process's branch entries to * mingle with the other process's entries during context switch. */ -static void power_pmu_sched_task (struct perf_event_context *ctx,bool sched_in) +static void power_pmu_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {if (!ppmu->bhrb_nr)return ;
arch/s390/kernel/perf_pai_crypto.c +1 -1
@@ -379,7 +379,7 @@ static int paicrypt_push_sample(void)/* Called on schedule-in and schedule-out. No access to event structure, * but for sampling only event CRYPTO_ALL is allowed. */ -static void paicrypt_sched_task (struct perf_event_context *ctx,bool sched_in) +static void paicrypt_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {/* We started with a clean page on event installation. So read out * results on schedule_out and if page was dirty, clear values.
arch/s390/kernel/perf_pai_ext.c +1 -1
@@ -471,7 +471,7 @@ static int paiext_push_sample(void)/* Called on schedule-in and schedule-out. No access to event structure, * but for sampling only event NNPA_ALL is allowed. */ -static void paiext_sched_task (struct perf_event_context *ctx,bool sched_in) +static void paiext_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {/* We started with a clean page on event installation. So read out * results on schedule_out and if page was dirty, clear values.
arch/x86/events/amd/brs.c +1 -1
@@ -384,7 +384,7 @@ static void amd_brs_poison_buffer(void)* On ctxswin, sched_in = true, called after the PMU has started * On ctxswout, sched_in = false, called before the PMU is stopped */ -void amd_pmu_brs_sched_task (struct perf_event_context *ctx,bool sched_in) +void amd_pmu_brs_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
arch/x86/events/amd/lbr.c +3 -3
@@ -352,7 +352,7 @@ void amd_pmu_lbr_add(struct perf_event *event) cpuc->br_sel = reg->reg; } -perf_sched_cb_inc (event->ctx->pmu); +perf_sched_cb_inc (event->pmu);if (!cpuc->lbr_users++ && !event->total_time_running)amd_pmu_lbr_reset ();
@@ -370,10 +370,10 @@ void amd_pmu_lbr_del(struct perf_event *event) cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 ); -perf_sched_cb_dec (event->ctx->pmu); +perf_sched_cb_dec (event->pmu); } -void amd_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in) +void amd_pmu_lbr_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
arch/x86/events/core.c +14 -30
92 lines not on this page · raw message
arch/x86/events/intel/core.c +10 -13
@@ -4536,8 +4536,6 @@ end:cpumask_set_cpu (cpu, &pmu->supported_cpus); cpuc->pmu = &pmu->pmu; -x86_pmu_update_cpu_context (&pmu->pmu, cpu); -return true; }
@@ -4671,17 +4669,17 @@ static void intel_pmu_cpu_dead(int cpu)cpumask_clear_cpu (cpu, &hybrid_pmu (cpuc->pmu)->supported_cpus); } -static void intel_pmu_sched_task (struct perf_event_context *ctx, +static void intel_pmu_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) { -intel_pmu_pebs_sched_task (ctx, sched_in); -intel_pmu_lbr_sched_task (ctx, sched_in); +intel_pmu_pebs_sched_task (pmu_ctx, sched_in); +intel_pmu_lbr_sched_task (pmu_ctx, sched_in); } -static void intel_pmu_swap_task_ctx (struct perf_event_context *prev, -struct perf_event_context *next) +static void intel_pmu_swap_task_ctx (struct perf_event_pmu_context *prev_epc, +struct perf_event_pmu_context *next_epc) { -intel_pmu_lbr_swap_task_ctx (prev, next); +intel_pmu_lbr_swap_task_ctx (prev_epc, next_epc); }static int intel_pmu_check_period (struct perf_event *event,u64 value)
@@ -4705,12 +4703,11 @@ static int intel_pmu_aux_output_match(struct perf_event *event)return is_intel_pt_event (event); } -static int intel_pmu_filter_match (struct perf_event *event) +static void intel_pmu_filter (struct pmu *pmu,int cpu,bool *ret) { -struct x86_hybrid_pmu *pmu =hybrid_pmu (event->pmu); -unsigned int cpu =smp_processor_id (); +struct x86_hybrid_pmu *hpmu =hybrid_pmu (pmu); -return cpumask_test_cpu (cpu, &pmu->supported_cpus); + *ret = !cpumask_test_cpu (cpu, &hpmu->supported_cpus); }PMU_FORMAT_ATTR (offcore_rsp,"config1:0-63" );
@@ -6412,7 +6409,7 @@ __init int intel_pmu_init(void)static_call_update (intel_pmu_set_topdown_event_period, &adl_set_topdown_event_period); - x86_pmu.filter_match = intel_pmu_filter_match; + x86_pmu.filter = intel_pmu_filter; x86_pmu.get_event_constraints = adl_get_event_constraints; x86_pmu.hw_config = adl_hw_config; x86_pmu.limit_period = spr_limit_period;
arch/x86/events/intel/ds.c +2 -2
@@ -1059,7 +1059,7 @@ static inline bool pebs_needs_sched_cb(struct cpu_hw_events *cpuc)return cpuc->n_pebs && (cpuc->n_pebs == cpuc->n_large_pebs); } -void intel_pmu_pebs_sched_task (struct perf_event_context *ctx,bool sched_in) +void intel_pmu_pebs_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -1167,7 +1167,7 @@ static void pebs_update_state(bool needed_cb, struct cpu_hw_events *cpuc, struct perf_event *event, bool add) { - struct pmu *pmu = event->ctx->pmu; + struct pmu *pmu = event->pmu;/* * Make sure we get updated with the first PEBS * event. It will trigger also during removal, but
arch/x86/events/intel/lbr.c +15 -15
@@ -515,21 +515,21 @@ static void __intel_pmu_lbr_save(void *ctx) cpuc->last_log_id = ++task_context_opt (ctx)->log_id; } -void intel_pmu_lbr_swap_task_ctx (struct perf_event_context *prev, -struct perf_event_context *next) +void intel_pmu_lbr_swap_task_ctx (struct perf_event_pmu_context *prev_epc, +struct perf_event_pmu_context *next_epc) {void *prev_ctx_data, *next_ctx_data; -swap (prev->task_ctx_data, next->task_ctx_data); +swap (prev_epc->task_ctx_data, next_epc->task_ctx_data);/* -* Architecture specific synchronization makes sense in -* case both prev->task_ctx_data and next->task_ctx_data +* Architecture specific synchronization makes sense in case +* both prev_epc->task_ctx_data and next_epc->task_ctx_data * pointers are allocated. */ - prev_ctx_data = next->task_ctx_data; - next_ctx_data = prev->task_ctx_data; + prev_ctx_data = next_epc->task_ctx_data; + next_ctx_data = prev_epc->task_ctx_data;if (!prev_ctx_data || !next_ctx_data)return ;
@@ -538,7 +538,7 @@ void intel_pmu_lbr_swap_task_ctx(struct perf_event_context *prev, task_context_opt(next_ctx_data)->lbr_callstack_users); } -void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in) +void intel_pmu_lbr_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);void *task_ctx;
@@ -551,7 +551,7 @@ void intel_pmu_lbr_sched_task(struct perf_event_context *ctx, bool sched_in)* the task was scheduled out, restore the stack. Otherwise flush * the LBR stack. */ - task_ctx = ctx ? ctx->task_ctx_data :NULL ; + task_ctx = pmu_ctx ? pmu_ctx->task_ctx_data :NULL ;if (task_ctx) {if (sched_in)__intel_pmu_lbr_restore (task_ctx);
@@ -587,8 +587,8 @@ void intel_pmu_lbr_add(struct perf_event *event) cpuc->br_sel = event->hw.branch_reg.reg; -if (branch_user_callstack (cpuc->br_sel) && event->ctx->task_ctx_data) -task_context_opt (event->ctx->task_ctx_data)->lbr_callstack_users++; +if (branch_user_callstack (cpuc->br_sel) && event->pmu_ctx->task_ctx_data) +task_context_opt (event->pmu_ctx->task_ctx_data)->lbr_callstack_users++;/* * Request pmu::sched_task() callback, which will fire inside the
@@ -611,7 +611,7 @@ void intel_pmu_lbr_add(struct perf_event *event)*/ if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 ) cpuc->lbr_pebs_users++; -perf_sched_cb_inc (event->ctx->pmu); +perf_sched_cb_inc (event->pmu);if (!cpuc->lbr_users++ && !event->total_time_running)intel_pmu_lbr_reset (); }
@@ -664,8 +664,8 @@ void intel_pmu_lbr_del(struct perf_event *event)return ;if (branch_user_callstack (cpuc->br_sel) && - event->ctx->task_ctx_data) -task_context_opt (event->ctx->task_ctx_data)->lbr_callstack_users--; + event->pmu_ctx->task_ctx_data) +task_context_opt (event->pmu_ctx->task_ctx_data)->lbr_callstack_users--;if (event->hw.flags &PERF_X86_EVENT_LBR_SELECT ) cpuc->lbr_select =0 ;
@@ -675,7 +675,7 @@ void intel_pmu_lbr_del(struct perf_event *event) cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 );WARN_ON_ONCE (cpuc->lbr_pebs_users <0 ); -perf_sched_cb_dec (event->ctx->pmu); +perf_sched_cb_dec (event->pmu); }static inline bool vlbr_exclude_host (void )
arch/x86/events/perf_event.h +14 -16
@@ -811,7 +811,7 @@ struct x86_pmu {void (*cpu_dead)(int cpu);void (*check_microcode)(void ); -void (*sched_task)(struct perf_event_context *ctx, +void (*sched_task)(struct perf_event_pmu_context *pmu_ctx,bool sched_in);/*
@@ -894,12 +894,12 @@ struct x86_pmu {int num_topdown_events;/* -* perf task context (i.e. struct perf_event_context::task_ctx_data) +* perf task context (i.e. struct perf_event_pmu_context::task_ctx_data) * switch helper to bridge calls from perf/core to perf/x86. * See struct pmu::swap_task_ctx() usage for examples; */ -void (*swap_task_ctx)(struct perf_event_context *prev, -struct perf_event_context *next); +void (*swap_task_ctx)(struct perf_event_pmu_context *prev_epc, +struct perf_event_pmu_context *next_epc);/* * AMD bits
@@ -925,7 +925,7 @@ struct x86_pmu {int (*aux_output_match) (struct perf_event *event); -int (*filter_match)(struct perf_event *event); +void (*filter)(struct pmu *pmu,int cpu,bool *ret);/* * Hybrid support *
@@ -1180,8 +1180,6 @@ int x86_pmu_handle_irq(struct pt_regs *regs);void x86_pmu_show_pmu_cap (int num_counters,int num_counters_fixed,u64 intel_ctrl); -void x86_pmu_update_cpu_context (struct pmu *pmu,int cpu); -extern struct event_constraint emptyconstraint;extern struct event_constraint unconstrained;
@@ -1306,7 +1304,7 @@ void amd_pmu_lbr_reset(void);void amd_pmu_lbr_read (void );void amd_pmu_lbr_add (struct perf_event *event);void amd_pmu_lbr_del (struct perf_event *event); -void amd_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in); +void amd_pmu_lbr_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in);void amd_pmu_lbr_enable_all (void );void amd_pmu_lbr_disable_all (void );int amd_pmu_lbr_hw_config (struct perf_event *event);
@@ -1330,7 +1328,7 @@ static inline void amd_pmu_brs_add(struct perf_event *event) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -perf_sched_cb_inc (event->ctx->pmu); +perf_sched_cb_inc (event->pmu); cpuc->lbr_users++;/* * No need to reset BRS because it is reset
@@ -1345,10 +1343,10 @@ static inline void amd_pmu_brs_del(struct perf_event *event) cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 ); - perf_sched_cb_dec(event->ctx->pmu); + perf_sched_cb_dec(event->pmu); } -void amd_pmu_brs_sched_task (struct perf_event_context *ctx,bool sched_in); +void amd_pmu_brs_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in);#else static inline int amd_brs_init (void ) {
@@ -1373,7 +1371,7 @@ static inline void amd_pmu_brs_del(struct perf_event *event) { } -static inline void amd_pmu_brs_sched_task (struct perf_event_context *ctx,bool sched_in) +static inline void amd_pmu_brs_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in) { }
@@ -1533,7 +1531,7 @@ void intel_pmu_pebs_enable_all(void);void intel_pmu_pebs_disable_all (void ); -void intel_pmu_pebs_sched_task (struct perf_event_context *ctx,bool sched_in); +void intel_pmu_pebs_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in);void intel_pmu_auto_reload_read (struct perf_event *event);
@@ -1541,10 +1539,10 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);void intel_ds_init (void ); -void intel_pmu_lbr_swap_task_ctx (struct perf_event_context *prev, -struct perf_event_context *next); +void intel_pmu_lbr_swap_task_ctx (struct perf_event_pmu_context *prev_epc, +struct perf_event_pmu_context *next_epc); -void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in); +void intel_pmu_lbr_sched_task (struct perf_event_pmu_context *pmu_ctx,bool sched_in);u64 lbr_from_signext_quirk_wr (u64 val);
drivers/perf/arm_pmu.c +7 -9
@@ -550,15 +550,14 @@ static void armpmu_disable(struct pmu *pmu)* microarchitecture, and aren't suitable for another. Thus, only match CPUs of * the same microarchitecture. */ -static int armpmu_filter_match (struct perf_event *event) +static bool armpmu_filter (struct pmu *pmu,int cpu) { -struct arm_pmu *armpmu =to_arm_pmu (event->pmu); -unsigned int cpu =smp_processor_id (); -int ret; +struct arm_pmu *armpmu =to_arm_pmu (pmu); +bool ret; ret =cpumask_test_cpu (cpu, &armpmu->supported_cpus); -if (ret && armpmu->filter_match) -return armpmu->filter_match (event); +if (ret && armpmu->filter) +return armpmu->filter (pmu, cpu);return ret; }
@@ -885,14 +884,13 @@ static struct arm_pmu *__armpmu_alloc(gfp_t flags) .start = armpmu_start, .stop = armpmu_stop, .read = armpmu_read, - .filter_match = armpmu_filter_match, + .filter = armpmu_filter, .attr_groups = pmu->attr_groups,/* * This is a CPU PMU potentially in a heterogeneous * configuration (e.g. big.LITTLE). This is not an uncore PMU, * and we have taken ctx sharing into account (e.g. with our -* pmu::filter_match callback and pmu::event_init group -* validation). +* pmu::filter callback and pmu::event_init group validation). */ .capabilities = PERF_PMU_CAP_HETEROGENEOUS_CPUS | PERF_PMU_CAP_EXTENDED_REGS, };
include/linux/perf/arm_pmu.h +1 -1
@@ -100,7 +100,7 @@ struct arm_pmu {void (*stop)(struct arm_pmu *);void (*reset)(void *);int (*map_event)(struct perf_event *event); -int (*filter_match)(struct perf_event *event); +bool (*filter)(struct pmu *pmu,int cpu);int num_events;bool secure_access;/* 32-bit ARM only */ #define ARMV8_PMUV3_MAX_COMMON_EVENTS 0x40
include/linux/perf_event.h +94 -31
237 lines not on this page · raw message
include/linux/sched.h +1 -1
@@ -1243,7 +1243,7 @@ struct task_struct {unsigned int futex_state;#endif #ifdef CONFIG_PERF_EVENTS -struct perf_event_context *perf_event_ctxp[perf_nr_task_contexts]; +struct perf_event_context *perf_event_ctxp;struct mutex perf_event_mutex;struct list_head perf_event_list;#endif
kernel/events/core.c +999 -959
2,259 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.