perf: Save PMU specific data in task_struct
Subsystems (4)
RFC PATCH v2 1/7 perf: Save PMU specific data in task_struct
From: Kan Liang <[email protected]>
Some PMU specific data has to be saved/restored during context switch, e.g. LBR call stack data. Currently, the data is saved in event context structure, but only for per-process event. For system-wide event, because of missing the LBR call stack data after context switch, LBR callstacks are always shorter in comparison to per-process mode.
For example,
Per-process mode:
$perf record --call-graph lbr -- taskset -c 0 ./tchain_edit
- 99.90% 99.86% tchain_edit tchain_edit [.] f3
99.86% _start
__libc_start_main
generic_start_main
main
f1
- f2
f3System-wide mode: $perf record --call-graph lbr -a -- taskset -c 0 ./tchain_edit
- 99.88% 99.82% tchain_edit tchain_edit [.] f3
- 62.02% main
f1
f2
f3
- 28.83% f1
- f2
f3
- 28.83% f1
- f2
f3
- 8.88% generic_start_main
main
f1
f2
f3It isn't practical to simply allocate the data for system-wide event in CPU context structure for all tasks. We have no idea which CPU a task will be scheduled to. The duplicated LBR data has to be maintained on every CPU context structure. That's a huge waste. Otherwise, the LBR data still lost if the task is scheduled to another CPU.
Save the pmu specific data in task_struct. The size of pmu specific data is 788 bytes for LBR call stack. Usually, the overall amount of threads doesn't exceed a few thousands. For 10K threads, keeping LBR data would consume additional ~8MB. The additional space will only be allocated during LBR call stack monitoring. It will be released when the monitoring is finished.
Furthermore, moving task_ctx_data from perf_event_context to task_struct can reduce complexity and make things clearer. E.g. perf doesn't need to swap task_ctx_data on optimized context switch path.
This patch set is just the first step. There could be other optimization/extension on top of this patch set. E.g. for cgroup profiling, perf just needs to save/store the LBR call stack information for tasks in specific cgroup. That could reduce the additional space. Also, the LBR call stack can be available for software events, or allow even debugging use cases, like LBRs on crash later.
The size of pmu specific data is saved in struct perf_ctx_data. It's required when child task allocates the space.
The refcount in struct perf_ctx_data is used to track the users of pmu specific data.
---
Changes since V1:
- Remove perf_ctx_data_lock. Use existing perf_event_mutex.
- Change type of refcount to refcount_t
include/linux/perf_event.h | 28 ++++++++++++++++++++++++++++ include/linux/sched.h | 2 ++ kernel/events/core.c | 1 + 3 files changed, 31 insertions(+)
include/linux/perf_event.h +28 -0
@@ -821,6 +821,34 @@ struct perf_event_context {struct rcu_head rcu_head; }; +/** +* struct perf_ctx_data - PMU specific data for a task +* @rcu_head: To avoid the race on free PMU specific data +* @refcount: To track users +* @data_size: Size of PMU specific data +* @data: PMU specific data +* +* Currently, the struct is only used in Intel LBR call stack mode to +* save/restore the call stack of a task on context switches. +* The data only be allocated when Intel LBR call stack mode is enabled. +* The data will be freed when the mode is disabled. The rcu_head is +* used to prevent the race on free the data. +* The content of the data will only be accessed in context switch, which +* should be protected by rcu_read_lock(). +* +* Careful: Struct perf_ctx_data is added as a pointor in struct task_struct. +* When system-wide Intel LBR call stack mode is enabled, a buffer with +* constant size will be allocated for each task. +* Also, system memory consumption can further grow when the size of +* struct perf_ctx_data enlarges. +*/ +struct perf_ctx_data { +struct rcu_head rcu_head; +refcount_t refcount; +size_t data_size; +void *data; +}; +/* * Number of contexts where an event can trigger: * task, softirq, hardirq, nmi.
include/linux/sched.h +2 -0
@@ -46,6 +46,7 @@ struct mempolicy;struct nameidata ;struct nsproxy ;struct perf_event_context ; +struct perf_ctx_data ;struct pid_namespace ;struct pipe_inode_info ;struct rcu_node ;
@@ -1059,6 +1060,7 @@ struct task_struct {struct perf_event_context *perf_event_ctxp[perf_nr_task_contexts];struct mutex perf_event_mutex;struct list_head perf_event_list; +struct perf_ctx_data __rcu *perf_ctx_data;#endif #ifdef CONFIG_DEBUG_PREEMPT unsigned long preempt_disable_ip;
kernel/events/core.c +1 -0
@@ -12321,6 +12321,7 @@ int perf_event_init_task(struct task_struct *child)memset (child->perf_event_ctxp,0 ,sizeof (child->perf_event_ctxp));mutex_init (&child->perf_event_mutex);INIT_LIST_HEAD (&child->perf_event_list); + child->perf_ctx_data =NULL ;for_each_task_context_nr (ctxn) { ret =perf_event_init_context (child, ctxn);
Signature · 1 line
2.7.4
RFC PATCH v2 2/7 perf: Init/fini PMU specific data
From: Kan Liang <[email protected]>
The PMU specific data for the monitored tasks only be allocated during LBR call stack monitoring.
When a LBR call stack event is accounted, the perf_ctx_data for related tasks will be allocated/initialized by init_perf_ctx_data(). When a LBR call stack event is unaccounted, the perf_ctx_data for related tasks will be detached/freed by fini_task_ctx_data().
Perf supports both per-process event and system-wide event.
- For per-process event, perf only allocates the perf_ctx_data for current task.
If the allocation fails, perf error out for per-process event.
- For system-wide event, perf has to allocate the perf_ctx_data for both existing tasks and upcoming tasks.
The allocation for the existing tasks is done in perf_event_alloc(). The allocation for new tasks will be done in perf_event_fork(). If any allocation fails, perf doesn't error out for system-wide event. A debug message will be dumped to system log instead. LBR callstack may be cutoff for the task which doesn't have the space allocated.
---
Changes since V1:
- Merge patch 2 and 3 of V1. Remove several helpers e.g. alloc/init/fini_task_ctx_data_rcu().
- Move the allocations out of spin lock
- Remove tasklist_lock. Use RCU iteration of the tasklist
- Retrieve total number of threads from global nr_threads
kernel/events/core.c | 325 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 325 insertions(+)
kernel/events/core.c +325 -0
@@ -47,6 +47,7 @@#include <linux/parser.h> #include <linux/sched/clock.h> #include <linux/sched/mm.h> +#include <linux/sched/stat.h> #include <linux/proc_ns.h> #include <linux/mount.h>
@@ -387,6 +388,10 @@ static atomic_t nr_switch_events __read_mostly;static atomic_t nr_ksymbol_events__read_mostly ;static atomic_t nr_bpf_events__read_mostly ; +/* Track the number of system-wide event which requires pmu specific data */ +static atomic_t nr_task_data_sys_wide_events; +static DEFINE_RAW_SPINLOCK (task_data_sys_wide_events_lock); +static LIST_HEAD (pmus);static DEFINE_MUTEX (pmus_lock);static struct srcu_struct pmus_srcu;
@@ -4440,6 +4445,256 @@ static void unaccount_freq_event(void)atomic_dec (&nr_freq_events); } +static int +alloc_perf_ctx_data (size_t ctx_size,gfp_t flags, +struct perf_ctx_data **task_ctx_data) +{ +struct perf_ctx_data *ctx_data; + + ctx_data =kzalloc (sizeof (struct perf_ctx_data ), flags); +if (!ctx_data) +return -ENOMEM ; + + ctx_data->data =kzalloc (ctx_size, flags); +if (!ctx_data->data) { +kfree (ctx_data); +return -ENOMEM ; + } + + ctx_data->data_size = ctx_size; + *task_ctx_data = ctx_data; + +return 0 ; +} + +static void +free_perf_ctx_data (struct perf_ctx_data *ctx_data) +{ +kfree (ctx_data->data); +kfree (ctx_data); +} + +static void +free_perf_ctx_data_rcu (struct rcu_head *rcu_head) +{ +struct perf_ctx_data *ctx_data; + + ctx_data = container_of(rcu_head, struct perf_ctx_data, rcu_head); +free_perf_ctx_data (ctx_data); +} + +static int +init_task_ctx_data (struct task_struct *task,size_t ctx_size) +{ +struct perf_ctx_data *ctx_data; + +if (alloc_perf_ctx_data (ctx_size,GFP_KERNEL , &ctx_data)) +return -ENOMEM ; + +mutex_lock (&task->perf_event_mutex); + +if (task->perf_ctx_data) { +refcount_inc (&task->perf_ctx_data->refcount); +free_perf_ctx_data (ctx_data); + }else { +refcount_set (&ctx_data->refcount,1 ); +/* System-wide event is active as well */ +if (atomic_read (&nr_task_data_sys_wide_events)) +refcount_inc (&ctx_data->refcount); +rcu_assign_pointer (task->perf_ctx_data, ctx_data); + } + +mutex_unlock (&task->perf_event_mutex); +return 0 ; +} + +static int +init_system_wide_ctx_data (size_t ctx_size) +{ +struct perf_ctx_data **data; +struct task_struct *g, *p; +unsigned long flags; +int i, num_thread, pos =0 , nr_new_tasks =0 ; + +/* Retrieve total number of threads */ + num_thread = nr_threads; + + data =kcalloc (num_thread,sizeof (*data),GFP_KERNEL ); +if (!data) { +printk_once (KERN_DEBUG +"Failed to allocate space for LBR callstack. " +"The LBR callstack for all tasks may be cutoff.\n" ); +return -ENOMEM ; + } + +/* Do a quick allocation in first round with GFP_ATOMIC. */ +for (i =0 ; i < num_thread; i++) { +if (alloc_perf_ctx_data (ctx_size,GFP_ATOMIC , &data[i])) +break ; + } + num_thread = i; + +raw_spin_lock_irqsave (&task_data_sys_wide_events_lock, flags); + +/* +* Allocate perf_ctx_data for all existing threads with the first +* system-wide event. +* The perf_ctx_data for new threads will be allocated in +* perf_event_fork(). +*/ +if (atomic_inc_return (&nr_task_data_sys_wide_events) >1 ) + goto unlock; + +rcu_read_lock (); +for_each_process_thread (g, p) { +mutex_lock (&p->perf_event_mutex); +if (p->perf_ctx_data) { +/* +* The perf_ctx_data for this thread may has been +* allocated by per-task event. +* Only update refcount for the case. +*/ +refcount_inc (&p->perf_ctx_data->refcount); +mutex_unlock (&p->perf_event_mutex); +continue ; + } + +if (pos < num_thread) { +refcount_set (&data[pos]->refcount,1 ); +rcu_assign_pointer (p->perf_ctx_data, data[pos++]); + }else { +/* +* There may be some new threads created, +* when we allocate space. +* Track the number in nr_new_tasks. +*/ + nr_new_tasks++; + } +mutex_unlock (&p->perf_event_mutex); + } +rcu_read_unlock (); + +raw_spin_unlock_irqrestore (&task_data_sys_wide_events_lock, flags); + +if (!nr_new_tasks) + goto end; + +/* +* Try again to allocate the space for the left tasks. +*/ +for (i =0 ; i < nr_new_tasks; i++) { +if (alloc_perf_ctx_data (ctx_size,GFP_KERNEL , &data[i])) +break ; + } + num_thread = i; + pos =0 ; + +raw_spin_lock_irqsave (&task_data_sys_wide_events_lock, flags); + +rcu_read_lock (); +for_each_process_thread (g, p) { + +mutex_lock (&p->perf_event_mutex); +if (p->perf_ctx_data) { +mutex_unlock (&p->perf_event_mutex); +continue ; + } + +if (pos < num_thread) { +refcount_set (&data[pos]->refcount,1 ); +rcu_assign_pointer (p->perf_ctx_data, data[pos]); + } + pos++; +mutex_unlock (&p->perf_event_mutex); + } +rcu_read_unlock (); + +if (pos >= num_thread) { +printk_once (KERN_DEBUG +"Failed to allocate space for LBR callstack. " +"The LBR callstack for some tasks may be cutoff.\n" ); + } +unlock: +raw_spin_unlock_irqrestore (&task_data_sys_wide_events_lock, flags); + +end: +for (; pos < num_thread; pos++) +free_perf_ctx_data (data[pos]); + +kfree (data); +return 0 ; +} + +static int +init_perf_ctx_data (struct perf_event *event) +{ +struct task_struct *task = event->hw.target; +size_t ctx_size = event->pmu->task_ctx_size; + +if (task) +return init_task_ctx_data (task, ctx_size); +else +return init_system_wide_ctx_data (ctx_size); +} + +/** +* Detach perf_ctx_data RCU pointer for a task +* @task: Target Task +* @force: Unconditionally free perf_ctx_data +* +* If force is set, free perf_ctx_data unconditionally. +* Otherwise, free perf_ctx_data when there are no users. +* Lock is required to sync the writers of perf_ctx_data RCU pointer +*/ +static void +fini_task_ctx_data (struct task_struct *task,bool force) +{ +struct perf_ctx_data *ctx_data; + +mutex_lock (&task->perf_event_mutex); + + ctx_data = task->perf_ctx_data; +if (!ctx_data) + goto unlock; + +if (!force && !refcount_dec_and_test (&ctx_data->refcount)) + goto unlock; + +RCU_INIT_POINTER (task->perf_ctx_data,NULL ); +call_rcu (&ctx_data->rcu_head, free_perf_ctx_data_rcu); + +unlock: +mutex_unlock (&task->perf_event_mutex); +} + +static void fini_system_wide_ctx_data (void ) +{ +struct task_struct *g, *p; +unsigned long flags; + +raw_spin_lock_irqsave (&task_data_sys_wide_events_lock, flags); +if (!atomic_dec_and_test (&nr_task_data_sys_wide_events)) + goto unlock; + +rcu_read_lock (); +for_each_process_thread (g, p) +fini_task_ctx_data (p, false); +rcu_read_unlock (); + +unlock: +raw_spin_unlock_irqrestore (&task_data_sys_wide_events_lock, flags); +} + +static void fini_perf_ctx_data (struct perf_event *event) +{ +struct task_struct *task = event->hw.target; + +if (task) +fini_task_ctx_data (task, false); +else +fini_system_wide_ctx_data (); +} +static void unaccount_event (struct perf_event *event) {bool dec = false;
@@ -4471,6 +4726,8 @@ static void unaccount_event(struct perf_event *event)atomic_dec (&nr_ksymbol_events);if (event->attr.bpf_event)atomic_dec (&nr_bpf_events); +if (event->attach_state &PERF_ATTACH_TASK_DATA ) +fini_perf_ctx_data (event);if (dec) {if (!atomic_add_unless (&perf_sched_count,-1 ,1 ))
@@ -7297,10 +7554,69 @@ static void perf_event_task(struct task_struct *task, task_ctx); } +/* +* Allocate data for a new task when profiling system-wide +* events which require PMU specific data +*/ +static void perf_event_alloc_task_data (struct task_struct *child, +struct task_struct *parent) +{ +struct perf_ctx_data *ctx_data; +size_t ctx_size =0 ; +unsigned long flags; + +if (!atomic_read (&nr_task_data_sys_wide_events)) +return ; + +rcu_read_lock (); + ctx_data =rcu_dereference (parent->perf_ctx_data); +if (ctx_data) + ctx_size = ctx_data->data_size; +rcu_read_unlock (); + +if (!ctx_size) +return ; + +if (alloc_perf_ctx_data (ctx_size,GFP_KERNEL , &ctx_data)) +return ; + +raw_spin_lock_irqsave (&task_data_sys_wide_events_lock, flags); + +/* +* System-wide event may be unaccount when attaching the perf_ctx_data. +* For example, +* CPU A CPU B +* perf_event_alloc_task_data(): +* read(nr_task_data_events) +* fini_system_wide_ctx_data() +* alloc_perf_ctx_data() +* rcu_assign_pointer(perf_ctx_data); +* +* The perf_ctx_data may never be freed until the task is terminated. +*/ +if (unlikely (!atomic_read (&nr_task_data_sys_wide_events))) { +free_perf_ctx_data (ctx_data); + goto unlock; + } + +mutex_lock (&child->perf_event_mutex); +if (child->perf_ctx_data) { +free_perf_ctx_data (ctx_data); + }else { +refcount_set (&ctx_data->refcount,1 ); +rcu_assign_pointer (child->perf_ctx_data, ctx_data); + } +mutex_unlock (&child->perf_event_mutex); + +unlock: +raw_spin_unlock_irqrestore (&task_data_sys_wide_events_lock, flags); +} +void perf_event_fork (struct task_struct *task) {perf_event_task (task,NULL ,1 );perf_event_namespaces (task); +perf_event_alloc_task_data (task, current); }/*
@@ -10826,11 +11142,18 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,if (err) goto err_callchain_buffer; +if ((event->attach_state &PERF_ATTACH_TASK_DATA ) && +init_perf_ctx_data (event)) + goto err_task_ctx_data; +/* symmetric to unaccount_event() in _free_event() */ account_event (event);return event; +err_task_ctx_data: +if (!event->parent && (event->attr.sample_type &PERF_SAMPLE_CALLCHAIN )) +put_callchain_buffers (); err_callchain_buffer:if (!event->parent) {if (event->attr.sample_type &PERF_SAMPLE_CALLCHAIN )
@@ -11892,6 +12215,8 @@ void perf_event_exit_task(struct task_struct *child)* At this point we need to send EXIT events to cpu contexts. */ perf_event_task (child,NULL ,0 ); + +fini_task_ctx_data (child, true); } static void perf_free_event(struct perf_event *event,
Signature · 1 line
2.7.4
Peter Zijlstra
maintainerRe: [RFC PATCH V2 2/7] perf: Init/fini PMU specific data
Quoted 9 lines · On Fri, Jan 03, 2020 at 11:39:19AM -0800, [email protected] wrote:
From: Kan Liang <[email protected]>
The PMU specific data for the monitored tasks only be allocated during LBR call stack monitoring.
When a LBR call stack event is accounted, the
perf_ctx_datafor related tasks will be allocated/initialized byinit_perf_ctx_data(). When a LBR call stack event is unaccounted, theperf_ctx_datafor related tasks will be detached/freed byfini_task_ctx_data().
I hate that 'fini' name. We don't use that anywhere else.
Quoted 31 lines
kernel/events/core.c +31
-0+raw_spin_lock_irqsave (&task_data_sys_wide_events_lock, flags); + +/* +* Allocate perf_ctx_data for all existing threads with the first +* system-wide event. +* The perf_ctx_data for new threads will be allocated in +* perf_event_fork(). +*/ +if (atomic_inc_return (&nr_task_data_sys_wide_events) >1 ) + goto unlock; + +rcu_read_lock (); +for_each_process_thread (g, p) { +mutex_lock (&p->perf_event_mutex); +if (p->perf_ctx_data) { +/* +* The perf_ctx_data for this thread may has been +* allocated by per-task event. +* Only update refcount for the case. +*/ +refcount_inc (&p->perf_ctx_data->refcount); +mutex_unlock (&p->perf_event_mutex); +continue ; + } + +if (pos < num_thread) { +refcount_set (&data[pos]->refcount,1 ); +rcu_assign_pointer (p->perf_ctx_data, data[pos++]); + }else { +/* +* There may be some new threads created,
kernel/events/core.c static void unaccount_freq_event(void) · line 4577
Quoted 10 lines
kernel/events/core.c +10
-0+* when we allocate space. +* Track the number in nr_new_tasks. +*/ + nr_new_tasks++; + } +mutex_unlock (&p->perf_event_mutex); + } +rcu_read_unlock (); + +raw_spin_unlock_irqrestore (&task_data_sys_wide_events_lock, flags);
Still NAK. That's some mightly broken code there.
Re: [RFC PATCH V2 2/7] perf: Init/fini PMU specific data
Quoted 24 lines
Quoted 24 lines
kernel/events/core.c +24
-0+rcu_read_lock (); +for_each_process_thread (g, p) { +mutex_lock (&p->perf_event_mutex); +if (p->perf_ctx_data) { +/* +* The perf_ctx_data for this thread may has been +* allocated by per-task event. +* Only update refcount for the case. +*/ +refcount_inc (&p->perf_ctx_data->refcount); +mutex_unlock (&p->perf_event_mutex); +continue ; + } + +if (pos < num_thread) { +refcount_set (&data[pos]->refcount,1 ); +rcu_assign_pointer (p->perf_ctx_data, data[pos++]); + }else { +/* +* There may be some new threads created, +* when we allocate space. +* Track the number in nr_new_tasks. +*/ + nr_new_tasks++;
Quoted 8 lines
kernel/events/core.c static void unaccount_freq_event(void) · line 4577
Quoted 6 lines
kernel/events/core.c +6
-0+ } +mutex_unlock (&p->perf_event_mutex); + } +rcu_read_unlock (); + +raw_spin_unlock_irqrestore (&task_data_sys_wide_events_lock, flags);Still NAK. That's some mightly broken code there.
Yes, Kan you cannot use a mutex (sleeping) inside rcu_read_lock(). Can perf_event_mutex be a spin lock?
-Andi
Peter Zijlstra
maintainerRe: [RFC PATCH V2 2/7] perf: Init/fini PMU specific data
Quoted 33 lines · Andi Kleen
Quoted 24 lines
Quoted 24 lines
kernel/events/core.c +24
-0+rcu_read_lock (); +for_each_process_thread (g, p) { +mutex_lock (&p->perf_event_mutex); +if (p->perf_ctx_data) { +/* +* The perf_ctx_data for this thread may has been +* allocated by per-task event. +* Only update refcount for the case. +*/ +refcount_inc (&p->perf_ctx_data->refcount); +mutex_unlock (&p->perf_event_mutex); +continue ; + } + +if (pos < num_thread) { +refcount_set (&data[pos]->refcount,1 ); +rcu_assign_pointer (p->perf_ctx_data, data[pos++]); + }else { +/* +* There may be some new threads created, +* when we allocate space. +* Track the number in nr_new_tasks. +*/ + nr_new_tasks++;Quoted 8 lines
kernel/events/core.c static void unaccount_freq_event(void) · line 4577
Quoted 6 lines
kernel/events/core.c +6
-0+ } +mutex_unlock (&p->perf_event_mutex); + } +rcu_read_unlock (); + +raw_spin_unlock_irqrestore (&task_data_sys_wide_events_lock, flags);Still NAK. That's some mightly broken code there.
Quoted 2 lines
Yes, Kan you cannot use a mutex (sleeping) inside
rcu_read_lock(). Canperf_event_mutexbe a spin lock?
Or insize that raw_spin_lock. And last time I expressly said to not do what whole tasklist iteration under a spinlock.
Re: [RFC PATCH V2 2/7] perf: Init/fini PMU specific data
Quoted 34 lines · Peter Zijlstra
Quoted 33 lines · Andi Kleen
Quoted 32 lines
Quoted 20 lines
kernel/events/core.c +20
-0+rcu_read_lock (); +for_each_process_thread (g, p) { +mutex_lock (&p->perf_event_mutex); +if (p->perf_ctx_data) { +/* +* The perf_ctx_data for this thread may has been +* allocated by per-task event. +* Only update refcount for the case. +*/ +refcount_inc (&p->perf_ctx_data->refcount); +mutex_unlock (&p->perf_event_mutex); +continue ; + } + +if (pos < num_thread) { +refcount_set (&data[pos]->refcount,1 ); +rcu_assign_pointer (p->perf_ctx_data, data[pos++]); + }else { +/* +* There may be some new threads created, kernel/events/core.c static void unaccount_freq_event(void) · line 4577
Quoted 10 lines
kernel/events/core.c +10
-0+* when we allocate space. +* Track the number in nr_new_tasks. +*/ + nr_new_tasks++; + } +mutex_unlock (&p->perf_event_mutex); + } +rcu_read_unlock (); + +raw_spin_unlock_irqrestore (&task_data_sys_wide_events_lock, flags);Still NAK. That's some mightly broken code there.
Quoted 4 lines
Quoted 2 lines
Yes, Kan you cannot use a mutex (sleeping) inside
rcu_read_lock(). Canperf_event_mutexbe a spin lock?Or insize that
raw_spin_lock.
The task_data_sys_wide_events_lock is a global lock. I think we just need per-task lock here.
I think I will add a new dedicate per-task raw_spin_lock for perf_ctx_data here.
Quoted 3 lines
And last time I expressly said to not do
what whole tasklist iteration under a spinlock.
We need an indicator to tell if the assignment for all existing tasks has been finished. Because we have to wait before processing the cases as below.
- Allocate the space for new threads. If we don't wait the assignments finished, we cannot tell if the perf_ctx_data is allocated by previous per-task event, or this system-wide event. The refcount may double count.
- There may be two or more system-wide events at the same time. When we are allocating the space for the first event, the second one may start profiling if we don't wait. The LBR shorten issue still exists.
- We have to serialize assignment and free.
If we cannot use spinlock to serialize the cases here, can we set a state when the assignment is finished, and use wait_var_event() in the cases as above?
Thanks,
Kan
RFC PATCH v2 3/7 perf: Supply task information to sched_task()
In reply to [email protected]From: Kan Liang <[email protected]>
To save/restore LBR call stack data in system-wide mode, the task_struct information is required.
Extend the parameters of sched_task() to supply task_struct information.
When schedule in, the LBR call stack data for new task will be restored. When schedule out, the LBR call stack data for old task will be saved. Only need to pass the required task_struct information.
---
No changes since V1
arch/powerpc/perf/core-book3s.c | 8 ++++++-- arch/x86/events/core.c | 5 +++-- arch/x86/events/intel/core.c | 4 ++-- arch/x86/events/intel/lbr.c | 3 ++- arch/x86/events/perf_event.h | 5 +++-- include/linux/perf_event.h | 2 +- kernel/events/core.c | 2 +- 7 files changed, 18 insertions(+), 11 deletions(-)
arch/powerpc/perf/core-book3s.c +6 -2
@@ -126,7 +126,10 @@ static unsigned long ebb_switch_in(bool ebb, struct cpu_hw_events *cpuhw)static inline void power_pmu_bhrb_enable (struct perf_event *event) {}static inline void power_pmu_bhrb_disable (struct perf_event *event) {} -static void power_pmu_sched_task (struct perf_event_context *ctx,bool sched_in) {} +static void power_pmu_sched_task (struct perf_event_context *ctx, +struct task_struct *task,bool sched_in) +{ +}static inline void power_pmu_bhrb_read (struct perf_event *event,struct cpu_hw_events *cpuhw) {}static void pmao_restore_workaround (bool ebb) { }#endif /* CONFIG_PPC32 */
@@ -403,7 +406,8 @@ static void power_pmu_bhrb_disable(struct perf_event *event)/* Called from ctxsw to prevent one process's branch entries to * mingle with the other process's entries during context switch. */ -static void power_pmu_sched_task (struct perf_event_context *ctx,bool sched_in) +static void power_pmu_sched_task (struct perf_event_context *ctx, +struct task_struct *task,bool sched_in) {if (!ppmu->bhrb_nr)return ;
arch/x86/events/core.c +3 -2
@@ -2237,10 +2237,11 @@ static const struct attribute_group *x86_pmu_attr_groups[] = { NULL, }; -static void x86_pmu_sched_task (struct perf_event_context *ctx,bool sched_in) +static void x86_pmu_sched_task (struct perf_event_context *ctx, +struct task_struct *task,bool sched_in) {if (x86_pmu.sched_task) - x86_pmu.sched_task (ctx, sched_in); + x86_pmu.sched_task (ctx, task, sched_in); } static void x86_pmu_swap_task_ctx(struct perf_event_context *prev,
arch/x86/events/intel/core.c +2 -2
@@ -3814,10 +3814,10 @@ static void intel_pmu_cpu_dead(int cpu) }static void intel_pmu_sched_task (struct perf_event_context *ctx, -bool sched_in) +struct task_struct *task,bool sched_in) {intel_pmu_pebs_sched_task (ctx, sched_in); -intel_pmu_lbr_sched_task (ctx, sched_in); +intel_pmu_lbr_sched_task (ctx, task, sched_in); } static void intel_pmu_swap_task_ctx(struct perf_event_context *prev,
arch/x86/events/intel/lbr.c +2 -1
@@ -440,7 +440,8 @@ void intel_pmu_lbr_swap_task_ctx(struct perf_event_context *prev, next_ctx_data->lbr_callstack_users); } -void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in) +void intel_pmu_lbr_sched_task (struct perf_event_context *ctx, +struct task_struct *task,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);struct x86_perf_task_context *task_ctx;
arch/x86/events/perf_event.h +3 -2
@@ -639,7 +639,7 @@ struct x86_pmu {void (*check_microcode)(void );void (*sched_task)(struct perf_event_context *ctx, -bool sched_in); +struct task_struct *task,bool sched_in);/* * Intel Arch Perfmon v2+
@@ -1027,7 +1027,8 @@ void intel_ds_init(void);void intel_pmu_lbr_swap_task_ctx (struct perf_event_context *prev,struct perf_event_context *next); -void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,bool sched_in); +void intel_pmu_lbr_sched_task (struct perf_event_context *ctx, +struct task_struct *task,bool sched_in);u64 lbr_from_signext_quirk_wr (u64 val);
include/linux/perf_event.h +1 -1
@@ -406,7 +406,7 @@ struct pmu {* context-switches callback */ void (*sched_task) (struct perf_event_context *ctx, -bool sched_in); +struct task_struct *task,bool sched_in);/* * PMU specific data size */
kernel/events/core.c +1 -1
@@ -3336,7 +3336,7 @@ static void perf_pmu_sched_task(struct task_struct *prev,perf_ctx_lock (cpuctx, cpuctx->task_ctx);perf_pmu_disable (pmu); - pmu->sched_task (cpuctx->task_ctx, sched_in); + pmu->sched_task (cpuctx->task_ctx, sched_in ? next : prev, sched_in);perf_pmu_enable (pmu);perf_ctx_unlock (cpuctx, cpuctx->task_ctx);
Signature · 1 line
2.7.4
RFC PATCH v2 4/7 perf/x86/lbr: Fix shorter LBRs call stacks for system-wide mode
In reply to [email protected]From: Kan Liang <[email protected]>
In system-wide mode, LBR callstacks are shorter in comparison to per-process mode.
LBR MSRs are reset during context switch in system-wide mode. For LBR call stack, the LBRs should be always saved/restored during context switch.
Use the space in task_struct to save/restore the LBR call stack data.
For system-wide event, it's unnecessagy to update the lbr_callstack_users for each threads. Add a variable in x86_pmu to indicate if the system-wide event is active.
---
No changes since V1
arch/x86/events/intel/lbr.c | 57 ++++++++++++++++++++++++++++++++++---------- arch/x86/events/perf_event.h | 1 + 2 files changed, 45 insertions(+), 13 deletions(-)
arch/x86/events/intel/lbr.c +44 -13
@@ -337,6 +337,12 @@ static inline u64 rdlbr_to(unsigned int idx)return val; } +static bool has_lbr_callstack_users (struct x86_perf_task_context *task_ctx) +{ +return task_ctx->lbr_callstack_users || + x86_pmu.lbr_callstack_users; +} +static void __intel_pmu_lbr_restore (struct x86_perf_task_context *task_ctx) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -344,7 +350,7 @@ static void __intel_pmu_lbr_restore(struct x86_perf_task_context *task_ctx)unsigned lbr_idx, mask;u64 tos; -if (task_ctx->lbr_callstack_users ==0 || +if (!has_lbr_callstack_users (task_ctx) || task_ctx->lbr_stack_state ==LBR_NONE ) {intel_pmu_lbr_reset ();return ;
@@ -392,7 +398,7 @@ static void __intel_pmu_lbr_save(struct x86_perf_task_context *task_ctx)u64 tos, from;int i; -if (task_ctx->lbr_callstack_users ==0 ) { +if (!has_lbr_callstack_users (task_ctx)) { task_ctx->lbr_stack_state =LBR_NONE ;return ; }
@@ -445,6 +451,7 @@ void intel_pmu_lbr_sched_task(struct perf_event_context *ctx, {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);struct x86_perf_task_context *task_ctx; +struct perf_ctx_data *ctx_data;if (!cpuc->lbr_users)return ;
@@ -454,15 +461,18 @@ void intel_pmu_lbr_sched_task(struct perf_event_context *ctx,* the task was scheduled out, restore the stack. Otherwise flush * the LBR stack. */ - task_ctx = ctx ? ctx->task_ctx_data :NULL ; +rcu_read_lock (); + ctx_data =rcu_dereference (task->perf_ctx_data); + task_ctx = ctx_data ? (struct x86_perf_task_context *) ctx_data->data :NULL ;if (task_ctx) {if (sched_in)__intel_pmu_lbr_restore (task_ctx);else __intel_pmu_lbr_save (task_ctx); +rcu_read_unlock ();return ; } - +rcu_read_unlock ();/* * Since a context switch can flip the address space and LBR entries * are not tagged with an identifier, we need to wipe the LBR, even for
@@ -481,16 +491,27 @@ static inline bool branch_user_callstack(unsigned br_sel)void intel_pmu_lbr_add (struct perf_event *event) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -struct x86_perf_task_context *task_ctx;if (!x86_pmu.lbr_nr)return ; cpuc->br_sel = event->hw.branch_reg.reg; -if (branch_user_callstack (cpuc->br_sel) && event->ctx->task_ctx_data) { - task_ctx = event->ctx->task_ctx_data; - task_ctx->lbr_callstack_users++; +if (branch_user_callstack (cpuc->br_sel)) { +if (event->attach_state &PERF_ATTACH_TASK ) { +struct x86_perf_task_context *task_ctx; +struct task_struct *task = event->hw.target; +struct perf_ctx_data *ctx_data; + +rcu_read_lock (); + ctx_data =rcu_dereference (task->perf_ctx_data); +if (ctx_data) { + task_ctx = (struct x86_perf_task_context *)ctx_data->data; + task_ctx->lbr_callstack_users++; + } +rcu_read_unlock (); + }else + x86_pmu.lbr_callstack_users++; }/*
@@ -522,15 +543,25 @@ void intel_pmu_lbr_add(struct perf_event *event)void intel_pmu_lbr_del (struct perf_event *event) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -struct x86_perf_task_context *task_ctx;if (!x86_pmu.lbr_nr)return ; -if (branch_user_callstack (cpuc->br_sel) && - event->ctx->task_ctx_data) { - task_ctx = event->ctx->task_ctx_data; - task_ctx->lbr_callstack_users--; +if (branch_user_callstack (cpuc->br_sel)) { +if (event->attach_state &PERF_ATTACH_TASK ) { +struct task_struct *task = event->hw.target; +struct x86_perf_task_context *task_ctx; +struct perf_ctx_data *ctx_data; + +rcu_read_lock (); + ctx_data =rcu_dereference (task->perf_ctx_data); +if (ctx_data) { + task_ctx = (struct x86_perf_task_context *)ctx_data->data; + task_ctx->lbr_callstack_users--; + } +rcu_read_unlock (); + }else + x86_pmu.lbr_callstack_users--; }if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 )
arch/x86/events/perf_event.h +1 -0
@@ -674,6 +674,7 @@ struct x86_pmu {int lbr_nr;/* hardware stack size */ u64 lbr_sel_mask;/* LBR_SELECT valid bits */ const int *lbr_sel_map;/* lbr_select mappings */ +u64 lbr_callstack_users;/* lbr callstack system wide users */ bool lbr_double_abort;/* duplicated lbr aborts */ bool lbr_pt_coexist;/* (LBR|BTS) may coexist with PT */
Signature · 1 line
2.7.4
RFC PATCH v2 5/7 perf/x86: Remove swap_task_ctx()
In reply to [email protected]From: Kan Liang <[email protected]>
The pmu specific data is saved in task_struct now. It doesn't need to swap between context.
Remove swap_task_ctx() support.
---
No changes since V1
arch/x86/events/core.c | 8 -------- arch/x86/events/intel/core.c | 7 ------- arch/x86/events/intel/lbr.c | 23 ----------------------- arch/x86/events/perf_event.h | 11 ----------- 4 files changed, 49 deletions(-)
arch/x86/events/core.c +0 -8
@@ -2244,13 +2244,6 @@ static void x86_pmu_sched_task(struct perf_event_context *ctx, x86_pmu.sched_task (ctx, task, sched_in); } -static void x86_pmu_swap_task_ctx (struct perf_event_context *prev, -struct perf_event_context *next) -{ -if (x86_pmu.swap_task_ctx) - x86_pmu.swap_task_ctx (prev, next); -} -void perf_check_microcode (void ) {if (x86_pmu.check_microcode)
@@ -2305,7 +2298,6 @@ static struct pmu pmu = { .event_idx = x86_pmu_event_idx, .sched_task = x86_pmu_sched_task, .task_ctx_size = sizeof(struct x86_perf_task_context), - .swap_task_ctx = x86_pmu_swap_task_ctx, .check_period = x86_pmu_check_period, .aux_output_match = x86_pmu_aux_output_match,
arch/x86/events/intel/core.c +0 -7
@@ -3820,12 +3820,6 @@ static void intel_pmu_sched_task(struct perf_event_context *ctx,intel_pmu_lbr_sched_task (ctx, task, sched_in); } -static void intel_pmu_swap_task_ctx (struct perf_event_context *prev, -struct perf_event_context *next) -{ -intel_pmu_lbr_swap_task_ctx (prev, next); -} -static int intel_pmu_check_period (struct perf_event *event,u64 value) {return intel_pmu_has_bts_period (event, value) ? -EINVAL :0 ;
@@ -3961,7 +3955,6 @@ static __initconst const struct x86_pmu intel_pmu = { .guest_get_msrs = intel_guest_get_msrs, .sched_task = intel_pmu_sched_task, - .swap_task_ctx = intel_pmu_swap_task_ctx, .check_period = intel_pmu_check_period,
arch/x86/events/intel/lbr.c +0 -23
@@ -423,29 +423,6 @@ static void __intel_pmu_lbr_save(struct x86_perf_task_context *task_ctx) cpuc->last_log_id = ++task_ctx->log_id; } -void intel_pmu_lbr_swap_task_ctx (struct perf_event_context *prev, -struct perf_event_context *next) -{ -struct x86_perf_task_context *prev_ctx_data, *next_ctx_data; - -swap (prev->task_ctx_data, next->task_ctx_data); - -/* -* Architecture specific synchronization makes sense in -* case both prev->task_ctx_data and next->task_ctx_data -* pointers are allocated. -*/ - - prev_ctx_data = next->task_ctx_data; - next_ctx_data = prev->task_ctx_data; - -if (!prev_ctx_data || !next_ctx_data) -return ; - -swap (prev_ctx_data->lbr_callstack_users, - next_ctx_data->lbr_callstack_users); -} -void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,struct task_struct *task,bool sched_in) {
arch/x86/events/perf_event.h +0 -11
@@ -684,14 +684,6 @@ struct x86_pmu {atomic_t lbr_exclusive[x86_lbr_exclusive_max];/* -* perf task context (i.e. struct perf_event_context::task_ctx_data) -* switch helper to bridge calls from perf/core to perf/x86. -* See struct pmu::swap_task_ctx() usage for examples; -*/ -void (*swap_task_ctx)(struct perf_event_context *prev, -struct perf_event_context *next); - -/* * AMD bits */ unsigned int amd_nb_constraints : 1;
@@ -1025,9 +1017,6 @@ void intel_pmu_store_pebs_lbrs(struct pebs_lbr *lbr);void intel_ds_init (void ); -void intel_pmu_lbr_swap_task_ctx (struct perf_event_context *prev, -struct perf_event_context *next); -void intel_pmu_lbr_sched_task (struct perf_event_context *ctx,struct task_struct *task,bool sched_in);
Signature · 1 line
2.7.4
RFC PATCH v2 6/7 perf: Clean up pmu specific data
In reply to [email protected]From: Kan Liang <[email protected]>
The pmu specific data is saved in task_struct now. Remove it from event context structure.
Remove swap_task_ctx() as well.
---
No changes since V1
include/linux/perf_event.h | 11 ---------- kernel/events/core.c | 53 ++-------------------------------------------- 2 files changed, 2 insertions(+), 62 deletions(-)
include/linux/perf_event.h +0 -11
@@ -413,16 +413,6 @@ struct pmu {size_t task_ctx_size;/* -* PMU specific parts of task perf event context (i.e. ctx->task_ctx_data) -* can be synchronized using this function. See Intel LBR callstack support -* implementation and Perf core context switch handling callbacks for usage -* examples. -*/ -void (*swap_task_ctx) (struct perf_event_context *prev, -struct perf_event_context *next); -/* optional */ - -/* * Set up pmu-private data structures for an AUX area */ void *(*setup_aux) (struct perf_event *event, void **pages,
@@ -817,7 +807,6 @@ struct perf_event_context {#ifdef CONFIG_CGROUP_PERF int nr_cgroups;/* cgroup evts */ #endif -void *task_ctx_data;/* pmu specific data */ struct rcu_head rcu_head; };
kernel/events/core.c +2 -51
116 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH v2 7/7 perf: Clean up event context from sched_task()
In reply to [email protected]From: Kan Liang <[email protected]>
No one uses event context in sched_task() anymore.
Remove the event context parameters from sched_task(). ctx_lock is useless as well.
---
No changes since V1
arch/powerpc/perf/core-book3s.c | 8 ++------ arch/x86/events/core.c | 5 ++--- arch/x86/events/intel/core.c | 7 +++---- arch/x86/events/intel/ds.c | 2 +- arch/x86/events/intel/lbr.c | 3 +-- arch/x86/events/perf_event.h | 8 +++----- include/linux/perf_event.h | 3 +-- kernel/events/core.c | 4 +--- 8 files changed, 14 insertions(+), 26 deletions(-)
arch/powerpc/perf/core-book3s.c +2 -6
@@ -126,10 +126,7 @@ static unsigned long ebb_switch_in(bool ebb, struct cpu_hw_events *cpuhw)static inline void power_pmu_bhrb_enable (struct perf_event *event) {}static inline void power_pmu_bhrb_disable (struct perf_event *event) {} -static void power_pmu_sched_task (struct perf_event_context *ctx, -struct task_struct *task,bool sched_in) -{ -} +static void power_pmu_sched_task (struct task_struct *task,bool sched_in) {}static inline void power_pmu_bhrb_read (struct perf_event *event,struct cpu_hw_events *cpuhw) {}static void pmao_restore_workaround (bool ebb) { }#endif /* CONFIG_PPC32 */
@@ -406,8 +403,7 @@ static void power_pmu_bhrb_disable(struct perf_event *event)/* Called from ctxsw to prevent one process's branch entries to * mingle with the other process's entries during context switch. */ -static void power_pmu_sched_task (struct perf_event_context *ctx, -struct task_struct *task,bool sched_in) +static void power_pmu_sched_task (struct task_struct *task,bool sched_in) {if (!ppmu->bhrb_nr)return ;
arch/x86/events/core.c +2 -3
@@ -2237,11 +2237,10 @@ static const struct attribute_group *x86_pmu_attr_groups[] = { NULL, }; -static void x86_pmu_sched_task (struct perf_event_context *ctx, -struct task_struct *task,bool sched_in) +static void x86_pmu_sched_task (struct task_struct *task,bool sched_in) {if (x86_pmu.sched_task) - x86_pmu.sched_task (ctx, task, sched_in); + x86_pmu.sched_task (task, sched_in); }void perf_check_microcode (void )
arch/x86/events/intel/core.c +3 -4
@@ -3813,11 +3813,10 @@ static void intel_pmu_cpu_dead(int cpu)intel_cpuc_finish (&per_cpu (cpu_hw_events, cpu)); } -static void intel_pmu_sched_task (struct perf_event_context *ctx, -struct task_struct *task,bool sched_in) +static void intel_pmu_sched_task (struct task_struct *task,bool sched_in) { -intel_pmu_pebs_sched_task (ctx, sched_in); -intel_pmu_lbr_sched_task (ctx, task, sched_in); +intel_pmu_pebs_sched_task (sched_in); +intel_pmu_lbr_sched_task (task, sched_in); }static int intel_pmu_check_period (struct perf_event *event,u64 value)
arch/x86/events/intel/ds.c +1 -1
@@ -908,7 +908,7 @@ static inline bool pebs_needs_sched_cb(struct cpu_hw_events *cpuc)return cpuc->n_pebs && (cpuc->n_pebs == cpuc->n_large_pebs); } -void intel_pmu_pebs_sched_task (struct perf_event_context *ctx,bool sched_in) +void intel_pmu_pebs_sched_task (bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
arch/x86/events/intel/lbr.c +1 -2
@@ -423,8 +423,7 @@ static void __intel_pmu_lbr_save(struct x86_perf_task_context *task_ctx) cpuc->last_log_id = ++task_ctx->log_id; } -void intel_pmu_lbr_sched_task (struct perf_event_context *ctx, -struct task_struct *task,bool sched_in) +void intel_pmu_lbr_sched_task (struct task_struct *task,bool sched_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);struct x86_perf_task_context *task_ctx;
arch/x86/events/perf_event.h +3 -5
@@ -638,8 +638,7 @@ struct x86_pmu {void (*cpu_dead)(int cpu);void (*check_microcode)(void ); -void (*sched_task)(struct perf_event_context *ctx, -struct task_struct *task,bool sched_in); +void (*sched_task)(struct task_struct *task,bool sched_in);/* * Intel Arch Perfmon v2+
@@ -1009,7 +1008,7 @@ void intel_pmu_pebs_enable_all(void);void intel_pmu_pebs_disable_all (void ); -void intel_pmu_pebs_sched_task (struct perf_event_context *ctx,bool sched_in); +void intel_pmu_pebs_sched_task (bool sched_in);void intel_pmu_auto_reload_read (struct perf_event *event);
@@ -1017,8 +1016,7 @@ void intel_pmu_store_pebs_lbrs(struct pebs_lbr *lbr);void intel_ds_init (void ); -void intel_pmu_lbr_sched_task (struct perf_event_context *ctx, -struct task_struct *task,bool sched_in); +void intel_pmu_lbr_sched_task (struct task_struct *task,bool sched_in);u64 lbr_from_signext_quirk_wr (u64 val);
include/linux/perf_event.h +1 -2
@@ -405,8 +405,7 @@ struct pmu {/* * context-switches callback */ -void (*sched_task) (struct perf_event_context *ctx, -struct task_struct *task,bool sched_in); +void (*sched_task) (struct task_struct *task,bool sched_in);/* * PMU specific data size */
kernel/events/core.c +1 -3
@@ -3318,13 +3318,11 @@ static void perf_pmu_sched_task(struct task_struct *prev,if (WARN_ON_ONCE (!pmu->sched_task))continue ; -perf_ctx_lock (cpuctx, cpuctx->task_ctx);perf_pmu_disable (pmu); - pmu->sched_task (cpuctx->task_ctx, sched_in ? next : prev, sched_in); + pmu->sched_task (sched_in ? next : prev, sched_in);perf_pmu_enable (pmu); -perf_ctx_unlock (cpuctx, cpuctx->task_ctx); } }
Signature · 1 line
2.7.4