perf: Per-pmu-per-cpu contexts
Commit-ID: 108b02cfce04ee90b0a07ee0b104baffd39f5934 Gitweb: http://git.kernel.org/tip/108b02cfce04ee90b0a07ee0b104baffd39f5934 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 6 Sep 2010 14:32:03 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 9 Sep 2010 20:46:32 +0200
perf: Per-pmu-per-cpu contexts
Allocate per-cpu contexts per pmu.
LKML-Reference: <new-submission>
--- include/linux/perf_event.h | 4 +- kernel/perf_event.c | 178 +++++++++++++++++++++++++++----------------- 2 files changed, 113 insertions(+), 69 deletions(-)
include/linux/perf_event.h +3 -1
@@ -570,7 +570,8 @@ struct perf_event;struct pmu {struct list_head entry; -int *pmu_disable_count; +int *__percpu pmu_disable_count; +struct perf_cpu_context *__percpu pmu_cpu_context;/* * Fully disable/enable this PMU, can be used to protect from the PMI
@@ -808,6 +809,7 @@ struct perf_event {* Used as a container for task events and CPU events as well: */ struct perf_event_context { +struct pmu *pmu;/* * Protect the states of the events in the list, * nr_active, and the list:
kernel/perf_event.c +110 -68
449 lines not on this page · raw message
Quoted 9 lines · tip-bot for Peter Zijlstra
kernel/perf_event.c +2
-1@@ -3745,18 +3757,20 @@ static void perf_event_task_ctx(struct perf_event_context *ctx,static void perf_event_task_event (struct perf_task_event *task_event) { -struct perf_cpu_context *cpuctx;struct perf_event_context *ctx = task_event->task_ctx; +struct perf_cpu_context *cpuctx; +struct pmu *pmu;
kernel/perf_event.c static void perf_event_task_ctx(struct perf_event_context *ctx, · line 3770
Quoted 10 lines
kernel/perf_event.c +5
-3-rcu_read_lock (); - cpuctx = &get_cpu_var (perf_cpu_context); -perf_event_task_ctx (&cpuctx->ctx, task_event); +rcu_read_lock_sched (); +list_for_each_entry_rcu (pmu, &pmus, entry) { + cpuctx =this_cpu_ptr (pmu->pmu_cpu_context); +perf_event_task_ctx (&cpuctx->ctx, task_event); + }if (!ctx) ctx =rcu_dereference (current->perf_event_ctxp);
So, you say below that it works because synchronize_srcu(), that waits for qs after touching pmus, implies synchronize_sched(), right?
And I guess you picked rcu_read_lock_sched() here because that preempt_disable() at the same time.
That looks complicated but I guess that works.
That said there is also this rcu_dereference(current->perf_event_ctxp). Now, this ctx is released after srcu barrier right? So this should be srcu_dereference(). But then you seem to actually use rcu_read_lock_sched() as it's compatible, so this should be rcu_dereference_sched() ?
With the current state, rcu will whine.
Moreover there seem to be too much game between the different rcu flavours here, and that breaks the reviewers parsing.
Quoted 10 lines · Frederic Weisbecker
Quoted 9 lines · tip-bot for Peter Zijlstra
kernel/perf_event.c +2
-1@@ -3745,18 +3757,20 @@ static void perf_event_task_ctx(struct perf_event_context *ctx,static void perf_event_task_event (struct perf_task_event *task_event) { -struct perf_cpu_context *cpuctx;struct perf_event_context *ctx = task_event->task_ctx; +struct perf_cpu_context *cpuctx; +struct pmu *pmu;
Quoted 15 lines
kernel/perf_event.c static void perf_event_task_ctx(struct perf_event_context *ctx, · line 3770
Quoted 10 lines
kernel/perf_event.c +5
-3-rcu_read_lock (); - cpuctx = &get_cpu_var (perf_cpu_context); -perf_event_task_ctx (&cpuctx->ctx, task_event); +rcu_read_lock_sched (); +list_for_each_entry_rcu (pmu, &pmus, entry) { + cpuctx =this_cpu_ptr (pmu->pmu_cpu_context); +perf_event_task_ctx (&cpuctx->ctx, task_event); + }if (!ctx) ctx =rcu_dereference (current->perf_event_ctxp);So, you say below that it works because
synchronize_srcu(), that waits for qs after touching pmus, impliessynchronize_sched(), right?
Ook... My current plans to fold SRCU into TREE_RCU would invalidate this assumption.
Maybe we need some sort of primitive that concurrently waits for multiple types of RCU grace periods?
Thanx, Paul
Quoted 14 lines
And I guess you picked
rcu_read_lock_sched()here because thatpreempt_disable()at the same time.That looks complicated but I guess that works.
That said there is also this
rcu_dereference(current->perf_event_ctxp). Now, this ctx is released after srcu barrier right? So this should besrcu_dereference(). But then you seem to actually usercu_read_lock_sched()as it's compatible, so this should bercu_dereference_sched()?With the current state, rcu will whine.
Moreover there seem to be too much game between the different rcu flavours here, and that breaks the reviewers parsing.
Peter Zijlstra
maintainerQuoted 8 lines · Paul E. McKenney
Quoted 2 lines
So, you say below that it works because
synchronize_srcu(), that waits for qs after touching pmus, impliessynchronize_sched(), right?Ook... My current plans to fold SRCU into
TREE_RCUwould invalidate this assumption.Maybe we need some sort of primitive that concurrently waits for multiple types of RCU grace periods?
Nah, but I was thinking that any kind of preemptible rcu sync would imply a sched rcu sync.
If not strictly implied I'd have no problem simply writing:
synchronize_rcu_sched(); synchronize_srcu();
Peter Zijlstra
maintainerQuoted 10 lines · Frederic Weisbecker
Quoted 9 lines · tip-bot for Peter Zijlstra
kernel/perf_event.c +2
-1@@ -3745,18 +3757,20 @@ static void perf_event_task_ctx(struct perf_event_context *ctx,static void perf_event_task_event (struct perf_task_event *task_event) { -struct perf_cpu_context *cpuctx;struct perf_event_context *ctx = task_event->task_ctx; +struct perf_cpu_context *cpuctx; +struct pmu *pmu;
Quoted 15 lines
kernel/perf_event.c static void perf_event_task_ctx(struct perf_event_context *ctx, · line 3770
Quoted 10 lines
kernel/perf_event.c +5
-3-rcu_read_lock (); - cpuctx = &get_cpu_var (perf_cpu_context); -perf_event_task_ctx (&cpuctx->ctx, task_event); +rcu_read_lock_sched (); +list_for_each_entry_rcu (pmu, &pmus, entry) { + cpuctx =this_cpu_ptr (pmu->pmu_cpu_context); +perf_event_task_ctx (&cpuctx->ctx, task_event); + }if (!ctx) ctx =rcu_dereference (current->perf_event_ctxp);So, you say below that it works because
synchronize_srcu(), that waits for qs after touching pmus, impliessynchronize_sched(), right?
yep.
Quoted 2 lines
And I guess you picked
rcu_read_lock_sched()here because thatpreempt_disable()at the same time.
Mostly because preemption is already disabled there, and sync_srcu() was implemented using sync_rcu_sched() primitives.
Quoted 1 line
That looks complicated but I guess that works.
Yeah, similar to the event lists which we protect with a mutex and a spinlock, hold either to traverse, hold both to modify.
Depending on the situation we need to traverse the pmu list preemptible -- for example when we need to take the above mentioned mutex (see perf_event_exit_cpu_context), or non-preemptible, the above. So we need to guard it using two different flavours of RCU too.
Quoted 4 lines
That said there is also this
rcu_dereference(current->perf_event_ctxp). Now, this ctx is released after srcu barrier right? So this should besrcu_dereference(). But then you seem to actually usercu_read_lock_sched()as it's compatible, so this should bercu_dereference_sched()?
The task context is released using call_rcu(), and should be done under rcu_read_lock(), I guess we should hold both rcu_read_lock() and rcu_read_lock_sched() there to be correct.
Quoted 3 lines
With the current state, rcu will whine.
Moreover there seem to be too much game between the different rcu flavours here, and that breaks the reviewers parsing.
:-)
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Paul E. McKenney
Quoted 2 lines
So, you say below that it works because
synchronize_srcu(), that waits for qs after touching pmus, impliessynchronize_sched(), right?
Quoted 8 lines
Quoted 5 lines
Ook... My current plans to fold SRCU into
TREE_RCUwould invalidate this assumption.Maybe we need some sort of primitive that concurrently waits for multiple types of RCU grace periods?
Nah, but I was thinking that any kind of preemptible rcu sync would imply a sched rcu sync.
Ah!
Although disabling interrupts will block preemptible RCU grace periods in current implementations (but please don't rely on this!), disabling preemption will -not- block preemptible RCU grace periods, even given current TREE_PREEMPT_RCU and TINY_PREEMPT_RCU implementations. Current SRCU grace periods are blocked by disabling preemption, but folding it into the tree/tiny implementations would make SRCU grace periods be no longer blocked by disabling preemption.
This might change if RCU priority boosting is enabled, due to RCU grace-period computation and callback invocation moving to a kthread, but I won't have the guts to make TREE_RCU use kthread by default for some time. (Probably a year or so trouble-free experience with RCU priority boosting/kthreads.)
Quoted 4 lines
If not strictly implied I'd have no problem simply writing:
synchronize_rcu_sched(); synchronize_srcu();
If that works for you, then we are set! The only reason to introduce a combined primitive would be if the latency of the above was too large.
Thanx, Paul