perf: Fixes on event accounting
Subsystems (2)
cover perf: Fixes on event accounting
Peter, Jiri,
Here is a proposed set of fixes after the discussion we had about the last changes in the event accounting code.
Thanks,
Frederic
---
Frederic Weisbecker (3):
perf: Rollback callchain buffer refcount under the callchain mutex
perf: Account freq events globally
nohz: Include local CPU in full dynticks global kick
kernel/events/callchain.c | 3 ++- kernel/events/core.c | 19 ++++++++----------- kernel/time/tick-sched.c | 1 + 3 files changed, 11 insertions(+), 12 deletions(-)
PATCH 1/3 perf: Rollback callchain buffer refcount under the callchain mutex
When we fail to allocate the callchain buffers, we rollback the refcount we did and return from get_callchain_buffers().
However we take the refcount and allocate under the callchain lock but the rollback is done outside the lock.
As a result, while we rollback, some concurrent callchain user may call get_callchain_buffers(), see the non-zero refcount and give up because the buffers are NULL without itself retrying the allocation.
The consequences aren't that bad but that behaviour looks weird enough and it's better to give their chances to the following callchain users where we failed.
--- kernel/events/callchain.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/events/callchain.c +2 -1
@@ -116,10 +116,11 @@ int get_callchain_buffers(void) err =alloc_callchain_buffers (); exit: -mutex_unlock (&callchain_mutex);if (err)atomic_dec (&nr_callchain_events); +mutex_unlock (&callchain_mutex); +return err; }
Signature · 1 line
1.7.5.4
Re: [PATCH 1/3] perf: Rollback callchain buffer refcount under the callchain mutex
Quoted 14 lines · Frederic Weisbecker
When we fail to allocate the callchain buffers, we rollback the refcount we did and return from
get_callchain_buffers().However we take the refcount and allocate under the callchain lock but the rollback is done outside the lock.
As a result, while we rollback, some concurrent callchain user may call
get_callchain_buffers(), see the non-zero refcount and give up because the buffers are NULL without itself retrying the allocation.The consequences aren't that bad but that behaviour looks weird enough and it's better to give their chances to the following callchain users where we failed.
Quoted 5 lines
Reported-by Jiri Olsa <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Jiri Olsa <[email protected]>
tip-bot for Frederic Weisbecker
merge bot[tip:perf/core] perf: Roll back callchain buffer refcount under the callchain mutex
In reply to Frederic Weisbeckerperf: Roll back callchain buffer refcount under the callchain mutex
Commit-ID: fc3b86d673e41ac66b4ba5b75a90c2fcafb90089 Gitweb: http://git.kernel.org/tip/fc3b86d673e41ac66b4ba5b75a90c2fcafb90089 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 2 Aug 2013 18:29:54 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 16 Aug 2013 17:55:50 +0200
perf: Roll back callchain buffer refcount under the callchain mutex
When we fail to allocate the callchain buffers, we roll back the refcount we did and return from get_callchain_buffers().
However we take the refcount and allocate under the callchain lock but the rollback is done outside the lock.
As a result, while we roll back, some concurrent callchain user may call get_callchain_buffers(), see the non-zero refcount and give up because the buffers are NULL without itself retrying the allocation.
The consequences aren't that bad but that behaviour looks weird enough and it's better to give their chances to the following callchain users where we failed.
--- kernel/events/callchain.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/events/callchain.c +2 -1
@@ -116,10 +116,11 @@ int get_callchain_buffers(void) err =alloc_callchain_buffers (); exit: -mutex_unlock (&callchain_mutex);if (err)atomic_dec (&nr_callchain_events); +mutex_unlock (&callchain_mutex); +return err; }
PATCH 2/3 perf: Account freq events globally
In reply to Frederic WeisbeckerFreq events may not always be affine to a particular CPU. As such, account_event_cpu() may crash if we account per cpu a freq event that has event->cpu == -1.
To solve this, lets account freq events globally. In practice this doesn't change much the picture because perf tools create per-task perf events with one event per CPU by default. Profiling a single CPU is usually a corner case so there is no much point in optimizing things that way.
--- kernel/events/core.c | 19 ++++++++----------- 1 files changed, 8 insertions(+), 11 deletions(-)
kernel/events/core.c +8 -11
@@ -141,11 +141,11 @@ enum event_type_t {struct static_key_deferred perf_sched_events__read_mostly ; static DEFINE_PER_CPU(atomic_t, perf_cgroup_events); static DEFINE_PER_CPU(atomic_t, perf_branch_stack_events); -static DEFINE_PER_CPU(atomic_t, perf_freq_events);static atomic_t nr_mmap_events__read_mostly ;static atomic_t nr_comm_events__read_mostly ;static atomic_t nr_task_events__read_mostly ; +static atomic_t nr_freq_events__read_mostly ;static LIST_HEAD (pmus);static DEFINE_MUTEX (pmus_lock);
@@ -1871,9 +1871,6 @@ static int __perf_install_in_context(void *info)perf_pmu_enable (cpuctx->ctx.pmu);perf_ctx_unlock (cpuctx, task_ctx); -if (atomic_read (&__get_cpu_var (perf_freq_events))) -tick_nohz_full_kick (); -return 0 ; }
@@ -2811,7 +2808,7 @@ done:#ifdef CONFIG_NO_HZ_FULL bool perf_event_can_stop_tick (void ) { -if (atomic_read (&__get_cpu_var (perf_freq_events)) || +if (atomic_read (&nr_freq_events) ||__this_cpu_read (perf_throttled_count))return false;else
@@ -3140,9 +3137,6 @@ static void unaccount_event_cpu(struct perf_event *event, int cpu) }if (is_cgroup_event (event))atomic_dec (&per_cpu (perf_cgroup_events, cpu)); - -if (event->attr.freq) -atomic_dec (&per_cpu (perf_freq_events, cpu)); }static void unaccount_event (struct perf_event *event)
@@ -3158,6 +3152,8 @@ static void unaccount_event(struct perf_event *event)atomic_dec (&nr_comm_events);if (event->attr.task)atomic_dec (&nr_task_events); +if (event->attr.freq) +atomic_dec (&nr_freq_events);if (is_cgroup_event (event))static_key_slow_dec_deferred (&perf_sched_events);if (has_branch_stack (event))
@@ -6479,9 +6475,6 @@ static void account_event_cpu(struct perf_event *event, int cpu) }if (is_cgroup_event (event))atomic_inc (&per_cpu (perf_cgroup_events, cpu)); - -if (event->attr.freq) -atomic_inc (&per_cpu (perf_freq_events, cpu)); }static void account_event (struct perf_event *event)
@@ -6497,6 +6490,10 @@ static void account_event(struct perf_event *event)atomic_inc (&nr_comm_events);if (event->attr.task)atomic_inc (&nr_task_events); +if (event->attr.freq) { +if (atomic_inc_return (&nr_freq_events) ==1 ) +tick_nohz_full_kick_all (); + }if (has_branch_stack (event))static_key_slow_inc (&perf_sched_events.key);if (is_cgroup_event (event))
Signature · 1 line
1.7.5.4
Re: [PATCH 2/3] perf: Account freq events globally
Quoted 10 lines · Frederic Weisbecker
Freq events may not always be affine to a particular CPU. As such,
account_event_cpu()may crash if we account per cpu a freq event that hasevent->cpu== -1.To solve this, lets account freq events globally. In practice this doesn't change much the picture because perf tools create per-task perf events with one event per CPU by default. Profiling a single CPU is usually a corner case so there is no much point in optimizing things that way.
Quoted 9 lines
Reported-by Jiri Olsa <[email protected]>Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Jiri Olsa <[email protected]>Cc Namhyung Kim <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Stephane Eranian <[email protected]>
no more OOPSes ;-)
tip-bot for Frederic Weisbecker
merge bot[tip:perf/core] perf: Account freq events globally
In reply to Frederic Weisbeckerperf: Account freq events globally
Commit-ID: 948b26b6ddd08a57cb95ebb0dc96fde2edd5c383 Gitweb: http://git.kernel.org/tip/948b26b6ddd08a57cb95ebb0dc96fde2edd5c383 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 2 Aug 2013 18:29:55 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 16 Aug 2013 17:55:51 +0200
perf: Account freq events globally
Freq events may not always be affine to a particular CPU. As such, account_event_cpu() may crash if we account per cpu a freq event that has event->cpu == -1.
To solve this, lets account freq events globally. In practice this doesn't change much the picture because perf tools create per-task perf events with one event per CPU by default. Profiling a single CPU is usually a corner case so there is no much point in optimizing things that way.
--- kernel/events/core.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-)
kernel/events/core.c +8 -11
@@ -141,11 +141,11 @@ enum event_type_t {struct static_key_deferred perf_sched_events__read_mostly ; static DEFINE_PER_CPU(atomic_t, perf_cgroup_events); static DEFINE_PER_CPU(atomic_t, perf_branch_stack_events); -static DEFINE_PER_CPU(atomic_t, perf_freq_events);static atomic_t nr_mmap_events__read_mostly ;static atomic_t nr_comm_events__read_mostly ;static atomic_t nr_task_events__read_mostly ; +static atomic_t nr_freq_events__read_mostly ;static LIST_HEAD (pmus);static DEFINE_MUTEX (pmus_lock);
@@ -1871,9 +1871,6 @@ static int __perf_install_in_context(void *info)perf_pmu_enable (cpuctx->ctx.pmu);perf_ctx_unlock (cpuctx, task_ctx); -if (atomic_read (&__get_cpu_var (perf_freq_events))) -tick_nohz_full_kick (); -return 0 ; }
@@ -2811,7 +2808,7 @@ done:#ifdef CONFIG_NO_HZ_FULL bool perf_event_can_stop_tick (void ) { -if (atomic_read (&__get_cpu_var (perf_freq_events)) || +if (atomic_read (&nr_freq_events) ||__this_cpu_read (perf_throttled_count))return false;else
@@ -3140,9 +3137,6 @@ static void unaccount_event_cpu(struct perf_event *event, int cpu) }if (is_cgroup_event (event))atomic_dec (&per_cpu (perf_cgroup_events, cpu)); - -if (event->attr.freq) -atomic_dec (&per_cpu (perf_freq_events, cpu)); }static void unaccount_event (struct perf_event *event)
@@ -3158,6 +3152,8 @@ static void unaccount_event(struct perf_event *event)atomic_dec (&nr_comm_events);if (event->attr.task)atomic_dec (&nr_task_events); +if (event->attr.freq) +atomic_dec (&nr_freq_events);if (is_cgroup_event (event))static_key_slow_dec_deferred (&perf_sched_events);if (has_branch_stack (event))
@@ -6489,9 +6485,6 @@ static void account_event_cpu(struct perf_event *event, int cpu) }if (is_cgroup_event (event))atomic_inc (&per_cpu (perf_cgroup_events, cpu)); - -if (event->attr.freq) -atomic_inc (&per_cpu (perf_freq_events, cpu)); }static void account_event (struct perf_event *event)
@@ -6507,6 +6500,10 @@ static void account_event(struct perf_event *event)atomic_inc (&nr_comm_events);if (event->attr.task)atomic_inc (&nr_task_events); +if (event->attr.freq) { +if (atomic_inc_return (&nr_freq_events) ==1 ) +tick_nohz_full_kick_all (); + }if (has_branch_stack (event))static_key_slow_inc (&perf_sched_events.key);if (is_cgroup_event (event))
PATCH 3/3 nohz: Include local CPU in full dynticks global kick
In reply to Frederic Weisbeckertick_nohz_full_kick_all() is useful to notify all full dynticks CPUs that there is a system state change to checkout before re-evaluating the need for the tick.
Unfortunately this is implemented using smp_call_function_many() that ignores the local CPU. This CPU also needs to re-evaluate the tick.
on_each_cpu_mask() is not useful either because we don't want to re-evaluate the tick state in place but asynchronously from an IPI to avoid messing up with any random locking scenario.
Sol lets call tick_nohz_full_kick() from tick_nohz_full_kick_all() so that the usual irq work takes care of it.
--- kernel/time/tick-sched.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
kernel/time/tick-sched.c +1 -0
@@ -244,6 +244,7 @@ void tick_nohz_full_kick_all(void)preempt_disable ();smp_call_function_many (nohz_full_mask, nohz_full_kick_ipi,NULL , false); +tick_nohz_full_kick ();preempt_enable (); }
Signature · 1 line
1.7.5.4
tip-bot for Frederic Weisbecker
merge bot[tip:timers/nohz] nohz: Include local CPU in full dynticks global kick
nohz: Include local CPU in full dynticks global kick
Commit-ID: c2e7fcf53c3cb02b4ada1c66a9bc8a4d97d58aba Gitweb: http://git.kernel.org/tip/c2e7fcf53c3cb02b4ada1c66a9bc8a4d97d58aba Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 2 Aug 2013 18:29:56 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 16 Aug 2013 17:55:33 +0200
nohz: Include local CPU in full dynticks global kick
tick_nohz_full_kick_all() is useful to notify all full dynticks CPUs that there is a system state change to checkout before re-evaluating the need for the tick.
Unfortunately this is implemented using smp_call_function_many() that ignores the local CPU. This CPU also needs to re-evaluate the tick.
on_each_cpu_mask() is not useful either because we don't want to re-evaluate the tick state in place but asynchronously from an IPI to avoid messing up with any random locking scenario.
So lets call tick_nohz_full_kick() from tick_nohz_full_kick_all() so that the usual irq work takes care of it.
--- kernel/time/tick-sched.c | 1 + 1 file changed, 1 insertion(+)
kernel/time/tick-sched.c +1 -0
@@ -246,6 +246,7 @@ void tick_nohz_full_kick_all(void)preempt_disable ();smp_call_function_many (tick_nohz_full_mask, nohz_full_kick_ipi,NULL , false); +tick_nohz_full_kick ();preempt_enable (); }