perf: Fixing throttling related WARN
Subsystems (2)
cover perf: Fixing throttling related WARN
hi,
we managed to trigger perf WARN (described in patch 2), but only on AMD system with 48 CPUs.
I believe the reason is a race in throttling code and I tried to fix it. So far my testing looks ok, but I suspect I broke something else.. ;-)
thanks for comments,
jirka
--- arch/x86/kernel/cpu/perf_event_intel.c | 4 +++- include/linux/perf_event.h | 2 +- kernel/events/core.c | 67 +++++++++++++++++++++++++++++++++++++++---------------------------- 3 files changed, 43 insertions(+), 30 deletions(-)
PATCH 1/2 perf x86: Make intel_pmu_enable_all to enable only active events
Currently the intel_pmu_enable_all enables all possible events, which is not allways desired. One case (there'll be probably more) is:
- event hits throttling threshold
- NMI stops event
- intel_pmu_enable_all starts it back on the NMI exit
--- arch/x86/kernel/cpu/perf_event_intel.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/perf_event_intel.c +3 -1
@@ -912,11 +912,13 @@ static void intel_pmu_disable_all(void)static void intel_pmu_enable_all (int added) {struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); +u64 active_mask = *((u64 *) cpuc->active_mask);intel_pmu_pebs_enable_all ();intel_pmu_lbr_enable_all ();wrmsrl (MSR_CORE_PERF_GLOBAL_CTRL , - x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask); + x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask + & active_mask);if (test_bit (INTEL_PMC_IDX_FIXED_BTS , cpuc->active_mask)) { struct perf_event *event =
Signature · 1 line
1.7.11.7
Peter Zijlstra
maintainerRe: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events
Quoted 13 lines · Jiri Olsa
Currently the
intel_pmu_enable_allenables all possible events, which is not allways desired. One case (there'll be probably more) is:- event hits throttling threshold
- NMI stops event
-intel_pmu_enable_allstarts it back on the NMI exit+++
b/arch/x86/kernel/cpu/perf_event_intel.carch/x86/kernel/cpu/perf_event_intel.c +0
-0@@ -912,11 +912,13 @@ static void intel_pmu_disable_all(void)static void intel_pmu_enable_all (int added)
arch/x86/kernel/cpu/perf_event_intel.c static void intel_pmu_disable_all(void) · line 921
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel.c +3
-1{struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); +u64 active_mask = *((u64 *) cpuc->active_mask);intel_pmu_pebs_enable_all ();intel_pmu_lbr_enable_all ();wrmsrl (MSR_CORE_PERF_GLOBAL_CTRL , - x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask); + x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask + & active_mask);
Garh.. you made my head hurt :-)
I think its a NOP; this is the global ctrl register but intel_pmu_disable_event() writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.
Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events
Quoted 27 lines · Peter Zijlstra
Quoted 13 lines · Jiri Olsa
Currently the
intel_pmu_enable_allenables all possible events, which is not allways desired. One case (there'll be probably more) is:- event hits throttling threshold
- NMI stops event
-intel_pmu_enable_allstarts it back on the NMI exit+++
b/arch/x86/kernel/cpu/perf_event_intel.carch/x86/kernel/cpu/perf_event_intel.c +0
-0@@ -912,11 +912,13 @@ static void intel_pmu_disable_all(void)static void intel_pmu_enable_all (int added)arch/x86/kernel/cpu/perf_event_intel.c static void intel_pmu_disable_all(void) · line 921
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel.c +3
-1{struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); +u64 active_mask = *((u64 *) cpuc->active_mask);intel_pmu_pebs_enable_all ();intel_pmu_lbr_enable_all ();wrmsrl (MSR_CORE_PERF_GLOBAL_CTRL , - x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask); + x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask + & active_mask);Garh.. you made my head hurt :-)
Quoted 3 lines
I think its a NOP; this is the global ctrl register but
intel_pmu_disable_event()writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.
hum, I might have misread the doc.. strange the test showed this result
jirka
Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events
In reply to Peter ZijlstraQuoted 4 lines
I think its a NOP; this is the global ctrl register but
intel_pmu_disable_event()writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.
Yes the hardware ANDs the various enable bits in the different registers.
-andi
Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events
Quoted 8 lines · Andi Kleen
Quoted 4 lines
I think its a NOP; this is the global ctrl register but
intel_pmu_disable_event()writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.Yes the hardware ANDs the various enable bits in the different registers.
Andi is correct. There is a logical AND between GLOBAL_CTRL and the individual PERFEVTCTL.EN bits. If the EN bit is zero, then the counter does not count. That also applies to fixed counters.
Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events
Quoted 6 lines · Stephane Eranian
Quoted 5 lines · Andi Kleen
Quoted 4 lines
I think its a NOP; this is the global ctrl register but
intel_pmu_disable_event()writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.
Quoted 6 lines
Quoted 3 lines
Yes the hardware ANDs the various enable bits in the different registers.
Andi is correct. There is a logical AND between
GLOBAL_CTRLand the individual PERFEVTCTL.EN bits. If the EN bit is zero, then the counter does not count. That also applies to fixed counters.
right, peter mentioned I could have seen already queded NMI comming in for that event..
jirka
Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events
Hi,
Quoted 10 lines · Jiri Olsa
Quoted 9 lines · Stephane Eranian
Quoted 8 lines · Andi Kleen
Quoted 4 lines
I think its a NOP; this is the global ctrl register but
intel_pmu_disable_event()writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.Yes the hardware ANDs the various enable bits in the different registers.
Quoted 7 lines
Quoted 3 lines
Andi is correct. There is a logical AND between
GLOBAL_CTRLand the individual PERFEVTCTL.EN bits. If the EN bit is zero, then the counter does not count. That also applies to fixed counters.right, peter mentioned I could have seen already queded NMI comming in for that event..
Yeah, I think you can have one NMI pending because it came while you were already servicing an NMI interrupt. Though, this is very unlikely.
Peter Zijlstra
maintainerRe: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events
Quoted 3 lines · Stephane Eranian
Yeah, I think you can have one NMI pending because it came while you were already servicing an NMI interrupt. Though, this is very unlikely.
Yeah exceedingly rare. I'm also fairly certain I've had NMIs right after clearing EN bits. Although I cannot remember if that was on Intel or AMD hardware, nor on what specific model :/
PATCH 2/2 perf: Move throtling flag to perf_event_context
In reply to Jiri OlsaThe current throttling code triggers WARN below via following workload (only hit on AMD machine with 48 CPUs):
# while [ 1 ]; do perf record perf bench sched messaging; done
WARNING: at arch/x86/kernel/cpu/perf_event.c:1054 x86_pmu_start+0xc6/0x100() SNIP Call Trace: <IRQ> [<ffffffff815f62d6>] dump_stack+0x19/0x1b [<ffffffff8105f531>] warn_slowpath_common+0x61/0x80 [<ffffffff8105f60a>] warn_slowpath_null+0x1a/0x20 [<ffffffff810213a6>] x86_pmu_start+0xc6/0x100 [<ffffffff81129dd2>] perf_adjust_freq_unthr_context.part.75+0x182/0x1a0 [<ffffffff8112a058>] perf_event_task_tick+0xc8/0xf0 [<ffffffff81093221>] scheduler_tick+0xd1/0x140 [<ffffffff81070176>] update_process_times+0x66/0x80 [<ffffffff810b9565>] tick_sched_handle.isra.15+0x25/0x60 [<ffffffff810b95e1>] tick_sched_timer+0x41/0x60 [<ffffffff81087c24>] __run_hrtimer+0x74/0x1d0 [<ffffffff810b95a0>] ? tick_sched_handle.isra.15+0x60/0x60 [<ffffffff81088407>] hrtimer_interrupt+0xf7/0x240 [<ffffffff81606829>] smp_apic_timer_interrupt+0x69/0x9c [<ffffffff8160569d>] apic_timer_interrupt+0x6d/0x80 <EOI> [<ffffffff81129f74>] ? __perf_event_task_sched_in+0x184/0x1a0 [<ffffffff814dd937>] ? kfree_skbmem+0x37/0x90 [<ffffffff815f2c47>] ? __slab_free+0x1ac/0x30f [<ffffffff8118143d>] ? kfree+0xfd/0x130 [<ffffffff81181622>] kmem_cache_free+0x1b2/0x1d0 [<ffffffff814dd937>] kfree_skbmem+0x37/0x90 [<ffffffff814e03c4>] consume_skb+0x34/0x80 [<ffffffff8158b057>] unix_stream_recvmsg+0x4e7/0x820 [<ffffffff814d5546>] sock_aio_read.part.7+0x116/0x130 [<ffffffff8112c10c>] ? __perf_sw_event+0x19c/0x1e0 [<ffffffff814d5581>] sock_aio_read+0x21/0x30 [<ffffffff8119a5d0>] do_sync_read+0x80/0xb0 [<ffffffff8119ac85>] vfs_read+0x145/0x170 [<ffffffff8119b699>] SyS_read+0x49/0xa0 [<ffffffff810df516>] ? __audit_syscall_exit+0x1f6/0x2a0 [<ffffffff81604a19>] system_call_fastpath+0x16/0x1b ---[ end trace 622b7e226c4a766a ]---
The reason is race in perf_event_task_tick throttling code. The race flow (simplified code):
- perf_throttled_count is per cpu variable and is CPU throttling flag, here starting with 0
- perf_throttled_seq is sequence/domain for allowed count of interrupts within the tick, gets increased each tick
on single CPU (CPU bounded event):
... workload
perf_event_task_tick:
|
| T0 inc(perf_throttled_seq)
| T1 throttled = xchg(perf_throttled_count, 0) == 0
tick gets interrupted:
... event gets throttled under new seq ...
T2 last NMI comes, event is throttled - inc(perf_throttled_count)
back to tick:
| perf_adjust_freq_unthr_context:
|
| T3 unthrottling is skiped for event (throttled == 0)
| T4 event is stop and started via freq adjustment
|
tick ends
... workload
... no sample is hit for event ...
perf_event_task_tick:
|
| T5 throttled = xchg(perf_throttled_count, 0) != 0 (from T2)
| T6 unthrottling is done on event (interrupts == MAX_INTERRUPTS)
| event is already started (from T4) -> WARN
Fixing this by:
- moving the per cpu 'throttle' flag into the event's ctx so the flag is context specific and thus not disturbed (changed) by others
- checking the 'throttled' under pmu disabled code again
- removing perf_throttled_seq as it's no longer needed
- keeping perf_throttled_count as perf_event_can_stop_tick flag
--- include/linux/perf_event.h | 2 +- kernel/events/core.c | 67 +++++++++++++++++++++++++++------------------- 2 files changed, 40 insertions(+), 29 deletions(-)
include/linux/perf_event.h +1 -1
@@ -167,7 +167,6 @@ struct hw_perf_event {u64 sample_period;u64 last_period;local64_t period_left; -u64 interrupts_seq;u64 interrupts;u64 freq_time_stamp;
@@ -494,6 +493,7 @@ struct perf_event_context {int nr_cgroups;/* cgroup evts */ int nr_branch_stack;/* branch_stack evt */ struct rcu_head rcu_head; +int *__percpu throttled; };/*
kernel/events/core.c +39 -28
167 lines not on this page · raw message
Signature · 1 line
1.7.11.7
Peter Zijlstra
maintainerRe: [PATCH 2/2] perf: Move throtling flag to perf_event_context
Quoted 46 lines · Jiri Olsa
The current throttling code triggers WARN below via following workload (only hit on AMD machine with 48 CPUs):
# while [ 1 ]; do perf record perf bench sched messaging; doneThe reason is race in
perf_event_task_tickthrottling code. The race flow (simplified code):-
perf_throttled_countis per cpu variable and is CPU throttling flag, here starting with 0
-perf_throttled_seqis sequence/domain for allowed count of interrupts within the tick, gets increased each tickon single CPU (CPU bounded event):
... workload
perf_event_task_tick:
|
| T0inc(perf_throttled_seq)
| T1 throttled =xchg(perf_throttled_count, 0)== 0
tick gets interrupted:... event gets throttled under new seq ...
T2 last NMI comes, event is throttled -
inc(perf_throttled_count)back to tick:
|perf_adjust_freq_unthr_context:
|
| T3 unthrottling is skiped for event (throttled == 0)
| T4 event is stop and started via freq adjustment
|
tick ends... workload
... no sample is hit for event ...
perf_event_task_tick:
|
| T5 throttled =xchg(perf_throttled_count, 0)!= 0 (from T2)
| T6 unthrottling is done on event (interrupts ==MAX_INTERRUPTS)
| event is already started (from T4) -> WARN
Quoted 3 lines
Fixing this by:
- moving the per cpu 'throttle' flag into the event's ctx so the flag is context specific and thus not disturbed (changed) by others
This is an 'unrelated' change purely meant to optimize the tick handler to not iterate as many events, right? It isn't actually part of the solution for the above issue afaict. Hence I think it should be separated into a separate patch. Esp. so because this seems to be the bulk of the below patch due to the extra allocations/error paths etc.
That said, thinking about it, we might want to use a llist and queue all throttled events on it from __perf_event_overflow(). That way we can limit the iteration to all throttled events; a much better proposition still.
Quoted 1 line
- checking the 'throttled' under pmu disabled code again
Might as well remove that and only keep hwc->interrupts == MAX_INTERRUPTS; that should be sufficient I think.
In either case, we could have not gotten here on the first tick and a second tick is needed to unthrottle us.
Quoted 1 line
- removing
perf_throttled_seqas it's no longer needed
I'm not entirely sure how we got here and your changelog is short of clues; that said, I think I agree.
Quoted 1 line
- keeping
perf_throttled_countasperf_event_can_stop_tickflag
... uhu.
Anyway, would something like the below not be a similar fix? It avoids the entire situation afaic.
---
kernel/events/core.c +1 -1
@@ -2712,7 +2712,7 @@ static void perf_adjust_freq_unthr_context(struct perf_event_context *ctx, hwc = &event->hw; -if (needs_unthr && hwc->interrupts ==MAX_INTERRUPTS ) { +if (hwc->interrupts ==MAX_INTERRUPTS ) { hwc->interrupts =0 ;perf_log_throttle (event,1 ); event->pmu->start (event,0 );