Skip to contentSkip to conversation

perf: Fixing throttling related WARN

RFC In review Jiri Olsa 4 participants 11 messages Aug 13, 2013 – Aug 19, 2013

Subsystems (2)
lore
Share & tools
lore

Jiri Olsa

author
#0 lore raw

cover perf: Fixing throttling related WARN

hi,
we managed to trigger perf WARN (described in patch 2), but only on AMD system with 48 CPUs.

I believe the reason is a race in throttling code and I tried to fix it. So far my testing looks ok, but I suspect I broke something else.. ;-)

thanks for comments,
jirka

Cc Corey Ashford <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 arch/x86/kernel/cpu/perf_event_intel.c |  4 +++-
 include/linux/perf_event.h             |  2 +-
 kernel/events/core.c                   | 67 +++++++++++++++++++++++++++++++++++++++----------------------------
 3 files changed, 43 insertions(+), 30 deletions(-)
author
#1 lore raw

PATCH 1/2 perf x86: Make intel_pmu_enable_all to enable only active events

Currently the intel_pmu_enable_all enables all possible events, which is not allways desired. One case (there'll be probably more) is:

- event hits throttling threshold
- NMI stops event
- intel_pmu_enable_all starts it back on the NMI exit

Cc Corey Ashford <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 arch/x86/kernel/cpu/perf_event_intel.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/perf_event_intel.c +3 -1
index a45d8d4..360e7a0 100644
@@ -912,11 +912,13 @@ static void intel_pmu_disable_all(void)
 static void intel_pmu_enable_all(int added)
 {
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+	u64 active_mask = *((u64*) cpuc->active_mask);
 
 	intel_pmu_pebs_enable_all();
 	intel_pmu_lbr_enable_all();
 	wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL,
-			x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask);
+			x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask
+			& active_mask);
 
 	if (test_bit(INTEL_PMC_IDX_FIXED_BTS, cpuc->active_mask)) {
 		struct perf_event *event =
Signature · 1 line
1.7.11.7
maintainer
#2 lore raw

Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events

Quoted 13 lines · Jiri Olsa

Currently the intel_pmu_enable_all enables all possible events, which is not allways desired. One case (there'll be probably more) is:

- event hits throttling threshold
- NMI stops event
- intel_pmu_enable_all starts it back on the NMI exit

+++ b/arch/x86/kernel/cpu/perf_event_intel.c

arch/x86/kernel/cpu/perf_event_intel.c +0 -0
@@ -912,11 +912,13 @@ static void intel_pmu_disable_all(void)
 static void intel_pmu_enable_all(int added)

arch/x86/kernel/cpu/perf_event_intel.c static void intel_pmu_disable_all(void) · line 921

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel.c +3 -1
 {
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+	u64 active_mask = *((u64*) cpuc->active_mask);
 
 	intel_pmu_pebs_enable_all();
 	intel_pmu_lbr_enable_all();
 	wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL,
-			x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask);
+			x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask
+			& active_mask);

Garh.. you made my head hurt :-)

I think its a NOP; this is the global ctrl register but intel_pmu_disable_event() writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.

author
#3 lore raw

Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events

Quoted 27 lines · Peter Zijlstra
Quoted 13 lines · Jiri Olsa

Currently the intel_pmu_enable_all enables all possible events, which is not allways desired. One case (there'll be probably more) is:

- event hits throttling threshold
- NMI stops event
- intel_pmu_enable_all starts it back on the NMI exit

+++ b/arch/x86/kernel/cpu/perf_event_intel.c

arch/x86/kernel/cpu/perf_event_intel.c +0 -0
@@ -912,11 +912,13 @@ static void intel_pmu_disable_all(void)
 static void intel_pmu_enable_all(int added)

arch/x86/kernel/cpu/perf_event_intel.c static void intel_pmu_disable_all(void) · line 921

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel.c +3 -1
 {
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+	u64 active_mask = *((u64*) cpuc->active_mask);
 
 	intel_pmu_pebs_enable_all();
 	intel_pmu_lbr_enable_all();
 	wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL,
-			x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask);
+			x86_pmu.intel_ctrl & ~cpuc->intel_ctrl_guest_mask
+			& active_mask);

Garh.. you made my head hurt :-)

Quoted 3 lines

I think its a NOP; this is the global ctrl register but intel_pmu_disable_event() writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.

hum, I might have misread the doc.. strange the test showed this result

jirka

#4 lore raw

Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events

In reply to Peter Zijlstra
Quoted 4 lines

I think its a NOP; this is the global ctrl register but intel_pmu_disable_event() writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.

Yes the hardware ANDs the various enable bits in the different registers.

-andi

#5 lore raw

Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events

Quoted 8 lines · Andi Kleen
Quoted 4 lines

I think its a NOP; this is the global ctrl register but intel_pmu_disable_event() writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.

Yes the hardware ANDs the various enable bits in the different registers.

Andi is correct. There is a logical AND between GLOBAL_CTRL and the individual PERFEVTCTL.EN bits. If the EN bit is zero, then the counter does not count. That also applies to fixed counters.

author
#6 lore raw

Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events

Quoted 6 lines · Stephane Eranian
Quoted 5 lines · Andi Kleen
Quoted 4 lines

I think its a NOP; this is the global ctrl register but intel_pmu_disable_event() writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.

Quoted 6 lines
Quoted 3 lines

Yes the hardware ANDs the various enable bits in the different registers.

Andi is correct. There is a logical AND between GLOBAL_CTRL and the individual PERFEVTCTL.EN bits. If the EN bit is zero, then the counter does not count. That also applies to fixed counters.

right, peter mentioned I could have seen already queded NMI comming in for that event..

jirka

#7 lore raw

Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events

Hi,

Quoted 10 lines · Jiri Olsa
Quoted 9 lines · Stephane Eranian
Quoted 8 lines · Andi Kleen
Quoted 4 lines

I think its a NOP; this is the global ctrl register but intel_pmu_disable_event() writes PERFEVTSELx.EN = 0, so even if you enable it in the global mask, the event should still be disabled.

Yes the hardware ANDs the various enable bits in the different registers.

Quoted 7 lines
Quoted 3 lines

Andi is correct. There is a logical AND between GLOBAL_CTRL and the individual PERFEVTCTL.EN bits. If the EN bit is zero, then the counter does not count. That also applies to fixed counters.

right, peter mentioned I could have seen already queded NMI comming in for that event..

Yeah, I think you can have one NMI pending because it came while you were already servicing an NMI interrupt. Though, this is very unlikely.

maintainer
#8 lore raw

Re: [PATCH 1/2] perf x86: Make intel_pmu_enable_all to enable only active events

Quoted 3 lines · Stephane Eranian

Yeah, I think you can have one NMI pending because it came while you were already servicing an NMI interrupt. Though, this is very unlikely.

Yeah exceedingly rare. I'm also fairly certain I've had NMIs right after clearing EN bits. Although I cannot remember if that was on Intel or AMD hardware, nor on what specific model :/

author
#9 lore raw

PATCH 2/2 perf: Move throtling flag to perf_event_context

In reply to Jiri Olsa

The current throttling code triggers WARN below via following workload (only hit on AMD machine with 48 CPUs):

  # while [ 1 ]; do perf record perf bench sched messaging; done
  WARNING: at arch/x86/kernel/cpu/perf_event.c:1054 x86_pmu_start+0xc6/0x100()
  SNIP
  Call Trace:
   <IRQ>  [<ffffffff815f62d6>] dump_stack+0x19/0x1b
   [<ffffffff8105f531>] warn_slowpath_common+0x61/0x80
   [<ffffffff8105f60a>] warn_slowpath_null+0x1a/0x20
   [<ffffffff810213a6>] x86_pmu_start+0xc6/0x100
   [<ffffffff81129dd2>] perf_adjust_freq_unthr_context.part.75+0x182/0x1a0
   [<ffffffff8112a058>] perf_event_task_tick+0xc8/0xf0
   [<ffffffff81093221>] scheduler_tick+0xd1/0x140
   [<ffffffff81070176>] update_process_times+0x66/0x80
   [<ffffffff810b9565>] tick_sched_handle.isra.15+0x25/0x60
   [<ffffffff810b95e1>] tick_sched_timer+0x41/0x60
   [<ffffffff81087c24>] __run_hrtimer+0x74/0x1d0
   [<ffffffff810b95a0>] ? tick_sched_handle.isra.15+0x60/0x60
   [<ffffffff81088407>] hrtimer_interrupt+0xf7/0x240
   [<ffffffff81606829>] smp_apic_timer_interrupt+0x69/0x9c
   [<ffffffff8160569d>] apic_timer_interrupt+0x6d/0x80
   <EOI>  [<ffffffff81129f74>] ? __perf_event_task_sched_in+0x184/0x1a0
   [<ffffffff814dd937>] ? kfree_skbmem+0x37/0x90
   [<ffffffff815f2c47>] ? __slab_free+0x1ac/0x30f
   [<ffffffff8118143d>] ? kfree+0xfd/0x130
   [<ffffffff81181622>] kmem_cache_free+0x1b2/0x1d0
   [<ffffffff814dd937>] kfree_skbmem+0x37/0x90
   [<ffffffff814e03c4>] consume_skb+0x34/0x80
   [<ffffffff8158b057>] unix_stream_recvmsg+0x4e7/0x820
   [<ffffffff814d5546>] sock_aio_read.part.7+0x116/0x130
   [<ffffffff8112c10c>] ? __perf_sw_event+0x19c/0x1e0
   [<ffffffff814d5581>] sock_aio_read+0x21/0x30
   [<ffffffff8119a5d0>] do_sync_read+0x80/0xb0
   [<ffffffff8119ac85>] vfs_read+0x145/0x170
   [<ffffffff8119b699>] SyS_read+0x49/0xa0
   [<ffffffff810df516>] ? __audit_syscall_exit+0x1f6/0x2a0
   [<ffffffff81604a19>] system_call_fastpath+0x16/0x1b
  ---[ end trace 622b7e226c4a766a ]---

The reason is race in perf_event_task_tick throttling code. The race flow (simplified code):

- perf_throttled_count is per cpu variable and is CPU throttling flag, here starting with 0
- perf_throttled_seq is sequence/domain for allowed count of interrupts within the tick, gets increased each tick

on single CPU (CPU bounded event):

... workload

perf_event_task_tick:
|
| T0 inc(perf_throttled_seq)
| T1 throttled = xchg(perf_throttled_count, 0) == 0
tick gets interrupted:

... event gets throttled under new seq ...

T2 last NMI comes, event is throttled - inc(perf_throttled_count)

back to tick:
| perf_adjust_freq_unthr_context:
|
| T3 unthrottling is skiped for event (throttled == 0)
| T4 event is stop and started via freq adjustment
|
tick ends

... workload
... no sample is hit for event ...

perf_event_task_tick:
|
| T5 throttled = xchg(perf_throttled_count, 0) != 0 (from T2)
| T6 unthrottling is done on event (interrupts == MAX_INTERRUPTS)
| event is already started (from T4) -> WARN

Fixing this by:
- moving the per cpu 'throttle' flag into the event's ctx so the flag is context specific and thus not disturbed (changed) by others
- checking the 'throttled' under pmu disabled code again
- removing perf_throttled_seq as it's no longer needed
- keeping perf_throttled_count as perf_event_can_stop_tick flag

Cc Corey Ashford <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 include/linux/perf_event.h |  2 +-
 kernel/events/core.c       | 67 +++++++++++++++++++++++++++-------------------
 2 files changed, 40 insertions(+), 29 deletions(-)
include/linux/perf_event.h +1 -1
index c43f6ea..d0c04fb 100644
@@ -167,7 +167,6 @@ struct hw_perf_event {
 	u64				sample_period;
 	u64				last_period;
 	local64_t			period_left;
-	u64                             interrupts_seq;
 	u64				interrupts;
 
 	u64				freq_time_stamp;
@@ -494,6 +493,7 @@ struct perf_event_context {
 	int				nr_cgroups;	 /* cgroup evts */
 	int				nr_branch_stack; /* branch_stack evt */
 	struct rcu_head			rcu_head;
+	int * __percpu			throttled;
 };
 
 /*
kernel/events/core.c +39 -28

167 lines not on this page · raw message

Signature · 1 line
1.7.11.7
maintainer
#10 lore raw

Re: [PATCH 2/2] perf: Move throtling flag to perf_event_context

Quoted 46 lines · Jiri Olsa

The current throttling code triggers WARN below via following workload (only hit on AMD machine with 48 CPUs):

  # while [ 1 ]; do perf record perf bench sched messaging; done

The reason is race in perf_event_task_tick throttling code. The race flow (simplified code):

- perf_throttled_count is per cpu variable and is CPU throttling flag, here starting with 0
- perf_throttled_seq is sequence/domain for allowed count of interrupts within the tick, gets increased each tick

on single CPU (CPU bounded event):

... workload

perf_event_task_tick:
|
| T0 inc(perf_throttled_seq)
| T1 throttled = xchg(perf_throttled_count, 0) == 0
tick gets interrupted:

... event gets throttled under new seq ...

T2 last NMI comes, event is throttled - inc(perf_throttled_count)

back to tick:
| perf_adjust_freq_unthr_context:
|
| T3 unthrottling is skiped for event (throttled == 0)
| T4 event is stop and started via freq adjustment
|
tick ends

... workload
... no sample is hit for event ...

perf_event_task_tick:
|
| T5 throttled = xchg(perf_throttled_count, 0) != 0 (from T2)
| T6 unthrottling is done on event (interrupts == MAX_INTERRUPTS)
| event is already started (from T4) -> WARN

Quoted 3 lines

Fixing this by:
- moving the per cpu 'throttle' flag into the event's ctx so the flag is context specific and thus not disturbed (changed) by others

This is an 'unrelated' change purely meant to optimize the tick handler to not iterate as many events, right? It isn't actually part of the solution for the above issue afaict. Hence I think it should be separated into a separate patch. Esp. so because this seems to be the bulk of the below patch due to the extra allocations/error paths etc.

That said, thinking about it, we might want to use a llist and queue all throttled events on it from __perf_event_overflow(). That way we can limit the iteration to all throttled events; a much better proposition still.

Quoted 1 line

- checking the 'throttled' under pmu disabled code again

Might as well remove that and only keep hwc->interrupts == MAX_INTERRUPTS; that should be sufficient I think.

In either case, we could have not gotten here on the first tick and a second tick is needed to unthrottle us.

Quoted 1 line

- removing perf_throttled_seq as it's no longer needed

I'm not entirely sure how we got here and your changelog is short of clues; that said, I think I agree.

Quoted 1 line

- keeping perf_throttled_count as perf_event_can_stop_tick flag

... uhu.

Anyway, would something like the below not be a similar fix? It avoids the entire situation afaic.

---

kernel/events/core.c +1 -1
index e82e700..8fffd18 100644
@@ -2712,7 +2712,7 @@ static void perf_adjust_freq_unthr_context(struct perf_event_context *ctx,
 
 		hwc = &event->hw;
 
-		if (needs_unthr && hwc->interrupts == MAX_INTERRUPTS) {
+		if (hwc->interrupts == MAX_INTERRUPTS) {
 			hwc->interrupts = 0;
 			perf_log_throttle(event, 1);
 			event->pmu->start(event, 0);