Skip to contentSkip to conversation

perf_event: fix a cgroup switch warning

RFC PATCH In review Cong Wang 2 participants 3 messages May 14, 2019

Subsystems (2)
lore
Share & tools
lore

Cong Wang

author
#0 lore raw

RFC PATCH 1/1 perf_event: fix a cgroup switch warning

We have been consistently triggering the warning WARN_ON_ONCE(cpuctx->cgrp) in perf_cgroup_switch() for a rather long time, although we still have no clue on how to reproduce it.

Looking into the code, it seems the only possibility here is that the process calling perf_event_open() with a cgroup target exits before the process in the target cgroup exits but after it gains CPU to run. This is because we use the atomic counter perf_cgroup_events as an indication of whether cgroup perf event has enabled or not, which is inaccurate, illustrated as below:

CPU 0 CPU 1
// open perf events with a cgroup
// target for all CPU's
perf_event_open():
account_event_cpu()
// perf_cgroup_events == 1

				// Schedule in a process in the target cgroup
				perf_cgroup_switch()
perf_event_release_kernel():
  unaccount_event_cpu()
  // perf_cgroup_events == 0
				// schedule out
				// but perf_cgroup_sched_out() is skipped
				// cpuctx->cgrp left as non-NULL

				// schedule in another process
				perf_cgroup_switch() // WARN triggerred

The proposed fix is kinda ugly, as it adds a flag in each process to indicate whether this process has to go through perf_cgroup_sched_out() when perf_cgroup_events is false negative. The other possible fix is to force a reschedule on each target CPU before decreasing the counter perf_cgroup_events, but this is expensive.

Suggestions? Thoughts?

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Namhyung Kim <[email protected]>
---
 include/linux/sched.h | 3 +++
 kernel/events/core.c  | 5 ++++-
 2 files changed, 7 insertions(+), 1 deletion(-)
include/linux/sched.h +3 -0
index a2cd15855bad..835bdf15f92c 100644
@@ -733,6 +733,9 @@ struct task_struct {
 	/* to be used once the psi infrastructure lands upstream. */
 	unsigned			use_memdelay:1;
 #endif
+#ifdef CONFIG_PERF_EVENTS
+	unsigned			perf_cgroup_sched_in:1;
+#endif
 
 	unsigned long			atomic_flags; /* Flags requiring atomic access. */
 
kernel/events/core.c +4 -1
index abbd4b3b96c2..9b86b043018e 100644
@@ -817,6 +817,7 @@ static void perf_cgroup_switch(struct task_struct *task, int mode)
 			 * to event_filter_match() in event_sched_out()
 			 */
 			cpuctx->cgrp = NULL;
+			task->perf_cgroup_sched_in = 0;
 		}
 
 		if (mode & PERF_CGROUP_SWIN) {
@@ -831,6 +832,7 @@ static void perf_cgroup_switch(struct task_struct *task, int mode)
 			cpuctx->cgrp = perf_cgroup_from_task(task,
 							     &cpuctx->ctx);
 			cpu_ctx_sched_in(cpuctx, EVENT_ALL, task);
+			task->perf_cgroup_sched_in = 1;
 		}
 		perf_pmu_enable(cpuctx->ctx.pmu);
 		perf_ctx_unlock(cpuctx, cpuctx->task_ctx);
@@ -3233,7 +3235,8 @@ void __perf_event_task_sched_out(struct task_struct *task,
 	 * to check if we have to switch out PMU state.
 	 * cgroup event are system-wide mode only
 	 */
-	if (atomic_read(this_cpu_ptr(&perf_cgroup_events)))
+	if (atomic_read(this_cpu_ptr(&perf_cgroup_events)) ||
+	    task->perf_cgroup_sched_in)
 		perf_cgroup_sched_out(task, next);
 }
 
Signature · 1 line
2.21.0
maintainer
#1 lore raw
Quoted 17 lines · Cong Wang

We have been consistently triggering the warning WARN_ON_ONCE(cpuctx->cgrp) in perf_cgroup_switch() for a rather long time, although we still have no clue on how to reproduce it.

Looking into the code, it seems the only possibility here is that the process calling perf_event_open() with a cgroup target exits before the process in the target cgroup exits but after it gains CPU to run. This is because we use the atomic counter perf_cgroup_events as an indication of whether cgroup perf event has enabled or not, which is inaccurate, illustrated as below:

CPU 0 CPU 1
// open perf events with a cgroup
// target for all CPU's
perf_event_open():
account_event_cpu()
// perf_cgroup_events == 1

Quoted 8 lines
				// Schedule in a process in the target cgroup
				perf_cgroup_switch()
perf_event_release_kernel():
  unaccount_event_cpu()
  // perf_cgroup_events == 0
				// schedule out
				// but perf_cgroup_sched_out() is skipped
				// cpuctx->cgrp left as non-NULL
				which implies we observed:
				'perf_cgroup_events == 0'
Quoted 2 lines
				// schedule in another process
				perf_cgroup_switch() // WARN triggerred
				which implies we observed:
				'perf_cgroup_events == 1'

Which is impossible. It _might_ have been possible if the out and in happened on different CPUs. But then I'm not sure that is enough to trigger the problem.

Quoted 1 line

The proposed fix is kinda ugly,

Yes :-)

Quoted 1 line

Suggestions? Thoughts?

At perf_event_release time, when it is the last cgroup event, there should not be any cgroup events running anymore, so ideally perf_cgroup_switch() would not set state.

Furthermore; list_update_cgroup_event() will actually clear cpuctx->cgrp on removal of the last cgroup event.

Also; perf_cgroup_switch() will WARN when there are not in fact any cgroup events at all. I would expect that WARN to trigger too in your scneario. But you're not seeing that?

I do however note that that check seems racy; we do that without holding the ctx_lock.

author
#2 lore raw
Quoted 34 lines · Peter Zijlstra
Quoted 17 lines · Cong Wang

We have been consistently triggering the warning WARN_ON_ONCE(cpuctx->cgrp) in perf_cgroup_switch() for a rather long time, although we still have no clue on how to reproduce it.

Looking into the code, it seems the only possibility here is that the process calling perf_event_open() with a cgroup target exits before the process in the target cgroup exits but after it gains CPU to run. This is because we use the atomic counter perf_cgroup_events as an indication of whether cgroup perf event has enabled or not, which is inaccurate, illustrated as below:

CPU 0 CPU 1
// open perf events with a cgroup
// target for all CPU's
perf_event_open():
account_event_cpu()
// perf_cgroup_events == 1

Quoted 8 lines
                              // Schedule in a process in the target cgroup
                              perf_cgroup_switch()
perf_event_release_kernel():
  unaccount_event_cpu()
  // perf_cgroup_events == 0
                              // schedule out
                              // but perf_cgroup_sched_out() is skipped
                              // cpuctx->cgrp left as non-NULL

which implies we observed:
'perf_cgroup_events == 0'

Quoted 2 lines

// schedule in another process perf_cgroup_switch() // WARN triggerred

Quoted 7 lines

which implies we observed:
'perf_cgroup_events == 1'

Which is impossible. It _might_ have been possible if the out and in happened on different CPUs. But then I'm not sure that is enough to trigger the problem.

Good catch, but this just needs one more perf_event_open(), right? :)

Quoted 5 lines
Quoted 2 lines

The proposed fix is kinda ugly,

Yes :-)

Quoted 8 lines
Quoted 1 line

Suggestions? Thoughts?

At perf_event_release time, when it is the last cgroup event, there should not be any cgroup events running anymore, so ideally perf_cgroup_switch() would not set state.

Furthermore; list_update_cgroup_event() will actually clear cpuctx->cgrp on removal of the last cgroup event.

Ah, yes, this probably explains why it is harder to trigger than I expected.

Quoted 4 lines

Also; perf_cgroup_switch() will WARN when there are not in fact any cgroup events at all. I would expect that WARN to trigger too in your scneario. But you're not seeing that?

Not sure if I follow you, but if there is no cgroup event, cgrp_cpuctx_list should be empty, right?

From the stack traces I can't tell, what I can tell is that we use cgroup events in most cases.

Quoted 3 lines

I do however note that that check seems racy; we do that without holding the ctx_lock.

Hmm? perf_ctx_lock() is taken in perf_cgroup_switch(), so I think locking is fine.

Thanks.