sysctl: Panic on RCU stall and schedule while atomic

It is not always easy to define the cause of an RCU stall just by analysing the RCU stall messages, mainly when the problem is caused by the indirect starvation of rcu threads. For example, when preempt_rcu is not awakened due to the starvation of a timer softirq.

We have been hard coding panic() in the RCU stall functions for some time while testing the kernel-rt. But this is not possible in some scenarios, like when supporting customers.

This patch implements the sysctl kerner.panic_on_rcu_stall. If set to 1, the system will panic() when an RCU stall takes place, enabling the capture of a vmcore. The vmcore provides a way to analyze all kernel/tasks states, helping out to point to the culprit and the solution for the stall.

The kerner.panic_on_rcu_stall sysctl is disabled by default.

Quoted 9 lines · Daniel Bristot de Oliveira

It is not always easy to define the cause of an RCU stall just by analysing the RCU stall messages, mainly when the problem is caused by the indirect starvation of rcu threads. For example, when preempt_rcu is not awakened due to the starvation of a timer softirq.

We have been hard coding panic() in the RCU stall functions for some time while testing the kernel-rt. But this is not possible in some scenarios, like when supporting customers.

Quoted 7 lines

This patch implements the sysctl kerner.panic_on_rcu_stall. If set to 1, the system will panic() when an RCU stall takes place, enabling the capture of a vmcore. The vmcore provides a way to analyze all kernel/tasks states, helping out to point to the culprit and the solution for the stall.

The kerner.panic_on_rcu_stall sysctl is disabled by default.

s/kerner/kernel/ (here and in the previous paragraph).

Also, even though it's only two lines, please consider creating a static function wrapping the if and panic, to avoid duplication.

With those changes,

Reviewed-by Josh Triplett <[email protected]>

Sorry, realized something else a moment after sending: I don't think this will build if you use the tiny RCU implementation. That implementation *does* support tracing, and if you enable tracing, you'll have CONFIG_RCU_STALL_COMMON=y, but you won't build tree.c where the variable definition lives. So, the sysctl code will reference a variable that doesn't exist.

On 05/31/2016 04:23 PM, Josh Triplett wrote:

Hi Josh,

Quoted 6 lines

Sorry, realized something else a moment after sending: I don't think this will build if you use the tiny RCU implementation. That implementation *does* support tracing, and if you enable tracing, you'll have CONFIG_RCU_STALL_COMMON=y, but you won't build tree.c where the variable definition lives. So, the sysctl code will reference a variable that doesn't exist.

Good catch!

[root@f23 linux]# egrep '(TINY_RCU|RCU_TRACE)' .config CONFIG_TINY_RCU=y
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_TRACE=y
[root@f23 linux]# make
...
kernel/built-in.o:(.data+0x4688): undefined reference to `sysctl_panic_on_rcu_stall' Makefile:937: recipe for target 'vmlinux' failed
make: *** [vmlinux] Error 1

How about this change in the condition:

It worked fine here, but I may be missing something, so I better check with the experts :-).

Thanks for reviewing the patch set!

-- Daniel

Currently, a schedule while atomic error prints the stack trace to the kernel log and the system continue running.

Although it is possible to collect the kernel log messages and analyse it, often more information are needed. Furthermore, keep the system running is not always the best choice. For example, when the preempt count underflows the system will not stop to complain about scheduling while atomic, so the kernel log can wraparound overwriting the first stack trace, tuning the analysis even more difficult.

This patch implements the kernel.panic_on_sched_in_atomic sysctl to help out on these more complex situations.

When kernel.panic_on_sched_in_atomic is set to 1, the kernel will panic() in the schedule while atomic detection.

The default value for the sysctl is 0, maintaining the current behavior.

Cc Ingo Molnar <[email protected]>

Cc Peter Zijlstra <[email protected]>

Reviewed-by Arnaldo Carvalho de Melo <[email protected]>

Signed-off-by Daniel Bristot de Oliveira <[email protected]>

 Documentation/sysctl/kernel.txt | 13 +++++++++++++
 include/linux/kernel.h          |  1 +
 kernel/sched/core.c             |  7 +++++++
 kernel/sysctl.c                 |  9 +++++++++
 4 files changed, 30 insertions(+)

Documentation/sysctl/kernel.txt +13 -0

index 29ec4bb..0b176bb 100644
@@ -59,6 +59,7 @@ show up in /proc/sys/kernel:
 - panic_on_unrecovered_nmi
 - panic_on_warn
 - panic_on_rcu_stall
+- panic_on_sched_in_atomic
 - perf_cpu_time_max_percent
 - perf_event_paranoid
 - perf_event_max_stack

@@ -630,6 +631,18 @@ is useful to define the root cause of RCU stalls using kdump.
 
 ==============================================================
 
+panic_on_sched_in_atomic:
+
+When set to 1, calls panic() in the schedule while in atomic detection.
+This is useful get a vmcore to inspect the root cause of a schedule()
+call in the atomic context.
+
+0: do not panic() on scheduling while in atomic, default behavior.
+
+1: panic() after printing schedule while in atomic messages.
+
+==============================================================
+
 perf_cpu_time_max_percent:
 
 Hints to the kernel how much CPU time it should be allowed to

include/linux/kernel.h +1 -0

index c420821..e4d9804 100644
@@ -452,6 +452,7 @@ extern int panic_on_unrecovered_nmi;
 extern int panic_on_io_nmi;
 extern int panic_on_warn;
 extern int sysctl_panic_on_rcu_stall;
+extern int sysctl_panic_on_sched_in_atomic;
 extern int sysctl_panic_on_stackoverflow;
 
 extern bool crash_kexec_post_notifiers;

kernel/sched/core.c +7 -0

index 7f2cae4..602f978 100644
@@ -152,6 +152,11 @@ __read_mostly int scheduler_running;
  */
 int sysctl_sched_rt_runtime = 950000;
 
+/*
+ * panic on scheduling while atomic
+ */
+__read_mostly int sysctl_panic_on_sched_in_atomic = 0;
+
 /* cpus with isolated domains */
 cpumask_var_t cpu_isolated_map;
 

@@ -3146,6 +3151,8 @@ static noinline void __schedule_bug(struct task_struct *prev)
 		pr_cont("\n");
 	}
 #endif
+	if (sysctl_panic_on_sched_in_atomic)
+		panic("scheduling while atomic\n");
 	dump_stack();
 	add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
 }

kernel/sysctl.c +9 -0

index d3b93aa..f0a984c 100644
@@ -1216,6 +1216,15 @@ static struct ctl_table kern_table[] = {
 		.extra2		= &one,
 	},
 #endif
+	{
+		.procname	= "panic_on_sched_in_atomic",
+		.data		= &sysctl_panic_on_sched_in_atomic,
+		.maxlen		= sizeof(sysctl_panic_on_sched_in_atomic),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= &zero,
+		.extra2		= &one,
+	},
 	{ }
 };
 

Signature · 1 line

2.5.5

Currently, a schedule while atomic error prints the stack trace to the kernel log and the system continue running.

Although it is possible to collect the kernel log messages and analyse it, often more information are needed. Furthermore, keep the system running is not always the best choice. For example, when the preempt count underflows the system will not stop to complain about scheduling while atomic, so the kernel log can wraparound overwriting the first stack trace, tuning the analysis even more difficult.

This patch implements the kernel.panic_on_sched_in_atomic sysctl to help out on these more complex situations.

When kernel.panic_on_sched_in_atomic is set to 1, the kernel will panic() in the schedule while atomic detection.

Do we really need more panic_on_* knobs? Can't we re-purpose panic_on_warn for this?

This patch series implements two kernel.panic_on_* like sysctl:

kernel.panic_on_rcu_stall:
panic() on RCU Stall detection.

kernel.panic_on_sched_in_atomic:
panic() on schedule while atomic detection.

These sysctls are useful to capture a vmcore when is not possible to recompile the kernel to include the panic() call. For instance when supporting enterprise users.

Daniel Bristot de Oliveira (2):
rcu: sysctl: Panic on RCU Stall
sched: sysctl: Panic on scheduling while atomic

 Documentation/sysctl/kernel.txt | 25 +++++++++++++++++++++++++
 include/linux/kernel.h          |  2 ++
 kernel/rcu/tree.c               |  8 ++++++++
 kernel/sched/core.c             |  7 +++++++
 kernel/sysctl.c                 | 20 ++++++++++++++++++++
 5 files changed, 62 insertions(+)