Skip to contentSkip to conversation

hung_task: Allow printing warnings every check interval

PATCH In review Dmitry Safonov 2 participants 6 messages Jul 24, 2019 – Jul 26, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 hung_task: Allow printing warnings every check interval

Hung task detector has one timeout and has two associated actions on it:
- issuing warnings with names and stacks of blocked tasks
- panic()

We want switches to panic (and reboot) if there's a task in uninterruptible sleep for some minutes - at that moment something ugly has happened and the box needs a reboot.
But we also want to detect conditions that are "out of range" or approaching the point of failure. Under such conditions we want to issue an "early warning" of an impending failure, minutes before the switch is going to panic.

Those "early warnings" serve a purpose while monitoring the network infrastructure. Those are also valuable on post-mortem analysis, when the logs from userspace applications aren't enough. Furthermore, we have a test pool of long-running duts that are constantly under close to real-world load for weeks. And such early warnings allowed to figure out some bottle necks without much engineer work intervention.

There are also not yet upstream patches for other kinds of "early warnings" as prints whenever a mutex/semaphore is released after being held for long time, but those patches are much more intricate and have their runtime cost.

It seems rather easy to add printing tasks and their stacks for notification and debugging purposes into hung task detector without complicating the code or major cost (prints are with KERN_INFO loglevel and so don't go on console, only into dmesg log).

Since commit a2e514453861 ("kernel/hung_task.c: allow to set checking interval separately from timeout") it's possible to set checking interval for hung task detector with hung_task_check_interval_secs.

Provide hung_task_interval_warnings sysctl that allows printing hung tasks every detection interval. It's not ratelimited, so the root should be cautious configuring it.

Cc Andrew Morton <[email protected]>
Cc Dmitry Vyukov <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jonathan Corbet <[email protected]>
Cc Tetsuo Handa <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc "Peter Zijlstra (Intel)" <[email protected]>
Cc Vasiliy Khoruzhick <[email protected]>
---
 Documentation/admin-guide/sysctl/kernel.rst | 20 ++++++++-
 include/linux/sched/sysctl.h                |  1 +
 kernel/hung_task.c                          | 50 ++++++++++++++-------
 kernel/sysctl.c                             |  8 ++++
 4 files changed, 62 insertions(+), 17 deletions(-)
Documentation/admin-guide/sysctl/kernel.rst +18 -2
index 032c7cd3cede..2e36620ec1e4 100644
@@ -45,6 +45,7 @@ show up in /proc/sys/kernel:
 - hung_task_timeout_secs
 - hung_task_check_interval_secs
 - hung_task_warnings
+- hung_task_interval_warnings
 - hyperv_record_panic_msg
 - kexec_load_disabled
 - kptr_restrict
@@ -383,14 +384,29 @@ Possible values to set are in range {0..LONG_MAX/HZ}.
 hung_task_warnings:
 ===================
 
-The maximum number of warnings to report. During a check interval
-if a hung task is detected, this value is decreased by 1.
+The maximum number of warnings to report. If after timeout a hung
+task is present, this value is decreased by 1 every check interval,
+producing a warning.
 When this value reaches 0, no more warnings will be reported.
 This file shows up if CONFIG_DETECT_HUNG_TASK is enabled.
 
 -1: report an infinite number of warnings.
 
 
+hung_task_interval_warnings:
+===================
+
+The same as hung_task_warnings, but set the number of interval
+warnings to be issued about detected hung tasks during check
+interval. That will produce warnings *before* the timeout happens.
+If a hung task is detected during check interval, this value is
+decreased by 1. When this value reaches 0, only timeout warnings
+will be reported.
+This file shows up if CONFIG_DETECT_HUNG_TASK is enabled.
+
+-1: report an infinite number of check interval warnings.
+
+
 hyperv_record_panic_msg:
 ========================
 
include/linux/sched/sysctl.h +1 -0
index d4f6215ee03f..89f55e914673 100644
@@ -12,6 +12,7 @@ extern unsigned int  sysctl_hung_task_panic;
 extern unsigned long sysctl_hung_task_timeout_secs;
 extern unsigned long sysctl_hung_task_check_interval_secs;
 extern int sysctl_hung_task_warnings;
+extern int sysctl_hung_task_interval_warnings;
 extern int proc_dohung_task_timeout_secs(struct ctl_table *table, int write,
 					 void __user *buffer,
 					 size_t *lenp, loff_t *ppos);
kernel/hung_task.c +35 -15

75 lines not on this page · raw message

kernel/sysctl.c +8 -0
index 078950d9605b..f12888971d66 100644
@@ -1147,6 +1147,14 @@ static struct ctl_table kern_table[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= &neg_one,
 	},
+	{
+		.procname	= "hung_task_interval_warnings",
+		.data		= &sysctl_hung_task_interval_warnings,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= &neg_one,
+	},
 #endif
 #ifdef CONFIG_RT_MUTEXES
 	{
Signature · 1 line
2.22.0
#1 lore raw
Quoted 11 lines · Dmitry Safonov

Hung task detector has one timeout and has two associated actions on it:
- issuing warnings with names and stacks of blocked tasks
- panic()

We want switches to panic (and reboot) if there's a task in uninterruptible sleep for some minutes - at that moment something ugly has happened and the box needs a reboot.
But we also want to detect conditions that are "out of range" or approaching the point of failure. Under such conditions we want to issue an "early warning" of an impending failure, minutes before the switch is going to panic.

Can't we do it by extending sysctl_hung_task_panic to accept values larger than 1, and decrease by one when at least one thread was reported by each check_hung_uninterruptible_tasks() check, and call panic() when sysctl_hung_task_panic reached to 0 (or maybe 1 is simpler) ?

Hmm, might have the same problem regarding how/when to reset the counter. If some userspace process can reset the counter, such process can trigger SysRq-c when some period expired...

Quoted 4 lines

It seems rather easy to add printing tasks and their stacks for notification and debugging purposes into hung task detector without complicating the code or major cost (prints are with KERN_INFO loglevel and so don't go on console, only into dmesg log).

Well, I don't think so. Might be noisy for systems without "quiet" kernel command line option, and we can't pass KERN_DEBUG to e.g. sched_show_task()...

#2 lore raw
Quoted 13 lines · Tetsuo Handa
Quoted 11 lines · Dmitry Safonov

Hung task detector has one timeout and has two associated actions on it:
- issuing warnings with names and stacks of blocked tasks
- panic()

We want switches to panic (and reboot) if there's a task in uninterruptible sleep for some minutes - at that moment something ugly has happened and the box needs a reboot.
But we also want to detect conditions that are "out of range" or approaching the point of failure. Under such conditions we want to issue an "early warning" of an impending failure, minutes before the switch is going to panic.

Quoted 8 lines

Can't we do it by extending sysctl_hung_task_panic to accept values larger than 1, and decrease by one when at least one thread was reported by each check_hung_uninterruptible_tasks() check, and call panic() when sysctl_hung_task_panic reached to 0 (or maybe 1 is simpler) ?

Hmm, might have the same problem regarding how/when to reset the counter. If some userspace process can reset the counter, such process can trigger SysRq-c when some period expired...

Yes, also current distributions already using the counter to print warnings number of times and then silently ignore. I.e., on my Arch Linux setup:
hung_task_warnings:10

Quoted 7 lines
Quoted 4 lines

It seems rather easy to add printing tasks and their stacks for notification and debugging purposes into hung task detector without complicating the code or major cost (prints are with KERN_INFO loglevel and so don't go on console, only into dmesg log).

Well, I don't think so. Might be noisy for systems without "quiet" kernel command line option, and we can't pass KERN_DEBUG to e.g. sched_show_task()...

Yes, that's why it's disabled by default (=0).
I tend to agree that printing with KERN_DEBUG may be better, but in my point of view the patch isn't enough justification for patching sched_show_task() and show_stack().

Thanks,
Dmitry

#3 lore raw
Quoted 4 lines · Dmitry Safonov

Yes, also current distributions already using the counter to print warnings number of times and then silently ignore. I.e., on my Arch Linux setup:
hung_task_warnings:10

You can propose changing the default value of hung_task_warnings to -1.

Current patch might be inconvenient because printk() from hung_task_warning(t, false) fails to go to consoles when that "t" was blocked for more than "timeout" seconds, for

	if (sysctl_hung_task_panic) {
		console_verbose();
		hung_task_show_lock = true;
		hung_task_call_panic = true;
	}

path which is intended to force printk() to go to consoles is ignored by

	/* Don't print warings twice */
	if (!sysctl_hung_task_interval_warnings)
		hung_task_warning(t, true);

when panic() should be called. (The vmcore would contain the printk() output which was not sent to consoles if kdump is configured. But vmcore is not always available.)

Quoted 4 lines

Yes, that's why it's disabled by default (=0).
I tend to agree that printing with KERN_DEBUG may be better, but in my point of view the patch isn't enough justification for patching sched_show_task() and show_stack().

You can propose sched_show_task_log_lvl() and show_stack_log_lvl() like show_trace_log_lvl().

I think that sysctl_hung_task_interval_warnings should not be decremented automatically. I guess that that variable should become a boolean which controls whether to report threads (with KERN_DEBUG level) which was blocked for more than sysctl_hung_task_check_interval_secs seconds (or a tristate which also controls whether the report should be sent to consoles (because KERN_DEBUG level likely prevents sending to consoles)), and hung_task_warning(t, false) should be called like

	if (time_is_after_jiffies(t->last_switch_time + timeout * HZ)) {
		if (sysctl_hung_task_interval_warnings)
			hung_task_warning(t, false);
		return;
	}

rather than

	if (sysctl_hung_task_interval_warnings)
		hung_task_warning(t, false);
	if (time_is_after_jiffies(t->last_switch_time + timeout * HZ))
		return;

.

#4 lore raw
Quoted 46 lines · Tetsuo Handa
Quoted 4 lines · Dmitry Safonov

Yes, also current distributions already using the counter to print warnings number of times and then silently ignore. I.e., on my Arch Linux setup:
hung_task_warnings:10

You can propose changing the default value of hung_task_warnings to -1.

Current patch might be inconvenient because printk() from hung_task_warning(t, false) fails to go to consoles when that "t" was blocked for more than "timeout" seconds, for

	if (sysctl_hung_task_panic) {
		console_verbose();
		hung_task_show_lock = true;
		hung_task_call_panic = true;
	}

path which is intended to force printk() to go to consoles is ignored by

	/* Don't print warings twice */
	if (!sysctl_hung_task_interval_warnings)
		hung_task_warning(t, true);

when panic() should be called. (The vmcore would contain the printk() output which was not sent to consoles if kdump is configured. But vmcore is not always available.)

Quoted 4 lines

Yes, that's why it's disabled by default (=0).
I tend to agree that printing with KERN_DEBUG may be better, but in my point of view the patch isn't enough justification for patching sched_show_task() and show_stack().

You can propose sched_show_task_log_lvl() and show_stack_log_lvl() like show_trace_log_lvl().

I think that sysctl_hung_task_interval_warnings should not be decremented automatically. I guess that that variable should become a boolean which controls whether to report threads (with KERN_DEBUG level) which was blocked for more than sysctl_hung_task_check_interval_secs seconds (or a tristate which also controls whether the report should be sent to consoles (because KERN_DEBUG level likely prevents sending to consoles)), and hung_task_warning(t, false) should be called like

	if (time_is_after_jiffies(t->last_switch_time + timeout * HZ)) {
		if (sysctl_hung_task_interval_warnings)
			hung_task_warning(t, false);
		return;
	}
Quoted 9 lines

rather than

	if (sysctl_hung_task_interval_warnings)
		hung_task_warning(t, false);
	if (time_is_after_jiffies(t->last_switch_time + timeout * HZ))
		return;

.

Well, another direction is to disassociate sysctl_hung_task_panic from sysctl_hung_task_timeout_secs. Since nobody would want to call panic() when a thread was blocked for only one second, allow sysctl_hung_task_panic to specify larger than 1, and interpret it as sysctl_hung_task_timeout_secs for calling panic(). Roughly speaking:

- if (sysctl_hung_task_panic) {
+ unsigned long panic_timeout = READ_ONCE(sysctl_hung_task_panic)
+ if (panic_timeout == 1 || (panic_timeout > 1 &&
+ (jiffies - t->last_switch_time) / HZ >= panic_timeout)) {
console_verbose();
hung_task_show_lock = true;
hung_task_call_panic = true;
}

If use of different loglevel is not a requirement for you, this would be the simplest.

#5 lore raw
Quoted 18 lines · Tetsuo Handa
Quoted 17 lines · Tetsuo Handa
Quoted 4 lines · Dmitry Safonov

Yes, also current distributions already using the counter to print warnings number of times and then silently ignore. I.e., on my Arch Linux setup:
hung_task_warnings:10

You can propose changing the default value of hung_task_warnings to -1.

Current patch might be inconvenient because printk() from hung_task_warning(t, false) fails to go to consoles when that "t" was blocked for more than "timeout" seconds, for

	if (sysctl_hung_task_panic) {
		console_verbose();
		hung_task_show_lock = true;
		hung_task_call_panic = true;
	}
Quoted 8 lines
Quoted 8 lines

path which is intended to force printk() to go to consoles is ignored by

	/* Don't print warings twice */
	if (!sysctl_hung_task_interval_warnings)
		hung_task_warning(t, true);

when panic() should be called. (The vmcore would contain the printk() output which was not sent to consoles if kdump is configured. But vmcore is not always available.)

Fair enough.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Yes, that's why it's disabled by default (=0).
I tend to agree that printing with KERN_DEBUG may be better, but in my point of view the patch isn't enough justification for patching sched_show_task() and show_stack().

You can propose sched_show_task_log_lvl() and show_stack_log_lvl() like show_trace_log_lvl().

I'll try, not sure how well it will go..

Quoted 14 lines
Quoted 14 lines

I think that sysctl_hung_task_interval_warnings should not be decremented automatically. I guess that that variable should become a boolean which controls whether to report threads (with KERN_DEBUG level) which was blocked for more than sysctl_hung_task_check_interval_secs seconds (or a tristate which also controls whether the report should be sent to consoles (because KERN_DEBUG level likely prevents sending to consoles)), and hung_task_warning(t, false) should be called like

	if (time_is_after_jiffies(t->last_switch_time + timeout * HZ)) {
		if (sysctl_hung_task_interval_warnings)
			hung_task_warning(t, false);
		return;
	}
Quoted 6 lines
Quoted 6 lines

rather than

	if (sysctl_hung_task_interval_warnings)
		hung_task_warning(t, false);
	if (time_is_after_jiffies(t->last_switch_time + timeout * HZ))
		return;

Good point, will do.

Quoted 15 lines

Well, another direction is to disassociate sysctl_hung_task_panic from sysctl_hung_task_timeout_secs. Since nobody would want to call panic() when a thread was blocked for only one second, allow sysctl_hung_task_panic to specify larger than 1, and interpret it as sysctl_hung_task_timeout_secs for calling panic(). Roughly speaking:

(quoted diff) +3 -1
-	if (sysctl_hung_task_panic) {
+	unsigned long panic_timeout = READ_ONCE(sysctl_hung_task_panic)
+	if (panic_timeout == 1 || (panic_timeout > 1 &&
+	     (jiffies - t->last_switch_time) / HZ >= panic_timeout)) {
 		console_verbose();
 		hung_task_show_lock = true;
 		hung_task_call_panic = true;
 	}
Quoted 1 line

If use of different loglevel is not a requirement for you, this would be the simplest.

No, we consider such messages as notifications/warnings, rather than complete failures. So, it would be better to hide them from console. They're also not rate-limited which is a bummer with slow serial consoles that we've on some devices (9600).

Thanks for the review,
Dmitry