Skip to contentSkip to conversation

Work around perf NMI-induced hangs

PATCH In review Dave Hansen 3 participants 7 messages May 29, 2013 – May 30, 2013

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v3 cover Work around perf NMI-induced hangs

Changes from v2:

2/4:
* Only warn on the longest NMIs. Don't print when over a threshhold.
* Output in ms as opposed to ns
4/4:
* Add some Documentation/ for the tracepoint
* keep tracepoint delta in a s64 instead of an int, and vall it 'delta_ns' instead of 'len'

Changes from v1:

* keep a running average instead of taking a single value for determining NMI lengths.
* Fixed some of the math converting from ns to/from percentages (it was backwards)
* Included nmi length tracepoint at end of series

--

If root or an unprivileged user runs 'perf top', my system hangs. If I'm lucky, I get a warning out to dmesg, along these lines:

	hrtimer: interrupt took 13915457 ns cpu: 132

or a hard-lockup message on occasion.

The proxmiate cause of this is that perf_event_nmi_handler() has been observed to take tens of ms on occasion. That needs to get fixed, and I'm working on tracking the root cause down.

But, These patches make the situation better: perf can no longer simply wedge the box, and we have a safe, controlled exit path when things go wrong.

#1 lore raw

PATCH v3 1/4 perf/x86: only print PMU state when also WARN()'ing

From: Dave Hansen <[email protected]>

intel_pmu_handle_irq() has a warning in it if it does too many loops. It is a WARN_ONCE(), but the perf_event_print_debug() call beneath it is unconditional. For the first warning, you get a nice backtrace and message, but subsequent ones just dump the PMU state with no leading messages. I doubt this is what was intended.

This patch will only print the PMU state when paired with the WARN_ON() text. It effectively open-codes WARN_ONCE()'s one-time-only logic.

My suspicion is that the code really just wants to make sure we do not sit in the loop and spit out a warning for every loop iteration after the 100th. From what I've seen, this is very unlikely to happen since we also clear the PMU state.

After this patch, instead of seeing the PMU state dumped each time, you will just see:

	[57494.894540] perf_event_intel: clearing PMU state on CPU#129
	[57579.539668] perf_event_intel: clearing PMU state on CPU#10
	[57587.137762] perf_event_intel: clearing PMU state on CPU#134
	[57623.039912] perf_event_intel: clearing PMU state on CPU#114
	[57644.559943] perf_event_intel: clearing PMU state on CPU#118
	...

---

 linux.git-davehans/arch/x86/kernel/cpu/perf_event_intel.c |    8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/perf_event_intel.c renamed from arch/x86/kernel/cpu/perf_event_intel.c~debug-perf-hangs +6 -2
diff -puN arch/x86/kernel/cpu/perf_event_intel.c~debug-perf-hangs arch/x86/kernel/cpu/perf_event_intel.c
@@ -1188,8 +1188,12 @@ static int intel_pmu_handle_irq(struct p
 again:
 	intel_pmu_ack_status(status);
 	if (++loops > 100) {
-		WARN_ONCE(1, "perfevents: irq loop stuck!\n");
-		perf_event_print_debug();
+		static bool warned = false;
+		if (!warned) {
+			WARN(1, "perfevents: irq loop stuck!\n");
+			perf_event_print_debug();
+			warned = true;
+		}
 		intel_pmu_reset();
 		goto done;
 	}

_

#2 lore raw

PATCH v3 2/4 x86: warn when NMI handlers take large amounts of time

In reply to Dave Hansen

From: Dave Hansen <[email protected]>

I have a system which is causing all kinds of problems. It has 8 NUMA nodes, and lots of cores that can fight over cachelines. If things are not working _perfectly_, then NMIs can take longer than expected.

If we get too many of them backed up to each other, we can easily end up in a situation where we are doing nothing *but* running NMIs. The biggest problem, though, is that this happens _silently_. You might be lucky to get an hrtimer warning, but most of the time system simply hangs.

This patch should at least give us some warning before we fall off the cliff. the warnings look like this:

	nmi_handle: perf_event_nmi_handler() took: 26095071 ns

The message is triggered whenever we notice the longest NMI we've seen to date. You can always view and reset this value via the debugfs interface if you like.

---

 linux.git-davehans/arch/x86/kernel/nmi.c |   28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)
arch/x86/kernel/nmi.c renamed from arch/x86/kernel/nmi.c~debug-nmi-timing +27 -1
diff -puN arch/x86/kernel/nmi.c~debug-nmi-timing arch/x86/kernel/nmi.c
@@ -14,6 +14,7 @@
 #include <linux/kprobes.h>
 #include <linux/kdebug.h>
 #include <linux/nmi.h>
+#include <linux/debugfs.h>
 #include <linux/delay.h>
 #include <linux/hardirq.h>
 #include <linux/slab.h>
@@ -82,6 +83,15 @@ __setup("unknown_nmi_panic", setup_unkno
 
 #define nmi_to_desc(type) (&nmi_desc[type])
 
+static u64 nmi_longest_ns = 1000 * 1000 * 1000;
+static int __init nmi_warning_debugfs(void)
+{
+	debugfs_create_u64("nmi_longest_ns", 0644,
+			arch_debugfs_dir, &nmi_longest_ns);
+	return 0;
+}
+fs_initcall(nmi_warning_debugfs);
+
 static int __kprobes nmi_handle(unsigned int type, struct pt_regs *regs, bool b2b)
 {
 	struct nmi_desc *desc = nmi_to_desc(type);
@@ -96,8 +106,24 @@ static int __kprobes nmi_handle(unsigned
 	 * can be latched at any given time.  Walk the whole list
 	 * to handle those situations.
 	 */
-	list_for_each_entry_rcu(a, &desc->head, list)
+	list_for_each_entry_rcu(a, &desc->head, list) {
+		u64 before, delta, whole_msecs;
+		int decimal_msecs;
+
+		before = local_clock();
 		handled += a->handler(type, regs);
+		delta = local_clock() - before;
+
+		if (delta < nmi_longest_ns)
+			continue;
+
+		nmi_longest_ns = delta;
+		whole_msecs = delta / (1000 * 1000);
+		decimal_msecs = (delta / 1000) % 1000;
+		printk_ratelimited(KERN_INFO
+			"INFO: NMI handler took too long to run: "
+			"%lld.%03d msecs\n", whole_msecs, decimal_msecs);
+	}
 
 	rcu_read_unlock();
 

_

#3 lore raw

Re: [v3][PATCH 2/4] x86: warn when NMI handlers take large amounts of time

Quoted 13 lines · Dave Hansen
arch/x86/kernel/nmi.c renamed from arch/x86/kernel/nmi.c~debug-nmi-timing +7 -1
@@ -96,8 +106,24 @@ static int __kprobes nmi_handle(unsigned
 	 * can be latched at any given time.  Walk the whole list
 	 * to handle those situations.
 	 */
-	list_for_each_entry_rcu(a, &desc->head, list)
+	list_for_each_entry_rcu(a, &desc->head, list) {
+		u64 before, delta, whole_msecs;
+		int decimal_msecs;
+
+		before = local_clock();
 		handled += a->handler(type, regs);
+		delta = local_clock() - before;
+

arch/x86/kernel/nmi.c static int __kprobes nmi_handle(unsigned · line 126

Quoted 10 lines
arch/x86/kernel/nmi.c renamed from arch/x86/kernel/nmi.c~debug-nmi-timing +10 -0
+		if (delta < nmi_longest_ns)
+			continue;
+
+		nmi_longest_ns = delta;
+		whole_msecs = delta / (1000 * 1000);
+		decimal_msecs = (delta / 1000) % 1000;
+		printk_ratelimited(KERN_INFO
+			"INFO: NMI handler took too long to run: "
+			"%lld.%03d msecs\n", whole_msecs, decimal_msecs);
+	}

One more thing, would it make sense to print the symbol name of the ->handler here? That way the message would be more actionable:

INFO: NMI handler (nmi_handler_foo) took too long to run: 12.613 msecs

(Note, printk() has a '%ps' format string extension to print function pointer symbolic names, so it's easy to add.)

Thanks,

Ingo

maintainer
#4 lore raw

Re: [v3][PATCH 2/4] x86: warn when NMI handlers take large amounts of time

In reply to Dave Hansen
Quoted 1 line · Dave Hansen

+static u64 nmi_longest_ns = 1000 * 1000 * 1000;

1s? Isn't that a tad too long?

FWIW there's NSEC_PER_{U,M,}SEC for such cases, I find those macros increase readability.

#5 lore raw

PATCH v3 3/4 perf: drop sample rate when sampling is too slow

In reply to Dave Hansen

From: Dave Hansen <[email protected]>

This patch keeps track of how long perf's NMI handler is taking, and also calculates how many samples perf can take a second. If the sample length times the expected max number of samples exceeds a configurable threshold, it drops the sample rate. This way, we don't have a runaway sampling process eating up the CPU.

This patch can tend to drop the sample rate down to level where perf doesn't work very well. *BUT* the alternative is that my system hangs because it spends all of its time handling NMIs.

I'll take a busted performance tool over an entire system that's busted and undebuggable any day.

BTW, my suspicion is that there's still an underlying bug here. Using the HPET instead of the TSC is definitely a contributing factor, but I suspect there are some other things going on. But, I can't go dig down on a bug like that with my machine hanging all the time.

---

 linux.git-davehans/Documentation/sysctl/kernel.txt  |   26 ++++++
 linux.git-davehans/arch/x86/kernel/cpu/perf_event.c |   12 ++-
 linux.git-davehans/include/linux/perf_event.h       |    7 +
 linux.git-davehans/kernel/events/core.c             |   78 ++++++++++++++++++++
 linux.git-davehans/kernel/sysctl.c                  |    9 ++
 5 files changed, 131 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/perf_event.c renamed from arch/x86/kernel/cpu/perf_event.c~drop-perf-samples-when-sampling-is-too-slow +11 -1
diff -puN arch/x86/kernel/cpu/perf_event.c~drop-perf-samples-when-sampling-is-too-slow arch/x86/kernel/cpu/perf_event.c
@@ -1249,10 +1249,20 @@ void perf_events_lapic_init(void)
 static int __kprobes
 perf_event_nmi_handler(unsigned int cmd, struct pt_regs *regs)
 {
+	int ret;
+	u64 start_clock;
+	u64 finish_clock;
+
 	if (!atomic_read(&active_events))
 		return NMI_DONE;
 
-	return x86_pmu.handle_irq(regs);
+	start_clock = local_clock();
+	ret = x86_pmu.handle_irq(regs);
+	finish_clock = local_clock();
+
+	perf_sample_event_took(finish_clock - start_clock);
+
+	return ret;
 }
 
 struct event_constraint emptyconstraint;
Documentation/sysctl/kernel.txt renamed from Documentation/sysctl/kernel.txt~drop-perf-samples-when-sampling-is-too-slow +26 -0
diff -puN Documentation/sysctl/kernel.txt~drop-perf-samples-when-sampling-is-too-slow Documentation/sysctl/kernel.txt
@@ -427,6 +427,32 @@ This file shows up if CONFIG_DEBUG_STACK
 
 ==============================================================
 
+perf_cpu_time_max_percent:
+
+Hints to the kernel how much CPU time it should be allowed to
+use to handle perf sampling events.  If the perf subsystem
+is informed that its samples are exceeding this limit, it
+will drop its sampling frequency to attempt to reduce its CPU
+usage.
+
+Some perf sampling happens in NMIs.  If these samples
+unexpectedly take too long to execute, the NMIs can become
+stacked up next to each other so much that nothing else is
+allowed to execute.
+
+0: disable the mechanism.  Do not monitor or correct perf's
+   sampling rate no matter how CPU time it takes.
+
+1-100: attempt to throttle perf's sample rate to this
+   percentage of CPU.  Note: the kernel calculates an
+   "expected" length of each sample event.  100 here means
+   100% of that expected length.  Even if this is set to
+   100, you may still see sample throttling if this
+   length is exceeded.  Set to 0 if you truly do not care
+   how much CPU is consumed.
+
+==============================================================
+
 
 pid_max:
 
include/linux/perf_event.h renamed from include/linux/perf_event.h~drop-perf-samples-when-sampling-is-too-slow +7 -0
diff -puN include/linux/perf_event.h~drop-perf-samples-when-sampling-is-too-slow include/linux/perf_event.h
@@ -696,10 +696,17 @@ static inline void perf_callchain_store(
 extern int sysctl_perf_event_paranoid;
 extern int sysctl_perf_event_mlock;
 extern int sysctl_perf_event_sample_rate;
+extern int sysctl_perf_cpu_time_max_percent;
+
+extern void perf_sample_event_took(u64 sample_len_ns);
 
 extern int perf_proc_update_handler(struct ctl_table *table, int write,
 		void __user *buffer, size_t *lenp,
 		loff_t *ppos);
+extern int perf_cpu_time_max_percent_handler(struct ctl_table *table, int write,
+		void __user *buffer, size_t *lenp,
+		loff_t *ppos);
+
 
 static inline bool perf_paranoid_tracepoint_raw(void)
 {
kernel/events/core.c renamed from kernel/events/core.c~drop-perf-samples-when-sampling-is-too-slow +78 -0

97 lines not on this page · raw message

kernel/sysctl.c renamed from kernel/sysctl.c~drop-perf-samples-when-sampling-is-too-slow +9 -0
diff -puN kernel/sysctl.c~drop-perf-samples-when-sampling-is-too-slow kernel/sysctl.c
@@ -1044,6 +1044,15 @@ static struct ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= perf_proc_update_handler,
 	},
+	{
+		.procname	= "perf_cpu_time_max_percent",
+		.data		= &sysctl_perf_cpu_time_max_percent,
+		.maxlen		= sizeof(sysctl_perf_cpu_time_max_percent),
+		.mode		= 0644,
+		.proc_handler	= perf_cpu_time_max_percent_handler,
+		.extra1		= &zero,
+		.extra2		= &one_hundred,
+	},
 #endif
 #ifdef CONFIG_KMEMCHECK
 	{

_

#6 lore raw

PATCH v3 4/4 x86: nmi length tracepoints

In reply to Dave Hansen

From: Dave Hansen <[email protected]>

This patch has been invaluable in my adventures finding issues in the perf NMI handler. I'm as big a fan of printk() as anybody is, but using printk() in NMIs is deadly when they're happening frequently.

Even hacking in trace_printk() ended up eating enough CPU to throw off some of the measurements I was making.

---

 linux.git-davehans/Documentation/trace/events-nmi.txt |   43 ++++++++++++++++++
 linux.git-davehans/arch/x86/kernel/nmi.c              |    9 ++-
 linux.git-davehans/include/trace/events/nmi.h         |   37 +++++++++++++++
 3 files changed, 87 insertions(+), 2 deletions(-)
arch/x86/kernel/nmi.c renamed from arch/x86/kernel/nmi.c~nmi-tracepoints +7 -2
diff -puN arch/x86/kernel/nmi.c~nmi-tracepoints arch/x86/kernel/nmi.c
@@ -30,6 +30,9 @@
 #include <asm/nmi.h>
 #include <asm/x86_init.h>
 
+#define CREATE_TRACE_POINTS
+#include <trace/events/nmi.h>
+
 struct nmi_desc {
 	spinlock_t lock;
 	struct list_head head;
@@ -108,11 +111,13 @@ static int __kprobes nmi_handle(unsigned
 	 */
 	list_for_each_entry_rcu(a, &desc->head, list) {
 		u64 before, delta, whole_msecs;
-		int decimal_msecs;
+		int decimal_msecs, thishandled;
 
 		before = local_clock();
-		handled += a->handler(type, regs);
+		thishandled = a->handler(type, regs);
+		handled += thishandled;
 		delta = local_clock() - before;
+		trace_nmi_handler(a->handler, (int)delta, thishandled);
 
 		if (delta < nmi_longest_ns)
 			continue;
Documentation/trace/events-nmi.txt new file +43 -0

43 lines not on this page · raw message

include/trace/events/nmi.h new file +37 -0
diff -puN /dev/null include/trace/events/nmi.h
@@ -0,0 +1,37 @@
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM nmi
+
+#if !defined(_TRACE_NMI_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_NMI_H
+
+#include <linux/ktime.h>
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(nmi_handler,
+
+	TP_PROTO(void *handler, s64 delta_ns, int handled),
+
+	TP_ARGS(handler, delta_ns, handled),
+
+	TP_STRUCT__entry(
+		__field(	void *,		handler	)
+		__field(	s64,		delta_ns)
+		__field(	int,		handled	)
+	),
+
+	TP_fast_assign(
+		__entry->handler = handler;
+		__entry->delta_ns = delta_ns;
+		__entry->handled = handled;
+	),
+
+	TP_printk("%ps() delta_ns: %lld handled: %d",
+		__entry->handler,
+		__entry->delta_ns,
+		__entry->handled)
+);
+
+#endif /* _TRACE_NMI_H */
+
+/* This part ust be outside protection */
+#include <trace/define_trace.h>

_