Skip to contentSkip to conversation

perf/x86/intel: Fix spurious NMI on fixed counter

PATCH In review [email protected] 6 participants 6 messages Jun 25, 2019 – Jul 13, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf/x86/intel: Fix spurious NMI on fixed counter

From: Kan Liang <[email protected]>

If a user first sample a PEBS event on a fixed counter, then sample a non-PEBS event on the same fixed counter on Icelake, it will trigger spurious NMI. For example,

perf record -e 'cycles:p' -a
perf record -e 'cycles' -a

The error message for spurious NMI.

[June 21 15:38] Uhhuh. NMI received for unknown reason 30 on CPU 2. [ +0.000000] Do you have a strange power saving mode enabled? [ +0.000000] Dazed and confused, but trying to continue

The issue was introduced by the following commit:

commit 6f55967ad9d9 ("perf/x86/intel: Fix race in intel_pmu_disable_event()")

The commit moves the intel_pmu_pebs_disable() after intel_pmu_disable_fixed(), which returns immediately.
The related bit of PEBS_ENABLE MSR will never be cleared for the fixed counter. Then a non-PEBS event runs on the fixed counter, but the bit on PEBS_ENABLE is still set, which trigger spurious NMI.

Check and disable PEBS for fixed counter after intel_pmu_disable_fixed().

Fixes 6f55967ad9d9 ("perf/x86/intel: Fix race in intel_pmu_disable_event()")
---
 arch/x86/events/intel/core.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
arch/x86/events/intel/core.c +3 -5
index 4377bf6a6f82..464316218b77 100644
@@ -2160,12 +2160,10 @@ static void intel_pmu_disable_event(struct perf_event *event)
 	cpuc->intel_ctrl_host_mask &= ~(1ull << hwc->idx);
 	cpuc->intel_cp_status &= ~(1ull << hwc->idx);
 
-	if (unlikely(hwc->config_base == MSR_ARCH_PERFMON_FIXED_CTR_CTRL)) {
+	if (unlikely(hwc->config_base == MSR_ARCH_PERFMON_FIXED_CTR_CTRL))
 		intel_pmu_disable_fixed(hwc);
-		return;
-	}
-
-	x86_pmu_disable_event(event);
+	else
+		x86_pmu_disable_event(event);
 
 	/*
 	 * Needs to be called after x86_pmu_disable_event,
Signature · 1 line
2.14.5
Acked-by on v1 1/1
#1 lore raw
Quoted 42 lines · On Tue, Jun 25, 2019 at 07:21:35AM -0700, [email protected] wrote:

From: Kan Liang <[email protected]>

If a user first sample a PEBS event on a fixed counter, then sample a non-PEBS event on the same fixed counter on Icelake, it will trigger spurious NMI. For example,

perf record -e 'cycles:p' -a
perf record -e 'cycles' -a

The error message for spurious NMI.

[June 21 15:38] Uhhuh. NMI received for unknown reason 30 on CPU 2. [ +0.000000] Do you have a strange power saving mode enabled? [ +0.000000] Dazed and confused, but trying to continue

The issue was introduced by the following commit:

commit 6f55967ad9d9 ("perf/x86/intel: Fix race in intel_pmu_disable_event()")

The commit moves the intel_pmu_pebs_disable() after intel_pmu_disable_fixed(), which returns immediately.
The related bit of PEBS_ENABLE MSR will never be cleared for the fixed counter. Then a non-PEBS event runs on the fixed counter, but the bit on PEBS_ENABLE is still set, which trigger spurious NMI.

Check and disable PEBS for fixed counter after intel_pmu_disable_fixed().

Fixes 6f55967ad9d9 ("perf/x86/intel: Fix race in intel_pmu_disable_event()")
---
 arch/x86/events/intel/core.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
arch/x86/events/intel/core.c +0 -0
index 4377bf6a6f82..464316218b77 100644
@@ -2160,12 +2160,10 @@ static void intel_pmu_disable_event(struct perf_event *event)
 	cpuc->intel_ctrl_host_mask &= ~(1ull << hwc->idx);
 	cpuc->intel_cp_status &= ~(1ull << hwc->idx);
 

arch/x86/events/intel/core.c static void intel_pmu_disable_event(struct perf_event *event) · line 2167

Quoted 10 lines
arch/x86/events/intel/core.c +3 -5
-	if (unlikely(hwc->config_base == MSR_ARCH_PERFMON_FIXED_CTR_CTRL)) {
+	if (unlikely(hwc->config_base == MSR_ARCH_PERFMON_FIXED_CTR_CTRL))
 		intel_pmu_disable_fixed(hwc);
-		return;
-	}
-
-	x86_pmu_disable_event(event);
+	else
+		x86_pmu_disable_event(event);
 

oops, I overlooed this, looks good

thanks,
jirka

Quoted 5 lines
arch/x86/events/intel/core.c +0 -0
 	/*
 	 * Needs to be called after x86_pmu_disable_event,
Signature · 1 line
2.14.5
#2 lore raw
Quoted 56 lines · Jiri Olsa
Quoted 42 lines · On Tue, Jun 25, 2019 at 07:21:35AM -0700, [email protected] wrote:

From: Kan Liang <[email protected]>

If a user first sample a PEBS event on a fixed counter, then sample a non-PEBS event on the same fixed counter on Icelake, it will trigger spurious NMI. For example,

perf record -e 'cycles:p' -a
perf record -e 'cycles' -a

The error message for spurious NMI.

[June 21 15:38] Uhhuh. NMI received for unknown reason 30 on CPU 2. [ +0.000000] Do you have a strange power saving mode enabled? [ +0.000000] Dazed and confused, but trying to continue

The issue was introduced by the following commit:

   commit 6f55967ad9d9 ("perf/x86/intel: Fix race in intel_pmu_disable_event()")

The commit moves the intel_pmu_pebs_disable() after intel_pmu_disable_fixed(), which returns immediately.
The related bit of PEBS_ENABLE MSR will never be cleared for the fixed counter. Then a non-PEBS event runs on the fixed counter, but the bit on PEBS_ENABLE is still set, which trigger spurious NMI.

Check and disable PEBS for fixed counter after intel_pmu_disable_fixed().

Fixes 6f55967ad9d9 ("perf/x86/intel: Fix race in intel_pmu_disable_event()")
---
  arch/x86/events/intel/core.c | 8 +++-----
  1 file changed, 3 insertions(+), 5 deletions(-)
arch/x86/events/intel/core.c +0 -0
index 4377bf6a6f82..464316218b77 100644
@@ -2160,12 +2160,10 @@ static void intel_pmu_disable_event(struct perf_event *event)
  	cpuc->intel_ctrl_host_mask &= ~(1ull << hwc->idx);
  	cpuc->intel_cp_status &= ~(1ull << hwc->idx);
 

arch/x86/events/intel/core.c static void intel_pmu_disable_event(struct perf_event *event) · line 2167

Quoted 10 lines
arch/x86/events/intel/core.c +3 -5
-	if (unlikely(hwc->config_base == MSR_ARCH_PERFMON_FIXED_CTR_CTRL)) {
+	if (unlikely(hwc->config_base == MSR_ARCH_PERFMON_FIXED_CTR_CTRL))
  		intel_pmu_disable_fixed(hwc);
-		return;
-	}
-
-	x86_pmu_disable_event(event);
+	else
+		x86_pmu_disable_event(event);
 

oops, I overlooed this, looks good

Quoted 5 lines

thanks,
jirka

Hi,

Could this fix be accepted?

Thanks
Jin Yao

Quoted 5 lines
Quoted 5 lines
arch/x86/events/intel/core.c +0 -0
  	/*
  	 * Needs to be called after x86_pmu_disable_event,
Signature · 1 line
2.14.5
maintainer
#3 lore raw
Quoted 31 lines · Yao
Quoted 28 lines · Jiri Olsa
Quoted 27 lines · On Tue, Jun 25, 2019 at 07:21:35AM -0700, [email protected] wrote:

From: Kan Liang <[email protected]>

If a user first sample a PEBS event on a fixed counter, then sample a non-PEBS event on the same fixed counter on Icelake, it will trigger spurious NMI. For example,

perf record -e 'cycles:p' -a
perf record -e 'cycles' -a

The error message for spurious NMI.

[June 21 15:38] Uhhuh. NMI received for unknown reason 30 on CPU 2. [ +0.000000] Do you have a strange power saving mode enabled? [ +0.000000] Dazed and confused, but trying to continue

The issue was introduced by the following commit:

   commit 6f55967ad9d9 ("perf/x86/intel: Fix race in intel_pmu_disable_event()")

The commit moves the intel_pmu_pebs_disable() after intel_pmu_disable_fixed(), which returns immediately.
The related bit of PEBS_ENABLE MSR will never be cleared for the fixed counter. Then a non-PEBS event runs on the fixed counter, but the bit on PEBS_ENABLE is still set, which trigger spurious NMI.

Check and disable PEBS for fixed counter after intel_pmu_disable_fixed().

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Fixes 6f55967ad9d9 ("perf/x86/intel: Fix race in intel_pmu_disable_event()")

oops, I overlooed this, looks good

Have it now, thanks!

#4 lore raw
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

oops, I overlooed this, looks good

Have it now, thanks!

Can Kan's patch please be merged asap and also put into stable for 5.2?

The regression causes crashes on Icelake when fixed counters are used in PEBS groups, and presumably also on Goldmont Plus.

I just had to debug this again until I realized it was the same problem.

-Andi

#5 lore raw
In reply to [email protected]
perf/x86/intel: Fix spurious NMI on fixed counter
Commit-ID:  e4557c1a46b0d32746bd309e1941914b5a6912b4
Gitweb:     https://git.kernel.org/tip/e4557c1a46b0d32746bd309e1941914b5a6912b4
Author:     Kan Liang <[email protected]>
AuthorDate: Tue, 25 Jun 2019 07:21:35 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sat, 13 Jul 2019 11:21:29 +0200

perf/x86/intel: Fix spurious NMI on fixed counter

If a user first sample a PEBS event on a fixed counter, then sample a non-PEBS event on the same fixed counter on Icelake, it will trigger spurious NMI. For example:

perf record -e 'cycles:p' -a
perf record -e 'cycles' -a

The error message for spurious NMI:

[June 21 15:38] Uhhuh. NMI received for unknown reason 30 on CPU 2.
[ +0.000000] Do you have a strange power saving mode enabled?
[ +0.000000] Dazed and confused, but trying to continue

The bug was introduced by the following commit:

commit 6f55967ad9d9 ("perf/x86/intel: Fix race in intel_pmu_disable_event()")

The commit moves the intel_pmu_pebs_disable() after intel_pmu_disable_fixed(), which returns immediately. The related bit of PEBS_ENABLE MSR will never be cleared for the fixed counter. Then a non-PEBS event runs on the fixed counter, but the bit on PEBS_ENABLE is still set, which triggers spurious NMIs.

Check and disable PEBS for fixed counters after intel_pmu_disable_fixed().

Cc Alexander Shishkin <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vince Weaver <[email protected]>
Fixes 6f55967ad9d9 ("perf/x86/intel: Fix race in intel_pmu_disable_event()")
---
 arch/x86/events/intel/core.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
arch/x86/events/intel/core.c +3 -5
index bda450ff51ee..9e911a96972b 100644
@@ -2161,12 +2161,10 @@ static void intel_pmu_disable_event(struct perf_event *event)
 	cpuc->intel_ctrl_host_mask &= ~(1ull << hwc->idx);
 	cpuc->intel_cp_status &= ~(1ull << hwc->idx);
 
-	if (unlikely(hwc->config_base == MSR_ARCH_PERFMON_FIXED_CTR_CTRL)) {
+	if (unlikely(hwc->config_base == MSR_ARCH_PERFMON_FIXED_CTR_CTRL))
 		intel_pmu_disable_fixed(hwc);
-		return;
-	}
-
-	x86_pmu_disable_event(event);
+	else
+		x86_pmu_disable_event(event);
 
 	/*
 	 * Needs to be called after x86_pmu_disable_event,