Skip to contentSkip to conversation

2.6.38-rc2: Uhhuh. NMI received for unknown reason 2d on CPU 0.

DISCUSSION George Spelvin 6 participants 20 messages Feb 1, 2011 – Feb 17, 2011

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Since upgrading to -rc2 (-rc3 is compiling right now), I've been getting complaints at irregular intervals. This didn't used to happen with 2.6.37.

It's an old crappy 1.6 GHz P4 (HP Pavilion) with an ASUS P4B266LA motherboard and a 2001 Award BIOS.

00:00.0 Host bridge [0600]: Intel Corporation 82845 845 [Brookdale] Chipset Host Bridge [8086:1a30] (rev 04) 00:01.0 PCI bridge [0604]: Intel Corporation 82845 845 [Brookdale] Chipset AGP Bridge [8086:1a31] (rev 04) 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 05)
00:1f.0 ISA bridge [0601]: Intel Corporation 82801BA ISA Bridge (LPC) [8086:2440] (rev 05)
00:1f.1 IDE interface [0101]: Intel Corporation 82801BA IDE U100 Controller [8086:244b] (rev 05) 00:1f.2 USB Controller [0c03]: Intel Corporation 82801BA/BAM USB Controller #1 [8086:2442] (rev 05) 00:1f.3 SMBus [0c05]: Intel Corporation 82801BA/BAM SMBus Controller [8086:2443] (rev 05)
00:1f.4 USB Controller [0c03]: Intel Corporation 82801BA/BAM USB Controller #1 [8086:2444] (rev 05) 00:1f.5 Multimedia audio controller [0401]: Intel Corporation 82801BA/BAM AC'97 Audio Controller [8086:2445] (rev 05) 01:00.0 VGA compatible controller [0300]: ATI Technologies Inc Radeon RV250 If [Radeon 9000] [1002:4966] (rev 01) 01:00.1 Display controller [0380]: ATI Technologies Inc Radeon RV250 [Radeon 9000] (Secondary) [1002:496e] (rev 01) 02:08.0 Ethernet controller [0200]: Intel Corporation 82801BA/BAM/CA/CAM Ethernet Controller [8086:2449] (rev 03) 02:09.0 FireWire (IEEE 1394) [0c00]: Texas Instruments TSB12LV26 IEEE-1394 Controller (Link) [104c:8020]

Should I bisect this, or does someone know what might be happening?

Thank you!

Jan 30 13:13:25 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 30 13:13:25 kernel: Do you have a strange power saving mode enabled?
Jan 30 13:13:25 kernel: Dazed and confused, but trying to continue
Jan 30 17:51:10 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 30 17:51:10 kernel: Do you have a strange power saving mode enabled?
Jan 30 17:51:10 kernel: Dazed and confused, but trying to continue
Jan 30 18:05:11 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Jan 30 18:05:11 kernel: Do you have a strange power saving mode enabled?
Jan 30 18:05:11 kernel: Dazed and confused, but trying to continue
Jan 30 18:19:16 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Jan 30 18:19:16 kernel: Do you have a strange power saving mode enabled?
Jan 30 18:19:16 kernel: Dazed and confused, but trying to continue
Jan 30 18:33:33 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 30 18:33:33 kernel: Do you have a strange power saving mode enabled?
Jan 30 18:33:33 kernel: Dazed and confused, but trying to continue
Jan 30 18:48:23 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 30 18:48:23 kernel: Do you have a strange power saving mode enabled?
Jan 30 18:48:23 kernel: Dazed and confused, but trying to continue
Jan 30 21:39:58 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 30 21:39:58 kernel: Do you have a strange power saving mode enabled?
Jan 30 21:39:58 kernel: Dazed and confused, but trying to continue
Jan 30 22:01:46 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 30 22:01:46 kernel: Do you have a strange power saving mode enabled?
Jan 30 22:01:46 kernel: Dazed and confused, but trying to continue
Jan 30 22:03:13 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 30 22:03:13 kernel: Do you have a strange power saving mode enabled?
Jan 30 22:03:13 kernel: Dazed and confused, but trying to continue
Jan 30 22:04:38 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Jan 30 22:04:38 kernel: Do you have a strange power saving mode enabled?
Jan 30 22:04:38 kernel: Dazed and confused, but trying to continue
Jan 30 22:06:03 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 30 22:06:03 kernel: Do you have a strange power saving mode enabled?
Jan 30 22:06:03 kernel: Dazed and confused, but trying to continue
Jan 30 22:07:23 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Jan 30 22:07:23 kernel: Do you have a strange power saving mode enabled?
Jan 30 22:07:23 kernel: Dazed and confused, but trying to continue
Jan 31 01:00:28 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Jan 31 01:00:28 kernel: Do you have a strange power saving mode enabled?
Jan 31 01:00:28 kernel: Dazed and confused, but trying to continue
Jan 31 03:00:02 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 31 03:00:02 kernel: Do you have a strange power saving mode enabled?
Jan 31 03:00:02 kernel: Dazed and confused, but trying to continue
Jan 31 06:27:52 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 31 06:27:52 kernel: Do you have a strange power saving mode enabled?
Jan 31 06:27:52 kernel: Dazed and confused, but trying to continue
Jan 31 07:36:54 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Jan 31 07:36:54 kernel: Do you have a strange power saving mode enabled?
Jan 31 07:36:54 kernel: Dazed and confused, but trying to continue
Jan 31 10:08:08 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 31 10:08:08 kernel: Do you have a strange power saving mode enabled?
Jan 31 10:08:08 kernel: Dazed and confused, but trying to continue
Jan 31 16:42:02 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Jan 31 16:42:02 kernel: Do you have a strange power saving mode enabled?
Jan 31 16:42:02 kernel: Dazed and confused, but trying to continue
Jan 31 20:05:21 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Jan 31 20:05:21 kernel: Do you have a strange power saving mode enabled?
Jan 31 20:05:21 kernel: Dazed and confused, but trying to continue
Feb  1 01:00:19 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 01:00:19 kernel: Do you have a strange power saving mode enabled?
Feb  1 01:00:19 kernel: Dazed and confused, but trying to continue
Feb  1 01:36:42 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 01:36:42 kernel: Do you have a strange power saving mode enabled?
Feb  1 01:36:42 kernel: Dazed and confused, but trying to continue
Feb  1 02:01:04 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 02:01:04 kernel: Do you have a strange power saving mode enabled?
Feb  1 02:01:04 kernel: Dazed and confused, but trying to continue
Feb  1 05:58:05 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 05:58:05 kernel: Do you have a strange power saving mode enabled?
Feb  1 05:58:05 kernel: Dazed and confused, but trying to continue
Feb  1 06:28:18 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 06:28:18 kernel: Do you have a strange power saving mode enabled?
Feb  1 06:28:18 kernel: Dazed and confused, but trying to continue
Feb  1 08:59:18 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 08:59:18 kernel: Do you have a strange power saving mode enabled?
Feb  1 08:59:18 kernel: Dazed and confused, but trying to continue
Feb  1 11:04:43 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Feb  1 11:04:43 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:04:43 kernel: Dazed and confused, but trying to continue
Feb  1 11:05:47 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Feb  1 11:05:47 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:05:47 kernel: Dazed and confused, but trying to continue
Feb  1 11:06:48 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Feb  1 11:06:48 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:06:48 kernel: Dazed and confused, but trying to continue
Feb  1 11:07:50 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 11:07:50 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:07:50 kernel: Dazed and confused, but trying to continue
Feb  1 11:08:52 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Feb  1 11:08:52 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:08:52 kernel: Dazed and confused, but trying to continue
Feb  1 11:09:54 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 11:09:54 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:09:54 kernel: Dazed and confused, but trying to continue
Feb  1 11:10:56 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Feb  1 11:10:56 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:10:56 kernel: Dazed and confused, but trying to continue
Feb  1 11:11:58 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 11:11:58 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:11:58 kernel: Dazed and confused, but trying to continue
Feb  1 11:13:00 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 11:13:00 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:13:00 kernel: Dazed and confused, but trying to continue
Feb  1 11:14:01 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Feb  1 11:14:01 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:14:01 kernel: Dazed and confused, but trying to continue
Feb  1 11:15:04 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 11:15:04 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:15:04 kernel: Dazed and confused, but trying to continue
Feb  1 11:16:05 kernel: Uhhuh. NMI received for unknown reason 3d on CPU 0.
Feb  1 11:16:05 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:16:05 kernel: Dazed and confused, but trying to continue
Feb  1 11:17:07 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 11:17:07 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:17:07 kernel: Dazed and confused, but trying to continue
Feb  1 11:18:33 kernel: Uhhuh. NMI received for unknown reason 2d on CPU 0.
Feb  1 11:18:33 kernel: Do you have a strange power saving mode enabled?
Feb  1 11:18:33 kernel: Dazed and confused, but trying to continue
#1 lore raw
Quoted 3 lines · George Spelvin

Since upgrading to -rc2 (-rc3 is compiling right now), I've been getting complaints at irregular intervals. This didn't used to happen with 2.6.37.

...

Quoted 4 lines

Should I bisect this, or does someone know what might be happening?

Thank you!

I fear it's known issue at moment, we're trying to resolve it. There is
an option -- to disable nmi_watchdog (nmi_watchdog=0 boot option).

But if you have a will or would like to help debug the problem -- mind to
try the patch below? Note the patch is ugly at moment and must *not* be running on non-P4 system (and I only compile-tested it so no guarantees at all, and I've CC'ed a couple of people as well)

Cyrill

---
 arch/x86/kernel/cpu/perf_event.c    |   12 +++++++++++-
 arch/x86/kernel/cpu/perf_event_p4.c |    8 +++++++-
 2 files changed, 18 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/perf_event.c +11 -1
Index: linux-2.6.git/arch/x86/kernel/cpu/perf_event.c
=====================================================================
@@ -1075,7 +1075,17 @@ static void x86_pmu_start(struct perf_ev
 
 	cpuc->events[idx] = event;
 	__set_bit(idx, cpuc->active_mask);
-	__set_bit(idx, cpuc->running);
+	if (1) {
+		/* running mask is shared across a core */
+		int leader_cpu;
+		struct cpu_hw_events *leader_cpuc;
+
+		leader_cpu	= cpumask_first(__get_cpu_var(cpu_sibling_map));
+		leader_cpuc	= &per_cpu(cpu_hw_events, leader_cpu);
+
+		__set_bit(idx, leader_cpuc->running);
+	} else
+		__set_bit(idx, cpuc->running);
 	x86_pmu.enable(event);
 	perf_event_update_userpage(event);
 }
arch/x86/kernel/cpu/perf_event_p4.c +7 -1
Index: linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
=====================================================================
@@ -907,8 +907,14 @@ static int p4_pmu_handle_irq(struct pt_r
 		int overflow;
 
 		if (!test_bit(idx, cpuc->active_mask)) {
+			int leader_cpu;
+			struct cpu_hw_events *leader_cpuc;
+
+			leader_cpu	= cpumask_first(__get_cpu_var(cpu_sibling_map));
+			leader_cpuc	= &per_cpu(cpu_hw_events, leader_cpu);
+
 			/* catch in-flight IRQs */
-			if (__test_and_clear_bit(idx, cpuc->running))
+			if (__test_and_clear_bit(idx, leader_cpuc->running))
 				handled++;
 			continue;
 		}
#2 lore raw
Quoted 10 lines · Cyrill Gorcunov
Quoted 3 lines · George Spelvin

Since upgrading to -rc2 (-rc3 is compiling right now), I've been getting complaints at irregular intervals. This didn't used to happen with 2.6.37.

...

Quoted 4 lines

Should I bisect this, or does someone know what might be happening?

Thank you!

Quoted 7 lines

I fear it's known issue at moment, we're trying to resolve it. There is
an option -- to disable nmi_watchdog (nmi_watchdog=0 boot option).

But if you have a will or would like to help debug the problem -- mind to
try the patch below? Note the patch is ugly at moment and must *not* be running on non-P4 system (and I only compile-tested it so no guarantees at all, and I've CC'ed a couple of people as well)

Unfortunately, I have not had success with patch below on my system. :-(

Cheers,
Don

#3 lore raw

On 02/01/2011 09:41 PM, Don Zickus wrote:
...

Quoted 5 lines

Unfortunately, I have not had success with patch below on my system. :-(

Cheers,
Don

You mean it didn't help?

Signature · 1 line
    Cyrill
#4 lore raw
Quoted 9 lines · Cyrill Gorcunov

On 02/01/2011 09:41 PM, Don Zickus wrote:
...

Quoted 5 lines

Unfortunately, I have not had success with patch below on my system. :-(

Cheers,
Don

You mean it didn't help?

Not that I noticed no.

Cheers,
Don

#5 lore raw

On 02/01/2011 09:51 PM, Don Zickus wrote:
...

Quoted 7 lines
Quoted 2 lines

You mean it didn't help?

Not that I noticed no.

Cheers,
Don

Thanks a huge for testing, Don! I'll check what else I can do.

Signature · 1 line
    Cyrill
#6 lore raw
In reply to Cyrill Gorcunov
Quoted 4 lines

But if you have a will or would like to help debug the problem -- mind to
try the patch below? Note the patch is ugly at moment and must *not* be running on non-P4 system (and I only compile-tested it so no guarantees at all, and I've CC'ed a couple of people as well)

Promising... After 32 minute of uptime, no NMI complaints so far.

I'll let it run overnight and see what happens.

Thank you very much!

#7 lore raw
Quoted 9 lines · George Spelvin
Quoted 4 lines

But if you have a will or would like to help debug the problem -- mind to
try the patch below? Note the patch is ugly at moment and must *not* be running on non-P4 system (and I only compile-tested it so no guarantees at all, and I've CC'ed a couple of people as well)

Promising... After 32 minute of uptime, no NMI complaints so far.

I'll let it run overnight and see what happens.

Great, thanks. Though the patch didn't help for Don, ie there is still an issue which needs to be resolved as well.

#8 lore raw
Quoted 11 lines · Cyrill Gorcunov
Quoted 9 lines · George Spelvin
Quoted 4 lines

 But if you have a will or would like to help debug the problem -- mind to try the patch below? Note the patch is ugly at moment and must *not* be running on non-P4 system (and I only compile-tested it so no guarantees at all, and I've CC'ed a couple of people as well)

Promising...  After 32 minute of uptime, no NMI complaints so far.

I'll let it run overnight and see what happens.

Quoted 8 lines

Great, thanks. Though the patch didn't help for Don, ie there is still an issue which needs to be resolved as well.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Ping on this problem, still seeing

Uhhuh. NMI received for unknown reason 3c on CPU 0. Do you have a strange power saving mode enabled? Dazed and confused, but trying to continue

on my Pentium-D system here with latest Linus head.

its sometimes 3c, sometimes 3d, I'm going to bisect and push for reverts if nobody still has any clue about how to fix this.

Dave.

#9 lore raw
Quoted 28 lines · Dave Airlie
Quoted 12 lines · Cyrill Gorcunov
Quoted 10 lines · George Spelvin
Quoted 5 lines

But if you have a will or would like to help debug the problem -- mind
to
try the patch below? Note the patch is ugly at moment and must *not* be running on non-P4 system (and I only compile-tested it so no guarantees at all, and I've CC'ed a couple of people as well)

Promising... After 32 minute of uptime, no NMI complaints so far.

I'll let it run overnight and see what happens.

Quoted 8 lines

Great, thanks. Though the patch didn't help for Don, ie there is still an issue which needs to be resolved as well.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ping on this problem, still seeing

Uhhuh. NMI received for unknown reason 3c on CPU 0. Do you have a strange power saving mode enabled? Dazed and confused, but trying to continue

Quoted 7 lines

on my Pentium-D system here with latest Linus head.

its sometimes 3c, sometimes 3d, I'm going to bisect and push for reverts if nobody still has any clue about how to fix this.

Dave.

We still trying to resolve it but without success yet. There is no easy way to revert it. One of the option might be to disable perf on p4 for a while. If this is acceptable -- i'll cook such patch and send it to Ingo. Hm?

#10 lore raw
Quoted 34 lines · Cyrill Gorcunov
Quoted 33 lines · Dave Airlie
Quoted 12 lines · Cyrill Gorcunov
Quoted 10 lines · George Spelvin
Quoted 5 lines

But if you have a will or would like to help debug the problem -- mind
to
try the patch below? Note the patch is ugly at moment and must *not* be running on non-P4 system (and I only compile-tested it so no guarantees at all, and I've CC'ed a couple of people as well)

Promising... After 32 minute of uptime, no NMI complaints so far.

I'll let it run overnight and see what happens.

Quoted 8 lines

Great, thanks. Though the patch didn't help for Don, ie there is still an issue which needs to be resolved as well.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ping on this problem, still seeing

Uhhuh. NMI received for unknown reason 3c on CPU 0. Do you have a strange power saving mode enabled? Dazed and confused, but trying to continue

on my Pentium-D system here with latest Linus head.

its sometimes 3c, sometimes 3d, I'm going to bisect and push for reverts if nobody still has any clue about how to fix this.

Quoted 7 lines
Quoted 2 lines

Dave.

We still trying to resolve it but without success yet. There is no easy way to revert it. One of the option might be to disable perf on p4 for a while. If this is acceptable -- i'll cook such patch and send it to Ingo. Hm?

That's not really acceptable - need to fix it or revert it to the last working state. Which commit broke it?

Thanks,

Ingo

#11 lore raw

On Wed, Feb 16, 2011 at 11:37 AM, Ingo Molnar <[email protected]> wrote:
...

Quoted 21 lines
Quoted 13 lines
Quoted 13 lines
Quoted 1 line

Ping on this problem, still seeing

Uhhuh. NMI received for unknown reason 3c on CPU 0. Do you have a strange power saving mode enabled? Dazed and confused, but trying to continue

on my Pentium-D system here with latest Linus head.

its sometimes 3c, sometimes 3d, I'm going to bisect and push for reverts if nobody still has any clue about how to fix this.

Quoted 7 lines
Quoted 2 lines

Dave.

We still trying to resolve it but without success yet. There is no easy way to revert it. One of the option might be to disable perf on p4 for a while. If this is acceptable -- i'll cook such patch and send it to Ingo. Hm?

Quoted 7 lines

That's not really acceptable - need to fix it or revert it to the last working state. Which commit broke it?

Thanks,

       Ingo

I can't say you the commit id after which unknown-nmi start happening (i'm out of git tree
at moment) but even then this commit should not be reverted since the problem is in
p4 code not in the rest of perf system.

I have two patches here (attached) and would really appreciate of their testing on HT machine
together with kgdb bootup tests enabled. Dave could you please?

#12 lore raw
Quoted 37 lines · Cyrill Gorcunov

On Wed, Feb 16, 2011 at 11:37 AM, Ingo Molnar <[email protected]> wrote:
...

Quoted 21 lines
Quoted 13 lines
Quoted 13 lines
Quoted 1 line

Ping on this problem, still seeing

Uhhuh. NMI received for unknown reason 3c on CPU 0. Do you have a strange power saving mode enabled? Dazed and confused, but trying to continue

on my Pentium-D system here with latest Linus head.

its sometimes 3c, sometimes 3d, I'm going to bisect and push for reverts if nobody still has any clue about how to fix this.

Quoted 7 lines
Quoted 2 lines

Dave.

We still trying to resolve it but without success yet. There is no easy way to revert it. One of the option might be to disable perf on p4 for a while. If this is acceptable -- i'll cook such patch and send it to Ingo. Hm?

Quoted 7 lines

That's not really acceptable - need to fix it or revert it to the last working state. Which commit broke it?

Thanks,

       Ingo

I can't say you the commit id after which unknown-nmi start happening (i'm out of git tree
at moment) but even then this commit should not be reverted since the problem is in
p4 code not in the rest of perf system.

Quoted 3 lines

I have two patches here (attached) and would really appreciate of their testing on HT machine
together with kgdb bootup tests enabled. Dave could you please?

Could these patches fix Dave's non-kgdb problem? Dave isnt using kgdb but is probably using perf which triggers NMIs? Dave, can you confirm that?

And it's a spurious NMI message, not actual lockup or other misbehavior, right?

Thanks,

Ingo

#13 lore raw
Quoted 48 lines · Ingo Molnar
Quoted 38 lines · Cyrill Gorcunov

On Wed, Feb 16, 2011 at 11:37 AM, Ingo Molnar <[email protected]> wrote:
...

Quoted 21 lines
Quoted 13 lines
Quoted 13 lines
Quoted 1 line

Ping on this problem, still seeing

Uhhuh. NMI received for unknown reason 3c on CPU 0. Do you have a strange power saving mode enabled? Dazed and confused, but trying to continue

on my Pentium-D system here with latest Linus head.

its sometimes 3c, sometimes 3d, I'm going to bisect and push for reverts if nobody still has any clue about how to fix this.

Quoted 7 lines
Quoted 2 lines

Dave.

We still trying to resolve it but without success yet. There is no easy way to revert it. One of the option might be to disable perf on p4 for a while. If this is acceptable -- i'll cook such patch and send it to Ingo. Hm?

Quoted 8 lines

That's not really acceptable - need to fix it or revert it to the last working
state. Which commit broke it?

Thanks,

Ingo

I can't say you the commit id after which unknown-nmi start happening (i'm out of git tree
at moment) but even then this commit should not be reverted since the problem is in
p4 code not in the rest of perf system.

Quoted 3 lines

I have two patches here (attached) and would really appreciate of their testing on HT machine
together with kgdb bootup tests enabled. Dave could you please?

Could these patches fix Dave's non-kgdb problem? Dave isnt using kgdb but is probably using perf which triggers NMIs? Dave, can you confirm that?

Quoted 7 lines

And it's a spurious NMI message, not actual lockup or other misbehavior, right?

Thanks,

Ingo

For nonkgdb case 'unflagged nmi fix' patch should be enough. i've tested it on non-ht machine by self. without it there is no lockup but only a message about unknown nmi.

for hr-machine with kgdb the things go harder, Don reported lockup on boot. The second patch might help but i cant test it (here i need help in testing)

#14 lore raw
Quoted 52 lines · Cyrill Gorcunov
Quoted 51 lines · Ingo Molnar
Quoted 38 lines · Cyrill Gorcunov

On Wed, Feb 16, 2011 at 11:37 AM, Ingo Molnar <[email protected]> wrote:
...

Quoted 21 lines
Quoted 13 lines
Quoted 13 lines
Quoted 1 line

Ping on this problem, still seeing

Uhhuh. NMI received for unknown reason 3c on CPU 0. Do you have a strange power saving mode enabled? Dazed and confused, but trying to continue

on my Pentium-D system here with latest Linus head.

its sometimes 3c, sometimes 3d, I'm going to bisect and push for reverts if nobody still has any clue about how to fix this.

Quoted 7 lines
Quoted 2 lines

Dave.

We still trying to resolve it but without success yet. There is no easy way to revert it. One of the option might be to disable perf on p4 for a while. If this is acceptable -- i'll cook such patch and send it to Ingo. Hm?

Quoted 8 lines

That's not really acceptable - need to fix it or revert it to the last working
state. Which commit broke it?

Thanks,

Ingo

I can't say you the commit id after which unknown-nmi start happening (i'm out of git tree
at moment) but even then this commit should not be reverted since the problem is in
p4 code not in the rest of perf system.

Quoted 3 lines

I have two patches here (attached) and would really appreciate of their testing on HT machine
together with kgdb bootup tests enabled. Dave could you please?

Could these patches fix Dave's non-kgdb problem? Dave isnt using kgdb but is probably using perf which triggers NMIs? Dave, can you confirm that?

And it's a spurious NMI message, not actual lockup or other misbehavior, right?

Quoted 8 lines
Quoted 4 lines

Thanks,

Ingo

For nonkgdb case 'unflagged nmi fix' patch should be enough. i've tested it on non-ht machine by self. without it there is no lockup but only a message about unknown nmi.

Ok, please submit it ASAP then - that ought to address the regression. Please Cc:
Dave to the patch.

Thanks,

Ingo

#15 lore raw

On Wed, Feb 16, 2011 at 1:09 PM, Ingo Molnar <[email protected]> wrote:
...

Quoted 5 lines
Quoted 4 lines

For nonkgdb case 'unflagged nmi fix' patch should be enough. i've tested it on non-ht machine by self. without it there is no lockup but only a message about unknown nmi.

Quoted 7 lines

Ok, please submit it ASAP then - that ought to address the regression. Please Cc:
Dave to the patch.

Thanks,

       Ingo

Ingo both patches are already in thread (attached to previous mail since I've web
access at moment). So just to be sure the 'unflagged' nmi patch is attached again,
this fix which should be aplied to -tip/master.

#16 lore raw

[tip:perf/urgent] perf, x86: P4 PMU: Fix spurious NMI messages

Commit-ID:  7d44ec193d95416d1342cdd86392a1eeb7461186
Gitweb:     http://git.kernel.org/tip/7d44ec193d95416d1342cdd86392a1eeb7461186
Author:     Cyrill Gorcunov <[email protected]>
AuthorDate: Wed, 16 Feb 2011 14:08:02 +0300
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 16 Feb 2011 12:26:12 +0100

perf, x86: P4 PMU: Fix spurious NMI messages

Several people have reported spurious unknown NMI messages on some P4 CPUs.

This patch fixes it by checking for an overflow (negative counter values) directly, instead of relying on the P4_CCCR_OVF bit.

LKML-Reference: <[email protected]>

---
 arch/x86/include/asm/perf_event_p4.h |    1 +
 arch/x86/kernel/cpu/perf_event_p4.c  |   11 ++++++++---
 2 files changed, 9 insertions(+), 3 deletions(-)
arch/x86/include/asm/perf_event_p4.h +1 -0
index e2f6a99..cc29086 100644
@@ -22,6 +22,7 @@
 
 #define ARCH_P4_CNTRVAL_BITS	(40)
 #define ARCH_P4_CNTRVAL_MASK	((1ULL << ARCH_P4_CNTRVAL_BITS) - 1)
+#define ARCH_P4_UNFLAGGED_BIT	((1ULL) << (ARCH_P4_CNTRVAL_BITS - 1))
 
 #define P4_ESCR_EVENT_MASK	0x7e000000U
 #define P4_ESCR_EVENT_SHIFT	25
arch/x86/kernel/cpu/perf_event_p4.c +8 -3
index f7a0993..ff751a9 100644
@@ -770,9 +770,14 @@ static inline int p4_pmu_clear_cccr_ovf(struct hw_perf_event *hwc)
 		return 1;
 	}
 
-	/* it might be unflagged overflow */
-	rdmsrl(hwc->event_base + hwc->idx, v);
-	if (!(v & ARCH_P4_CNTRVAL_MASK))
+	/*
+	 * In some circumstances the overflow might issue an NMI but did
+	 * not set P4_CCCR_OVF bit. Because a counter holds a negative value
+	 * we simply check for high bit being set, if it's cleared it means
+	 * the counter has reached zero value and continued counting before
+	 * real NMI signal was received:
+	 */
+	if (!(v & ARCH_P4_UNFLAGGED_BIT))
 		return 1;
 
 	return 0;
#17 lore raw
In reply to Dave Airlie
Quoted 10 lines

Ping on this problem, still seeing

Uhhuh. NMI received for unknown reason 3c on CPU 0. Do you have a strange power saving mode enabled? Dazed and confused, but trying to continue

on my Pentium-D system here with latest Linus head.

its sometimes 3c, sometimes 3d, I'm going to bisect and push for reverts if nobody still has any clue about how to fix this.

The second patch (not the one you quote) fixed it for me. Almost 8 days of uptime and no log spam.

It's appended below for your convenience. Are you using this unsuccessfully?

From: Cyrill Gorcunov <[email protected]>
Subject: [PATCH] perf, x86: P4 PMU -- Fix unflagged overflows test

A couple of people have reported an unknown NMI issue on p4 pmu. This patch should fix it.

---
 arch/x86/include/asm/perf_event_p4.h |    1 +
 arch/x86/kernel/cpu/perf_event_p4.c  |   11 ++++++++---
 2 files changed, 9 insertions(+), 3 deletions(-)
arch/x86/include/asm/perf_event_p4.h +1 -0
Index: linux-2.6.tip/arch/x86/include/asm/perf_event_p4.h
===================================================================
@@ -22,6 +22,7 @@
 
 #define ARCH_P4_CNTRVAL_BITS	(40)
 #define ARCH_P4_CNTRVAL_MASK	((1ULL << ARCH_P4_CNTRVAL_BITS) - 1)
+#define ARCH_P4_UNFLAGGED_BIT	((1ULL) << (ARCH_P4_CNTRVAL_BITS - 1))
 
 #define P4_ESCR_EVENT_MASK	0x7e000000U
 #define P4_ESCR_EVENT_SHIFT	25
arch/x86/kernel/cpu/perf_event_p4.c +8 -3
Index: linux-2.6.tip/arch/x86/kernel/cpu/perf_event_p4.c
===================================================================
@@ -770,9 +770,14 @@ static inline int p4_pmu_clear_cccr_ovf(
 		return 1;
 	}
 
-	/* it might be unflagged overflow */
-	rdmsrl(hwc->event_base + hwc->idx, v);
-	if (!(v & ARCH_P4_CNTRVAL_MASK))
+	/*
+	 * at some circumstances the overflow might issue NMI but did
+	 * not set P4_CCCR_OVF bit so since a counter holds a negative value
+	 * we simply check for high bit being set, if it's cleared it means
+	 * the counter has reached zero value and continued counting before
+	 * real NMI signal was received
+	 */
+	if (!(v & ARCH_P4_UNFLAGGED_BIT))
 		return 1;
 
 	return 0;
Tested-by
#18 lore raw
Quoted 3 lines

It's appended below for your convenience.  Are you using this unsuccessfully?

This patch quoted below fixes it for me.

No more spurious NMIs on my P4.

Quoted 56 lines

From: Cyrill Gorcunov <[email protected]>
Subject: [PATCH] perf, x86: P4 PMU -- Fix unflagged overflows test

A couple of people have reported an unknown NMI issue on p4 pmu. This patch should fix it.

---
 arch/x86/include/asm/perf_event_p4.h |    1 + arch/x86/kernel/cpu/perf_event_p4.c  |   11 ++++++++---

 2 files changed, 9 insertions(+), 3 deletions(-)

Index: linux-2.6.tip/arch/x86/include/asm/perf_event_p4.h
===================================================================

arch/x86/include/asm/perf_event_p4.h +1 -0
@@ -22,6 +22,7 @@
 
  #define ARCH_P4_CNTRVAL_BITS   (40)
  #define ARCH_P4_CNTRVAL_MASK   ((1ULL << ARCH_P4_CNTRVAL_BITS) - 1)
+#define ARCH_P4_UNFLAGGED_BIT  ((1ULL) << (ARCH_P4_CNTRVAL_BITS - 1))
 
  #define P4_ESCR_EVENT_MASK     0x7e000000U
  #define P4_ESCR_EVENT_SHIFT    25

Index: linux-2.6.tip/arch/x86/kernel/cpu/perf_event_p4.c
===================================================================

arch/x86/kernel/cpu/perf_event_p4.c +8 -3
@@ -770,9 +770,14 @@ static inline int p4_pmu_clear_cccr_ovf(
                return 1;
        }
 
-       /* it might be unflagged overflow */
-       rdmsrl(hwc->event_base + hwc->idx, v);
-       if (!(v & ARCH_P4_CNTRVAL_MASK))
+       /*
+        * at some circumstances the overflow might issue NMI but did
+        * not set P4_CCCR_OVF bit so since a counter holds a negative value
+        * we simply check for high bit being set, if it's cleared it means
+        * the counter has reached zero value and continued counting before
+        * real NMI signal was received
+        */
+       if (!(v & ARCH_P4_UNFLAGGED_BIT))
                return 1;
 
        return 0;
#19 lore raw
Quoted 10 lines · Dave Airlie
Quoted 3 lines

It's appended below for your convenience.  Are you using this unsuccessfully?

This patch quoted below fixes it for me.

No more spurious NMIs on my P4.

Thanks Dave! Ingo has merged it into -urgent branch, so it should reach mainline soon.