Skip to contentSkip to conversation

NMI reason 2d when running perf

DISCUSSION Jens Axboe 3 participants 3 messages Mar 17, 2023 – Mar 20, 2023

lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

Hi,

When running perf on my Dell R7525 on a running process, I get a ton of:

[  504.234782] Dazed and confused, but trying to continue
[  504.267843] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.267846] Dazed and confused, but trying to continue
[  504.335975] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.335977] Dazed and confused, but trying to continue
[  504.368031] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.368033] Dazed and confused, but trying to continue
[  504.371037] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.371038] Dazed and confused, but trying to continue
[  504.439165] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.439167] Dazed and confused, but trying to continue

spew in dmesg. The box has 2x7763 CPUS. This seems to be a recent regression, been using this box for a while and haven't seen this before. The test being traced is pinned to CPU 48. The box is currently running:

commit 6015b1aca1a233379625385feb01dd014aca60b5 (origin/master, origin/HEAD)

Author: Linus Torvalds <[email protected]>
Date:   Tue Mar 14 19:32:38 2023 -0700

sched_getaffinity: don't assume 'cpumask_size()' is fully initialized

with the pending block/io_uring branches merged in for testing.

Signature · 1 line
Jens Axboe
#1 lore raw
Quoted 16 lines · * Jens Axboe ([email protected]) wrote:

Hi,

When running perf on my Dell R7525 on a running process, I get a ton of:

[  504.234782] Dazed and confused, but trying to continue
[  504.267843] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.267846] Dazed and confused, but trying to continue
[  504.335975] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.335977] Dazed and confused, but trying to continue
[  504.368031] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.368033] Dazed and confused, but trying to continue
[  504.371037] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.371038] Dazed and confused, but trying to continue
[  504.439165] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.439167] Dazed and confused, but trying to continue
Quoted 4 lines

spew in dmesg. The box has 2x7763 CPUS. This seems to be a recent regression, been using this box for a while and haven't seen this before. The test being traced is pinned to CPU 48. The box is currently running:

I'm not sure this is new; I've seen this intermittently on AMD boxes for a few years. Always when running perf or related tools.

Dave

Quoted 11 lines

commit 6015b1aca1a233379625385feb01dd014aca60b5 (origin/master, origin/HEAD)

Author: Linus Torvalds <[email protected]>
Date:   Tue Mar 14 19:32:38 2023 -0700

sched_getaffinity: don't assume 'cpumask_size()' is fully initialized

with the pending block/io_uring branches merged in for testing.

Signature · 1 line
Jens Axboe
Signature · 4 lines
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    |       Running GNU/Linux       | Happy  \ 
\        dave @ treblig.org |                               | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/
#2 lore raw
In reply to Jens Axboe
Quoted 16 lines · Jens Axboe

Hi,

When running perf on my Dell R7525 on a running process, I get a ton of:

[  504.234782] Dazed and confused, but trying to continue
[  504.267843] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.267846] Dazed and confused, but trying to continue
[  504.335975] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.335977] Dazed and confused, but trying to continue
[  504.368031] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.368033] Dazed and confused, but trying to continue
[  504.371037] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.371038] Dazed and confused, but trying to continue
[  504.439165] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.439167] Dazed and confused, but trying to continue
Quoted 1 line

spew in dmesg. The box has 2x7763 CPUS.

Are you using IBS directly or indirectly (perf record, if run as root without explicit -e event, internally invokes IBS)? fwiw, Zen2 had similar hw issue where IBS can raise NMI without sample valid bit set while exiting from CC6 state[1]. But as per my knowledge, it's already fixed in Zen3.

[1]: https://lore.kernel.org/all/YFDSSxftYw9tCGC6@krava/

Quoted 13 lines

This seems to be a recent
regression, been using this box for a while and haven't seen this before. The test being traced is pinned to CPU 48. The box is currently running:

commit 6015b1aca1a233379625385feb01dd014aca60b5 (origin/master, origin/HEAD)

Author: Linus Torvalds <[email protected]>
Date:   Tue Mar 14 19:32:38 2023 -0700

sched_getaffinity: don't assume 'cpumask_size()' is fully initialized

with the pending block/io_uring branches merged in for testing.