Skip to contentSkip to conversation

nvme: machine check when running nvme subsystem-reset /dev/nvme0 against direct attach via PCIE slot

DISCUSSION Laurence Oberman 3 participants 8 messages Sep 26, 2024 – Oct 29, 2024 NVMe

lore
Share & tools
lore
#0 lore raw

Hi Keith
Hope all is well

Quick question (expected or not)

It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset /dev/nvme0" command to test resets.

On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices

1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)

Example of one working

Does not trap and land up as a machine-check

[ 2215.440468] pcieport 0000:10:01.1: AER: Multiple Uncorrected (Non-
Fatal) error received: 0000:12:13.0
[ 2215.440532] pcieport 0000:12:13.0: PCIe Bus Error:
severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester
ID)
[ 2215.440536] pcieport 0000:12:13.0:   device [10b5:8748] error
status/mask=00100000/00000000
[ 2215.440540] pcieport 0000:12:13.0:    [20] UnsupReq              
(First)
[ 2215.440544] pcieport 0000:12:13.0: AER:   TLP Header: 40009001
1000000f e9211000 12000000
[ 2215.441813] systemd-journald[2173]: Sent WATCHDOG=1 notification.
[ 2216.937498] {1}[Hardware Error]: Hardware error from APEI Generic
Hardware Error Source: 4
[ 2216.937505] {1}[Hardware Error]: event severity: info
[ 2216.937508] {1}[Hardware Error]:  Error 0, type: fatal
[ 2216.937511] {1}[Hardware Error]:  fru_text: PcieError
[ 2216.937514] {1}[Hardware Error]:   section_type: PCIe error
[ 2216.937515] {1}[Hardware Error]:   port_type: 4, root port
[ 2216.937517] {1}[Hardware Error]:   version: 0.2
[ 2216.937519] {1}[Hardware Error]:   command: 0x0407, status: 0x0010
[ 2216.937522] {1}[Hardware Error]:   device_id: 0000:10:01.1
[ 2216.937524] {1}[Hardware Error]:   slot: 3
[ 2216.937525] {1}[Hardware Error]:   secondary_bus: 0x11
[ 2216.937526] {1}[Hardware Error]:   vendor_id: 0x1022, device_id:
0x1453
[ 2216.937528] {1}[Hardware Error]:   class_code: 060400
[ 2216.937529] {1}[Hardware Error]:   bridge: secondary_status: 0x2000,
control: 0x0012
[ 2216.937530] {1}[Hardware Error]:   aer_uncor_status: 0x00000000,
aer_uncor_mask: 0x04500000
[ 2216.937532] {1}[Hardware Error]:   aer_uncor_severity: 0x004e2030
[ 2216.937532] {1}[Hardware Error]:   TLP Header: 00000000 00000000
00000000 00000000
[ 2216.937629] pcieport 0000:10:01.1: AER: aer_status: 0x00000000,
aer_mask: 0x04500000
[ 2216.937634] pcieport 0000:10:01.1: AER: aer_layer=Transaction Layer,
aer_agent=Receiver ID
[ 2216.937638] pcieport 0000:10:01.1: AER: aer_uncor_severity:
0x004e2030
[ 2216.937645] nvme nvme4: frozen state error detected, reset
controller
[ 2217.071095] nvme nvme10: frozen state error detected, reset
controller
[ 2217.096928] nvme nvme0: frozen state error detected, reset
controller
[ 2217.118947] nvme nvme18: frozen state error detected, reset
controller
[ 2217.138945] nvme nvme6: frozen state error detected, reset
controller
[ 2217.164918] nvme nvme14: frozen state error detected, reset
controller
[ 2217.186902] nvme nvme20: frozen state error detected, reset
controller
[ 2279.420266] nvme 0000:1a:00.0: Unable to change power state from
D3cold to D0, device inaccessible
[ 2279.420329] nvme nvme22: Disabling device after reset failure: -19
[ 2279.464727] pcieport 0000:12:13.0: AER: device recovery failed
[ 2279.464823] pcieport 0000:12:13.0: pciehp: pcie_do_write_cmd: no

response from device

Port resets and recovers

[ 2279.593196] pcieport 0000:10:01.1: AER: Root Port link has been
reset (0)
[ 2279.593699] nvme nvme4: restart after slot reset
[ 2279.593949] nvme nvme10: restart after slot reset
[ 2279.594222] nvme nvme0: restart after slot reset
[ 2279.594453] nvme nvme18: restart after slot reset
[ 2279.594728] nvme nvme6: restart after slot reset
[ 2279.594984] nvme nvme14: restart after slot reset
[ 2279.595226] nvme nvme20: restart after slot reset
[ 2279.595435] pcieport 0000:12:13.0: pciehp: Slot(19): Card present
[ 2279.595441] pcieport 0000:12:13.0: pciehp: Slot(19): Link Up
[ 2279.609081] nvme nvme4: Shutdown timeout set to 8 seconds
[ 2279.617532] nvme nvme0: Shutdown timeout set to 8 seconds
[ 2279.617533] nvme nvme14: Shutdown timeout set to 8 seconds
[ 2279.618028] nvme nvme6: Shutdown timeout set to 8 seconds
[ 2279.618207] nvme nvme18: Shutdown timeout set to 8 seconds
[ 2279.618290] nvme nvme10: Shutdown timeout set to 8 seconds
[ 2279.618308] nvme nvme20: Shutdown timeout set to 8 seconds
[ 2279.631961] nvme nvme4: 32/0/0 default/read/poll queues
[ 2279.643293] nvme nvme14: 32/0/0 default/read/poll queues
[ 2279.643372] nvme nvme0: 32/0/0 default/read/poll queues
[ 2279.644881] nvme nvme6: 32/0/0 default/read/poll queues
[ 2279.644966] nvme nvme10: 32/0/0 default/read/poll queues
[ 2279.645030] nvme nvme18: 32/0/0 default/read/poll queues
[ 2279.645132] nvme nvme20: 32/0/0 default/read/poll queues
[ 2279.645202] pcieport 0000:10:01.1: AER: device recovery successful

2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot

263.862919] mce: [Hardware Error]: CPU 12: Machine Check Exception: 5
Bank 6: ba00000000000e0b

[  263.862924] mce: [Hardware Error]: RIP !INEXACT!
10:<ffffffff8571dce4> {intel_idle+0x54/0x90}
[  263.862931] mce: [Hardware Error]: TSC 7a47d8d62ba6dd MISC 83100000 
[  263.862933] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1727384194
SOCKET 1 APIC 40 microcode d0003a5
[  263.862936] mce: [Hardware Error]: Run the above through 'mcelog --
ascii'
[  263.885254] mce: [Hardware Error]: Machine check: Processor context
corrupt
[  263.885259] Kernel panic - not syncing: Fatal machine check

Hardware event. This is not a software error.
CPU 0 BANK 0 TSC 7a47d8d62ba6dd
RIP !INEXACT! 10:ffffffff8571dce4
TIME 1727384194 Thu Sep 26 16:56:34 2024
MCG status:
MCi status:
Machine check not valid
Corrected error
MCA: No Error
STATUS 0 MCGSTATUS 0
CPUID Vendor Intel Family 6 Model 106 Step 6

RIP: intel_idle+0x54/0x90}

SOCKET 1 APIC 40 microcode d0003a5
Run the above through 'mcelog --ascii'
Machine check: Processor context corrupt

Regards
Laurence

#1 lore raw
Quoted 136 lines · Laurence Oberman

Hi Keith
Hope all is well

Quick question (expected or not)

It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset /dev/nvme0" command to test resets.

On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices

1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)

Example of one working

Does not trap and land up as a machine-check

[ 2215.440468] pcieport 0000:10:01.1: AER: Multiple Uncorrected (Non-
Fatal) error received: 0000:12:13.0
[ 2215.440532] pcieport 0000:12:13.0: PCIe Bus Error:
severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester
ID)
[ 2215.440536] pcieport 0000:12:13.0:   device [10b5:8748] error
status/mask=00100000/00000000
[ 2215.440540] pcieport 0000:12:13.0:    [20] UnsupReq              
(First)
[ 2215.440544] pcieport 0000:12:13.0: AER:   TLP Header: 40009001
1000000f e9211000 12000000
[ 2215.441813] systemd-journald[2173]: Sent WATCHDOG=1 notification.
[ 2216.937498] {1}[Hardware Error]: Hardware error from APEI Generic
Hardware Error Source: 4
[ 2216.937505] {1}[Hardware Error]: event severity: info
[ 2216.937508] {1}[Hardware Error]:  Error 0, type: fatal
[ 2216.937511] {1}[Hardware Error]:  fru_text: PcieError
[ 2216.937514] {1}[Hardware Error]:   section_type: PCIe error
[ 2216.937515] {1}[Hardware Error]:   port_type: 4, root port
[ 2216.937517] {1}[Hardware Error]:   version: 0.2
[ 2216.937519] {1}[Hardware Error]:   command: 0x0407, status: 0x0010
[ 2216.937522] {1}[Hardware Error]:   device_id: 0000:10:01.1
[ 2216.937524] {1}[Hardware Error]:   slot: 3
[ 2216.937525] {1}[Hardware Error]:   secondary_bus: 0x11
[ 2216.937526] {1}[Hardware Error]:   vendor_id: 0x1022, device_id:
0x1453
[ 2216.937528] {1}[Hardware Error]:   class_code: 060400
[ 2216.937529] {1}[Hardware Error]:   bridge: secondary_status: 0x2000,
control: 0x0012
[ 2216.937530] {1}[Hardware Error]:   aer_uncor_status: 0x00000000,
aer_uncor_mask: 0x04500000
[ 2216.937532] {1}[Hardware Error]:   aer_uncor_severity: 0x004e2030
[ 2216.937532] {1}[Hardware Error]:   TLP Header: 00000000 00000000
00000000 00000000
[ 2216.937629] pcieport 0000:10:01.1: AER: aer_status: 0x00000000,
aer_mask: 0x04500000
[ 2216.937634] pcieport 0000:10:01.1: AER: aer_layer=Transaction Layer,
aer_agent=Receiver ID
[ 2216.937638] pcieport 0000:10:01.1: AER: aer_uncor_severity:
0x004e2030
[ 2216.937645] nvme nvme4: frozen state error detected, reset
controller
[ 2217.071095] nvme nvme10: frozen state error detected, reset
controller
[ 2217.096928] nvme nvme0: frozen state error detected, reset
controller
[ 2217.118947] nvme nvme18: frozen state error detected, reset
controller
[ 2217.138945] nvme nvme6: frozen state error detected, reset
controller
[ 2217.164918] nvme nvme14: frozen state error detected, reset
controller
[ 2217.186902] nvme nvme20: frozen state error detected, reset
controller
[ 2279.420266] nvme 0000:1a:00.0: Unable to change power state from
D3cold to D0, device inaccessible
[ 2279.420329] nvme nvme22: Disabling device after reset failure: -19
[ 2279.464727] pcieport 0000:12:13.0: AER: device recovery failed
[ 2279.464823] pcieport 0000:12:13.0: pciehp: pcie_do_write_cmd: no

response from device

Port resets and recovers

[ 2279.593196] pcieport 0000:10:01.1: AER: Root Port link has been
reset (0)
[ 2279.593699] nvme nvme4: restart after slot reset
[ 2279.593949] nvme nvme10: restart after slot reset
[ 2279.594222] nvme nvme0: restart after slot reset
[ 2279.594453] nvme nvme18: restart after slot reset
[ 2279.594728] nvme nvme6: restart after slot reset
[ 2279.594984] nvme nvme14: restart after slot reset
[ 2279.595226] nvme nvme20: restart after slot reset
[ 2279.595435] pcieport 0000:12:13.0: pciehp: Slot(19): Card present
[ 2279.595441] pcieport 0000:12:13.0: pciehp: Slot(19): Link Up
[ 2279.609081] nvme nvme4: Shutdown timeout set to 8 seconds
[ 2279.617532] nvme nvme0: Shutdown timeout set to 8 seconds
[ 2279.617533] nvme nvme14: Shutdown timeout set to 8 seconds
[ 2279.618028] nvme nvme6: Shutdown timeout set to 8 seconds
[ 2279.618207] nvme nvme18: Shutdown timeout set to 8 seconds
[ 2279.618290] nvme nvme10: Shutdown timeout set to 8 seconds
[ 2279.618308] nvme nvme20: Shutdown timeout set to 8 seconds
[ 2279.631961] nvme nvme4: 32/0/0 default/read/poll queues
[ 2279.643293] nvme nvme14: 32/0/0 default/read/poll queues
[ 2279.643372] nvme nvme0: 32/0/0 default/read/poll queues
[ 2279.644881] nvme nvme6: 32/0/0 default/read/poll queues
[ 2279.644966] nvme nvme10: 32/0/0 default/read/poll queues
[ 2279.645030] nvme nvme18: 32/0/0 default/read/poll queues
[ 2279.645132] nvme nvme20: 32/0/0 default/read/poll queues
[ 2279.645202] pcieport 0000:10:01.1: AER: device recovery successful

2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot

263.862919] mce: [Hardware Error]: CPU 12: Machine Check Exception: 5
Bank 6: ba00000000000e0b

[  263.862924] mce: [Hardware Error]: RIP !INEXACT!
10:<ffffffff8571dce4> {intel_idle+0x54/0x90}
[  263.862931] mce: [Hardware Error]: TSC 7a47d8d62ba6dd MISC 83100000 
[  263.862933] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1727384194
SOCKET 1 APIC 40 microcode d0003a5
[  263.862936] mce: [Hardware Error]: Run the above through 'mcelog --
ascii'
[  263.885254] mce: [Hardware Error]: Machine check: Processor context
corrupt
[  263.885259] Kernel panic - not syncing: Fatal machine check

Hardware event. This is not a software error.
CPU 0 BANK 0 TSC 7a47d8d62ba6dd
RIP !INEXACT! 10:ffffffff8571dce4
TIME 1727384194 Thu Sep 26 16:56:34 2024
MCG status:
MCi status:
Machine check not valid
Corrected error
MCA: No Error
STATUS 0 MCGSTATUS 0
CPUID Vendor Intel Family 6 Model 106 Step 6

Quoted 10 lines
RIP: intel_idle+0x54/0x90}

SOCKET 1 APIC 40 microcode d0003a5
Run the above through 'mcelog --ascii'
Machine check: Processor context corrupt

Regards
Laurence

I think the Keith's email address is not correct. Adding the correct email address of Keith here.

BTW, Keith recently help fixed an issue in kernel v6.11 with nvme subsystem-reset command to ensure that we recover the nvme disk on PPC. On PPC architecture, we use EEH to recover the disk post subsystem-reset but yours is Intel and that uses AER for recovery. So I'm not sure if that same commit 210b1f6576e8("nvme-pci: do not directly handle subsys reset fallout") which was merged in kernel v6.11 causing a side effect on the Intel machine.

Would you please revert the above commit and see if that help fix the observed symptom on your Intel machine?

Thanks,
--Nilay

#2 lore raw
Quoted 177 lines · Nilay Shroff
Quoted 151 lines · Laurence Oberman

Hi Keith
Hope all is well

Quick question (expected or not)

It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset /dev/nvme0" command to test resets.

On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices

1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)

Example of one working

Does not trap and land up as a machine-check

[ 2215.440468] pcieport 0000:10:01.1: AER: Multiple Uncorrected
(Non-
Fatal) error received: 0000:12:13.0
[ 2215.440532] pcieport 0000:12:13.0: PCIe Bus Error:
severity=Uncorrected (Non-Fatal), type=Transaction Layer,
(Requester
ID)
[ 2215.440536] pcieport 0000:12:13.0:   device [10b5:8748] error
status/mask=00100000/00000000
(First)
[ 2215.440544] pcieport 0000:12:13.0: AER:   TLP Header: 40009001
1000000f e9211000 12000000
[ 2215.441813] systemd-journald[2173]: Sent WATCHDOG=1
notification.
[ 2216.937498] {1}[Hardware Error]: Hardware error from APEI
Generic
Hardware Error Source: 4
[ 2216.937505] {1}[Hardware Error]: event severity: info
[ 2216.937508] {1}[Hardware Error]:  Error 0, type: fatal
[ 2216.937511] {1}[Hardware Error]:  fru_text: PcieError
[ 2216.937514] {1}[Hardware Error]:   section_type: PCIe error
[ 2216.937515] {1}[Hardware Error]:   port_type: 4, root port
[ 2216.937517] {1}[Hardware Error]:   version: 0.2
[ 2216.937519] {1}[Hardware Error]:   command: 0x0407, status:
0x0010
[ 2216.937522] {1}[Hardware Error]:   device_id: 0000:10:01.1
[ 2216.937524] {1}[Hardware Error]:   slot: 3
[ 2216.937525] {1}[Hardware Error]:   secondary_bus: 0x11
[ 2216.937526] {1}[Hardware Error]:   vendor_id: 0x1022, device_id:
0x1453
[ 2216.937528] {1}[Hardware Error]:   class_code: 060400
[ 2216.937529] {1}[Hardware Error]:   bridge: secondary_status:
0x2000,
control: 0x0012
[ 2216.937530] {1}[Hardware Error]:   aer_uncor_status: 0x00000000,
aer_uncor_mask: 0x04500000
[ 2216.937532] {1}[Hardware Error]:   aer_uncor_severity:
0x004e2030
[ 2216.937532] {1}[Hardware Error]:   TLP Header: 00000000 00000000
00000000 00000000
[ 2216.937629] pcieport 0000:10:01.1: AER: aer_status: 0x00000000,
aer_mask: 0x04500000
[ 2216.937634] pcieport 0000:10:01.1: AER: aer_layer=Transaction
Layer,
aer_agent=Receiver ID
[ 2216.937638] pcieport 0000:10:01.1: AER: aer_uncor_severity:
0x004e2030
[ 2216.937645] nvme nvme4: frozen state error detected, reset
controller
[ 2217.071095] nvme nvme10: frozen state error detected, reset
controller
[ 2217.096928] nvme nvme0: frozen state error detected, reset
controller
[ 2217.118947] nvme nvme18: frozen state error detected, reset
controller
[ 2217.138945] nvme nvme6: frozen state error detected, reset
controller
[ 2217.164918] nvme nvme14: frozen state error detected, reset
controller
[ 2217.186902] nvme nvme20: frozen state error detected, reset
controller
[ 2279.420266] nvme 0000:1a:00.0: Unable to change power state from
D3cold to D0, device inaccessible
[ 2279.420329] nvme nvme22: Disabling device after reset failure: -
19
[ 2279.464727] pcieport 0000:12:13.0: AER: device recovery failed
[ 2279.464823] pcieport 0000:12:13.0: pciehp: pcie_do_write_cmd: no

response from device

Port resets and recovers

[ 2279.593196] pcieport 0000:10:01.1: AER: Root Port link has been
reset (0)
[ 2279.593699] nvme nvme4: restart after slot reset
[ 2279.593949] nvme nvme10: restart after slot reset
[ 2279.594222] nvme nvme0: restart after slot reset
[ 2279.594453] nvme nvme18: restart after slot reset
[ 2279.594728] nvme nvme6: restart after slot reset
[ 2279.594984] nvme nvme14: restart after slot reset
[ 2279.595226] nvme nvme20: restart after slot reset
[ 2279.595435] pcieport 0000:12:13.0: pciehp: Slot(19): Card
present
[ 2279.595441] pcieport 0000:12:13.0: pciehp: Slot(19): Link Up
[ 2279.609081] nvme nvme4: Shutdown timeout set to 8 seconds
[ 2279.617532] nvme nvme0: Shutdown timeout set to 8 seconds
[ 2279.617533] nvme nvme14: Shutdown timeout set to 8 seconds
[ 2279.618028] nvme nvme6: Shutdown timeout set to 8 seconds
[ 2279.618207] nvme nvme18: Shutdown timeout set to 8 seconds
[ 2279.618290] nvme nvme10: Shutdown timeout set to 8 seconds
[ 2279.618308] nvme nvme20: Shutdown timeout set to 8 seconds
[ 2279.631961] nvme nvme4: 32/0/0 default/read/poll queues
[ 2279.643293] nvme nvme14: 32/0/0 default/read/poll queues
[ 2279.643372] nvme nvme0: 32/0/0 default/read/poll queues
[ 2279.644881] nvme nvme6: 32/0/0 default/read/poll queues
[ 2279.644966] nvme nvme10: 32/0/0 default/read/poll queues
[ 2279.645030] nvme nvme18: 32/0/0 default/read/poll queues
[ 2279.645132] nvme nvme20: 32/0/0 default/read/poll queues
[ 2279.645202] pcieport 0000:10:01.1: AER: device recovery

successful

2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot

  263.862919] mce: [Hardware Error]: CPU 12: Machine Check
Exception: 5
Bank 6: ba00000000000e0b
[  263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90}
[  263.862931] mce: [Hardware Error]: TSC 7a47d8d62ba6dd MISC 83100000
[  263.862933] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1727384194
SOCKET 1 APIC 40 microcode d0003a5
[  263.862936] mce: [Hardware Error]: Run the above through 'mcelog --
ascii'
[  263.885254] mce: [Hardware Error]: Machine check: Processor context
corrupt
[  263.885259] Kernel panic - not syncing: Fatal machine check

Hardware event. This is not a software error.
CPU 0 BANK 0 TSC 7a47d8d62ba6dd
RIP !INEXACT! 10:ffffffff8571dce4
TIME 1727384194 Thu Sep 26 16:56:34 2024
MCG status:
MCi status:
Machine check not valid
Corrected error
MCA: No Error
STATUS 0 MCGSTATUS 0
CPUID Vendor Intel Family 6 Model 106 Step 6

Quoted 10 lines
RIP: intel_idle+0x54/0x90}

SOCKET 1 APIC 40 microcode d0003a5
Run the above through 'mcelog --ascii'
Machine check: Processor context corrupt

Regards
Laurence

I think the Keith's email address is not correct. Adding the correct email address of Keith here.

BTW, Keith recently help fixed an issue in kernel v6.11 with nvme subsystem-reset command to ensure
that we recover the nvme disk on PPC. On PPC architecture, we use EEH to recover the disk post
subsystem-reset but yours is Intel and that uses AER for recovery. So I'm not sure if that same
commit 210b1f6576e8("nvme-pci: do not directly handle subsys reset fallout") which was merged in
kernel v6.11 causing a side effect on the Intel machine.

Quoted 11 lines

Would you please revert the above commit and see if that help fix the observed symptom on your
Intel machine?

Thanks,
--Nilay

Hello Nilay
Thanks will try that.
Was your IBM PPC issue also only with direct attached PCIE slot based nvme.
Will report back after testing with the revert

#3 lore raw
Quoted 189 lines · Laurence Oberman
Quoted 177 lines · Nilay Shroff
Quoted 151 lines · Laurence Oberman

Hi Keith
Hope all is well

Quick question (expected or not)

It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset /dev/nvme0" command to test resets.

On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices

1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)

Example of one working

Does not trap and land up as a machine-check

[ 2215.440468] pcieport 0000:10:01.1: AER: Multiple Uncorrected
(Non-
Fatal) error received: 0000:12:13.0
[ 2215.440532] pcieport 0000:12:13.0: PCIe Bus Error:
severity=Uncorrected (Non-Fatal), type=Transaction Layer,
(Requester
ID)
[ 2215.440536] pcieport 0000:12:13.0:   device [10b5:8748] error
status/mask=00100000/00000000
(First)
[ 2215.440544] pcieport 0000:12:13.0: AER:   TLP Header: 40009001
1000000f e9211000 12000000
[ 2215.441813] systemd-journald[2173]: Sent WATCHDOG=1
notification.
[ 2216.937498] {1}[Hardware Error]: Hardware error from APEI
Generic
Hardware Error Source: 4
[ 2216.937505] {1}[Hardware Error]: event severity: info
[ 2216.937508] {1}[Hardware Error]:  Error 0, type: fatal
[ 2216.937511] {1}[Hardware Error]:  fru_text: PcieError
[ 2216.937514] {1}[Hardware Error]:   section_type: PCIe error
[ 2216.937515] {1}[Hardware Error]:   port_type: 4, root port
[ 2216.937517] {1}[Hardware Error]:   version: 0.2
[ 2216.937519] {1}[Hardware Error]:   command: 0x0407, status:
0x0010
[ 2216.937522] {1}[Hardware Error]:   device_id: 0000:10:01.1
[ 2216.937524] {1}[Hardware Error]:   slot: 3
[ 2216.937525] {1}[Hardware Error]:   secondary_bus: 0x11
[ 2216.937526] {1}[Hardware Error]:   vendor_id: 0x1022, device_id:
0x1453
[ 2216.937528] {1}[Hardware Error]:   class_code: 060400
[ 2216.937529] {1}[Hardware Error]:   bridge: secondary_status:
0x2000,
control: 0x0012
[ 2216.937530] {1}[Hardware Error]:   aer_uncor_status: 0x00000000,
aer_uncor_mask: 0x04500000
[ 2216.937532] {1}[Hardware Error]:   aer_uncor_severity:
0x004e2030
[ 2216.937532] {1}[Hardware Error]:   TLP Header: 00000000 00000000
00000000 00000000
[ 2216.937629] pcieport 0000:10:01.1: AER: aer_status: 0x00000000,
aer_mask: 0x04500000
[ 2216.937634] pcieport 0000:10:01.1: AER: aer_layer=Transaction
Layer,
aer_agent=Receiver ID
[ 2216.937638] pcieport 0000:10:01.1: AER: aer_uncor_severity:
0x004e2030
[ 2216.937645] nvme nvme4: frozen state error detected, reset
controller
[ 2217.071095] nvme nvme10: frozen state error detected, reset
controller
[ 2217.096928] nvme nvme0: frozen state error detected, reset
controller
[ 2217.118947] nvme nvme18: frozen state error detected, reset
controller
[ 2217.138945] nvme nvme6: frozen state error detected, reset
controller
[ 2217.164918] nvme nvme14: frozen state error detected, reset
controller
[ 2217.186902] nvme nvme20: frozen state error detected, reset
controller
[ 2279.420266] nvme 0000:1a:00.0: Unable to change power state from
D3cold to D0, device inaccessible
[ 2279.420329] nvme nvme22: Disabling device after reset failure: -
19
[ 2279.464727] pcieport 0000:12:13.0: AER: device recovery failed
[ 2279.464823] pcieport 0000:12:13.0: pciehp: pcie_do_write_cmd: no

response from device

Port resets and recovers

[ 2279.593196] pcieport 0000:10:01.1: AER: Root Port link has been
reset (0)
[ 2279.593699] nvme nvme4: restart after slot reset
[ 2279.593949] nvme nvme10: restart after slot reset
[ 2279.594222] nvme nvme0: restart after slot reset
[ 2279.594453] nvme nvme18: restart after slot reset
[ 2279.594728] nvme nvme6: restart after slot reset
[ 2279.594984] nvme nvme14: restart after slot reset
[ 2279.595226] nvme nvme20: restart after slot reset
[ 2279.595435] pcieport 0000:12:13.0: pciehp: Slot(19): Card
present
[ 2279.595441] pcieport 0000:12:13.0: pciehp: Slot(19): Link Up
[ 2279.609081] nvme nvme4: Shutdown timeout set to 8 seconds
[ 2279.617532] nvme nvme0: Shutdown timeout set to 8 seconds
[ 2279.617533] nvme nvme14: Shutdown timeout set to 8 seconds
[ 2279.618028] nvme nvme6: Shutdown timeout set to 8 seconds
[ 2279.618207] nvme nvme18: Shutdown timeout set to 8 seconds
[ 2279.618290] nvme nvme10: Shutdown timeout set to 8 seconds
[ 2279.618308] nvme nvme20: Shutdown timeout set to 8 seconds
[ 2279.631961] nvme nvme4: 32/0/0 default/read/poll queues
[ 2279.643293] nvme nvme14: 32/0/0 default/read/poll queues
[ 2279.643372] nvme nvme0: 32/0/0 default/read/poll queues
[ 2279.644881] nvme nvme6: 32/0/0 default/read/poll queues
[ 2279.644966] nvme nvme10: 32/0/0 default/read/poll queues
[ 2279.645030] nvme nvme18: 32/0/0 default/read/poll queues
[ 2279.645132] nvme nvme20: 32/0/0 default/read/poll queues
[ 2279.645202] pcieport 0000:10:01.1: AER: device recovery

successful

2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot

  263.862919] mce: [Hardware Error]: CPU 12: Machine Check
Exception: 5
Bank 6: ba00000000000e0b
[  263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90}
[  263.862931] mce: [Hardware Error]: TSC 7a47d8d62ba6dd MISC 83100000
[  263.862933] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1727384194
SOCKET 1 APIC 40 microcode d0003a5
[  263.862936] mce: [Hardware Error]: Run the above through 'mcelog --
ascii'
[  263.885254] mce: [Hardware Error]: Machine check: Processor context
corrupt
[  263.885259] Kernel panic - not syncing: Fatal machine check

Hardware event. This is not a software error.
CPU 0 BANK 0 TSC 7a47d8d62ba6dd
RIP !INEXACT! 10:ffffffff8571dce4
TIME 1727384194 Thu Sep 26 16:56:34 2024
MCG status:
MCi status:
Machine check not valid
Corrected error
MCA: No Error
STATUS 0 MCGSTATUS 0
CPUID Vendor Intel Family 6 Model 106 Step 6

Quoted 10 lines
RIP: intel_idle+0x54/0x90}

SOCKET 1 APIC 40 microcode d0003a5
Run the above through 'mcelog --ascii'
Machine check: Processor context corrupt

Regards
Laurence

I think the Keith's email address is not correct. Adding the correct email address of Keith here.

BTW, Keith recently help fixed an issue in kernel v6.11 with nvme subsystem-reset command to ensure
that we recover the nvme disk on PPC. On PPC architecture, we use EEH to recover the disk post
subsystem-reset but yours is Intel and that uses AER for recovery. So I'm not sure if that same
commit 210b1f6576e8("nvme-pci: do not directly handle subsys reset fallout") which was merged in
kernel v6.11 causing a side effect on the Intel machine.

Quoted 11 lines

Would you please revert the above commit and see if that help fix the observed symptom on your
Intel machine?

Thanks,
--Nilay

Quoted 6 lines

Hello Nilay
Thanks will try that.
Was your IBM PPC issue also only with direct attached PCIE slot based nvme.
Will report back after testing with the revert

On PPC, it doesn't matter whether NVMe disk is directly attached to PHB or attached through another PCIe bridge. On PPC we saw that when nvme subsystem- reset command is executed on an NVMe disk the EEH couldn't recover the disk and that' where the above commit (from Keith) helped get the disk recovered using EEH after the subsystem-reset command.

Thanks,
--Nilay

maintainer
#4 lore raw
In reply to Laurence Oberman
Quoted 2 lines · Laurence Oberman

It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset /dev/nvme0" command to test resets.

I really dislike that command. The side effects are overkill for the pci transport...

Quoted 9 lines

On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices

1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)

Example of one working

Does not trap and land up as a machine-check

<snip>

Quoted 6 lines

2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot

[  263.862919] mce: [Hardware Error]: CPU 12: Machine Check Exception: 5 Bank 6: ba00000000000e0b
[  263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90}

So this wasn't failing before 6.11? As Nilay mentioned, there are some changes on how nvme subsystem reset is handled. The main thing being this ioctl doesn't automatically trigger an nvme reset. I expected delayed recovery might happen, but machine checks are not expected. If this was working before, I can only guess right now that the previous behavior was accessing MMIO and config quicker and triggered a different error path. If you're successful with the PPC patch reverted, I would be interested to hear about it.

#5 lore raw
Quoted 29 lines · Keith Busch
Quoted 2 lines · Laurence Oberman

It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset /dev/nvme0" command to test resets.

I really dislike that command. The side effects are overkill for the pci
transport...

Quoted 9 lines

On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices

1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)

Example of one working

Does not trap and land up as a machine-check

<snip>

Quoted 8 lines

2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot

[  263.862919] mce: [Hardware Error]: CPU 12: Machine Check
Exception: 5 Bank 6: ba00000000000e0b
[  263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90}

Quoted 13 lines

So this wasn't failing before 6.11? As Nilay mentioned, there are some
changes on how nvme subsystem reset is handled. The main thing being this ioctl doesn't automatically trigger an nvme reset. I expected delayed recovery might happen, but machine checks are not expected. If
this was working before, I can only guess right now that the previous behavior was accessing MMIO and config quicker and triggered a different
error path. If you're successful with the PPC patch reverted, I would be
interested to hear about it.

Hello

Quick update about this.
I went back all the way to 6.8 and this still happens.
I started to think that these HPE servers were more susceptible to the machine checks on the PCIE state changes.

So I tested on a Lenovo and still had panics.
I do not think this is worth pursuing given that Keith already confirmed this is not recommended and way too heavy handed on the PCIE path.

I have told the reporter of this that they are not to use this type of fault injection on directly attached nvme devices.

Thanks
Laurence

#6 lore raw
Quoted 63 lines · Laurence Oberman
Quoted 31 lines · Keith Busch
Quoted 2 lines · Laurence Oberman

It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset /dev/nvme0" command to test resets.

I really dislike that command. The side effects are overkill for the
pci
transport...

Quoted 10 lines

On multiple servers I tested on two types of nvme attached devices
These are not the rootfs devices

1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)

Example of one working

Does not trap and land up as a machine-check

<snip>

Quoted 8 lines

2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot

[  263.862919] mce: [Hardware Error]: CPU 12: Machine Check
Exception: 5 Bank 6: ba00000000000e0b
[  263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90}

Quoted 16 lines

So this wasn't failing before 6.11? As Nilay mentioned, there are some
changes on how nvme subsystem reset is handled. The main thing being
this ioctl doesn't automatically trigger an nvme reset. I expected delayed recovery might happen, but machine checks are not expected. If
this was working before, I can only guess right now that the previous
behavior was accessing MMIO and config quicker and triggered a different
error path. If you're successful with the PPC patch reverted, I would
be
interested to hear about it.

Hello

Quick update about this.
I went back all the way to 6.8 and this still happens. I started to think that these HPE servers were more susceptible to the
machine checks on the PCIE state changes.

So I tested on a Lenovo and still had panics.
I do not think this is worth pursuing given that Keith already confirmed this is not recommended and way too heavy handed on the PCIE
path.

Quoted 7 lines

I have told the reporter of this that they are not to use this type of
fault injection on directly attached nvme devices.

Thanks
Laurence

Hello

Finishing this thread off but have a final question. Bottom line is certain server hardware sees the nvme reset command create a machine check for PCIE plugged NVME devices going back quite far in kernel versions, and we panic.

As Keith had said, that nvme reset command is too much impact

There is a final simple question for M2 connected NVME devices. Are these expected to auto-re-connect after an nvme reset is issued.

The complaint is the following

nvme subsystem-reset /dev/nvme0
Device is disconnected as expected but requires the following to reconnect

echo 1 > /sys/bus/pci/devices/0000:02:00.0/remove echo 1 > /sys/bus/pci/rescan

Then it is reconnected.

Thanks
Laurence

maintainer
#7 lore raw
Quoted 6 lines · Laurence Oberman

Finishing this thread off but have a final question. Bottom line is certain server hardware sees the nvme reset command create a machine check for PCIE plugged NVME devices going back quite far in kernel versions, and we panic.

As Keith had said, that nvme reset command is too much impact

Sure, it takes the PCIe link down, and handling for that, if at all, is platform specific.

Quoted 5 lines

There is a final simple question for M2 connected NVME devices. Are these expected to auto-re-connect after an nvme reset is issued.

The complaint is the following

Quoted 8 lines

nvme subsystem-reset /dev/nvme0
Device is disconnected as expected but requires the following to reconnect

echo 1 > /sys/bus/pci/devices/0000:02:00.0/remove echo 1 > /sys/bus/pci/rescan

Then it is reconnected.

For platforms that don't support link detected hotplug, that sequence should get the device back to usable.