nvme: machine check when running nvme subsystem-reset /dev/nvme0 against direct attach via PCIE slot
Hi Keith
Hope all is well
Quick question (expected or not)
It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset /dev/nvme0" command to test resets.
On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices
1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)
Example of one working
Does not trap and land up as a machine-check
[ 2215.440468] pcieport 0000:10:01.1: AER: Multiple Uncorrected (Non-
Fatal) error received: 0000:12:13.0
[ 2215.440532] pcieport 0000:12:13.0: PCIe Bus Error:
severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester
ID)
[ 2215.440536] pcieport 0000:12:13.0: device [10b5:8748] error
status/mask=00100000/00000000
[ 2215.440540] pcieport 0000:12:13.0: [20] UnsupReq
(First)
[ 2215.440544] pcieport 0000:12:13.0: AER: TLP Header: 40009001
1000000f e9211000 12000000
[ 2215.441813] systemd-journald[2173]: Sent WATCHDOG=1 notification.
[ 2216.937498] {1}[Hardware Error]: Hardware error from APEI Generic
Hardware Error Source: 4
[ 2216.937505] {1}[Hardware Error]: event severity: info
[ 2216.937508] {1}[Hardware Error]: Error 0, type: fatal
[ 2216.937511] {1}[Hardware Error]: fru_text: PcieError
[ 2216.937514] {1}[Hardware Error]: section_type: PCIe error
[ 2216.937515] {1}[Hardware Error]: port_type: 4, root port
[ 2216.937517] {1}[Hardware Error]: version: 0.2
[ 2216.937519] {1}[Hardware Error]: command: 0x0407, status: 0x0010
[ 2216.937522] {1}[Hardware Error]: device_id: 0000:10:01.1
[ 2216.937524] {1}[Hardware Error]: slot: 3
[ 2216.937525] {1}[Hardware Error]: secondary_bus: 0x11
[ 2216.937526] {1}[Hardware Error]: vendor_id: 0x1022, device_id:
0x1453
[ 2216.937528] {1}[Hardware Error]: class_code: 060400
[ 2216.937529] {1}[Hardware Error]: bridge: secondary_status: 0x2000,
control: 0x0012
[ 2216.937530] {1}[Hardware Error]: aer_uncor_status: 0x00000000,
aer_uncor_mask: 0x04500000
[ 2216.937532] {1}[Hardware Error]: aer_uncor_severity: 0x004e2030
[ 2216.937532] {1}[Hardware Error]: TLP Header: 00000000 00000000
00000000 00000000
[ 2216.937629] pcieport 0000:10:01.1: AER: aer_status: 0x00000000,
aer_mask: 0x04500000
[ 2216.937634] pcieport 0000:10:01.1: AER: aer_layer=Transaction Layer,
aer_agent=Receiver ID
[ 2216.937638] pcieport 0000:10:01.1: AER: aer_uncor_severity:
0x004e2030
[ 2216.937645] nvme nvme4: frozen state error detected, reset
controller
[ 2217.071095] nvme nvme10: frozen state error detected, reset
controller
[ 2217.096928] nvme nvme0: frozen state error detected, reset
controller
[ 2217.118947] nvme nvme18: frozen state error detected, reset
controller
[ 2217.138945] nvme nvme6: frozen state error detected, reset
controller
[ 2217.164918] nvme nvme14: frozen state error detected, reset
controller
[ 2217.186902] nvme nvme20: frozen state error detected, reset
controller
[ 2279.420266] nvme 0000:1a:00.0: Unable to change power state from
D3cold to D0, device inaccessible
[ 2279.420329] nvme nvme22: Disabling device after reset failure: -19
[ 2279.464727] pcieport 0000:12:13.0: AER: device recovery failed
[ 2279.464823] pcieport 0000:12:13.0: pciehp: pcie_do_write_cmd: noresponse from device
Port resets and recovers
[ 2279.593196] pcieport 0000:10:01.1: AER: Root Port link has been reset (0) [ 2279.593699] nvme nvme4: restart after slot reset [ 2279.593949] nvme nvme10: restart after slot reset [ 2279.594222] nvme nvme0: restart after slot reset [ 2279.594453] nvme nvme18: restart after slot reset [ 2279.594728] nvme nvme6: restart after slot reset [ 2279.594984] nvme nvme14: restart after slot reset [ 2279.595226] nvme nvme20: restart after slot reset [ 2279.595435] pcieport 0000:12:13.0: pciehp: Slot(19): Card present [ 2279.595441] pcieport 0000:12:13.0: pciehp: Slot(19): Link Up [ 2279.609081] nvme nvme4: Shutdown timeout set to 8 seconds [ 2279.617532] nvme nvme0: Shutdown timeout set to 8 seconds [ 2279.617533] nvme nvme14: Shutdown timeout set to 8 seconds [ 2279.618028] nvme nvme6: Shutdown timeout set to 8 seconds [ 2279.618207] nvme nvme18: Shutdown timeout set to 8 seconds [ 2279.618290] nvme nvme10: Shutdown timeout set to 8 seconds [ 2279.618308] nvme nvme20: Shutdown timeout set to 8 seconds [ 2279.631961] nvme nvme4: 32/0/0 default/read/poll queues [ 2279.643293] nvme nvme14: 32/0/0 default/read/poll queues [ 2279.643372] nvme nvme0: 32/0/0 default/read/poll queues [ 2279.644881] nvme nvme6: 32/0/0 default/read/poll queues [ 2279.644966] nvme nvme10: 32/0/0 default/read/poll queues [ 2279.645030] nvme nvme18: 32/0/0 default/read/poll queues [ 2279.645132] nvme nvme20: 32/0/0 default/read/poll queues [ 2279.645202] pcieport 0000:10:01.1: AER: device recovery successful
2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot
263.862919] mce: [Hardware Error]: CPU 12: Machine Check Exception: 5
Bank 6: ba00000000000e0b
[ 263.862924] mce: [Hardware Error]: RIP !INEXACT!
10:<ffffffff8571dce4> {intel_idle+0x54/0x90}
[ 263.862931] mce: [Hardware Error]: TSC 7a47d8d62ba6dd MISC 83100000
[ 263.862933] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1727384194
SOCKET 1 APIC 40 microcode d0003a5
[ 263.862936] mce: [Hardware Error]: Run the above through 'mcelog --
ascii'
[ 263.885254] mce: [Hardware Error]: Machine check: Processor context
corrupt
[ 263.885259] Kernel panic - not syncing: Fatal machine checkHardware event. This is not a software error.
CPU 0 BANK 0 TSC 7a47d8d62ba6dd
RIP !INEXACT! 10:ffffffff8571dce4
TIME 1727384194 Thu Sep 26 16:56:34 2024
MCG status:
MCi status:
Machine check not valid
Corrected error
MCA: No Error
STATUS 0 MCGSTATUS 0
CPUID Vendor Intel Family 6 Model 106 Step 6
RIP: intel_idle+0x54/0x90}
SOCKET 1 APIC 40 microcode d0003a5
Run the above through 'mcelog --ascii'
Machine check: Processor context corrupt
Regards
Laurence
Quoted 136 lines · Laurence Oberman
Hi Keith
Hope all is wellQuick question (expected or not)
It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset
/dev/nvme0" command to test resets.On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices
1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)
Example of one working
Does not trap and land up as a machine-check
[ 2215.440468] pcieport 0000:10:01.1: AER: Multiple Uncorrected (Non- Fatal) error received: 0000:12:13.0 [ 2215.440532] pcieport 0000:12:13.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) [ 2215.440536] pcieport 0000:12:13.0: device [10b5:8748] error status/mask=00100000/00000000 [ 2215.440540] pcieport 0000:12:13.0: [20] UnsupReq (First) [ 2215.440544] pcieport 0000:12:13.0: AER: TLP Header: 40009001 1000000f e9211000 12000000 [ 2215.441813] systemd-journald[2173]: Sent WATCHDOG=1 notification. [ 2216.937498] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 [ 2216.937505] {1}[Hardware Error]: event severity: info [ 2216.937508] {1}[Hardware Error]: Error 0, type: fatal [ 2216.937511] {1}[Hardware Error]: fru_text: PcieError [ 2216.937514] {1}[Hardware Error]: section_type: PCIe error [ 2216.937515] {1}[Hardware Error]: port_type: 4, root port [ 2216.937517] {1}[Hardware Error]: version: 0.2 [ 2216.937519] {1}[Hardware Error]: command: 0x0407, status: 0x0010 [ 2216.937522] {1}[Hardware Error]: device_id: 0000:10:01.1 [ 2216.937524] {1}[Hardware Error]: slot: 3 [ 2216.937525] {1}[Hardware Error]: secondary_bus: 0x11 [ 2216.937526] {1}[Hardware Error]: vendor_id: 0x1022, device_id: 0x1453 [ 2216.937528] {1}[Hardware Error]: class_code: 060400 [ 2216.937529] {1}[Hardware Error]: bridge: secondary_status: 0x2000, control: 0x0012 [ 2216.937530] {1}[Hardware Error]: aer_uncor_status: 0x00000000, aer_uncor_mask: 0x04500000 [ 2216.937532] {1}[Hardware Error]: aer_uncor_severity: 0x004e2030 [ 2216.937532] {1}[Hardware Error]: TLP Header: 00000000 00000000 00000000 00000000 [ 2216.937629] pcieport 0000:10:01.1: AER: aer_status: 0x00000000, aer_mask: 0x04500000 [ 2216.937634] pcieport 0000:10:01.1: AER: aer_layer=Transaction Layer, aer_agent=Receiver ID [ 2216.937638] pcieport 0000:10:01.1: AER: aer_uncor_severity: 0x004e2030 [ 2216.937645] nvme nvme4: frozen state error detected, reset controller [ 2217.071095] nvme nvme10: frozen state error detected, reset controller [ 2217.096928] nvme nvme0: frozen state error detected, reset controller [ 2217.118947] nvme nvme18: frozen state error detected, reset controller [ 2217.138945] nvme nvme6: frozen state error detected, reset controller [ 2217.164918] nvme nvme14: frozen state error detected, reset controller [ 2217.186902] nvme nvme20: frozen state error detected, reset controller [ 2279.420266] nvme 0000:1a:00.0: Unable to change power state from D3cold to D0, device inaccessible [ 2279.420329] nvme nvme22: Disabling device after reset failure: -19 [ 2279.464727] pcieport 0000:12:13.0: AER: device recovery failed [ 2279.464823] pcieport 0000:12:13.0: pciehp: pcie_do_write_cmd: noresponse from device
Port resets and recovers
[ 2279.593196] pcieport 0000:10:01.1: AER: Root Port link has been reset (0) [ 2279.593699] nvme nvme4: restart after slot reset [ 2279.593949] nvme nvme10: restart after slot reset [ 2279.594222] nvme nvme0: restart after slot reset [ 2279.594453] nvme nvme18: restart after slot reset [ 2279.594728] nvme nvme6: restart after slot reset [ 2279.594984] nvme nvme14: restart after slot reset [ 2279.595226] nvme nvme20: restart after slot reset [ 2279.595435] pcieport 0000:12:13.0: pciehp: Slot(19): Card present [ 2279.595441] pcieport 0000:12:13.0: pciehp: Slot(19): Link Up [ 2279.609081] nvme nvme4: Shutdown timeout set to 8 seconds [ 2279.617532] nvme nvme0: Shutdown timeout set to 8 seconds [ 2279.617533] nvme nvme14: Shutdown timeout set to 8 seconds [ 2279.618028] nvme nvme6: Shutdown timeout set to 8 seconds [ 2279.618207] nvme nvme18: Shutdown timeout set to 8 seconds [ 2279.618290] nvme nvme10: Shutdown timeout set to 8 seconds [ 2279.618308] nvme nvme20: Shutdown timeout set to 8 seconds [ 2279.631961] nvme nvme4: 32/0/0 default/read/poll queues [ 2279.643293] nvme nvme14: 32/0/0 default/read/poll queues [ 2279.643372] nvme nvme0: 32/0/0 default/read/poll queues [ 2279.644881] nvme nvme6: 32/0/0 default/read/poll queues [ 2279.644966] nvme nvme10: 32/0/0 default/read/poll queues [ 2279.645030] nvme nvme18: 32/0/0 default/read/poll queues [ 2279.645132] nvme nvme20: 32/0/0 default/read/poll queues [ 2279.645202] pcieport 0000:10:01.1: AER: device recovery successful2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot
263.862919] mce: [Hardware Error]: CPU 12: Machine Check Exception: 5
Bank 6:ba00000000000e0b[ 263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90} [ 263.862931] mce: [Hardware Error]: TSC 7a47d8d62ba6dd MISC 83100000 [ 263.862933] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1727384194 SOCKET 1 APIC 40 microcode d0003a5 [ 263.862936] mce: [Hardware Error]: Run the above through 'mcelog -- ascii' [ 263.885254] mce: [Hardware Error]: Machine check: Processor context corrupt [ 263.885259] Kernel panic - not syncing: Fatal machine checkHardware event. This is not a software error.
CPU 0 BANK 0 TSC7a47d8d62ba6dd
RIP !INEXACT! 10:ffffffff8571dce4
TIME 1727384194 Thu Sep 26 16:56:34 2024
MCG status:
MCi status:
Machine check not valid
Corrected error
MCA: No Error
STATUS 0 MCGSTATUS 0
CPUID Vendor Intel Family 6 Model 106 Step 6
Quoted 10 lines
RIP: intel_idle+0x54/0x90}SOCKET 1 APIC 40 microcode d0003a5
Run the above through 'mcelog --ascii'
Machine check: Processor context corruptRegards
Laurence
I think the Keith's email address is not correct. Adding the correct email address of Keith here.
BTW, Keith recently help fixed an issue in kernel v6.11 with nvme subsystem-reset command to ensure that we recover the nvme disk on PPC. On PPC architecture, we use EEH to recover the disk post subsystem-reset but yours is Intel and that uses AER for recovery. So I'm not sure if that same commit 210b1f6576e8("nvme-pci: do not directly handle subsys reset fallout") which was merged in kernel v6.11 causing a side effect on the Intel machine.
Would you please revert the above commit and see if that help fix the observed symptom on your Intel machine?
Thanks,
--Nilay
Quoted 177 lines · Nilay Shroff
Quoted 151 lines · Laurence Oberman
Hi Keith
Hope all is wellQuick question (expected or not)
It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset
/dev/nvme0" command to test resets.On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices
1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)
Example of one working
Does not trap and land up as a machine-check
[ 2215.440468] pcieport 0000:10:01.1: AER: Multiple Uncorrected (Non- Fatal) error received: 0000:12:13.0 [ 2215.440532] pcieport 0000:12:13.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) [ 2215.440536] pcieport 0000:12:13.0: device [10b5:8748] error status/mask=00100000/00000000 (First) [ 2215.440544] pcieport 0000:12:13.0: AER: TLP Header: 40009001 1000000f e9211000 12000000 [ 2215.441813] systemd-journald[2173]: Sent WATCHDOG=1 notification. [ 2216.937498] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 [ 2216.937505] {1}[Hardware Error]: event severity: info [ 2216.937508] {1}[Hardware Error]: Error 0, type: fatal [ 2216.937511] {1}[Hardware Error]: fru_text: PcieError [ 2216.937514] {1}[Hardware Error]: section_type: PCIe error [ 2216.937515] {1}[Hardware Error]: port_type: 4, root port [ 2216.937517] {1}[Hardware Error]: version: 0.2 [ 2216.937519] {1}[Hardware Error]: command: 0x0407, status: 0x0010 [ 2216.937522] {1}[Hardware Error]: device_id: 0000:10:01.1 [ 2216.937524] {1}[Hardware Error]: slot: 3 [ 2216.937525] {1}[Hardware Error]: secondary_bus: 0x11 [ 2216.937526] {1}[Hardware Error]: vendor_id: 0x1022, device_id: 0x1453 [ 2216.937528] {1}[Hardware Error]: class_code: 060400 [ 2216.937529] {1}[Hardware Error]: bridge: secondary_status: 0x2000, control: 0x0012 [ 2216.937530] {1}[Hardware Error]: aer_uncor_status: 0x00000000, aer_uncor_mask: 0x04500000 [ 2216.937532] {1}[Hardware Error]: aer_uncor_severity: 0x004e2030 [ 2216.937532] {1}[Hardware Error]: TLP Header: 00000000 00000000 00000000 00000000 [ 2216.937629] pcieport 0000:10:01.1: AER: aer_status: 0x00000000, aer_mask: 0x04500000 [ 2216.937634] pcieport 0000:10:01.1: AER: aer_layer=Transaction Layer, aer_agent=Receiver ID [ 2216.937638] pcieport 0000:10:01.1: AER: aer_uncor_severity: 0x004e2030 [ 2216.937645] nvme nvme4: frozen state error detected, reset controller [ 2217.071095] nvme nvme10: frozen state error detected, reset controller [ 2217.096928] nvme nvme0: frozen state error detected, reset controller [ 2217.118947] nvme nvme18: frozen state error detected, reset controller [ 2217.138945] nvme nvme6: frozen state error detected, reset controller [ 2217.164918] nvme nvme14: frozen state error detected, reset controller [ 2217.186902] nvme nvme20: frozen state error detected, reset controller [ 2279.420266] nvme 0000:1a:00.0: Unable to change power state from D3cold to D0, device inaccessible [ 2279.420329] nvme nvme22: Disabling device after reset failure: - 19 [ 2279.464727] pcieport 0000:12:13.0: AER: device recovery failed [ 2279.464823] pcieport 0000:12:13.0: pciehp: pcie_do_write_cmd: noresponse from device
Port resets and recovers
[ 2279.593196] pcieport 0000:10:01.1: AER: Root Port link has been reset (0) [ 2279.593699] nvme nvme4: restart after slot reset [ 2279.593949] nvme nvme10: restart after slot reset [ 2279.594222] nvme nvme0: restart after slot reset [ 2279.594453] nvme nvme18: restart after slot reset [ 2279.594728] nvme nvme6: restart after slot reset [ 2279.594984] nvme nvme14: restart after slot reset [ 2279.595226] nvme nvme20: restart after slot reset [ 2279.595435] pcieport 0000:12:13.0: pciehp: Slot(19): Card present [ 2279.595441] pcieport 0000:12:13.0: pciehp: Slot(19): Link Up [ 2279.609081] nvme nvme4: Shutdown timeout set to 8 seconds [ 2279.617532] nvme nvme0: Shutdown timeout set to 8 seconds [ 2279.617533] nvme nvme14: Shutdown timeout set to 8 seconds [ 2279.618028] nvme nvme6: Shutdown timeout set to 8 seconds [ 2279.618207] nvme nvme18: Shutdown timeout set to 8 seconds [ 2279.618290] nvme nvme10: Shutdown timeout set to 8 seconds [ 2279.618308] nvme nvme20: Shutdown timeout set to 8 seconds [ 2279.631961] nvme nvme4: 32/0/0 default/read/poll queues [ 2279.643293] nvme nvme14: 32/0/0 default/read/poll queues [ 2279.643372] nvme nvme0: 32/0/0 default/read/poll queues [ 2279.644881] nvme nvme6: 32/0/0 default/read/poll queues [ 2279.644966] nvme nvme10: 32/0/0 default/read/poll queues [ 2279.645030] nvme nvme18: 32/0/0 default/read/poll queues [ 2279.645132] nvme nvme20: 32/0/0 default/read/poll queues [ 2279.645202] pcieport 0000:10:01.1: AER: device recoverysuccessful
2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot
263.862919] mce: [Hardware Error]: CPU 12: Machine Check
Exception: 5
Bank 6:ba00000000000e0b
[ 263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90}
[ 263.862931] mce: [Hardware Error]: TSC7a47d8d62ba6ddMISC 83100000
[ 263.862933] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1727384194
SOCKET 1 APIC 40 microcode d0003a5
[ 263.862936] mce: [Hardware Error]: Run the above through 'mcelog --
ascii'
[ 263.885254] mce: [Hardware Error]: Machine check: Processor context
corrupt
[ 263.885259] Kernel panic - not syncing: Fatal machine checkHardware event. This is not a software error.
CPU 0 BANK 0 TSC7a47d8d62ba6dd
RIP !INEXACT! 10:ffffffff8571dce4
TIME 1727384194 Thu Sep 26 16:56:34 2024
MCG status:
MCi status:
Machine check not valid
Corrected error
MCA: No Error
STATUS 0 MCGSTATUS 0
CPUID Vendor Intel Family 6 Model 106 Step 6Quoted 10 lines
RIP: intel_idle+0x54/0x90}SOCKET 1 APIC 40 microcode d0003a5
Run the above through 'mcelog --ascii'
Machine check: Processor context corruptRegards
LaurenceI think the Keith's email address is not correct. Adding the correct email address of Keith here.
BTW, Keith recently help fixed an issue in kernel v6.11 with nvme subsystem-reset command to ensure
that we recover the nvme disk on PPC. On PPC architecture, we use EEH to recover the disk post
subsystem-reset but yours is Intel and that uses AER for recovery. So I'm not sure if that same
commit 210b1f6576e8("nvme-pci: do not directly handle subsys reset fallout") which was merged in
kernel v6.11 causing a side effect on the Intel machine.
Quoted 11 lines
Would you please revert the above commit and see if that help fix the observed symptom on your
Intel machine?Thanks,
--Nilay
Hello Nilay
Thanks will try that.
Was your IBM PPC issue also only with direct attached PCIE slot based nvme.
Will report back after testing with the revert
Quoted 189 lines · Laurence Oberman
Quoted 177 lines · Nilay Shroff
Quoted 151 lines · Laurence Oberman
Hi Keith
Hope all is wellQuick question (expected or not)
It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset
/dev/nvme0" command to test resets.On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices
1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)
Example of one working
Does not trap and land up as a machine-check
[ 2215.440468] pcieport 0000:10:01.1: AER: Multiple Uncorrected (Non- Fatal) error received: 0000:12:13.0 [ 2215.440532] pcieport 0000:12:13.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) [ 2215.440536] pcieport 0000:12:13.0: device [10b5:8748] error status/mask=00100000/00000000 (First) [ 2215.440544] pcieport 0000:12:13.0: AER: TLP Header: 40009001 1000000f e9211000 12000000 [ 2215.441813] systemd-journald[2173]: Sent WATCHDOG=1 notification. [ 2216.937498] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 [ 2216.937505] {1}[Hardware Error]: event severity: info [ 2216.937508] {1}[Hardware Error]: Error 0, type: fatal [ 2216.937511] {1}[Hardware Error]: fru_text: PcieError [ 2216.937514] {1}[Hardware Error]: section_type: PCIe error [ 2216.937515] {1}[Hardware Error]: port_type: 4, root port [ 2216.937517] {1}[Hardware Error]: version: 0.2 [ 2216.937519] {1}[Hardware Error]: command: 0x0407, status: 0x0010 [ 2216.937522] {1}[Hardware Error]: device_id: 0000:10:01.1 [ 2216.937524] {1}[Hardware Error]: slot: 3 [ 2216.937525] {1}[Hardware Error]: secondary_bus: 0x11 [ 2216.937526] {1}[Hardware Error]: vendor_id: 0x1022, device_id: 0x1453 [ 2216.937528] {1}[Hardware Error]: class_code: 060400 [ 2216.937529] {1}[Hardware Error]: bridge: secondary_status: 0x2000, control: 0x0012 [ 2216.937530] {1}[Hardware Error]: aer_uncor_status: 0x00000000, aer_uncor_mask: 0x04500000 [ 2216.937532] {1}[Hardware Error]: aer_uncor_severity: 0x004e2030 [ 2216.937532] {1}[Hardware Error]: TLP Header: 00000000 00000000 00000000 00000000 [ 2216.937629] pcieport 0000:10:01.1: AER: aer_status: 0x00000000, aer_mask: 0x04500000 [ 2216.937634] pcieport 0000:10:01.1: AER: aer_layer=Transaction Layer, aer_agent=Receiver ID [ 2216.937638] pcieport 0000:10:01.1: AER: aer_uncor_severity: 0x004e2030 [ 2216.937645] nvme nvme4: frozen state error detected, reset controller [ 2217.071095] nvme nvme10: frozen state error detected, reset controller [ 2217.096928] nvme nvme0: frozen state error detected, reset controller [ 2217.118947] nvme nvme18: frozen state error detected, reset controller [ 2217.138945] nvme nvme6: frozen state error detected, reset controller [ 2217.164918] nvme nvme14: frozen state error detected, reset controller [ 2217.186902] nvme nvme20: frozen state error detected, reset controller [ 2279.420266] nvme 0000:1a:00.0: Unable to change power state from D3cold to D0, device inaccessible [ 2279.420329] nvme nvme22: Disabling device after reset failure: - 19 [ 2279.464727] pcieport 0000:12:13.0: AER: device recovery failed [ 2279.464823] pcieport 0000:12:13.0: pciehp: pcie_do_write_cmd: noresponse from device
Port resets and recovers
[ 2279.593196] pcieport 0000:10:01.1: AER: Root Port link has been reset (0) [ 2279.593699] nvme nvme4: restart after slot reset [ 2279.593949] nvme nvme10: restart after slot reset [ 2279.594222] nvme nvme0: restart after slot reset [ 2279.594453] nvme nvme18: restart after slot reset [ 2279.594728] nvme nvme6: restart after slot reset [ 2279.594984] nvme nvme14: restart after slot reset [ 2279.595226] nvme nvme20: restart after slot reset [ 2279.595435] pcieport 0000:12:13.0: pciehp: Slot(19): Card present [ 2279.595441] pcieport 0000:12:13.0: pciehp: Slot(19): Link Up [ 2279.609081] nvme nvme4: Shutdown timeout set to 8 seconds [ 2279.617532] nvme nvme0: Shutdown timeout set to 8 seconds [ 2279.617533] nvme nvme14: Shutdown timeout set to 8 seconds [ 2279.618028] nvme nvme6: Shutdown timeout set to 8 seconds [ 2279.618207] nvme nvme18: Shutdown timeout set to 8 seconds [ 2279.618290] nvme nvme10: Shutdown timeout set to 8 seconds [ 2279.618308] nvme nvme20: Shutdown timeout set to 8 seconds [ 2279.631961] nvme nvme4: 32/0/0 default/read/poll queues [ 2279.643293] nvme nvme14: 32/0/0 default/read/poll queues [ 2279.643372] nvme nvme0: 32/0/0 default/read/poll queues [ 2279.644881] nvme nvme6: 32/0/0 default/read/poll queues [ 2279.644966] nvme nvme10: 32/0/0 default/read/poll queues [ 2279.645030] nvme nvme18: 32/0/0 default/read/poll queues [ 2279.645132] nvme nvme20: 32/0/0 default/read/poll queues [ 2279.645202] pcieport 0000:10:01.1: AER: device recoverysuccessful
2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot
263.862919] mce: [Hardware Error]: CPU 12: Machine Check
Exception: 5
Bank 6:ba00000000000e0b
[ 263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90}
[ 263.862931] mce: [Hardware Error]: TSC7a47d8d62ba6ddMISC 83100000
[ 263.862933] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1727384194
SOCKET 1 APIC 40 microcode d0003a5
[ 263.862936] mce: [Hardware Error]: Run the above through 'mcelog --
ascii'
[ 263.885254] mce: [Hardware Error]: Machine check: Processor context
corrupt
[ 263.885259] Kernel panic - not syncing: Fatal machine checkHardware event. This is not a software error.
CPU 0 BANK 0 TSC7a47d8d62ba6dd
RIP !INEXACT! 10:ffffffff8571dce4
TIME 1727384194 Thu Sep 26 16:56:34 2024
MCG status:
MCi status:
Machine check not valid
Corrected error
MCA: No Error
STATUS 0 MCGSTATUS 0
CPUID Vendor Intel Family 6 Model 106 Step 6Quoted 10 lines
RIP: intel_idle+0x54/0x90}SOCKET 1 APIC 40 microcode d0003a5
Run the above through 'mcelog --ascii'
Machine check: Processor context corruptRegards
LaurenceI think the Keith's email address is not correct. Adding the correct email address of Keith here.
BTW, Keith recently help fixed an issue in kernel v6.11 with nvme subsystem-reset command to ensure
that we recover the nvme disk on PPC. On PPC architecture, we use EEH to recover the disk post
subsystem-reset but yours is Intel and that uses AER for recovery. So I'm not sure if that same
commit 210b1f6576e8("nvme-pci: do not directly handle subsys reset fallout") which was merged in
kernel v6.11 causing a side effect on the Intel machine.Quoted 11 lines
Would you please revert the above commit and see if that help fix the observed symptom on your
Intel machine?Thanks,
--Nilay
Quoted 6 lines
Hello Nilay
Thanks will try that.
Was your IBM PPC issue also only with direct attached PCIE slot based nvme.
Will report back after testing with the revert
On PPC, it doesn't matter whether NVMe disk is directly attached to PHB or attached through another PCIe bridge. On PPC we saw that when nvme subsystem- reset command is executed on an NVMe disk the EEH couldn't recover the disk and that' where the above commit (from Keith) helped get the disk recovered using EEH after the subsystem-reset command.
Thanks,
--Nilay
Keith Busch
maintainerQuoted 2 lines · Laurence Oberman
It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset
/dev/nvme0" command to test resets.
I really dislike that command. The side effects are overkill for the pci transport...
Quoted 9 lines
On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices
1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)
Example of one working
Does not trap and land up as a machine-check
<snip>
Quoted 6 lines
2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot
[ 263.862919] mce: [Hardware Error]: CPU 12: Machine Check Exception: 5 Bank 6: ba00000000000e0b [ 263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90}
So this wasn't failing before 6.11? As Nilay mentioned, there are some changes on how nvme subsystem reset is handled. The main thing being this ioctl doesn't automatically trigger an nvme reset. I expected delayed recovery might happen, but machine checks are not expected. If this was working before, I can only guess right now that the previous behavior was accessing MMIO and config quicker and triggered a different error path. If you're successful with the PPC patch reverted, I would be interested to hear about it.
Quoted 29 lines · Keith Busch
Quoted 2 lines · Laurence Oberman
It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset
/dev/nvme0" command to test resets.I really dislike that command. The side effects are overkill for the pci
transport...Quoted 9 lines
On multiple servers I tested on two types of nvme attached devices These are not the rootfs devices
1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)
Example of one working
Does not trap and land up as a machine-check
<snip>
Quoted 8 lines
2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot
[ 263.862919] mce: [Hardware Error]: CPU 12: Machine Check
Exception: 5 Bank 6:ba00000000000e0b
[ 263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90}
Quoted 13 lines
So this wasn't failing before 6.11? As Nilay mentioned, there are some
changes on how nvme subsystem reset is handled. The main thing being this ioctl doesn't automatically trigger an nvme reset. I expected delayed recovery might happen, but machine checks are not expected. If
this was working before, I can only guess right now that the previous behavior was accessing MMIO and config quicker and triggered a different
error path. If you're successful with the PPC patch reverted, I would be
interested to hear about it.
Hello
Quick update about this.
I went back all the way to 6.8 and this still happens.
I started to think that these HPE servers were more susceptible to the machine checks on the PCIE state changes.
So I tested on a Lenovo and still had panics.
I do not think this is worth pursuing given that Keith already confirmed this is not recommended and way too heavy handed on the PCIE path.
I have told the reporter of this that they are not to use this type of fault injection on directly attached nvme devices.
Thanks
Laurence
Quoted 63 lines · Laurence Oberman
Quoted 31 lines · Keith Busch
Quoted 2 lines · Laurence Oberman
It was reported to Red Hat, seeing issues with using a "nvme subsystem-reset
/dev/nvme0" command to test resets.I really dislike that command. The side effects are overkill for the
pci
transport...Quoted 10 lines
On multiple servers I tested on two types of nvme attached devices
These are not the rootfs devices1. The front slot (hotplug) devices in a 2.5in format reset and after some time recover (what is expected)
Example of one working
Does not trap and land up as a machine-check
<snip>
Quoted 8 lines
2. Any kernel upstream latest 6.11, RHEL8 or RHEL9 causes a machine check and panics the box when its against a nvme in a PCIE slot
[ 263.862919] mce: [Hardware Error]: CPU 12: Machine Check
Exception: 5 Bank 6:ba00000000000e0b
[ 263.862924] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8571dce4> {intel_idle+0x54/0x90}Quoted 16 lines
So this wasn't failing before 6.11? As Nilay mentioned, there are some
changes on how nvme subsystem reset is handled. The main thing being
this ioctl doesn't automatically trigger an nvme reset. I expected delayed recovery might happen, but machine checks are not expected. If
this was working before, I can only guess right now that the previous
behavior was accessing MMIO and config quicker and triggered a different
error path. If you're successful with the PPC patch reverted, I would
be
interested to hear about it.Hello
Quick update about this.
I went back all the way to 6.8 and this still happens. I started to think that these HPE servers were more susceptible to the
machine checks on the PCIE state changes.So I tested on a Lenovo and still had panics.
I do not think this is worth pursuing given that Keith already confirmed this is not recommended and way too heavy handed on the PCIE
path.
Quoted 7 lines
I have told the reporter of this that they are not to use this type of
fault injection on directly attached nvme devices.Thanks
Laurence
Hello
Finishing this thread off but have a final question. Bottom line is certain server hardware sees the nvme reset command create a machine check for PCIE plugged NVME devices going back quite far in kernel versions, and we panic.
As Keith had said, that nvme reset command is too much impact
There is a final simple question for M2 connected NVME devices. Are these expected to auto-re-connect after an nvme reset is issued.
The complaint is the following
nvme subsystem-reset /dev/nvme0
Device is disconnected as expected but requires the following to reconnect
echo 1 > /sys/bus/pci/devices/0000:02:00.0/remove echo 1 > /sys/bus/pci/rescan
Then it is reconnected.
Thanks
Laurence
Keith Busch
maintainerQuoted 6 lines · Laurence Oberman
Finishing this thread off but have a final question. Bottom line is certain server hardware sees the nvme reset command create a machine check for PCIE plugged NVME devices going back quite far in kernel versions, and we panic.
As Keith had said, that nvme reset command is too much impact
Sure, it takes the PCIe link down, and handling for that, if at all, is platform specific.
Quoted 5 lines
There is a final simple question for M2 connected NVME devices. Are these expected to auto-re-connect after an nvme reset is issued.
The complaint is the following
Quoted 8 lines
nvme subsystem-reset
/dev/nvme0
Device is disconnected as expected but requires the following to reconnectecho 1 > /sys/bus/pci/devices/0000:02:00.0/remove echo 1 >
/sys/bus/pci/rescanThen it is reconnected.
For platforms that don't support link detected hotplug, that sequence should get the device back to usable.