Skip to contentSkip to conversation

Bug#1104670: linux-image-6.12.25-amd64: system does not shut down - GHES: Fatal hardware error

Intel-wired-lan Tim Connors 1 participant 1 message Apr 13 PCI

lore
Share & tools
lore
#0 lore raw
Quoted 30 lines · Aleksandr
Quoted 30 lines · Laurent Bonnaud

[...]

Quoted 1 line

- Previously the kernel would output an error in

/var/lib/systemd/pstore/ but would shutdown anyway.

Quoted 2 lines

- Now, with kernel 6.1.135-1, the shutdown is blocked as with

6.12.x kernels (see below).

Quoted 3 lines
<30>[  961.098671] systemd-shutdown[1]: Rebooting.
<6>[  961.098743] kvm: exiting hardware virtualization <6>[

961.361878] megaraid_sas 0000:17:00.0: megasas_disable_intr_fusion

is

Quoted 4 lines

called outbound_intr_mask:0x40000009 <6>[ 961.414526] ACPI: PM:
Preparing to enter system sleep state S5 <0>[ 963.828210] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error
Source: 5 <0>[ 963.828213] {1}[Hardware Error]: event severity:

fatal <0>[ 963.828214] {1}[Hardware Error]: Error 0, type: fatal

Quoted 2 lines
<0>[  963.828216] {1}[Hardware Error]:   section_type: PCIe error
<0>[  963.828216] {1}[Hardware Error]:   port_type: 0, PCIe end

point

Quoted 2 lines
<0>[  963.828217] {1}[Hardware Error]:   version: 3.0
<0>[  963.828218] {1}[Hardware Error]:   command: 0x0002, status:

0x0010

Quoted 4 lines
<0>[  963.828220] {1}[Hardware Error]:   device_id: 0000:01:00.1
<0>[  963.828221] {1}[Hardware Error]:   slot: 6
<0>[  963.828222] {1}[Hardware Error]:   secondary_bus: 0x00
<0>[  963.828223] {1}[Hardware Error]:   vendor_id: 0x8086,

device_id: 0x1563

Quoted 2 lines
<0>[  963.828224] {1}[Hardware Error]:   class_code: 020000
<0>[  963.828225] {1}[Hardware Error]:   aer_uncor_status:

0x00100000, aer_uncor_mask: 0x00018000

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
<0>[  963.828226] {1}[Hardware Error]:   aer_uncor_severity:

0x000ef010

Quoted 1 line
<0>[  963.828227] {1}[Hardware Error]:   TLP Header: 40000001

0000000f 90028090 00000000
[...]

It seems that this is a known bug in the BIOS of several Dell PowerEdge models including (in this case) the R540.

Yup, R730XD here.

Quoted 9 lines
Quoted 9 lines

A workaround was added to the tg3 driver
<https://git.kernel.org/linus/e0efe83ed325277bb70f9435d4d9fc70bebdcca8

and a similar change was proposed (but not accepted) in the i40e driver <https://lore.kernel.org/all/20241227035459.90602-1-
[email protected]/>.
On tihis system the erorr log points to a deivce handled by the ixgbe driver, and no workaround has been implemented for that.

Quoted 3 lines
Quoted 3 lines

Since this issue seems to affect multiple different NIC vendors and drivers, would it make more sense to implement this workaround as a PCI quirk?

It's not just network devices either.

<5>[965917.449277] sd 4:0:0:0: [sda] Synchronizing SCSI cache
<6>[965917.614364] [drm] PCIE GART of 256M enabled (table at
0x000000F47FF80000).
<6>[965917.820364] [drm] UVD and UVD ENC initialized successfully.
<6>[965917.921559] [drm] VCE initialized successfully.
<6>[965917.926574] amdgpu 0000:04:00.0: [drm] Cannot find any crtc or sizes
<6>[965917.934684] amdgpu 0000:04:00.0: [drm] Cannot find any crtc or sizes
<0>[965919.725575] {1}[Hardware Error]: Hardware error from APEI Generic
Hardware Error Source: 3
<0>[965919.725582] {1}[Hardware Error]: event severity: fatal
<0>[965919.725587] {1}[Hardware Error]:  Error 0, type: fatal
<0>[965919.725591] {1}[Hardware Error]:   section_type: PCIe error
<0>[965919.725595] {1}[Hardware Error]:   port_type: 1, legacy PCI end point
<0>[965919.725598] {1}[Hardware Error]:   version: 1.16
<0>[965919.725602] {1}[Hardware Error]:   command: 0x0407, status: 0x0010
<0>[965919.725607] {1}[Hardware Error]:   device_id: 0000:04:00.1
<0>[965919.725611] {1}[Hardware Error]:   slot: 0
<0>[965919.725614] {1}[Hardware Error]:   secondary_bus: 0x00
<0>[965919.725617] {1}[Hardware Error]:   vendor_id: 0x1002, device_id:
0xaae0
<0>[965919.725622] {1}[Hardware Error]:   class_code: 040300
<0>[965919.725625] {1}[Hardware Error]:   aer_cor_status: 0x00002000,
aer_cor_mask: 0x000031c0
<0>[965919.725630] {1}[Hardware Error]:   aer_uncor_status: 0x00100000,
aer_uncor_mask: 0x00010000
<0>[965919.725635] {1}[Hardware Error]:   aer_uncor_severity: 0x004e7030
<0>[965919.725638] {1}[Hardware Error]:   TLP Header: 40008001 00000a0f
96a121a0 00000000
<0>[965919.725646] GHES: Fatal hardware error but panic disabled
<0>[965919.725650] Kernel panic - not syncing: GHES: Fatal hardware error
<4>[965919.725662] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: P
  O       6.14.11-5-bpo12-pve #1
<4>[965919.725676] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
<4>[965919.725689] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS
2.19.0 12/12/2023
<4>[965919.725694] Call Trace:
<4>[965919.725700]  <NMI>
<4>[965919.725706]  dump_stack_lvl+0x27/0xa0
<4>[965919.725722]  dump_stack+0x10/0x20
<4>[965919.725729]  panic+0x358/0x3b0
<4>[965919.725742]  __ghes_panic+0x60/0x80
<4>[965919.725756]  ghes_notify_nmi+0x1d5/0x380
<4>[965919.725768]  nmi_handle.part.0+0x58/0x160
<4>[965919.725781]  default_do_nmi+0x131/0x170
<4>[965919.725792]  exc_nmi+0x1c4/0x290
<4>[965919.725799]  end_repeat_nmi+0xf/0x53
<4>[965919.725816] RIP: 0010:intel_idle+0x51/0x90
<4>[965919.725824] Code: 2d 80 ca 2b 00 eb 52 cc cc cc 48 89 f0 0f 1f 00 31
d2 48 89 d1 0f 01 c8 48 8b 06 a8 08 75 0b b9 01 00 00 00 4c 89 c0 0f 01 c9
<f0> 80 66 02 df f0 83 44 24 fc 00 48 8b 06 a8 08 74 0b 65 81 25 ea
<4>[965919.725830] RSP: 0018:ffffffff8ec03db0 EFLAGS: 00000046
<4>[965919.725837] RAX: 0000000000000020 RBX: ffff8aa2ffa44680 RCX:
0000000000000001
<4>[965919.725841] RDX: 0000000000000000 RSI: ffffffff8ec107c0 RDI:
0000000000000004
<4>[965919.725849] RBP: ffffffff8ec03df0 R08: 0000000000000020 R09:
0000000000000000
<4>[965919.725854] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000004
<4>[965919.725857] R13: ffffffff8ee86960 R14: ffffffff8ee86b18 R15:
0000000000000004
<4>[965919.725866]  ? intel_idle+0x51/0x90
<4>[965919.725873]  ? intel_idle+0x51/0x90
<4>[965919.725879]  </NMI>
<4>[965919.725882]  <TASK>
<4>[965919.725884]  ? cpuidle_enter_state+0x85/0x450
<4>[965919.725895]  cpuidle_enter+0x2e/0x50
<4>[965919.725908]  call_cpuidle+0x22/0x60
<4>[965919.725918]  do_idle+0x1de/0x240
<4>[965919.725925]  cpu_startup_entry+0x29/0x30
<4>[965919.725930]  rest_init+0xd0/0xd0
<4>[965919.725934]  start_kernel+0x779/0xb60
<4>[965919.725941]  ? load_ucode_intel_bsp+0x43/0xa0
<4>[965919.725952]  x86_64_start_reservations+0x18/0x30
<4>[965919.725961]  x86_64_start_kernel+0xbf/0x110
<4>[965919.725968]  common_startup_64+0x13e/0x141
<4>[965919.725980]  </TASK>
<0>[965919.726136] Kernel Offset: 0xb600000 from 0xffffffff81000000

(relocation range: 0xffffffff80000000-0xffffffffbfffffff)

04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X] (rev c7) (prog-if 00 [VGA controller])
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Lexa PRO [Radeon
540/540X/550/550X / RX 540X/550/550X]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 174
NUMA node: 0
IOMMU group: 29
Region 0: Memory at 38000000000 (64-bit, prefetchable) [size=2G] Region 2: Memory at 38080000000 (64-bit, prefetchable) [size=2M] Region 4: I/O ports at 2000 [size=256]
Region 5: Memory at 96a00000 (32-bit, non-prefetchable) [size=256K] Expansion ROM at 96a60000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: amdgpu
Kernel modules: amdgpu

04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP
Audio [Radeon RX 550 640SP / RX 560/560X]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin B routed to IRQ 172
NUMA node: 0
IOMMU group: 30
Region 0: Memory at 96a40000 (64-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel

Was completely idle and unused all boot session, and reboot was routine after patching. kernel 6.14.11-5-bpo12 from proxmox backports (so ubuntu backports, essentially).

Quoted 1 line

I support the idea of PCI workaround, but who will implement it ?