Lunar Lake: silent fatal platform resets caused by idle-exit core hang
Hello.
So, I've got frustrated enough with my system cold-resetting every few days to try to debug the issue, with some success.
TL;DR: cold reset on the cpuidle exit path.
It's *probably* not actually a kernel bug, but I don't know any better place to submit it (hoping that the Intel folks will take a look).
System:
- Lenovo ThinkPad X1 Carbon Gen 13 (21NS001ACD), BIOS N4BET75W (1.45).
- Intel Core Ultra 7 258V (Lunar Lake), microcode 0x126.
- Kernel 7.1.1, intel_idle in ACPI _CST mode.
- Reproduces even with intel_idle.max_cstate=2, so the enabled states are only POLL, C1_ACPI and C2_ACPI.
On boot after each crash, the kernel logs:
BERT: [Hardware Error]: Skipped 1 error records
I pulled /sys/firmware/acpi/tables/data/BERT by hand and decoded it with
https://github.com/intel/crashlog.
The region contains a full PMC crashlog (MCA, Punit, UNCORE, PCORE, PMC, PMC_RST, PMC_TRACE records). iclg summarizes the failure as:
CORE_TIMEOUT.SINGLE_STUCK_TRANSACTION.C898FH MCA.BANK3.INTERNAL_TIMER_ERROR.MSCOD_E184H CRASHLOG_REASON.PMC.10H CRASHLOG_REASON.PUNIT.20004H RESET_CAUSE.FIRMWARE_GLOBAL_RESET.FW_GBLRST_SCRATCH16 RESET_CAUSE.GLOBAL_RESET.PMC_FW
The important registers (one P-core has captured state):
mca.bank3.status = 0xbe000000e1840400
VAL|UC|EN|MISCV|ADDRV|PCC, MCACOD 0x0400 (internal timer /
retirement watchdog, "three-strike"), MSCOD 0xE184
mca.bank3.addr = mca.bank3.misc = arch_state.lip
= 0xffffffff9471bb3c (kernel text) super queue: exactly one stuck transaction (cacheline 0xC898F)
pmc_rst: gblrst_cause_0.pmc_fw=1, fw_gblrst_cause_0.scratch16=1,
gblrst_req_0 = pmc_fw|syspwr_flr|pchpwr_flr
The LIP: given that KASLR slide is 2MB-aligned, (lip mod 2M) is an invariant. Solving over System.map text symbols gives 14 candidates, of which only one can be related:
cpuidle_enter_state+0xbc
Disassembly:
+0x73 call __x86_indirect_thunk_rax ; target_state->enter()
... sched_clock_idle_wakeup_event / local_clock_noinstr /
sched_idle_set_state
+0xb6 call *pv_ops...irq_enable ; patched to STI at boot +0xbc test %r14d,%r14d ; <- LIP
So (my best guess) the core had already returned from MWAIT, executed STI on the idle-exit path, and then stopped retiring instructions. The retirement watchdog then fires, Punit reports CORE_TIMEOUT, and PMC firmware pulls a global platform reset.
The question is whether it is something known or rather something platform-specific that should be reported to Lenovo instead.
I can provide the raw BERT dump, and also I'd be happy to debug/test if requested.
Best regards,
Polina.
I think the chances of this being related to the Linux kernel are very small. (We've run Linux on this model and it has been fine.)
It would be interesting if booting with "idle=halt" makes the issue go away.
My guess is that you've got a flakey DIMM.
Does it pass an overnight memory test?
Unfortunately, I believe the DIMMs on this one are soldered onto the motherboard...
cheers,
-Len
Quoted 75 lines · Polina Vishneva
Hello.
So, I've got frustrated enough with my system cold-resetting every few days to try to debug the issue, with some success.
TL;DR: cold reset on the cpuidle exit path.
It's *probably* not actually a kernel bug, but I don't know any better place to submit it (hoping that the Intel folks will take a look).
System:
- Lenovo ThinkPad X1 Carbon Gen 13 (21NS001ACD), BIOS N4BET75W (1.45).
- Intel Core Ultra 7 258V (Lunar Lake), microcode0x126.
- Kernel 7.1.1,intel_idlein ACPI_CSTmode.
- Reproduces even with intel_idle.max_cstate=2, so the enabled states are only POLL,C1_ACPIandC2_ACPI.On boot after each crash, the kernel logs:
BERT: [Hardware Error]: Skipped 1 error records
I pulled
/sys/firmware/acpi/tables/data/BERTby hand and decoded it with
https://github.com/intel/crashlog.The region contains a full PMC crashlog (MCA, Punit, UNCORE, PCORE, PMC,
PMC_RST,PMC_TRACErecords). iclg summarizes the failure as:CORE_TIMEOUT.SINGLE_STUCK_TRANSACTION.C898FH MCA.BANK3.INTERNAL_TIMER_ERROR.MSCOD_E184H CRASHLOG_REASON.PMC.10H CRASHLOG_REASON.PUNIT.20004H RESET_CAUSE.FIRMWARE_GLOBAL_RESET.FW_GBLRST_SCRATCH16 RESET_CAUSE.GLOBAL_RESET.PMC_FW
The important registers (one P-core has captured state):
mca.bank3.status =
0xbe000000e1840400VAL|UC|EN|MISCV|ADDRV|PCC, MCACOD 0x0400 (internal timer / retirement watchdog, "three-strike"), MSCOD 0xE184 mca.bank3.addr = mca.bank3.misc = arch_state.lip = 0xffffffff9471bb3c (kernel text)super queue: exactly one stuck transaction (cacheline
0xC898F)
pmc_rst: gblrst_cause_0.pmc_fw=1, fw_gblrst_cause_0.scratch16=1,
gblrst_req_0= pmc_fw|syspwr_flr|pchpwr_flrThe LIP: given that KASLR slide is 2MB-aligned, (lip mod 2M) is an invariant. Solving over System.map text symbols gives 14 candidates, of which only one can be related:
cpuidle_enter_state+0xbcDisassembly:
+0x73 call
__x86_indirect_thunk_rax;target_state->enter()
...sched_clock_idle_wakeup_event/local_clock_noinstr/
sched_idle_set_state+0xb6 call *pv_ops...irq_enable ; patched to STI at boot +0xbc test %r14d,%r14d ; <- LIPSo (my best guess) the core had already returned from MWAIT, executed STI on the idle-exit path, and then stopped retiring instructions. The retirement watchdog then fires, Punit reports
CORE_TIMEOUT, and PMC firmware pulls a global platform reset.The question is whether it is something known or rather something platform-specific that should be reported to Lenovo instead.
I can provide the raw BERT dump, and also I'd be happy to debug/test if requested.
Best regards,
Polina.
Signature · 1 line
Len Brown, Intel Open Source Technology Center
A message that is not in the archive (sent off-list or not yet received): YQYPR01MB11542163CE28DD33BDC70AC73BDFF2@YQYPR01MB11542.CANPRD01.PROD.OUTLOOK.COM
Note - switched to my open-source friendly email account. Lenovo email sucks.
Quoted 9 lines
I think the chances of this being related to the Linux kernel are very small. (We've run Linux on this model and it has been fine.)
It would be interesting if booting with "idle=halt" makes the issue go away.
My guess is that you've got a flakey DIMM.
Does it pass an overnight memory test?
Quoted 79 lines
Unfortunately, I believe the DIMMs on this one are soldered onto the motherboard...
cc: Mark, who may have some insight....
cheers,
-LenQuoted 71 lines · Polina Vishneva
Hello.
So, I've got frustrated enough with my system cold-resetting every few days to try to debug the issue, with some success.
TL;DR: cold reset on the cpuidle exit path.
It's *probably* not actually a kernel bug, but I don't know any better place to submit it (hoping that the Intel folks will take a look).
System:
- Lenovo ThinkPad X1 Carbon Gen 13 (21NS001ACD), BIOS N4BET75W (1.45).
- Intel Core Ultra 7 258V (Lunar Lake), microcode0x126.
- Kernel 7.1.1,intel_idlein ACPI_CSTmode.
- Reproduces even with intel_idle.max_cstate=2, so the enabled states are only POLL,C1_ACPIandC2_ACPI.On boot after each crash, the kernel logs:
BERT: [Hardware Error]: Skipped 1 error records
I pulled
/sys/firmware/acpi/tables/data/BERTby hand and decoded it with
https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fintel%2Fcrashlog&data=05%7C02%7Cmpearson%40lenovo.com%7Cf3802edcfdf349bfdd3c08dedd3a20a6%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C639191438937666379%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=t09jXVwO9ptn39ez5fPlWlnPF7YtqtyuUwxvs484PUU%3D&reserved=0<https://github.com/intel/crashlog>.The region contains a full PMC crashlog (MCA, Punit, UNCORE, PCORE, PMC,
PMC_RST,PMC_TRACErecords). iclg summarizes the failure as:CORE_TIMEOUT.SINGLE_STUCK_TRANSACTION.C898FH MCA.BANK3.INTERNAL_TIMER_ERROR.MSCOD_E184H CRASHLOG_REASON.PMC.10H CRASHLOG_REASON.PUNIT.20004H RESET_CAUSE.FIRMWARE_GLOBAL_RESET.FW_GBLRST_SCRATCH16 RESET_CAUSE.GLOBAL_RESET.PMC_FW
The important registers (one P-core has captured state):
mca.bank3.status =
0xbe000000e1840400VAL|UC|EN|MISCV|ADDRV|PCC, MCACOD 0x0400 (internal timer / retirement watchdog, "three-strike"), MSCOD 0xE184 mca.bank3.addr = mca.bank3.misc = arch_state.lip = 0xffffffff9471bb3c (kernel text)super queue: exactly one stuck transaction (cacheline
0xC898F)
pmc_rst: gblrst_cause_0.pmc_fw=1, fw_gblrst_cause_0.scratch16=1,
gblrst_req_0= pmc_fw|syspwr_flr|pchpwr_flrThe LIP: given that KASLR slide is 2MB-aligned, (lip mod 2M) is an invariant. Solving over System.map text symbols gives 14 candidates, of which only one can be related:
cpuidle_enter_state+0xbcDisassembly:
+0x73 call
__x86_indirect_thunk_rax;target_state->enter()
...sched_clock_idle_wakeup_event/local_clock_noinstr/
sched_idle_set_state+0xb6 call *pv_ops...irq_enable ; patched to STI at boot +0xbc test %r14d,%r14d ; <- LIPSo (my best guess) the core had already returned from MWAIT, executed STI on the idle-exit path, and then stopped retiring instructions. The retirement watchdog then fires, Punit reports
CORE_TIMEOUT, and PMC firmware pulls a global platform reset.The question is whether it is something known or rather something platform-specific that should be reported to Lenovo instead.
I can provide the raw BERT dump, and also I'd be happy to debug/test if requested.
With the BERT error this looks like a HW issue. X1 Carbon is one of our more popular platforms and should run Linux really well (I used it myself quite a bit for 6 months). You probably need to call Lenovo support to organise an RMA.
If you want you can send me the kernel log and the EC and ME details (your BIOS version looks recent) and I'll have a look through and confirm there's nothing else funky going on.
I can share the certified Ubuntu or Fedora images if interested. I doubt it's worth burning the time on doing the whole reinstall dance but it can help when dealing with Lenovo support - they like to know the certified image fails too.
Mark