Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot

Hey Xen Team,

I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.

Environment
-----------
Xen:   4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1) dom0:  Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs CPU:   AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X, microcode 0x0a201210
Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems

The panic
---------
(XEN) [2026-10-19 08:58:18.727] Xen BUG at arch/x86/cpu/mcheck/mce_amd.c:100
(XEN) CPU:    2
(XEN) RIP:    e008:[<ffff82d04028a650>] mce_amd.c#mc_ec2type.part.0+0/0x10
(XEN) rbx: 0000000000000445   r12: 0000000000000011   r15:
0000000000000446
(XEN) Xen code around <ffff82d04028a650>:
(XEN)  90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90
(XEN) Xen call trace:
(XEN)    [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10 (XEN)    [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40 (XEN)    [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840 (XEN)    [<ffff82d04028a35c>] S amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
(XEN)    [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90 (XEN)    [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
(XEN)    [<ffff82d040201915>] S common_interrupt+0x135/0x150 (XEN)    [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140 (XEN)    [<ffff82d0402887cf>] S cpu_idle.c#acpi_processor_idle+0x1ef/0x650
(XEN)    [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130 (XEN) Panic on CPU 2

The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is the "unreachable" default branch of mc_ec2type() -- the extended error code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.

Across the eight occurrences, r12 appears to be the bank index from the loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS) and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.

Not a hardware defect
---------------------
- Started within days of upgrading two hosts from Debian bookworm (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no incident.
- Three different banks, so not a single failing block.
- Updating to the current Debian package (4.20.5-pre, built 2026-10-16) changed nothing.

Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks updating").

Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
Filter bogus hardware errors on Zen3 clients"), where bogus MCA values on Zen 3 are filtered rather than treated as errors.
https://ratatoskr.run/stable/2026/04/7143905

Diagnostics / workaround
------------------------
mce_verbosity=verbose yields nothing useful.
max_cstate=1 made no difference. mce=false is the only thing that stops it, and it is what I'm running now.

One more thing: the host does not actually reboot after the panic. "Reboot in five seconds" is printed, but the ring buffer continues with normal activity minutes later and the remaining 31 CPUs keep running. CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0 and guest vCPUs start losing timer interrupts permanently (RCU stalls with a frozen timer-softirq count while dynticks keeps advancing), so VMs hang silently hours or days later.

I've attached the full xl dmesg log of the last crash, and I'm happy to provide more if needed. I can also test patches.

Thanks for your work, and thanks in advance for taking a look.

Peter

Quoted 42 lines

Hey Xen Team,

I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.

Environment
-----------
Xen:   4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
       present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
dom0:  Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs CPU:   AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
       microcode 0x0a201210
Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems

The panic
---------
(XEN) [2026-10-19 08:58:18.727] Xen BUG at arch/x86/cpu/mcheck/mce_amd.c:100
(XEN) CPU:    2
(XEN) RIP:    e008:[<ffff82d04028a650>] mce_amd.c#mc_ec2type.part.0+0/0x10
(XEN) rbx: 0000000000000445   r12: 0000000000000011   r15:
0000000000000446
(XEN) Xen code around <ffff82d04028a650>:
(XEN)  90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90
(XEN) Xen call trace:
(XEN)    [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10 (XEN)    [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40 (XEN)    [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840 (XEN)    [<ffff82d04028a35c>] S amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
(XEN)    [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90 (XEN)    [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
(XEN)    [<ffff82d040201915>] S common_interrupt+0x135/0x150 (XEN)    [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140 (XEN)    [<ffff82d0402887cf>] S cpu_idle.c#acpi_processor_idle+0x1ef/0x650
(XEN)    [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130 (XEN) Panic on CPU 2

Quoted 3 lines

The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is the "unreachable" default branch of mc_ec2type() -- the extended error code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.

I think it's safe to just drop the BUG().

That code is literally worse than useless, because if nothing else it's just hiding the diagnostics.

I guess it means AMD have added new statuses.

Quoted 13 lines

Across the eight occurrences, r12 appears to be the bank index from the loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS) and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.

Not a hardware defect
---------------------
- Started within days of upgrading two hosts from Debian bookworm (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no incident.
- Three different banks, so not a single failing block.
- Updating to the current Debian package (4.20.5-pre, built 2026-10-16) changed nothing.

Quoted 7 lines

Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks updating").

Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
Filter bogus hardware errors on Zen3 clients"), where bogus MCA values on Zen 3 are filtered rather than treated as errors.
https://ratatoskr.run/stable/2026/04/7143905

From above, you're on a Vermeer, not a Cezanne, so I don't think that
Linux patch is relevant.

However, your observation about "fix mca_allbanks updating" quite possibly is.  That was backported into Xen 4.20.1 as 247b708b1400a, and Debian has 4.20.3

Are you able to experiment with that patch reverted?

Alternatively, are you able to experiment with the BUG() removed?

Quoted 5 lines

Diagnostics / workaround
------------------------
mce_verbosity=verbose yields nothing useful.
max_cstate=1 made no difference. mce=false is the only thing that stops it, and it is what I'm running now.

The fact that mce=false isn't causing machine resets means it's clearly something the hardware thinks isn't fatal.

But, I am concerned by the fact it's always on CPU 2, and multiple banks.  Is this true for both systems?

Quoted 7 lines

One more thing: the host does not actually reboot after the panic. "Reboot in five seconds" is printed, but the ring buffer continues with normal activity minutes later and the remaining 31 CPUs keep running. CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0 and guest vCPUs start losing timer interrupts permanently (RCU stalls with a frozen timer-softirq count while dynticks keeps advancing), so VMs hang silently hours or days later.

I've recently noticed that crashing with interrupts off doesn't reboot any more.  I thought it was something local to my development, but apparently not if you're seeing the same.

Quoted 4 lines

I've attached the full xl dmesg log of the last crash, and I'm happy to provide more if needed. I can also test patches.

Quoted 3 lines

Thanks for your work, and thanks in advance for taking a look.

Peter

Thankyou,

~Andrew

Hi Andrew,

Thanks a lot for the quick response.

Quoted 76 lines · Peter Deggendorfer

Quoted 42 lines

Hey Xen Team,

I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.

Environment
-----------
Xen:   4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1) dom0:  Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs CPU:   AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X, microcode 0x0a201210
Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems

The panic
---------
(XEN) [2026-10-19 08:58:18.727] Xen BUG at arch/x86/cpu/mcheck/mce_amd.c:100
(XEN) CPU:    2
(XEN) RIP:    e008:[<ffff82d04028a650>] mce_amd.c#mc_ec2type.part.0+0/0x10
(XEN) rbx: 0000000000000445   r12: 0000000000000011   r15:
0000000000000446
(XEN) Xen code around <ffff82d04028a650>:
(XEN)  90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90
(XEN) Xen call trace:
(XEN)    [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10 (XEN)    [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40 (XEN)    [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840 (XEN)    [<ffff82d04028a35c>] S amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
(XEN)    [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90 (XEN)    [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
(XEN)    [<ffff82d040201915>] S common_interrupt+0x135/0x150 (XEN)    [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140 (XEN)    [<ffff82d0402887cf>] S cpu_idle.c#acpi_processor_idle+0x1ef/0x650
(XEN)    [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130 (XEN) Panic on CPU 2

Quoted 3 lines

The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is the "unreachable" default branch of mc_ec2type() -- the extended error code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.

I think it's safe to just drop the BUG().

That code is literally worse than useless, because if nothing else it's just hiding the diagnostics.

I guess it means AMD have added new statuses.

Quoted 13 lines

Across the eight occurrences, r12 appears to be the bank index from the loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS) and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.

Not a hardware defect
---------------------
- Started within days of upgrading two hosts from Debian bookworm (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no incident.
- Three different banks, so not a single failing block.
- Updating to the current Debian package (4.20.5-pre, built 2026-10-16) changed nothing.

Quoted 7 lines

Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks updating").

Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
Filter bogus hardware errors on Zen3 clients"), where bogus MCA values on Zen 3 are filtered rather than treated as errors.
https://ratatoskr.run/stable/2026/04/7143905

 From above, you're on a Vermeer, not a Cezanne, so I don't think that Linux patch is relevant.

Quoted 7 lines

However, your observation about "fix mca_allbanks updating" quite possibly is.  That was backported into Xen 4.20.1 as 247b708b1400a, and Debian has 4.20.3

Are you able to experiment with that patch reverted?

Alternatively, are you able to experiment with the BUG() removed?

I went with dropping the BUG() rather than the revert. Reverting 247b708b1400a should stop any bank from being read at all, which is effectively what mce=false already does here, so I don't expect it to tell us anything new. I'll run it too if you'd like.

I'm not familiar with the Xen codebase, but I've built a package with mc_ec2type() changed to:

    if ( errorcode & MC_EC_TLB_TYPE )
        return MC_EC_TLB_TYPE;

    printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n", errorcode);

    return 0;

Crashes have been 12 hours to 2 weeks apart, so it may take a while before I can report back.

Quoted 7 lines

Quoted 7 lines

Diagnostics / workaround
------------------------
mce_verbosity=verbose yields nothing useful.
max_cstate=1 made no difference. mce=false is the only thing that stops it, and it is what I'm running now.

Quoted 5 lines

The fact that mce=false isn't causing machine resets means it's clearly something the hardware thinks isn't fatal.

But, I am concerned by the fact it's always on CPU 2, and multiple banks.  Is this true for both systems?

To clarify one thing from my report: it is actually a single host whose complete hardware (board, CPU, RAM) was replaced on 2026-10-07, keeping only the disks and thus the installation. The crashes happened both before and after that swap. 10 more hosts with same hardware still run on bookworm (xen 4.17) for years now without any incidents.

So yes, always CPU 2. The bank index varies: on the current hardware I have seen banks 23, 17 and 10 across four crashes, and banks 23 and 17 on the previous hardware.

Quoted 16 lines

Quoted 8 lines

One more thing: the host does not actually reboot after the panic. "Reboot in five seconds" is printed, but the ring buffer continues with normal activity minutes later and the remaining 31 CPUs keep running. CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0 and guest vCPUs start losing timer interrupts permanently (RCU stalls with a frozen timer-softirq count while dynticks keeps advancing), so VMs hang silently hours or days later.

I've recently noticed that crashing with interrupts off doesn't reboot any more.  I thought it was something local to my development, but apparently not if you're seeing the same.

Quoted 4 lines

I've attached the full xl dmesg log of the last crash, and I'm happy to provide more if needed. I can also test patches.

Quoted 6 lines

Quoted 3 lines

Thanks for your work, and thanks in advance for taking a look.

Peter

Thankyou,

~Andrew

I have full xl dmesg logs for almost all crashes. Happy to send any of them.

Peter

Quoted 103 lines

Hi Andrew,

Thanks a lot for the quick response.

Quoted 78 lines · Peter Deggendorfer

Quoted 43 lines

Hey Xen Team,

I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.

Environment
-----------
Xen:   4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1) dom0:  Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs CPU:   AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X, microcode 0x0a201210
Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems

The panic
---------
(XEN) [2026-10-19 08:58:18.727] Xen BUG at arch/x86/cpu/mcheck/mce_amd.c:100
(XEN) CPU:    2
(XEN) RIP:    e008:[<ffff82d04028a650>] mce_amd.c#mc_ec2type.part.0+0/0x10
(XEN) rbx: 0000000000000445   r12: 0000000000000011   r15:
0000000000000446
(XEN) Xen code around <ffff82d04028a650>:
(XEN)  90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90
(XEN) Xen call trace:
(XEN)    [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10 (XEN)    [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40 (XEN)    [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840 (XEN)    [<ffff82d04028a35c>] S amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
(XEN)    [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90 (XEN)    [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
(XEN)    [<ffff82d040201915>] S common_interrupt+0x135/0x150 (XEN)    [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
(XEN)    [<ffff82d0402887cf>] S cpu_idle.c#acpi_processor_idle+0x1ef/0x650
(XEN)    [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130 (XEN) Panic on CPU 2

Quoted 3 lines

The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is the "unreachable" default branch of mc_ec2type() -- the extended error code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.

I think it's safe to just drop the BUG().

That code is literally worse than useless, because if nothing else it's just hiding the diagnostics.

I guess it means AMD have added new statuses.

Quoted 14 lines

Across the eight occurrences, r12 appears to be the bank index from the loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS) and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.

Not a hardware defect
---------------------
- Started within days of upgrading two hosts from Debian bookworm (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no
   incident.
- Three different banks, so not a single failing block.
- Updating to the current Debian package (4.20.5-pre, built 2026-10-16) changed nothing.

Quoted 7 lines

Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks updating").

Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
Filter bogus hardware errors on Zen3 clients"), where bogus MCA values on Zen 3 are filtered rather than treated as errors.
https://ratatoskr.run/stable/2026/04/7143905

 From above, you're on a Vermeer, not a Cezanne, so I don't think that Linux patch is relevant.

Quoted 7 lines

However, your observation about "fix mca_allbanks updating" quite possibly is.  That was backported into Xen 4.20.1 as 247b708b1400a, and Debian has 4.20.3

Are you able to experiment with that patch reverted?

Alternatively, are you able to experiment with the BUG() removed?

I went with dropping the BUG() rather than the revert. Reverting 247b708b1400a should stop any bank from being read at all, which is effectively what mce=false already does here, so I don't expect it to tell us anything new. I'll run it too if you'd like.

I'm not familiar with the Xen codebase, but I've built a package with mc_ec2type() changed to:

    if ( errorcode & MC_EC_TLB_TYPE )
        return MC_EC_TLB_TYPE;

Quoted 8 lines

    printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n", errorcode);

    return 0;

Crashes have been 12 hours to 2 weeks apart, so it may take a while before I can report back.

After ~31 hours of uptime:

(XEN) [2026-10-25 04:06:27.260] MCE: unknown AMD error code 0

No panic, no wedged CPU, no RCU stalls, no hung guests. Host and all VMs running normally.

I also dumped /dev/mcelog in dom0 and got the following records:

00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000030: 0000 0000 0000 0000 0200 0000 0000 0000  ................ 00000040: 0010 0201 0200 0000 0000 0000 0200 0000  ................ 00000050: 1c01 0000 0000 0000 0000 0000 0000 0000  ................ 00000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 000000a0: 0000 0000 0000 0000 0200 0000 0000 0000  ................ 000000b0: 0017 0201 0200 0000 0000 0000 0200 0000  ................ 000000c0: 1c01 0000 0000 0000 0000 0000 0000 0000  ................ 000000d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

Quoted 20 lines

Quoted 7 lines

Quoted 7 lines

Diagnostics / workaround
------------------------
mce_verbosity=verbose yields nothing useful.
max_cstate=1 made no difference. mce=false is the only thing that stops it, and it is what I'm running now.

Quoted 5 lines

The fact that mce=false isn't causing machine resets means it's clearly something the hardware thinks isn't fatal.

But, I am concerned by the fact it's always on CPU 2, and multiple banks.  Is this true for both systems?

To clarify one thing from my report: it is actually a single host whose complete hardware (board, CPU, RAM) was replaced on 2026-10-07, keeping only the disks and thus the installation. The crashes happened both before and after that swap. 10 more hosts with same hardware still run on bookworm (xen 4.17) for years now without any incidents.

Quoted 3 lines

So yes, always CPU 2. The bank index varies: on the current hardware I have seen banks 23, 17 and 10 across four crashes, and banks 23 and 17 on the previous hardware.

Regarding the output of /dev/mcelog: if my decoding is right, three things stand out. MCi_STATUS is zero, i.e. these banks are reporting nothing at all -- no error, no garbage value.

Both records are again
from CPU 2, and bank 23 is one of the banks that appeared in the panics, bank 16 is new.

Quoted 153 lines

Quoted 101 lines

Hi Andrew,

Thanks a lot for the quick response.

Quoted 78 lines · Peter Deggendorfer

Quoted 43 lines

Hey Xen Team,

I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.

Environment
-----------
Xen:   4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
        present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
dom0:  Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs CPU:   AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
        microcode 0x0a201210
Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems

The panic
---------
(XEN) [2026-10-19 08:58:18.727] Xen BUG at arch/x86/cpu/mcheck/mce_amd.c:100
(XEN) CPU:    2
(XEN) RIP:    e008:[<ffff82d04028a650>] mce_amd.c#mc_ec2type.part.0+0/0x10
(XEN) rbx: 0000000000000445   r12: 0000000000000011   r15:
0000000000000446
(XEN) Xen code around <ffff82d04028a650>:
(XEN)  90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90
(XEN) Xen call trace:
(XEN)    [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10 (XEN)    [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40 (XEN)    [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840 (XEN)    [<ffff82d04028a35c>] S amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
(XEN)    [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90 (XEN)    [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
(XEN)    [<ffff82d040201915>] S common_interrupt+0x135/0x150 (XEN)    [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
(XEN)    [<ffff82d0402887cf>] S cpu_idle.c#acpi_processor_idle+0x1ef/0x650
(XEN)    [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130 (XEN) Panic on CPU 2

Quoted 3 lines

The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is the "unreachable" default branch of mc_ec2type() -- the extended error code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.

I think it's safe to just drop the BUG().

That code is literally worse than useless, because if nothing else it's just hiding the diagnostics.

I guess it means AMD have added new statuses.

Quoted 14 lines

Across the eight occurrences, r12 appears to be the bank index from the loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS) and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.

Not a hardware defect
---------------------
- Started within days of upgrading two hosts from Debian bookworm (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17
with no
   incident.
- Three different banks, so not a single failing block.
- Updating to the current Debian package (4.20.5-pre, built 2026-10-16) changed nothing.

Quoted 7 lines

Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks updating").

Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
Filter bogus hardware errors on Zen3 clients"), where bogus MCA values on Zen 3 are filtered rather than treated as errors.
https://ratatoskr.run/stable/2026/04/7143905

 From above, you're on a Vermeer, not a Cezanne, so I don't think that
Linux patch is relevant.

Quoted 7 lines

However, your observation about "fix mca_allbanks updating" quite possibly is.  That was backported into Xen 4.20.1 as 247b708b1400a, and Debian has 4.20.3

Are you able to experiment with that patch reverted?

Alternatively, are you able to experiment with the BUG() removed?

I went with dropping the BUG() rather than the revert. Reverting 247b708b1400a should stop any bank from being read at all, which is effectively what mce=false already does here, so I don't expect it to tell us anything new. I'll run it too if you'd like.

I'm not familiar with the Xen codebase, but I've built a package with mc_ec2type() changed to:

    if ( errorcode & MC_EC_TLB_TYPE )
        return MC_EC_TLB_TYPE;

Quoted 8 lines

    printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n",
errorcode);

    return 0;

Crashes have been 12 hours to 2 weeks apart, so it may take a while before I can report back.

After ~31 hours of uptime:

(XEN) [2026-10-25 04:06:27.260] MCE: unknown AMD error code 0

No panic, no wedged CPU, no RCU stalls, no hung guests. Host and all VMs running normally.

I also dumped /dev/mcelog in dom0 and got the following records:

00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000030: 0000 0000 0000 0000 0200 0000 0000 0000  ................ 00000040: 0010 0201 0200 0000 0000 0000 0200 0000  ................ 00000050: 1c01 0000 0000 0000 0000 0000 0000 0000  ................ 00000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 00000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................ 000000a0: 0000 0000 0000 0000 0200 0000 0000 0000  ................ 000000b0: 0017 0201 0200 0000 0000 0000 0200 0000  ................ 000000c0: 1c01 0000 0000 0000 0000 0000 0000 0000  ................ 000000d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

Quoted 18 lines

Quoted 11 lines

Quoted 6 lines

Diagnostics / workaround
------------------------
mce_verbosity=verbose yields nothing useful.
max_cstate=1 made no difference. mce=false is the only thing that stops it, and it is what I'm running now.

The fact that mce=false isn't causing machine resets means it's clearly something the hardware thinks isn't fatal.

But, I am concerned by the fact it's always on CPU 2, and multiple banks.  Is this true for both systems?

To clarify one thing from my report: it is actually a single host whose complete hardware (board, CPU, RAM) was replaced on 2026-10-07, keeping only the disks and thus the installation. The crashes happened both before and after that swap. 10 more hosts with same hardware still run on bookworm (xen 4.17) for years now without any incidents.

Quoted 6 lines

Quoted 3 lines

So yes, always CPU 2. The bank index varies: on the current hardware I have seen banks 23, 17 and 10 across four crashes, and banks 23 and 17 on the previous hardware.

Regarding the output of /dev/mcelog: if my decoding is right, three things stand out. MCi_STATUS is zero, i.e. these banks are reporting nothing at all -- no error, no garbage value.

Well, things can't quite be that way: If MCi_STATUS was zero, the first check in mc_amd_recoverable_scan() would prevent mc_ec2type() from being reached in the first place. Its caller also checks MCi_STATUS_VAL to be set.

What I find puzzling is this:

     errorcode = status & (MCi_STATUS_MCA | MCi_STATUS_MSEC);
     ectype = mc_ec2type(errorcode);

With errorcode (and mc_ec2type()'s parameter type) being uint16_t, what use is it to include MCi_STATUS_MSEC in the mask? (Yet that's likely unrelated to the issue at hand.) Considering how mc_ec2type() works, I see no reason why status couldn't be passed to it directly (with the parameter type widened to uint32_t). Then the BUG() there could perhaps become BUG_ON(errorcode).

Quoted 3 lines

Both records are again
from CPU 2, and bank 23 is one of the banks that appeared in the panics, bank 16 is new.

Isn't it possible that the lack of 532988dccc19 ("x86/mcheck: fix mca_allbanks updating") simply hid the issue previously?

Am 26.10.2026 um 11:53 schrieb Jan Beulich:

Quoted 14 lines · Peter Deggendorfer

Quoted 3 lines

Regarding the output of /dev/mcelog: if my decoding is right, three things stand out. MCi_STATUS is zero, i.e. these banks are reporting nothing at all -- no error, no garbage value.

Well, things can't quite be that way: If MCi_STATUS was zero, the first check in mc_amd_recoverable_scan() would prevent mc_ec2type() from being reached in the first place. Its caller also checks MCi_STATUS_VAL to be set.

What I find puzzling is this:

     errorcode = status & (MCi_STATUS_MCA | MCi_STATUS_MSEC);
     ectype = mc_ec2type(errorcode);

Quoted 6 lines

With errorcode (and mc_ec2type()'s parameter type) being uint16_t, what use is it to include MCi_STATUS_MSEC in the mask? (Yet that's likely unrelated to the issue at hand.) Considering how mc_ec2type() works, I see no reason why status couldn't be passed to it directly (with the parameter type widened to uint32_t). Then the BUG() there could perhaps become BUG_ON(errorcode).

You're right, my conclusion was wrong.

One more data point, from an hour ago:

(XEN) [2026-10-26 13:13:42.050] MCE: unknown AMD error code 0x1000

So it isn't always 0. The uint16_t truncation might explain the first (MSEC bits dropped), but 0x1000 is a non-zero code that still matches none of MC_EC_BUS_TYPE / MC_EC_MEM_TYPE / MC_EC_TLB_TYPE.

/dev/mcelog right afterwards showed three records, banks 17, 14 and 16, all CPU 2 (status zero, as above). Banks seen so far across panics and log lines: 10, 14, 16, 17, 23.

With the BUG() dropped, the host has now logged this twice with no panic, no wedged CPU and no hung guests. Everything is running normally.

Regarding passing status as uint32_t and BUG_ON(errorcode): with today's 0x1000, bits 0-31 are non-zero but none of the type bits is set, so that BUG_ON would still trigger. So unless 0x1000 is a known type that mc_ec2type() should handle, I think the poller shouldn't BUG on an unclassifiable code at all.

Quoted 5 lines

Quoted 3 lines

Both records are again
from CPU 2, and bank 23 is one of the banks that appeared in the panics, bank 16 is new.

Isn't it possible that the lack of 532988dccc19 ("x86/mcheck: fix mca_allbanks updating") simply hid the issue previously?

That is exactly what I suspect. This host, and 10 more with identical hardware, ran for years on Xen 4.17 without a single incident. The first panic came within days of upgrading this one to 4.20; the others are still on 4.17 and still fine. If no bank was being collected before, nothing could have reached mc_ec2type() -- so the condition may well have been there the whole time, just never looked at.

Would it help if I logged the full MCi_STATUS of the first read, i.e. in mcheck_mca_logout() right after the MCi_STATUS_VAL check, together with CPU and bank number?

Quoted 39 lines

Am 26.10.2026 um 11:53 schrieb Jan Beulich:

Quoted 14 lines · Peter Deggendorfer

Quoted 3 lines

Regarding the output of /dev/mcelog: if my decoding is right, three things stand out. MCi_STATUS is zero, i.e. these banks are reporting nothing at all -- no error, no garbage value.

Well, things can't quite be that way: If MCi_STATUS was zero, the first check in mc_amd_recoverable_scan() would prevent mc_ec2type() from being reached in the first place. Its caller also checks MCi_STATUS_VAL to be set.

What I find puzzling is this:

      errorcode = status & (MCi_STATUS_MCA | MCi_STATUS_MSEC);
      ectype = mc_ec2type(errorcode);

Quoted 6 lines

With errorcode (and mc_ec2type()'s parameter type) being uint16_t, what use is it to include MCi_STATUS_MSEC in the mask? (Yet that's likely unrelated to the issue at hand.) Considering how mc_ec2type() works, I see no reason why status couldn't be passed to it directly (with the parameter type widened to uint32_t). Then the BUG() there could perhaps become BUG_ON(errorcode).

You're right, my conclusion was wrong.

One more data point, from an hour ago:

(XEN) [2026-10-26 13:13:42.050] MCE: unknown AMD error code 0x1000

So it isn't always 0. The uint16_t truncation might explain the first (MSEC bits dropped), but 0x1000 is a non-zero code that still matches none of MC_EC_BUS_TYPE / MC_EC_MEM_TYPE / MC_EC_TLB_TYPE.

/dev/mcelog right afterwards showed three records, banks 17, 14 and 16, all CPU 2 (status zero, as above). Banks seen so far across panics and log lines: 10, 14, 16, 17, 23.

With the BUG() dropped, the host has now logged this twice with no panic, no wedged CPU and no hung guests. Everything is running normally.

Quoted 5 lines

Regarding passing status as uint32_t and BUG_ON(errorcode): with today's 0x1000, bits 0-31 are non-zero but none of the type bits is set, so that BUG_ON would still trigger. So unless 0x1000 is a known type that mc_ec2type() should handle, I think the poller shouldn't BUG on an unclassifiable code at all.

I disagree. That code, aiui, is to decide whether something the hardware reported as uncorrectable can still be recovered from. Seeing an error code of unknown type therefore has to be treated as fatal. Whether that is by keeping that BUG() or by some other means is secondary; continuing to run Xen is putting all of the system's data at risk, even if in your particular instance(s) you didn't observe further anomalies.

Quoted 12 lines

Quoted 5 lines

Quoted 3 lines

Both records are again
from CPU 2, and bank 23 is one of the banks that appeared in the panics, bank 16 is new.

Isn't it possible that the lack of 532988dccc19 ("x86/mcheck: fix mca_allbanks updating") simply hid the issue previously?

That is exactly what I suspect. This host, and 10 more with identical hardware, ran for years on Xen 4.17 without a single incident. The first panic came within days of upgrading this one to 4.20; the others are still on 4.17 and still fine. If no bank was being collected before, nothing could have reached mc_ec2type() -- so the condition may well have been there the whole time, just never looked at.

Quoted 3 lines

Would it help if I logged the full MCi_STATUS of the first read, i.e. in mcheck_mca_logout() right after the MCi_STATUS_VAL check, together with CPU and bank number?

Yes please.

As requested, I added a printk to mcheck_mca_logout(), and the BUG() in mc_ec2type() is still replaced. These are the only changes against the Debian package:

xen/arch/x86/cpu/mcheck/mce_amd.c +1 -1

@@ -97,7 +97,7 @@
      if ( errorcode & MC_EC_TLB_TYPE )
          return MC_EC_TLB_TYPE;
      /* Unreached */
-    BUG();
+    printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n", errorcode);
      return 0;
  }
 

xen/arch/x86/cpu/mcheck/mce.c +3 -0

@@ -304,3 +304,6 @@
          if ( !(status & MCi_STATUS_VAL) )
              continue; /* this bank has no valid telemetry */
 
+        if ( who == MCA_POLLER )
+            printk(XENLOG_WARNING "MCE: CPU%u bank %d status %#"PRIx64"\n",
+                   smp_processor_id(), i, status);

We now have some data points:

(XEN) [2026-10-27 13:42:10.588] MCE: CPU2 bank 23 status 0x830824448b48ffff (XEN) [2026-10-27 18:19:55.593] MCE: CPU2 bank 23 status 0xffffffffffffffff (XEN) [2026-10-27 22:11:10.596] MCE: CPU2 bank 17 status 0xffff832036a40048 (XEN) [2026-10-27 22:11:10.596] MCE: unknown AMD error code 0x48 (XEN) [2026-10-28 00:33:55.597] MCE: CPU2 bank 10 status 0xffff82d04022a8af (XEN) [2026-10-28 05:26:40.599] MCE: CPU2 bank 17 status 0xffff82d0402408e3 (XEN) [2026-10-29 23:17:25.618] MCE: CPU2 bank 17 status 0xffff82d04022a8af (XEN) [2026-10-30 03:25:10.622] MCE: CPU2 bank 23 status 0xffffffffffffffff (XEN) [2026-10-30 10:05:55.625] MCE: CPU2 bank 10 status 0xffffffffffffffff (XEN) [2026-10-31 00:58:10.625] MCE: CPU2 bank 17 status 0xffff888100b872e8 (XEN) [2026-10-31 00:58:10.625] MCE: unknown AMD error code 0x72e8 (XEN) [2026-10-31 04:17:55.632] MCE: CPU2 bank 23 status 0xffff831f00006000 (XEN) [2026-10-31 04:17:55.632] MCE: unknown AMD error code 0x6000

Only 3 of these would have hit the BUG(); the others have UC clear or an error code that happens to contain a type bit.

The values look like pointers rather than status registers:

  ffff82d040......  Xen text/data -- the prefix of every RIP in my earlier panic dumps
  ffff831f........ / ffff8320........  same range as rsp in those dumps (e.g. ffff831fe6b3fc38, ffff832036a87c38) ffff888100b872e8  dom0 Linux range?

Two things that make me doubt these are real status values at all:
0xffff82d04022a8af appears twice, 47 hours apart, for two different banks (10 and 17). And all timestamps are congruent mod 15 seconds, i.e. this is the poller hitting it on a small fraction of passes.

I don't know where the read goes wrong.

/dev/mcelog records match the log lines one-to-one (same banks, same order, all CPU 2), but carry status zero in every case. The record's status comes from the second read in mca_init_bank(), before mcabank_clear(), so that second read returned 0.

Quoted 45 lines

As requested, I added a printk to mcheck_mca_logout(), and the BUG() in mc_ec2type() is still replaced. These are the only changes against the Debian package:

xen/arch/x86/cpu/mcheck/mce_amd.c +1 -1

@@ -97,7 +97,7 @@
       if ( errorcode & MC_EC_TLB_TYPE )
           return MC_EC_TLB_TYPE;
       /* Unreached */
-    BUG();
+    printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n", errorcode);
       return 0;
   }
 

xen/arch/x86/cpu/mcheck/mce.c +3 -0

@@ -304,3 +304,6 @@
           if ( !(status & MCi_STATUS_VAL) )
               continue; /* this bank has no valid telemetry */
 
+        if ( who == MCA_POLLER )
+            printk(XENLOG_WARNING "MCE: CPU%u bank %d status %#"PRIx64"\n",
+                   smp_processor_id(), i, status);

We now have some data points:

(XEN) [2026-10-27 13:42:10.588] MCE: CPU2 bank 23 status 0x830824448b48ffff (XEN) [2026-10-27 18:19:55.593] MCE: CPU2 bank 23 status 0xffffffffffffffff (XEN) [2026-10-27 22:11:10.596] MCE: CPU2 bank 17 status 0xffff832036a40048 (XEN) [2026-10-27 22:11:10.596] MCE: unknown AMD error code 0x48 (XEN) [2026-10-28 00:33:55.597] MCE: CPU2 bank 10 status 0xffff82d04022a8af (XEN) [2026-10-28 05:26:40.599] MCE: CPU2 bank 17 status 0xffff82d0402408e3 (XEN) [2026-10-29 23:17:25.618] MCE: CPU2 bank 17 status 0xffff82d04022a8af (XEN) [2026-10-30 03:25:10.622] MCE: CPU2 bank 23 status 0xffffffffffffffff (XEN) [2026-10-30 10:05:55.625] MCE: CPU2 bank 10 status 0xffffffffffffffff (XEN) [2026-10-31 00:58:10.625] MCE: CPU2 bank 17 status 0xffff888100b872e8 (XEN) [2026-10-31 00:58:10.625] MCE: unknown AMD error code 0x72e8 (XEN) [2026-10-31 04:17:55.632] MCE: CPU2 bank 23 status 0xffff831f00006000 (XEN) [2026-10-31 04:17:55.632] MCE: unknown AMD error code 0x6000

Only 3 of these would have hit the BUG(); the others have UC clear or an error code that happens to contain a type bit.

Quoted 7 lines

The values look like pointers rather than status registers:

  ffff82d040......  Xen text/data -- the prefix of every RIP in my earlier panic dumps
  ffff831f........ / ffff8320........  same range as rsp in those dumps (e.g. ffff831fe6b3fc38, ffff832036a87c38) ffff888100b872e8  dom0 Linux range?

Plus the very first of the entries above is a pointer value rotated by 16 bits.

Quoted 4 lines

Two things that make me doubt these are real status values at all:
0xffff82d04022a8af appears twice, 47 hours apart, for two different banks (10 and 17). And all timestamps are congruent mod 15 seconds, i.e. this is the poller hitting it on a small fraction of passes.

There's no question that the values are garbage. We just need to find out why that is. As we have to explore all (remote) possibilities, first of all - you don't play any games involving the XEN_MC_msrinject hypercall, do you? For the immediate purpose here, and since ...

Quoted 1 line

I don't know where the read goes wrong.

... I can't make sense of the behavior either, I'm inclined to suggest that you add this hunk to your debugging patch:

xen/arch/x86/cpu/mcheck/mce.h +0 -1

@@ -79,7 +79,6 @@
  static inline uint64_t mca_rdmsr(unsigned int msr)
  {
      uint64_t val;
-    if (intpose_lookup(smp_processor_id(), msr, &val) == NULL)
          rdmsrl(msr, val);
      return val;
  }

Quoted 5 lines

/dev/mcelog records match the log lines one-to-one (same banks, same order, all CPU 2), but carry status zero in every case. The record's status comes from the second read in mca_init_bank(), before mcabank_clear(), so that second read returned 0.

Based on this observation, another thing you may want to try is extend the existing hunk you have to

xen/arch/x86/cpu/mcheck/mce.c +4 -0

@@ -304,6 +304,17 @@
           if ( !(status & MCi_STATUS_VAL) )
               continue; /* this bank has no valid telemetry */
 
+        if ( who == MCA_POLLER )
+        {
+            printk("MCE: CPU%u bank %d status %#"PRIx64"\n",
+                   smp_processor_id(), i, status);

Am 02.11.2026 um 13:54 schrieb Jan Beulich:

Quoted 55 lines · Peter Deggendorfer

Quoted 45 lines

As requested, I added a printk to mcheck_mca_logout(), and the BUG() in mc_ec2type() is still replaced. These are the only changes against the Debian package:

xen/arch/x86/cpu/mcheck/mce_amd.c +1 -1

@@ -97,7 +97,7 @@
       if ( errorcode & MC_EC_TLB_TYPE )
           return MC_EC_TLB_TYPE;
       /* Unreached */
-    BUG();
+    printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n", errorcode);
       return 0;
   }
 

xen/arch/x86/cpu/mcheck/mce.c +3 -0

@@ -304,3 +304,6 @@
           if ( !(status & MCi_STATUS_VAL) )
               continue; /* this bank has no valid telemetry */
 
+        if ( who == MCA_POLLER )
+            printk(XENLOG_WARNING "MCE: CPU%u bank %d status %#"PRIx64"\n",
+                   smp_processor_id(), i, status);

We now have some data points:

(XEN) [2026-10-27 13:42:10.588] MCE: CPU2 bank 23 status 0x830824448b48ffff (XEN) [2026-10-27 18:19:55.593] MCE: CPU2 bank 23 status 0xffffffffffffffff (XEN) [2026-10-27 22:11:10.596] MCE: CPU2 bank 17 status 0xffff832036a40048 (XEN) [2026-10-27 22:11:10.596] MCE: unknown AMD error code 0x48 (XEN) [2026-10-28 00:33:55.597] MCE: CPU2 bank 10 status 0xffff82d04022a8af (XEN) [2026-10-28 05:26:40.599] MCE: CPU2 bank 17 status 0xffff82d0402408e3 (XEN) [2026-10-29 23:17:25.618] MCE: CPU2 bank 17 status 0xffff82d04022a8af (XEN) [2026-10-30 03:25:10.622] MCE: CPU2 bank 23 status 0xffffffffffffffff (XEN) [2026-10-30 10:05:55.625] MCE: CPU2 bank 10 status 0xffffffffffffffff (XEN) [2026-10-31 00:58:10.625] MCE: CPU2 bank 17 status 0xffff888100b872e8 (XEN) [2026-10-31 00:58:10.625] MCE: unknown AMD error code 0x72e8 (XEN) [2026-10-31 04:17:55.632] MCE: CPU2 bank 23 status 0xffff831f00006000 (XEN) [2026-10-31 04:17:55.632] MCE: unknown AMD error code 0x6000

Only 3 of these would have hit the BUG(); the others have UC clear or an error code that happens to contain a type bit.

Quoted 7 lines

The values look like pointers rather than status registers:

  ffff82d040......  Xen text/data -- the prefix of every RIP in my earlier panic dumps
  ffff831f........ / ffff8320........  same range as rsp in those dumps (e.g. ffff831fe6b3fc38, ffff832036a87c38) ffff888100b872e8  dom0 Linux range?

Plus the very first of the entries above is a pointer value rotated by 16 bits.

Quoted 8 lines

Quoted 4 lines

Two things that make me doubt these are real status values at all:
0xffff82d04022a8af appears twice, 47 hours apart, for two different banks (10 and 17). And all timestamps are congruent mod 15 seconds, i.e. this is the poller hitting it on a small fraction of passes.

There's no question that the values are garbage. We just need to find out why that is. As we have to explore all (remote) possibilities, first of all - you don't play any games involving the XEN_MC_msrinject hypercall, do you? For the immediate purpose here, and since ...

No, I never used XEN_MC_msrinject hypercall on this host; it's a plain Debian trixie installation.

Quoted 4 lines

Quoted 1 line

I don't know where the read goes wrong.

... I can't make sense of the behavior either, I'm inclined to suggest that you add this hunk to your debugging patch:

xen/arch/x86/cpu/mcheck/mce.h line 84

Quoted 10 lines

xen/arch/x86/cpu/mcheck/mce.h +0 -1

@@ -79,7 +79,6 @@
  static inline uint64_t mca_rdmsr(unsigned int msr)
  {
      uint64_t val;
-    if (intpose_lookup(smp_processor_id(), msr, &val) == NULL)
          rdmsrl(msr, val);
      return val;
  }

Quoted 18 lines

Quoted 4 lines

/dev/mcelog records match the log lines one-to-one (same banks, same order, all CPU 2), but carry status zero in every case. The record's status comes from the second read in mca_init_bank(), before mcabank_clear(), so that second read returned 0.

Based on this observation, another thing you may want to try is extend the existing hunk you have to

xen/arch/x86/cpu/mcheck/mce.c +4 -0

@@ -304,6 +304,17 @@
           if ( !(status & MCi_STATUS_VAL) )
               continue; /* this bank has no valid telemetry */
 
+        if ( who == MCA_POLLER )
+        {
+            printk("MCE: CPU%u bank %d status %#"PRIx64"\n",
+                   smp_processor_id(), i, status);

xen/arch/x86/cpu/mcheck/mce.c line 320

Quoted 10 lines

xen/arch/x86/cpu/mcheck/mce.c +7 -0

+            status = mca_rdmsr(MSR_IA32_MCx_STATUS(i));
+            printk("MCE: CPU%u bank %d -> %#"PRIx64"\n",
+                   smp_processor_id(), i, status);
+            if ( !(status & MCi_STATUS_VAL) )
+                continue;
+        }
+
          /*
           * For Intel Latest CPU CMCI/MCE Handler caller, we need to
           * decide whether to clear bank by MCi_STATUS bit value such as

Since 2026-11-03 the host runs Xen 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1) with your two debugging hunks (intpose_lookup() removed from mca_rdmsr(), re-read in the poller) and your "x86/MCE: reduce redundant code" series (both patches). In 7 days:

(XEN) [2026-11-03 13:54:50.378] MCE: CPU2 bank 25 status 0x8000001fe67e1163 (XEN) [2026-11-03 13:54:50.378] MCE: CPU2 bank 25 -> 0
(XEN) [2026-11-03 14:52:20.360] MCE: CPU2 bank 7 status 0x8000000080000000 (XEN) [2026-11-03 14:52:20.360] MCE: CPU2 bank 7 -> 0
(XEN) [2026-11-03 20:01:20.264] MCE: CPU2 bank 25 status 0xffff82d04022a8ef (XEN) [2026-11-03 20:01:20.264] MCE: CPU2 bank 25 -> 0
(XEN) [2026-11-05 14:31:19.276] MCE: CPU2 bank 25 status 0x8000001fe67e1163 (XEN) [2026-11-05 14:31:19.276] MCE: CPU2 bank 25 -> 0
(XEN) [2026-11-06 11:27:18.812] MCE: CPU2 bank 24 status 0xffff832036a754c0 (XEN) [2026-11-06 11:27:18.812] MCE: CPU2 bank 24 -> 0
(XEN) [2026-11-06 18:26:48.671] MCE: CPU2 bank 7 status 0xffff831fe6b154c0 (XEN) [2026-11-06 18:26:48.671] MCE: CPU2 bank 7 -> 0
(XEN) [2026-11-07 06:31:48.345] MCE: CPU2 bank 16 status 0xffff8313f11d2010 (XEN) [2026-11-07 06:31:48.345] MCE: CPU2 bank 16 -> 0
(XEN) [2026-11-07 09:41:48.243] MCE: CPU2 bank 16 status 0xffffffffffffffff (XEN) [2026-11-07 09:41:48.243] MCE: CPU2 bank 16 -> 0
(XEN) [2026-11-07 21:32:17.875] MCE: CPU2 bank 7 status 0xffff83203a5ea2d0 (XEN) [2026-11-07 21:32:17.875] MCE: CPU2 bank 7 -> 0
(XEN) [2026-11-08 06:25:47.633] MCE: CPU2 bank 16 status 0xffffffffffffffff (XEN) [2026-11-08 06:25:47.633] MCE: CPU2 bank 16 -> 0

(XEN) [2026-11-08 15:11:17.390] MCE: CPU2 bank 16 status 0xffff82d040588f80 (XEN) [2026-11-08 15:11:17.390] MCE: CPU2 bank 16 -> 0
(XEN) [2026-11-09 07:29:46.980] MCE: CPU2 bank 16 status 0xffffffffffffffff (XEN) [2026-11-09 07:29:46.980] MCE: CPU2 bank 16 -> 0
(XEN) [2026-11-09 10:49:46.880] MCE: CPU2 bank 7 status 0xffffffffffffffff (XEN) [2026-11-09 10:49:46.880] MCE: CPU2 bank 7 -> 0
(XEN) [2026-11-09 13:09:16.822] MCE: CPU2 bank 7 status 0xffffffffffffffff (XEN) [2026-11-09 13:09:16.822] MCE: CPU2 bank 7 -> 0

We now have some data points:

(XEN) [2026-10-27 13:42:10.588] MCE: CPU2 bank 23 status 0x830824448b48ffff (XEN) [2026-10-27 18:19:55.593] MCE: CPU2 bank 23 status 0xffffffffffffffff (XEN) [2026-10-27 22:11:10.596] MCE: CPU2 bank 17 status 0xffff832036a40048 (XEN) [2026-10-27 22:11:10.596] MCE: unknown AMD error code 0x48 (XEN) [2026-10-28 00:33:55.597] MCE: CPU2 bank 10 status 0xffff82d04022a8af (XEN) [2026-10-28 05:26:40.599] MCE: CPU2 bank 17 status 0xffff82d0402408e3 (XEN) [2026-10-29 23:17:25.618] MCE: CPU2 bank 17 status 0xffff82d04022a8af (XEN) [2026-10-30 03:25:10.622] MCE: CPU2 bank 23 status 0xffffffffffffffff (XEN) [2026-10-30 10:05:55.625] MCE: CPU2 bank 10 status 0xffffffffffffffff (XEN) [2026-10-31 00:58:10.625] MCE: CPU2 bank 17 status 0xffff888100b872e8 (XEN) [2026-10-31 00:58:10.625] MCE: unknown AMD error code 0x72e8 (XEN) [2026-10-31 04:17:55.632] MCE: CPU2 bank 23 status 0xffff831f00006000 (XEN) [2026-10-31 04:17:55.632] MCE: unknown AMD error code 0x6000

Only 3 of these would have hit the BUG(); the others have UC clear or an error code that happens to contain a type bit.

The values look like pointers rather than status registers:

  ffff82d040......  Xen text/data -- the prefix of every RIP in my earlier panic dumps
  ffff831f........ / ffff8320........  same range as rsp in those dumps (e.g. ffff831fe6b3fc38, ffff832036a87c38) ffff888100b872e8  dom0 Linux range?
  ffffffffffffffff  looks like a failed read?

Two things that make me doubt these are real status values at all:
0xffff82d04022a8af appears twice, 47 hours apart, for two different banks (10 and 17). And all timestamps are congruent mod 15 seconds, i.e. this is the poller hitting it on a small fraction of passes.

I don't know where the read goes wrong.

/dev/mcelog records match the log lines one-to-one (same banks, same order, all CPU 2), but carry status zero in every case. The record's status comes from the second read in mca_init_bank(), before mcabank_clear(), so that second read returned 0.

Hey Xen Team,

I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.

Environment
-----------
Xen:   4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
       present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
dom0:  Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs CPU:   AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
       microcode 0x0a201210
Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems

The panic
---------
(XEN) [2026-10-19 08:58:18.727] Xen BUG at arch/x86/cpu/mcheck/ mce_amd.c:100 (XEN) CPU:    2
(XEN) RIP:    e008:[<ffff82d04028a650>] mce_amd.c#mc_ec2type.part.0+0/0x10 (XEN) rbx: 0000000000000445   r12: 0000000000000011   r15: 0000000000000446 (XEN) Xen code around <ffff82d04028a650>:
(XEN)  90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90 (XEN) Xen call trace:
(XEN)    [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10 (XEN)    [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40 (XEN)    [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840 (XEN)    [<ffff82d04028a35c>] S amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100 (XEN)    [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90 (XEN)    [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
(XEN)    [<ffff82d040201915>] S common_interrupt+0x135/0x150 (XEN)    [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140 (XEN)    [<ffff82d0402887cf>] S cpu_idle.c#acpi_processor_idle+0x1ef/0x650 (XEN)    [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130 (XEN) Panic on CPU 2

The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is the "unreachable" default branch of mc_ec2type() -- the extended error code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.

Across the eight occurrences, r12 appears to be the bank index from the loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS) and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.

Not a hardware defect
---------------------
- Started within days of upgrading two hosts from Debian bookworm (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no incident.
- Three different banks, so not a single failing block.
- Updating to the current Debian package (4.20.5-pre, built 2026-10-16) changed nothing.

Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks updating").

Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
Filter bogus hardware errors on Zen3 clients"), where bogus MCA values on Zen 3 are filtered rather than treated as errors.
https://ratatoskr.run/stable/2026/04/7143905

Diagnostics / workaround
------------------------
mce_verbosity=verbose yields nothing useful.
max_cstate=1 made no difference. mce=false is the only thing that stops it, and it is what I'm running now.

One more thing: the host does not actually reboot after the panic. "Reboot in five seconds" is printed, but the ring buffer continues with normal activity minutes later and the remaining 31 CPUs keep running. CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0 and guest vCPUs start losing timer interrupts permanently (RCU stalls with a frozen timer-softirq count while dynticks keeps advancing), so VMs hang silently hours or days later.

I've attached the full xl dmesg log of the last crash, and I'm happy to provide more if needed. I can also test patches.

Thanks for your work, and thanks in advance for taking a look.

Peter

[39230.481809] [Hardware Error]: System Fatal error.
[39230.481813] [Hardware Error]: CPU:1 (19:21:2) MC12_STATUS[Over|UE|MiscV|AddrV|PCC|SyndV|UECC|Deferred|Poison|Scrub]: 0xffffffffb8616890
[39230.481823] [Hardware Error]: Error Addr: 0x0000000000000000
[39230.481825] [Hardware Error]: IPID: 0x0000000000000000, Syndrome: 0x0000000000000000
[39230.481830] [Hardware Error]: Bank 12 is reserved.
[39230.481832] [Hardware Error]: cache level: RESV, tx: INSN