cxl/mce: Only act on uncorrected memory errors
Subsystems (3)
PATCH 1/1 cxl/mce: Only act on uncorrected memory errors
cxl_handle_mce() offlines the aliased page of an ELC region on any record with a usable address; it does not check MCI_STATUS_UC or filter non-memory errors. uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters on mce->severity and leaves corrected errors untouched. cxl_handle_mce() has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired via memory_failure().
Corrected errors do reach the chain: machine_check_poll() logs them via the same mce_gen_pool_process() path that feeds x86_mce_decoder_chain, and cxl_extended_linear_cache_resize() extends p->res to cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.
Filter the record as nfit_handle_mce() does. Commit fc08a4703a41 ("acpi, nfit: Fix the memory error check in nfit_handle_mce()") and commit 5d96c9342c23 ("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler calls memory_failure() rather than recording a bad block.
mce_is_correctable() is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined on MCE_AR_SEVERITY, where kill_me_maybe() owns the reported page but nothing owns the alias.
---
Reproduced under QEMU/vng. QEMU's cxl-type3 does not emulate the HMAT extended-linear address_mode bit, so ELC was forced locally for testing via a one-line debug hack in cxl_region_probe() (not part of this patch):
if (!p->cache_size && p->res) p->cache_size = resource_size(p->res) / 2;
Steps:
1. Boot with an Intel CPU model under TCG (KVM host-passthrough will
otherwise leak the host's real vendor ID, and AMD/SMCA takes a
different mce_usable_address() path than the one under test):
vng -v -r ./arch/x86/boot/bzImage --disable-kvm --qemu-opts='-cpu Skylake-Server-v4,+mce,+mca -m 4G -machine q35,cxl=on -object memory-backend-ram,id=cxl-mem0,size=512M -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.0 -device cxl-rp,port=0,bus=cxl.0,id=root_port0,chassis=0,slot=0 -device cxl-type3,bus=root_port0,volatile-memdev=cxl-mem0,id=cxl-mem-device0 -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=512M'
2. modprobe mce-inject
$cxl list -M
$cxl list -D
$cxl create-region -m mem0 -d decoder0.0 -w 1 -g 256 -t ram $dmesg | grep "DEBUG: forced cache_size"
$modprobe device_dax
$modprobe kmem
$ls /sys/bus/dax/devices/
$daxctl reconfigure-device dax0.0 --mode=system-ram $lsmem
it will show as below:
dax0.0
[ 42.857642] Fallback order for Node 0: 0
[ 42.857828] Built 1 zonelists, mobility grouping on. Total pages: 1010805
[ 42.858446] Policy zone: Normal
[
{
"chardev":"dax0.0",
"size":536870912,
"target_node":0,
"align":2097152,
"mode":"system-ram",
"online_memblocks":4,
"total_memblocks":4,
"movable":true }
]
reconfigured 1 device
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff 2G online yes 0-15 0x0000000100000000-0x000000017fffffff 2G online yes 32-47 0x0000000190000000-0x00000001afffffff 512M online yes 50-53
Memory block size: 128M Total online memory: 4.5G Total offline memory: 0B
3. Load the injector if not loaded earlier and derive the two MCi_STATUS values.
modprobe mce-inject
MCi_STATUS bit layout used here (arch/x86/include/asm/mce.h):
bit 63 VAL - record valid
bit 61 UC - uncorrected (0 = corrected error under test)
bit 60 EN - error reporting enabled
bit 59 MISCV - MCi_MISC valid
bit 58 ADDRV - MCi_ADDR valid
bits[15:0] MCACOD - bit 7 set = memory-error signature, per
mce_is_memory_error()'s Intel branch
python3 -c "
VAL, UC, EN, MISCV, ADDRV = 1<<63, 1<<61, 1<<60, 1<<59, 1<<58
MCACOD_MEM = 1<<7 # memory error signature
ce = VAL | EN | MISCV | ADDRV | MCACOD_MEM
uc = ce | UC
print(f'CE status = {hex(ce)}')
print(f'UC status = {hex(uc)}')"
# CE status = 0x9c00000000000080
# UC status = 0xbc00000000000080
MCi_MISC: address-mode field, bits[8:6], must be 2 (physical):
python3 -c "print(hex(2 << 6))"
# misc = 0x80
4. $ cd /sys/kernel/debug/mce-inject
$echo sw > flags
$echo 0x9C00000000000080 > status
$echo 0x80 > misc
$echo 0x190010000 > addr
$echo 9 > bank
[ 221.813395] mce: [Hardware Error]: Machine check events logged [ 221.814716] cxl_mce_debug: entered status=0x9c00000000000080 addr=0x190010000 cache_size=0x10000000 res=[mem 0x190000000-0x1afffffff flags 0x200] usable=1 [ 221.816200] cxl_mce_debug: spa=0x190010000 contains=1 [ 221.817089] cxl_mce_debug: spa_alias=0x1a0010000 pfn=0x1a0010 pfn_valid=1 [ 221.817655] cxl_region region0: Offlining aliased SPA address0: 0x1a0010000 [ 221.821504] Memory failure: 0x1a0010: recovery action for free buddy page: Recovered [ 221.823903] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9: 9c00000000000080 [ 221.824617] mce: [Hardware Error]: TSC a605ccda40 ADDR 190010000 MISC 80 [ 221.824997] mce: [Hardware Error]: PROCESSOR 0:50654 TIME 1786375908 SOCKET 0 APIC 0 microcode 1 root@virtme-ng:/sys/kernel/debug/mce-inject# root@virtme-ng:/sys/kernel/debug/mce-inject# grep HardwareCorrupted /proc/meminfo HardwareCorrupted: 4 kB
CE without this patch:
cxl_region region0: Offlining aliased SPA address0: 0x1a0010000 Memory failure: 0x1a0010: recovery action for free buddy page: Recovered
HardwareCorrupted: 4 kB
-------------------------------------
CE with this patch:
(no "Offlining aliased SPA" message logged)
HardwareCorrupted: 0 kB
$cd /sys/kernel/debug/mce-inject
$echo sw > flags
$echo 0x9C00000000000080 > status
$echo 0x80 > misc
$echo 0x190010000 > addr
$echo 9 > bank
addr bank cpu flags ipid misc README status synd
[ 61.681659] mce: [Hardware Error]: Machine check events logged [ 61.683442] cxl_mce_debug: entered status=0x9c00000000000080 addr=0x190010000 cache_size=0x10000000 res=[mem 0x190000000-0x1afffffff flags 0x200] usable=1 [ 61.684528] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9: 9c00000000000080 [ 61.684974] mce: [Hardware Error]: TSC 2eeae32fe0 ADDR 190010000 MISC 80 [ 61.688880] mce: [Hardware Error]: PROCESSOR 0:50654 TIME 1786377156 SOCKET 0 APIC 0 microcode 1 [ 86.504456] clocksource: Watchdog remote CPU 11 read timed out
$grep HardwareCorrupted /proc/meminfo
HardwareCorrupted: 0 kB
-------------------------
For UC, same steps only status bit information will change :
$cd /sys/kernel/debug/mce-inject
$echo sw > flags
$echo 0xbc00000000000080 > status
$echo 0x80 > misc
$echo 0x190010000 > addr
$echo 9 > bank
$dmesg | grep cxl_mce_debug
$grep HardwareCorrupted /proc/meminfo
[ 313.398747] mce: [Hardware Error]: Machine check events logged [ 313.400518] cxl_mce_debug: entered status=0xbc00000000000080 addr=0x190010000 cache_size=0x10000000 res=[mem 0x190000000-0x1afffffff flags 0x200] usable=1 [ 313.401727] cxl_mce_debug: spa=0x190010000 contains=1 [ 313.402157] cxl_mce_debug: spa_alias=0x1a0010000 pfn=0x1a0010 pfn_valid=1 [ 313.402663] cxl_region region0: Offlining aliased SPA address0: 0x1a0010000 [ 313.405999] Memory failure: 0x1a0010: recovery action for free buddy page: Recovered [ 313.408375] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9: bc00000000000080 [ 313.408999] mce: [Hardware Error]: TSC e9dd2869e0 ADDR 190010000 MISC 80 [ 313.409367] mce: [Hardware Error]: PROCESSOR 0:50654 TIME 1786377750 SOCKET 0 APIC 0 microcode 1
Patched, UC: alias still offlined, confirming uncorrected handling is unchanged by this patch.
cxl_region region0: Offlining aliased SPA address0: 0x1a0010000
Memory failure: 0x1a0010: recovery action for free buddy page: Recovered
HardwareCorrupted: 4 kBdrivers/cxl/core/mce.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/cxl/core/mce.c +8 -1
@@ -18,7 +18,14 @@ static int cxl_handle_mce(struct notifier_block *nb, unsigned long val,u64 spa, spa_alias;unsigned long pfn; -if (!mce || !mce_usable_address (mce)) +if (!mce) +return NOTIFY_DONE ; + +/* Only uncorrected memory errors warrant taking down the alias page */ +if (!mce_is_memory_error (mce) ||mce_is_correctable (mce)) +return NOTIFY_DONE ; + +if (!mce_usable_address (mce))return NOTIFY_DONE ; spa = mce->addr &MCI_ADDR_PHYSADDR ;
base-commit: 7098e9cd98a05c0c5de2fae0c2465f9d966fdd07
Signature · 1 line
2.43.0
Quoted 30 lines · Shaikh Kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
cxl_handle_mce()offlines the aliased page of an ELC region on any record with a usable address; it does not checkMCI_STATUS_UCor filter non-memory errors.uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters onmce->severityand leaves corrected errors untouched.cxl_handle_mce()has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired viamemory_failure().Corrected errors do reach the chain:
machine_check_poll()logs them via the samemce_gen_pool_process()path that feedsx86_mce_decoder_chain, andcxl_extended_linear_cache_resize()extendsp->resto cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.Filter the record as
nfit_handle_mce()does. Commitfc08a4703a41("acpi, nfit: Fix the memory error check innfit_handle_mce()") and commit5d96c9342c23("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler callsmemory_failure()rather than recording a bad block.
mce_is_correctable()is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined onMCE_AR_SEVERITY, wherekill_me_maybe()owns the reported page but nothing owns the alias.
Quoted 4 lines
Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")Signed-off-by Shaikh Kamaluddin <[email protected]>---
This looks good to me, so:
If you have the time, could you also share a script that does the below testing on the list? It may be possible to integrate into the CXL testing suite (see https://github.com/pmem/ndctl.git), though the QEMU usage may throw a wrench in that. Even if it's not possible, having the tests out there for people to run would help with any future breakage.
Thanks,
Ben
Quoted 175 lines
Reproduced under QEMU/vng. QEMU's cxl-type3 does not emulate the HMAT extended-linear
address_modebit, so ELC was forced locally for testing via a one-line debug hack incxl_region_probe()(not part of this patch):if (!p->cache_size && p->res) p->cache_size = resource_size(p->res) / 2;Steps:
1. Boot with an Intel CPU model under TCG (KVM host-passthrough will
otherwise leak the host's real vendor ID, and AMD/SMCA takes a
differentmce_usable_address()path than the one under test):
vng -v -r./arch/x86/boot/bzImage--disable-kvm --qemu-opts='-cpu Skylake-Server-v4,+mce,+mca -m 4G -machine q35,cxl=on -object memory-backend-ram,id=cxl-mem0,size=512M -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.0 -device cxl-rp,port=0,bus=cxl.0,id=root_port0,chassis=0,slot=0 -device cxl-type3,bus=root_port0,volatile-memdev=cxl-mem0,id=cxl-mem-device0 -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=512M'2. modprobe mce-inject
$cxl list -M
$cxl list -D
$cxl create-region -m mem0 -d decoder0.0 -w 1 -g 256 -t ram $dmesg | grep "DEBUG: forcedcache_size"
$modprobedevice_dax
$modprobe kmem
$ls/sys/bus/dax/devices/
$daxctl reconfigure-device dax0.0 --mode=system-ram $lsmem
it will show as below:
dax0.0[ 42.857642] Fallback order for Node 0: 0 [ 42.857828] Built 1 zonelists, mobility grouping on. Total pages: 1010805 [ 42.858446] Policy zone: Normal [ { "chardev":"dax0.0", "size":536870912, "target_node":0, "align":2097152, "mode":"system-ram", "online_memblocks":4, "total_memblocks":4, "movable":true}
]
reconfigured 1 device
RANGE SIZE STATE REMOVABLE BLOCK0x0000000000000000-0x000000007fffffff 2G online yes 0-15 0x0000000100000000-0x000000017fffffff 2G online yes 32-47 0x0000000190000000-0x00000001afffffff 512M online yes 50-53Memory block size: 128M Total online memory: 4.5G Total offline memory: 0B3. Load the injector if not loaded earlier and derive the two
MCi_STATUSvalues.modprobe mce-inject
MCi_STATUSbit layout used here (arch/x86/include/asm/mce.h):
bit 63 VAL - record valid
bit 61 UC - uncorrected (0 = corrected error under test)
bit 60 EN - error reporting enabled
bit 59 MISCV -MCi_MISCvalid
bit 58 ADDRV -MCi_ADDRvalid
bits[15:0] MCACOD - bit 7 set = memory-error signature, per
mce_is_memory_error()'s Intel branchpython3 -c "
VAL, UC, EN, MISCV, ADDRV = 1<<63, 1<<61, 1<<60, 1<<59, 1<<58
MCACOD_MEM= 1<<7 # memory error signature
ce = VAL | EN | MISCV | ADDRV |MCACOD_MEM
uc = ce | UC
print(f'CE status = {hex(ce)}')
print(f'UC status = {hex(uc)}')"
# CE status =0x9c00000000000080
# UC status =0xbc00000000000080
MCi_MISC: address-mode field, bits[8:6], must be 2 (physical):python3 -c "print(hex(2 << 6))"
# misc =0x804. $ cd
/sys/kernel/debug/mce-inject
$echo sw > flags
$echo0x9C00000000000080> status
$echo0x80> misc
$echo0x190010000> addr
$echo 9 > bank[ 221.813395] mce: [Hardware Error]: Machine check events logged [ 221.814716] cxl_mce_debug: entered status=0x9c00000000000080 addr=0x190010000 cache_size=0x10000000 res=[mem 0x190000000-0x1afffffff flags 0x200] usable=1 [ 221.816200] cxl_mce_debug: spa=0x190010000 contains=1 [ 221.817089] cxl_mce_debug: spa_alias=0x1a0010000 pfn=0x1a0010 pfn_valid=1 [ 221.817655] cxl_region region0: Offlining aliased SPA address0: 0x1a0010000 [ 221.821504] Memory failure: 0x1a0010: recovery action for free buddy page: Recovered [ 221.823903] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9: 9c00000000000080 [ 221.824617] mce: [Hardware Error]: TSC a605ccda40 ADDR 190010000 MISC 80 [ 221.824997] mce: [Hardware Error]: PROCESSOR 0:50654 TIME 1786375908 SOCKET 0 APIC 0 microcode 1 root@virtme-ng:/sys/kernel/debug/mce-inject# root@virtme-ng:/sys/kernel/debug/mce-inject# grep HardwareCorrupted /proc/meminfo HardwareCorrupted: 4 kBCE without this patch:
cxl_regionregion0: Offlining aliased SPA address0:0x1a0010000Memory failure:0x1a0010: recovery action for free buddy page: Recovered
HardwareCorrupted: 4 kB
-------------------------------------
CE with this patch:
(no "Offlining aliased SPA" message logged)
HardwareCorrupted: 0 kB$cd
/sys/kernel/debug/mce-inject
$echo sw > flags
$echo0x9C00000000000080> status
$echo0x80> misc
$echo0x190010000> addr
$echo 9 > bank
addr bank cpu flags ipid misc README status synd[ 61.681659] mce: [Hardware Error]: Machine check events logged [ 61.683442] cxl_mce_debug: entered status=0x9c00000000000080 addr=0x190010000 cache_size=0x10000000 res=[mem 0x190000000-0x1afffffff flags 0x200] usable=1 [ 61.684528] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9: 9c00000000000080 [ 61.684974] mce: [Hardware Error]: TSC 2eeae32fe0 ADDR 190010000 MISC 80 [ 61.688880] mce: [Hardware Error]: PROCESSOR 0:50654 TIME 1786377156 SOCKET 0 APIC 0 microcode 1 [ 86.504456] clocksource: Watchdog remote CPU 11 read timed out$grep HardwareCorrupted
/proc/meminfo
HardwareCorrupted: 0 kB
-------------------------
For UC, same steps only status bit information will change :
$cd/sys/kernel/debug/mce-inject
$echo sw > flags
$echo0xbc00000000000080> status
$echo0x80> misc
$echo0x190010000> addr
$echo 9 > bank
$dmesg | grepcxl_mce_debug
$grep HardwareCorrupted/proc/meminfo[ 313.398747] mce: [Hardware Error]: Machine check events logged [ 313.400518] cxl_mce_debug: entered status=0xbc00000000000080 addr=0x190010000 cache_size=0x10000000 res=[mem 0x190000000-0x1afffffff flags 0x200] usable=1 [ 313.401727] cxl_mce_debug: spa=0x190010000 contains=1 [ 313.402157] cxl_mce_debug: spa_alias=0x1a0010000 pfn=0x1a0010 pfn_valid=1 [ 313.402663] cxl_region region0: Offlining aliased SPA address0: 0x1a0010000 [ 313.405999] Memory failure: 0x1a0010: recovery action for free buddy page: Recovered [ 313.408375] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9: bc00000000000080 [ 313.408999] mce: [Hardware Error]: TSC e9dd2869e0 ADDR 190010000 MISC 80 [ 313.409367] mce: [Hardware Error]: PROCESSOR 0:50654 TIME 1786377750 SOCKET 0 APIC 0 microcode 1Patched, UC: alias still offlined, confirming uncorrected handling is unchanged by this patch.
cxl_region region0: Offlining aliased SPA address0: 0x1a0010000 Memory failure: 0x1a0010: recovery action for free buddy page: Recovered HardwareCorrupted: 4 kBdrivers/cxl/core/mce.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/cxl/core/mce.c +8
-1@@ -18,7 +18,14 @@ static int cxl_handle_mce(struct notifier_block *nb, unsigned long val,u64 spa, spa_alias;unsigned long pfn; -if (!mce || !mce_usable_address (mce)) +if (!mce) +return NOTIFY_DONE ; + +/* Only uncorrected memory errors warrant taking down the alias page */ +if (!mce_is_memory_error (mce) ||mce_is_correctable (mce)) +return NOTIFY_DONE ; + +if (!mce_usable_address (mce))return NOTIFY_DONE ; spa = mce->addr &MCI_ADDR_PHYSADDR ;base-commit:
7098e9cd98a05c0c5de2fae0c2465f9d966fdd07--
2.43.0
Quoted 31 lines · Benjamin
Quoted 30 lines · Shaikh Kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
cxl_handle_mce()offlines the aliased page of an ELC region on any record with a usable address; it does not checkMCI_STATUS_UCor filter non-memory errors.uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters onmce->severityand leaves corrected errors untouched.cxl_handle_mce()has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired viamemory_failure().Corrected errors do reach the chain:
machine_check_poll()logs them via the samemce_gen_pool_process()path that feedsx86_mce_decoder_chain, andcxl_extended_linear_cache_resize()extendsp->resto cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.Filter the record as
nfit_handle_mce()does. Commitfc08a4703a41("acpi, nfit: Fix the memory error check innfit_handle_mce()") and commit5d96c9342c23("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler callsmemory_failure()rather than recording a bad block.
mce_is_correctable()is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined onMCE_AR_SEVERITY, wherekill_me_maybe()owns the reported page but nothing owns the alias.
Quoted 8 lines
Quoted 4 lines
Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")Signed-off-by Shaikh Kamaluddin <[email protected]>---
This looks good to me, so:
Reviewed-by Ben Cheatham <[email protected]>
Thanks for review!
Quoted 4 lines
If you have the time, could you also share a script that does the below testing on the list? It may be possible to integrate into the CXL testing suite (see https://github.com/pmem/ndctl.git), though the QEMU usage may throw a wrench in that. Even if it's not possible, having the tests out there for people to run would help with any future breakage.
Happy to share it. One question on where it'd fit best: ndctl (github.com/pmem/ndctl.git) as you mentioned, or drivers/cxl's own tools/testing/cxl/ in-tree? I'm open to either, or proposing it in both if that's useful - happy to follow your lead on which is the better home for it.
Thanks,
Shaikh
Quoted 179 lines
Thanks,
BenQuoted 175 lines
Reproduced under QEMU/vng. QEMU's cxl-type3 does not emulate the HMAT extended-linear
address_modebit, so ELC was forced locally for testing via a one-line debug hack incxl_region_probe()(not part of this patch):if (!p->cache_size && p->res) p->cache_size = resource_size(p->res) / 2;Steps:
1. Boot with an Intel CPU model under TCG (KVM host-passthrough will
otherwise leak the host's real vendor ID, and AMD/SMCA takes a
differentmce_usable_address()path than the one under test):
vng -v -r./arch/x86/boot/bzImage--disable-kvm --qemu-opts='-cpu Skylake-Server-v4,+mce,+mca -m 4G -machine q35,cxl=on -object memory-backend-ram,id=cxl-mem0,size=512M -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.0 -device cxl-rp,port=0,bus=cxl.0,id=root_port0,chassis=0,slot=0 -device cxl-type3,bus=root_port0,volatile-memdev=cxl-mem0,id=cxl-mem-device0 -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=512M'2. modprobe mce-inject
$cxl list -M
$cxl list -D
$cxl create-region -m mem0 -d decoder0.0 -w 1 -g 256 -t ram $dmesg | grep "DEBUG: forcedcache_size"
$modprobedevice_dax
$modprobe kmem
$ls/sys/bus/dax/devices/
$daxctl reconfigure-device dax0.0 --mode=system-ram $lsmem
it will show as below:
dax0.0[ 42.857642] Fallback order for Node 0: 0 [ 42.857828] Built 1 zonelists, mobility grouping on. Total pages: 1010805 [ 42.858446] Policy zone: Normal [ { "chardev":"dax0.0", "size":536870912, "target_node":0, "align":2097152, "mode":"system-ram", "online_memblocks":4, "total_memblocks":4, "movable":true}
]
reconfigured 1 device
RANGE SIZE STATE REMOVABLE BLOCK0x0000000000000000-0x000000007fffffff 2G online yes 0-15 0x0000000100000000-0x000000017fffffff 2G online yes 32-47 0x0000000190000000-0x00000001afffffff 512M online yes 50-53Memory block size: 128M Total online memory: 4.5G Total offline memory: 0B3. Load the injector if not loaded earlier and derive the two
MCi_STATUSvalues.modprobe mce-inject
MCi_STATUSbit layout used here (arch/x86/include/asm/mce.h):
bit 63 VAL - record valid
bit 61 UC - uncorrected (0 = corrected error under test)
bit 60 EN - error reporting enabled
bit 59 MISCV -MCi_MISCvalid
bit 58 ADDRV -MCi_ADDRvalid
bits[15:0] MCACOD - bit 7 set = memory-error signature, per
mce_is_memory_error()'s Intel branchpython3 -c "
VAL, UC, EN, MISCV, ADDRV = 1<<63, 1<<61, 1<<60, 1<<59, 1<<58
MCACOD_MEM= 1<<7 # memory error signature
ce = VAL | EN | MISCV | ADDRV |MCACOD_MEM
uc = ce | UC
print(f'CE status = {hex(ce)}')
print(f'UC status = {hex(uc)}')"
# CE status =0x9c00000000000080
# UC status =0xbc00000000000080
MCi_MISC: address-mode field, bits[8:6], must be 2 (physical):python3 -c "print(hex(2 << 6))"
# misc =0x804. $ cd
/sys/kernel/debug/mce-inject
$echo sw > flags
$echo0x9C00000000000080> status
$echo0x80> misc
$echo0x190010000> addr
$echo 9 > bank[ 221.813395] mce: [Hardware Error]: Machine check events logged [ 221.814716] cxl_mce_debug: entered status=0x9c00000000000080 addr=0x190010000 cache_size=0x10000000 res=[mem 0x190000000-0x1afffffff flags 0x200] usable=1 [ 221.816200] cxl_mce_debug: spa=0x190010000 contains=1 [ 221.817089] cxl_mce_debug: spa_alias=0x1a0010000 pfn=0x1a0010 pfn_valid=1 [ 221.817655] cxl_region region0: Offlining aliased SPA address0: 0x1a0010000 [ 221.821504] Memory failure: 0x1a0010: recovery action for free buddy page: Recovered [ 221.823903] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9: 9c00000000000080 [ 221.824617] mce: [Hardware Error]: TSC a605ccda40 ADDR 190010000 MISC 80 [ 221.824997] mce: [Hardware Error]: PROCESSOR 0:50654 TIME 1786375908 SOCKET 0 APIC 0 microcode 1 root@virtme-ng:/sys/kernel/debug/mce-inject# root@virtme-ng:/sys/kernel/debug/mce-inject# grep HardwareCorrupted /proc/meminfo HardwareCorrupted: 4 kBCE without this patch:
cxl_regionregion0: Offlining aliased SPA address0:0x1a0010000Memory failure:0x1a0010: recovery action for free buddy page: Recovered
HardwareCorrupted: 4 kB
-------------------------------------
CE with this patch:
(no "Offlining aliased SPA" message logged)
HardwareCorrupted: 0 kB$cd
/sys/kernel/debug/mce-inject
$echo sw > flags
$echo0x9C00000000000080> status
$echo0x80> misc
$echo0x190010000> addr
$echo 9 > bank
addr bank cpu flags ipid misc README status synd[ 61.681659] mce: [Hardware Error]: Machine check events logged [ 61.683442] cxl_mce_debug: entered status=0x9c00000000000080 addr=0x190010000 cache_size=0x10000000 res=[mem 0x190000000-0x1afffffff flags 0x200] usable=1 [ 61.684528] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9: 9c00000000000080 [ 61.684974] mce: [Hardware Error]: TSC 2eeae32fe0 ADDR 190010000 MISC 80 [ 61.688880] mce: [Hardware Error]: PROCESSOR 0:50654 TIME 1786377156 SOCKET 0 APIC 0 microcode 1 [ 86.504456] clocksource: Watchdog remote CPU 11 read timed out$grep HardwareCorrupted
/proc/meminfo
HardwareCorrupted: 0 kB
-------------------------
For UC, same steps only status bit information will change :
$cd/sys/kernel/debug/mce-inject
$echo sw > flags
$echo0xbc00000000000080> status
$echo0x80> misc
$echo0x190010000> addr
$echo 9 > bank
$dmesg | grepcxl_mce_debug
$grep HardwareCorrupted/proc/meminfo[ 313.398747] mce: [Hardware Error]: Machine check events logged [ 313.400518] cxl_mce_debug: entered status=0xbc00000000000080 addr=0x190010000 cache_size=0x10000000 res=[mem 0x190000000-0x1afffffff flags 0x200] usable=1 [ 313.401727] cxl_mce_debug: spa=0x190010000 contains=1 [ 313.402157] cxl_mce_debug: spa_alias=0x1a0010000 pfn=0x1a0010 pfn_valid=1 [ 313.402663] cxl_region region0: Offlining aliased SPA address0: 0x1a0010000 [ 313.405999] Memory failure: 0x1a0010: recovery action for free buddy page: Recovered [ 313.408375] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9: bc00000000000080 [ 313.408999] mce: [Hardware Error]: TSC e9dd2869e0 ADDR 190010000 MISC 80 [ 313.409367] mce: [Hardware Error]: PROCESSOR 0:50654 TIME 1786377750 SOCKET 0 APIC 0 microcode 1Patched, UC: alias still offlined, confirming uncorrected handling is unchanged by this patch.
cxl_region region0: Offlining aliased SPA address0: 0x1a0010000 Memory failure: 0x1a0010: recovery action for free buddy page: Recovered HardwareCorrupted: 4 kBdrivers/cxl/core/mce.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/cxl/core/mce.c +8
-1@@ -18,7 +18,14 @@ static int cxl_handle_mce(struct notifier_block *nb, unsigned long val,u64 spa, spa_alias;unsigned long pfn; -if (!mce || !mce_usable_address (mce)) +if (!mce) +return NOTIFY_DONE ; + +/* Only uncorrected memory errors warrant taking down the alias page */ +if (!mce_is_memory_error (mce) ||mce_is_correctable (mce)) +return NOTIFY_DONE ; + +if (!mce_usable_address (mce))return NOTIFY_DONE ; spa = mce->addr &MCI_ADDR_PHYSADDR ;base-commit:
7098e9cd98a05c0c5de2fae0c2465f9d966fdd07--
2.43.0
Quoted 47 lines · shaikh kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Quoted 31 lines · Benjamin
Quoted 30 lines · Shaikh Kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
cxl_handle_mce()offlines the aliased page of an ELC region on any record with a usable address; it does not checkMCI_STATUS_UCor filter non-memory errors.uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters onmce->severityand leaves corrected errors untouched.cxl_handle_mce()has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired viamemory_failure().Corrected errors do reach the chain:
machine_check_poll()logs them via the samemce_gen_pool_process()path that feedsx86_mce_decoder_chain, andcxl_extended_linear_cache_resize()extendsp->resto cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.Filter the record as
nfit_handle_mce()does. Commitfc08a4703a41("acpi, nfit: Fix the memory error check innfit_handle_mce()") and commit5d96c9342c23("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler callsmemory_failure()rather than recording a bad block.
mce_is_correctable()is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined onMCE_AR_SEVERITY, wherekill_me_maybe()owns the reported page but nothing owns the alias.Quoted 8 lines
Quoted 4 lines
Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")Signed-off-by Shaikh Kamaluddin <[email protected]>---
This looks good to me, so:
Reviewed-by Ben Cheatham <[email protected]>Thanks for review!
Quoted 4 lines
If you have the time, could you also share a script that does the below testing on the list? It may be possible to integrate into the CXL testing suite (see https://github.com/pmem/ndctl.git), though the QEMU usage may throw a wrench in that. Even if it's not possible, having the tests out there for people to run would help with any future breakage.
Quoted 6 lines
Happy to share it. One question on where it'd fit best: ndctl (github.com/pmem/ndctl.git) as you mentioned, or drivers/cxl's own
tools/testing/cxl/in-tree? I'm open to either, or proposing it in both if that's useful - happy to follow your lead on which is the better home for it.
If looks like there's already some mock functions for extended linear cache in tools/testing/cxl, so I'd recommending trying to put it there to begin with. It may require updates to ndctl after the fact to run the test(s) as well. If that looks too involved then sending the script out to the list standalone should be fine. I don't know if anyone will pick it up, but it'll be searchable on lore if anyone wants to test this.
Thanks,
Ben
Alison Schofield
maintainerQuoted 55 lines · Benjamin
Quoted 47 lines · shaikh kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Quoted 31 lines · Benjamin
Quoted 30 lines · Shaikh Kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
cxl_handle_mce()offlines the aliased page of an ELC region on any record with a usable address; it does not checkMCI_STATUS_UCor filter non-memory errors.uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters onmce->severityand leaves corrected errors untouched.cxl_handle_mce()has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired viamemory_failure().Corrected errors do reach the chain:
machine_check_poll()logs them via the samemce_gen_pool_process()path that feedsx86_mce_decoder_chain, andcxl_extended_linear_cache_resize()extendsp->resto cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.Filter the record as
nfit_handle_mce()does. Commitfc08a4703a41("acpi, nfit: Fix the memory error check innfit_handle_mce()") and commit5d96c9342c23("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler callsmemory_failure()rather than recording a bad block.
mce_is_correctable()is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined onMCE_AR_SEVERITY, wherekill_me_maybe()owns the reported page but nothing owns the alias.Quoted 8 lines
Quoted 4 lines
Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")Signed-off-by Shaikh Kamaluddin <[email protected]>---
This looks good to me, so:
Reviewed-by Ben Cheatham <[email protected]>Thanks for review!
Quoted 4 lines
If you have the time, could you also share a script that does the below testing on the list? It may be possible to integrate into the CXL testing suite (see https://github.com/pmem/ndctl.git), though the QEMU usage may throw a wrench in that. Even if it's not possible, having the tests out there for people to run would help with any future breakage.
Quoted 6 lines
Happy to share it. One question on where it'd fit best: ndctl (github.com/pmem/ndctl.git) as you mentioned, or drivers/cxl's own
tools/testing/cxl/in-tree? I'm open to either, or proposing it in both if that's useful - happy to follow your lead on which is the better home for it.
Quoted 5 lines
If looks like there's already some mock functions for extended linear cache in
tools/testing/cxl, so I'd recommending trying to put it there to begin with. It may require updates to ndctl after the fact to run the test(s) as well. If that looks too involved then sending the script out to the list standalone should be fine. I don't know if anyone will pick it up, but it'll be searchable on lore if anyone wants to test this.
This sounded interesting! I gave it a try with cxl_test and mce-inject, and it looks like this can be tested without the QEMU CXL topology or the forced cache_size hack.
I built with CONFIG_X86_MCE_INJECT=m and loaded cxl_test with its existing ELC support:
# modprobe cxl_test extended_linear_cache=1
# cxl list -R
[
{
"region":"region0",
"resource":70300293136384,
"size":1073741824,
"extended_linear_cache_size":536870912,
"type":"ram",
"interleave_ways":2,
"interleave_granularity":4096,
"decode_state":"commit",
"locked":false
}
]Using 0x3ff010010000 as the injected SPA, I first injected the corrected error from your example. On the patched kernel there was no CXL offlining message, as expected.
I then changed only the status to the uncorrectable case:
# cd /sys/kernel/debug/mce-inject # echo sw > flags # echo 0xbc00000000000080 > status # echo 0x80 > misc # echo 0x3ff010010000 > addr # echo 9 > bank
and got:
cxl_region region0: Offlining aliased SPA address0: 0x3ff030010000
Memory failure: 0x3ff030010: memory outside kernel control
mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9: bc00000000000080
mce: [Hardware Error]: TSC 23c5a5a9188 ADDR 3ff010010000 MISC 80
mce: [Hardware Error]: PROCESSOR 0:50657 TIME 1786573477 SOCKET 0 APIC 0 microcode 5003302
So the cxl_test ELC plus mce-inject looks sufficient to exercise the
path: the CE is ignored with the patch, while the UC reaches the alias offlining path and computes the expected alias.
The "memory outside kernel control" is because I did not put the aliased memory into system RAM for this quick test.
This would be a test case addition for the the cxl-elc.sh unit test. It seems like a tiny, close-the-barn-door-after-the-horse-got-out, kind of test case, but maybe not? Maybe it opens the door to more things can do we mce-inject elsewhere?
I'll leave that to Shaikh if they want to add the new test case.
-- Alison
Quoted 3 lines
Thanks,
Ben
Quoted 114 lines · Alison Schofield
Quoted 55 lines · Benjamin
Quoted 47 lines · shaikh kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Quoted 31 lines · Benjamin
Quoted 30 lines · Shaikh Kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
cxl_handle_mce()offlines the aliased page of an ELC region on any record with a usable address; it does not checkMCI_STATUS_UCor filter non-memory errors.uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters onmce->severityand leaves corrected errors untouched.cxl_handle_mce()has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired viamemory_failure().Corrected errors do reach the chain:
machine_check_poll()logs them via the samemce_gen_pool_process()path that feedsx86_mce_decoder_chain, andcxl_extended_linear_cache_resize()extendsp->resto cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.Filter the record as
nfit_handle_mce()does. Commitfc08a4703a41("acpi, nfit: Fix the memory error check innfit_handle_mce()") and commit5d96c9342c23("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler callsmemory_failure()rather than recording a bad block.
mce_is_correctable()is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined onMCE_AR_SEVERITY, wherekill_me_maybe()owns the reported page but nothing owns the alias.Quoted 8 lines
Quoted 4 lines
Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")Signed-off-by Shaikh Kamaluddin <[email protected]>---
This looks good to me, so:
Reviewed-by Ben Cheatham <[email protected]>Thanks for review!
Quoted 4 lines
If you have the time, could you also share a script that does the below testing on the list? It may be possible to integrate into the CXL testing suite (see https://github.com/pmem/ndctl.git), though the QEMU usage may throw a wrench in that. Even if it's not possible, having the tests out there for people to run would help with any future breakage.
Quoted 6 lines
Happy to share it. One question on where it'd fit best: ndctl (github.com/pmem/ndctl.git) as you mentioned, or drivers/cxl's own
tools/testing/cxl/in-tree? I'm open to either, or proposing it in both if that's useful - happy to follow your lead on which is the better home for it.Quoted 5 lines
If looks like there's already some mock functions for extended linear cache in
tools/testing/cxl, so I'd recommending trying to put it there to begin with. It may require updates to ndctl after the fact to run the test(s) as well. If that looks too involved then sending the script out to the list standalone should be fine. I don't know if anyone will pick it up, but it'll be searchable on lore if anyone wants to test this.This sounded interesting! I gave it a try with
cxl_testand mce-inject, and it looks like this can be tested without the QEMU CXL topology or the forcedcache_sizehack.I built with
CONFIG_X86_MCE_INJECT=mand loadedcxl_testwith its existing ELC support:# modprobe cxl_test extended_linear_cache=1 # cxl list -R [ { "region":"region0", "resource":70300293136384, "size":1073741824, "extended_linear_cache_size":536870912, "type":"ram", "interleave_ways":2, "interleave_granularity":4096, "decode_state":"commit", "locked":false } ]Using
0x3ff010010000as the injected SPA, I first injected the corrected error from your example. On the patched kernel there was no CXL offlining message, as expected.I then changed only the status to the uncorrectable case:
# cd /sys/kernel/debug/mce-inject # echo sw > flags # echo 0xbc00000000000080 > status # echo 0x80 > misc # echo 0x3ff010010000 > addr # echo 9 > bankand got:
cxl_regionregion0: Offlining aliased SPA address0:0x3ff030010000
Memory failure:0x3ff030010: memory outside kernel control
mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9:bc00000000000080
mce: [Hardware Error]: TSC23c5a5a9188ADDR3ff010010000MISC 80
mce: [Hardware Error]: PROCESSOR 0:50657 TIME 1786573477 SOCKET 0 APIC 0 microcode 5003302So the
cxl_testELC plus mce-inject looks sufficient to exercise the
path: the CE is ignored with the patch, while the UC reaches the alias offlining path and computes the expected alias.The "memory outside kernel control" is because I did not put the aliased memory into system RAM for this quick test.
Quoted 8 lines
This would be a test case addition for the the
cxl-elc.shunit test. It seems like a tiny, close-the-barn-door-after-the-horse-got-out, kind of test case, but maybe not? Maybe it opens the door to more things can do we mce-inject elsewhere?I'll leave that to Shaikh if they want to add the new test case.
-- Alison
Hi Alison,
Thanks for trying this. This is very helpful.
I had initially started with the same cxl_test + mce-inject approach before moving to the vng/QEMU CXL Type-3 setup.
On the current cxl/next tree, I was first blocked while building tools/testing/cxl with LLVM/ld.lld. modpost was failing on wrapped CXL symbols, for example:
.export_symbol section references '__wrap_devm_cxl_add_rch_dport', but it does not seem to be an export symbol
.export_symbol section references '__wrap_devm_cxl_add_dport_by_dev', but it does not seem to be an export symbol
.export_symbol section references '__wrap_cxl_await_media_ready', but it does not seem to be an export symbol
There were similar failures for some of the decoder/CDAT wrappers.
This appears to be the same issue addressed by the patch currently under review:
[PATCH] tools/testing/cxl: Don't wrap cxl_core's own exported symbols
https://lore.kernel.org/linux-cxl/[email protected]/
The patch avoids globally wrapping the CXL core symbols that are also defined/exported by cxl_core, and instead applies those wrappers only to the modules that need them.
With that patch applied, tools/testing/cxl builds successfully for me. However, I still hit a runtime crash when loading the ELC setup:
# modprobe cxl_test extended_linear_cache=1
BUG: unable to handle page fault for address: 0000000000003358 #PF: supervisor read access in kernel mode RIP: __alloc_frozen_pages_noprof+0x12e/0x320 CR2: 0000000000003358 Workqueue: async async_run_entry_fn
So the build issue and this runtime crash appear to be separate problems. The runtime failure is what led me to use the vng/QEMU CXL Type-3 setup for validating the MCE change, where I was able to exercise both the CE and UC cases.
Since you were able to run:
modprobe cxl_test extended_linear_cache=1
+
mce-inject
successfully, could you please share the kernel configuration and any patches you have on top of cxl/next? That would help me compare the working setup with mine and identify what I am still missing on the cxl_test side.
Your result also confirms that, once I get this setup stable, extending the existing cxl-elc.sh test looks like the right direction for regression coverage. My plan would be to derive the SPA and expected alias dynamically from the ELC region, inject a CE and verify that the alias-offlining path is not reached, then inject a UC and verify that the expected aliased SPA is still offlined.
This is a small regression test for the current issue, but exercising mce-inject through the CXL test infrastructure may also provide useful coverage for other CXL RAS/MCE paths in the future.
For the current patch validation, I still think the vng/QEMU Type-3 setup is useful as an end-to-end test since it can exercise the actual CXL region and system-RAM memory_failure() path, while cxl_test + mce-inject looks better suited for the lightweight automated regression test.
Thanks,
Shaikh
Quoted 4 lines
Quoted 4 lines
Thanks,
Ben
Alison Schofield
maintainerQuoted 144 lines · shaikh kamaluddin
Quoted 114 lines · Alison Schofield
Quoted 55 lines · Benjamin
Quoted 47 lines · shaikh kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Quoted 31 lines · Benjamin
Quoted 30 lines · Shaikh Kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
cxl_handle_mce()offlines the aliased page of an ELC region on any record with a usable address; it does not checkMCI_STATUS_UCor filter non-memory errors.uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters onmce->severityand leaves corrected errors untouched.cxl_handle_mce()has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired viamemory_failure().Corrected errors do reach the chain:
machine_check_poll()logs them via the samemce_gen_pool_process()path that feedsx86_mce_decoder_chain, andcxl_extended_linear_cache_resize()extendsp->resto cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.Filter the record as
nfit_handle_mce()does. Commitfc08a4703a41("acpi, nfit: Fix the memory error check innfit_handle_mce()") and commit5d96c9342c23("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler callsmemory_failure()rather than recording a bad block.
mce_is_correctable()is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined onMCE_AR_SEVERITY, wherekill_me_maybe()owns the reported page but nothing owns the alias.Quoted 8 lines
Quoted 4 lines
Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")Signed-off-by Shaikh Kamaluddin <[email protected]>---
This looks good to me, so:
Reviewed-by Ben Cheatham <[email protected]>Thanks for review!
Quoted 4 lines
If you have the time, could you also share a script that does the below testing on the list? It may be possible to integrate into the CXL testing suite (see https://github.com/pmem/ndctl.git), though the QEMU usage may throw a wrench in that. Even if it's not possible, having the tests out there for people to run would help with any future breakage.
Quoted 6 lines
Happy to share it. One question on where it'd fit best: ndctl (github.com/pmem/ndctl.git) as you mentioned, or drivers/cxl's own
tools/testing/cxl/in-tree? I'm open to either, or proposing it in both if that's useful - happy to follow your lead on which is the better home for it.Quoted 5 lines
If looks like there's already some mock functions for extended linear cache in
tools/testing/cxl, so I'd recommending trying to put it there to begin with. It may require updates to ndctl after the fact to run the test(s) as well. If that looks too involved then sending the script out to the list standalone should be fine. I don't know if anyone will pick it up, but it'll be searchable on lore if anyone wants to test this.This sounded interesting! I gave it a try with
cxl_testand mce-inject, and it looks like this can be tested without the QEMU CXL topology or the forcedcache_sizehack.I built with
CONFIG_X86_MCE_INJECT=mand loadedcxl_testwith its existing ELC support:# modprobe cxl_test extended_linear_cache=1 # cxl list -R [ { "region":"region0", "resource":70300293136384, "size":1073741824, "extended_linear_cache_size":536870912, "type":"ram", "interleave_ways":2, "interleave_granularity":4096, "decode_state":"commit", "locked":false } ]Using
0x3ff010010000as the injected SPA, I first injected the corrected error from your example. On the patched kernel there was no CXL offlining message, as expected.I then changed only the status to the uncorrectable case:
# cd /sys/kernel/debug/mce-inject # echo sw > flags # echo 0xbc00000000000080 > status # echo 0x80 > misc # echo 0x3ff010010000 > addr # echo 9 > bankand got:
cxl_regionregion0: Offlining aliased SPA address0:0x3ff030010000
Memory failure:0x3ff030010: memory outside kernel control
mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9:bc00000000000080
mce: [Hardware Error]: TSC23c5a5a9188ADDR3ff010010000MISC 80
mce: [Hardware Error]: PROCESSOR 0:50657 TIME 1786573477 SOCKET 0 APIC 0 microcode 5003302So the
cxl_testELC plus mce-inject looks sufficient to exercise the
path: the CE is ignored with the patch, while the UC reaches the alias offlining path and computes the expected alias.The "memory outside kernel control" is because I did not put the aliased memory into system RAM for this quick test.
Quoted 8 lines
This would be a test case addition for the the
cxl-elc.shunit test. It seems like a tiny, close-the-barn-door-after-the-horse-got-out, kind of test case, but maybe not? Maybe it opens the door to more things can do we mce-inject elsewhere?I'll leave that to Shaikh if they want to add the new test case.
-- Alison
Hi Alison,
Thanks for trying this. This is very helpful.
I had initially started with the same
cxl_test+ mce-inject approach before moving to the vng/QEMU CXL Type-3 setup.On the current cxl/next tree, I was first blocked while building
tools/testing/cxlwith LLVM/ld.lld. modpost was failing on wrapped CXL symbols, for example:.export_symbol section references '
__wrap_devm_cxl_add_rch_dport', but it does not seem to be an export symbol.export_symbol section references '
__wrap_devm_cxl_add_dport_by_dev', but it does not seem to be an export symbol.export_symbol section references '
__wrap_cxl_await_media_ready', but it does not seem to be an export symbolThere were similar failures for some of the decoder/CDAT wrappers.
Quoted 7 lines
This appears to be the same issue addressed by the patch currently under review:
[PATCH]
tools/testing/cxl: Don't wrap cxl_core's own exported symbolshttps://lore.kernel.org/linux-cxl/[email protected]/
The patch avoids globally wrapping the CXL core symbols that are also defined/exported by
cxl_core, and instead applies those wrappers only to the modules that need them.
Hi Shaikh,
I applied the patch to the base commit included in the patch:
Quoted 1 line
Quoted 1 line
base-commit:
7098e9cd98a05c0c5de2fae0c2465f9d966fdd07
I don't use LLVM toolchain, but I do see Richards patch you refer to. We'll need to get that queued for 7.4.
Quoted 10 lines
With that patch applied,
tools/testing/cxlbuilds successfully for me. However, I still hit a runtime crash when loading the ELC setup:# modprobe cxl_test extended_linear_cache=1BUG: unable to handle page fault for address: 0000000000003358 #PF: supervisor read access in kernel mode RIP: __alloc_frozen_pages_noprof+0x12e/0x320 CR2: 0000000000003358 Workqueue: async async_run_entry_fn
So there must be something different about your cxl-test address space. I have
region resource: 0x3ff010000000 region size: 0x40000000 ELC size: 0x20000000
You cannot even load w elc = 1 !
Does "modprobe cxl_test" work for you?
If yes, how far can you get? Does cxl-topology.sh work for you? I wouldn't have you try the whole suite since you are telling me cxl-elc.sh already fails.
Maybe we need to look at where cxl test is loaded for you and why the ELC address shennanigans fail in your environment.
Let me know what you see.
Thanks,
Alison
Quoted 25 lines
So the build issue and this runtime crash appear to be separate problems. The runtime failure is what led me to use the vng/QEMU CXL Type-3 setup for validating the MCE change, where I was able to exercise both the CE and UC cases.
Since you were able to run:
modprobe
cxl_testextended_linear_cache=1+mce-inject
successfully, could you please share the kernel configuration and any patches you have on top of cxl/next? That would help me compare the working setup with mine and identify what I am still missing on the
cxl_testside.Your result also confirms that, once I get this setup stable, extending the existing
cxl-elc.shtest looks like the right direction for regression coverage. My plan would be to derive the SPA and expected alias dynamically from the ELC region, inject a CE and verify that the alias-offlining path is not reached, then inject a UC and verify that the expected aliased SPA is still offlined.This is a small regression test for the current issue, but exercising mce-inject through the CXL test infrastructure may also provide useful coverage for other CXL RAS/MCE paths in the future.
For the current patch validation, I still think the vng/QEMU Type-3 setup is useful as an end-to-end test since it can exercise the actual CXL region and system-RAM
memory_failure()path, whilecxl_test+ mce-inject looks better suited for the lightweight automated regression test.Thanks,
ShaikhQuoted 4 lines
Quoted 4 lines
Thanks,
Ben
Quoted 186 lines · Alison Schofield
Quoted 144 lines · shaikh kamaluddin
Quoted 114 lines · Alison Schofield
Quoted 55 lines · Benjamin
Quoted 47 lines · shaikh kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Quoted 31 lines · Benjamin
Quoted 30 lines · Shaikh Kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
cxl_handle_mce()offlines the aliased page of an ELC region on any record with a usable address; it does not checkMCI_STATUS_UCor filter non-memory errors.uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters onmce->severityand leaves corrected errors untouched.cxl_handle_mce()has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired viamemory_failure().Corrected errors do reach the chain:
machine_check_poll()logs them via the samemce_gen_pool_process()path that feedsx86_mce_decoder_chain, andcxl_extended_linear_cache_resize()extendsp->resto cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.Filter the record as
nfit_handle_mce()does. Commitfc08a4703a41("acpi, nfit: Fix the memory error check innfit_handle_mce()") and commit5d96c9342c23("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler callsmemory_failure()rather than recording a bad block.
mce_is_correctable()is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined onMCE_AR_SEVERITY, wherekill_me_maybe()owns the reported page but nothing owns the alias.Quoted 8 lines
Quoted 4 lines
Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")Signed-off-by Shaikh Kamaluddin <[email protected]>---
This looks good to me, so:
Reviewed-by Ben Cheatham <[email protected]>Thanks for review!
Quoted 4 lines
If you have the time, could you also share a script that does the below testing on the list? It may be possible to integrate into the CXL testing suite (see https://github.com/pmem/ndctl.git), though the QEMU usage may throw a wrench in that. Even if it's not possible, having the tests out there for people to run would help with any future breakage.
Quoted 6 lines
Happy to share it. One question on where it'd fit best: ndctl (github.com/pmem/ndctl.git) as you mentioned, or drivers/cxl's own
tools/testing/cxl/in-tree? I'm open to either, or proposing it in both if that's useful - happy to follow your lead on which is the better home for it.Quoted 5 lines
If looks like there's already some mock functions for extended linear cache in
tools/testing/cxl, so I'd recommending trying to put it there to begin with. It may require updates to ndctl after the fact to run the test(s) as well. If that looks too involved then sending the script out to the list standalone should be fine. I don't know if anyone will pick it up, but it'll be searchable on lore if anyone wants to test this.This sounded interesting! I gave it a try with
cxl_testand mce-inject, and it looks like this can be tested without the QEMU CXL topology or the forcedcache_sizehack.I built with
CONFIG_X86_MCE_INJECT=mand loadedcxl_testwith its existing ELC support:# modprobe cxl_test extended_linear_cache=1 # cxl list -R [ { "region":"region0", "resource":70300293136384, "size":1073741824, "extended_linear_cache_size":536870912, "type":"ram", "interleave_ways":2, "interleave_granularity":4096, "decode_state":"commit", "locked":false } ]Using
0x3ff010010000as the injected SPA, I first injected the corrected error from your example. On the patched kernel there was no CXL offlining message, as expected.I then changed only the status to the uncorrectable case:
# cd /sys/kernel/debug/mce-inject # echo sw > flags # echo 0xbc00000000000080 > status # echo 0x80 > misc # echo 0x3ff010010000 > addr # echo 9 > bankand got:
cxl_regionregion0: Offlining aliased SPA address0:0x3ff030010000
Memory failure:0x3ff030010: memory outside kernel control
mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 9:bc00000000000080
mce: [Hardware Error]: TSC23c5a5a9188ADDR3ff010010000MISC 80
mce: [Hardware Error]: PROCESSOR 0:50657 TIME 1786573477 SOCKET 0 APIC 0 microcode 5003302So the
cxl_testELC plus mce-inject looks sufficient to exercise the
path: the CE is ignored with the patch, while the UC reaches the alias offlining path and computes the expected alias.The "memory outside kernel control" is because I did not put the aliased memory into system RAM for this quick test.
Quoted 8 lines
This would be a test case addition for the the
cxl-elc.shunit test. It seems like a tiny, close-the-barn-door-after-the-horse-got-out, kind of test case, but maybe not? Maybe it opens the door to more things can do we mce-inject elsewhere?I'll leave that to Shaikh if they want to add the new test case.
-- Alison
Hi Alison,
Thanks for trying this. This is very helpful.
I had initially started with the same
cxl_test+ mce-inject approach before moving to the vng/QEMU CXL Type-3 setup.On the current cxl/next tree, I was first blocked while building
tools/testing/cxlwith LLVM/ld.lld. modpost was failing on wrapped CXL symbols, for example:.export_symbol section references '
__wrap_devm_cxl_add_rch_dport', but it does not seem to be an export symbol.export_symbol section references '
__wrap_devm_cxl_add_dport_by_dev', but it does not seem to be an export symbol.export_symbol section references '
__wrap_cxl_await_media_ready', but it does not seem to be an export symbolThere were similar failures for some of the decoder/CDAT wrappers.
Quoted 7 lines
This appears to be the same issue addressed by the patch currently under review:
[PATCH]
tools/testing/cxl: Don't wrap cxl_core's own exported symbolshttps://lore.kernel.org/linux-cxl/[email protected]/
The patch avoids globally wrapping the CXL core symbols that are also defined/exported by
cxl_core, and instead applies those wrappers only to the modules that need them.Hi Shaikh,
I applied the patch to the base commit included in the patch:
Quoted 1 line
Quoted 1 line
base-commit:
7098e9cd98a05c0c5de2fae0c2465f9d966fdd07I don't use LLVM toolchain, but I do see Richards patch you refer to. We'll need to get that queued for 7.4.
Quoted 10 lines
With that patch applied,
tools/testing/cxlbuilds successfully for me. However, I still hit a runtime crash when loading the ELC setup:# modprobe cxl_test extended_linear_cache=1BUG: unable to handle page fault for address: 0000000000003358 #PF: supervisor read access in kernel mode RIP: __alloc_frozen_pages_noprof+0x12e/0x320 CR2: 0000000000003358 Workqueue: async async_run_entry_fnSo there must be something different about your cxl-test address space. I have
region resource: 0x3ff010000000 region size: 0x40000000 ELC size: 0x20000000You cannot even load w elc = 1 !
Does "modprobe
cxl_test" work for you?
If yes, how far can you get? Doescxl-topology.shwork for you? I wouldn't have you try the whole suite since you are telling mecxl-elc.shalready fails.
Quoted 5 lines
Maybe we need to look at where cxl test is loaded for you and why the ELC address shennanigans fail in your environment.
Let me know what you see.
Hi Alison,
I found the cause of the cxl_test runtime crash.
My virtme-ng guest has only NUMA node 0 available and online:
# cat /sys/devices/system/node/possible 0
# cat /sys/devices/system/node/online 0
The crash I was seeing matches the issue fixed by Davidlohr's patch:
https://lore.kernel.org/linux-cxl/[email protected]/T/#m60009c527f2017c031dad319366418578fcab1fd
cxl/test: Map mock device nodes to an online node
The test code has changed since that patch was posted. The original patch updates the three set_dev_node() sites in cxl_mem_init(), while in current cxl/next the corresponding paths are under cxl_type3_mem_init().
I applied the equivalent change there:
set_dev_node(&pdev->dev, numa_map_to_online_node(i % 2)); to all three Type-3 memdev paths. With that applied, the repeated crash is gone and this now succeeds:
# modprobe cxl_test extended_linear_cache=1
I can also get a committed test region:
# cxl list -R
[
{
"region":"region1",
"resource":1031060586496,
"size":1073741824,
"type":"ram",
"interleave_ways":2,
"interleave_granularity":4096,
"decode_state":"commit"
}
]So the earlier failure was not related to the ELC address-space layout itself; it was the mock memdev being assigned to NUMA node 1 on my single-node guest.
I also noticed current cxl/next has a new set_dev_node(&pdev->dev, i % 2) in cxl_type2_mem_init(). NR_CXL_TYPE2_ACCEL is currently 1, so it does not hit this problem today, but I mentioned it on Davidlohr's patch thread as a possible consistency/future-proofing update.
I have not yet checked why my cxl list -R output does not show extended_linear_cache_size like yours. I will look at that separately now that cxl_test loads cleanly.
Thanks,
Shaikh
Quoted 30 lines
Thanks,
AlisonQuoted 25 lines
So the build issue and this runtime crash appear to be separate problems. The runtime failure is what led me to use the vng/QEMU CXL Type-3 setup for validating the MCE change, where I was able to exercise both the CE and UC cases.
Since you were able to run:
modprobe
cxl_testextended_linear_cache=1+mce-inject
successfully, could you please share the kernel configuration and any patches you have on top of cxl/next? That would help me compare the working setup with mine and identify what I am still missing on the
cxl_testside.Your result also confirms that, once I get this setup stable, extending the existing
cxl-elc.shtest looks like the right direction for regression coverage. My plan would be to derive the SPA and expected alias dynamically from the ELC region, inject a CE and verify that the alias-offlining path is not reached, then inject a UC and verify that the expected aliased SPA is still offlined.This is a small regression test for the current issue, but exercising mce-inject through the CXL test infrastructure may also provide useful coverage for other CXL RAS/MCE paths in the future.
For the current patch validation, I still think the vng/QEMU Type-3 setup is useful as an end-to-end test since it can exercise the actual CXL region and system-RAM
memory_failure()path, whilecxl_test+ mce-inject looks better suited for the lightweight automated regression test.Thanks,
ShaikhQuoted 4 lines
Quoted 4 lines
Thanks,
Ben
Quoted 55 lines · Benjamin
Quoted 47 lines · shaikh kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Quoted 31 lines · Benjamin
Quoted 30 lines · Shaikh Kamaluddin
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
cxl_handle_mce()offlines the aliased page of an ELC region on any record with a usable address; it does not checkMCI_STATUS_UCor filter non-memory errors.uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters onmce->severityand leaves corrected errors untouched.cxl_handle_mce()has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired viamemory_failure().Corrected errors do reach the chain:
machine_check_poll()logs them via the samemce_gen_pool_process()path that feedsx86_mce_decoder_chain, andcxl_extended_linear_cache_resize()extendsp->resto cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.Filter the record as
nfit_handle_mce()does. Commitfc08a4703a41("acpi, nfit: Fix the memory error check innfit_handle_mce()") and commit5d96c9342c23("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler callsmemory_failure()rather than recording a bad block.
mce_is_correctable()is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined onMCE_AR_SEVERITY, wherekill_me_maybe()owns the reported page but nothing owns the alias.Quoted 8 lines
Quoted 4 lines
Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")Signed-off-by Shaikh Kamaluddin <[email protected]>---
This looks good to me, so:
Reviewed-by Ben Cheatham <[email protected]>Thanks for review!
Quoted 4 lines
If you have the time, could you also share a script that does the below testing on the list? It may be possible to integrate into the CXL testing suite (see https://github.com/pmem/ndctl.git), though the QEMU usage may throw a wrench in that. Even if it's not possible, having the tests out there for people to run would help with any future breakage.
Quoted 6 lines
Happy to share it. One question on where it'd fit best: ndctl (github.com/pmem/ndctl.git) as you mentioned, or drivers/cxl's own
tools/testing/cxl/in-tree? I'm open to either, or proposing it in both if that's useful - happy to follow your lead on which is the better home for it.
Quoted 8 lines
If looks like there's already some mock functions for extended linear cache in
tools/testing/cxl, so I'd recommending trying to put it there to begin with. It may require updates to ndctl after the fact to run the test(s) as well. If that looks too involved then sending the script out to the list standalone should be fine. I don't know if anyone will pick it up, but it'll be searchable on lore if anyone wants to test this.Thanks,
Ben
Thanks Ben.
I took a closer look at the existing tools/testing/cxl ELC support. It already provides the mock ELC topology and uses the real CXL MCE notifier path, so based on Alison's testing it looks like no additional CE/UC-specific mock support should be required there.
I had initially tried the same cxl_test + mce-inject approach, but hit the current tools/testing/cxl build/wrap issue. With the proposed build fix applied, it builds for me now, although I am still debugging a crash when running the ELC setup. I am comparing my setup with Alison's working configuration before making any test changes.
Once that is resolved, my current plan is to extend the existing ndctl/test/cxl-elc.sh test to:
derive the SPA and expected alias dynamically from the ELC region, inject a CE and verify the alias-offlining path is not reached, and inject a UC and verify the expected alias-offlining path is still reached.
So I will first confirm that the existing tools/testing/cxl infrastructure is sufficient as-is, then share the cxl-elc.sh changes here for review before sending a separate ndctl patch.
Thanks,
Shaikh
Alison Schofield
maintainerQuoted 23 lines · Shaikh Kamaluddin
cxl_handle_mce()offlines the aliased page of an ELC region on any record with a usable address; it does not checkMCI_STATUS_UCor filter non-memory errors.uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters onmce->severityand leaves corrected errors untouched.cxl_handle_mce()has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired viamemory_failure().Corrected errors do reach the chain:
machine_check_poll()logs them via the samemce_gen_pool_process()path that feedsx86_mce_decoder_chain, andcxl_extended_linear_cache_resize()extendsp->resto cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.Filter the record as
nfit_handle_mce()does. Commitfc08a4703a41("acpi, nfit: Fix the memory error check innfit_handle_mce()") and commit5d96c9342c23("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler callsmemory_failure()rather than recording a bad block.
Quoted 7 lines
mce_is_correctable()is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined onMCE_AR_SEVERITY, wherekill_me_maybe()owns the reported page but nothing owns the alias.Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")
Thanks for the patch Shaikh,
The patch looks good to me, and the reproduction details show that this has been thoroughly investigated.
The commit log is harder to follow than the code change itself. It seems that too much of the investigation has leaked into the commit log which makes it very dense and not very clarifying. For what is ultimately a small and straightforward behavioral fix, the log spends a lot of time narrating the code paths that establish the problem, while the important impact, a reduction in usable memory, gets lost.
The commit log should introduce the code change and make it easier to understand. I'd prefer the usual behavioral progression: describe what happens today, why that is wrong and its impact, then state how the patch fixes it. Also, please spell out Extended Linear Cache (ELC) on first use.
The subject can describe the behavior being fixed more directly as well.
For example: (if you use this example, you need to proofread it, to be sure I actually captured this correctly ;))
cxl/mce: Avoid alias page retirement for corrected errors
cxl_handle_mce() offlines the aliased page of an Extended Linear Cache (ELC) region for any MCE with a usable address in the region. This includes corrected errors and non-memory errors, needlessly reducing usable memory.
Restrict ELC MCE handling to uncorrected memory errors. Use mce_is_correctable() so that MCE_AR_SEVERITY errors continue to retire the alias when the reported page is handled elsewhere.
more below...
Quoted 2 lines
Signed-off-by Shaikh Kamaluddin <[email protected]>---
The detailed investigation and reproduction below the scissors is useful and should stay there.
snip
Quoted 7 lines
drivers/cxl/core/mce.c +0
-0@@ -18,7 +18,14 @@ static int cxl_handle_mce(struct notifier_block *nb, unsigned long val,u64 spa, spa_alias;
drivers/cxl/core/mce.c static int cxl_handle_mce(struct notifier_block *nb, unsigned long val, · line 27
Quoted 10 lines
drivers/cxl/core/mce.c +7
-1unsigned long pfn; -if (!mce || !mce_usable_address (mce)) +if (!mce) +return NOTIFY_DONE ; + +/* Only uncorrected memory errors warrant taking down the alias page */ +if (!mce_is_memory_error (mce) ||mce_is_correctable (mce)) +return NOTIFY_DONE ; +
The comment restates what the if condition says. I think the comment can simply be dropped.
Quoted 9 lines
drivers/cxl/core/mce.c +1
-0+if (!mce_usable_address (mce))return NOTIFY_DONE ; spa = mce->addr &MCI_ADDR_PHYSADDR ;base-commit:
7098e9cd98a05c0c5de2fae0c2465f9d966fdd07Signature · 1 line
2.43.0
Quoted 61 lines · Alison Schofield
Quoted 23 lines · Shaikh Kamaluddin
cxl_handle_mce()offlines the aliased page of an ELC region on any record with a usable address; it does not checkMCI_STATUS_UCor filter non-memory errors.uc_decode_notifier(), the equivalent handler for plain memory on the same chain at the same priority, filters onmce->severityand leaves corrected errors untouched.cxl_handle_mce()has no such gate, so a corrected error - which the generic handler ignores - still causes the alias to be permanently retired viamemory_failure().Corrected errors do reach the chain:
machine_check_poll()logs them via the samemce_gen_pool_process()path that feedsx86_mce_decoder_chain, andcxl_extended_linear_cache_resize()extendsp->resto cover the DRAM half of the ELC pair, so a routine DRAM CE carries an address inside the region resource.Filter the record as
nfit_handle_mce()does. Commitfc08a4703a41("acpi, nfit: Fix the memory error check innfit_handle_mce()") and commit5d96c9342c23("acpi/nfit, x86/mce: Handle only uncorrectable machine checks") established this filter for an equivalent handler on the same notifier chain; the consequence here is more severe, as the CXL handler callsmemory_failure()rather than recording a bad block.Quoted 7 lines
mce_is_correctable()is used instead of copying uc_decode_notifier()'s AO/DEFERRED test because the alias must still be offlined onMCE_AR_SEVERITY, wherekill_me_maybe()owns the reported page but nothing owns the alias.Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")Thanks for the patch Shaikh,
The patch looks good to me, and the reproduction details show that this has been thoroughly investigated.
The commit log is harder to follow than the code change itself. It seems that too much of the investigation has leaked into the commit log which makes it very dense and not very clarifying. For what is ultimately a small and straightforward behavioral fix, the log spends a lot of time narrating the code paths that establish the problem, while the important impact, a reduction in usable memory, gets lost.
The commit log should introduce the code change and make it easier to understand. I'd prefer the usual behavioral progression: describe what happens today, why that is wrong and its impact, then state how the patch fixes it. Also, please spell out Extended Linear Cache (ELC) on first use.
The subject can describe the behavior being fixed more directly as well.
For example: (if you use this example, you need to proofread it, to be sure I actually captured this correctly ;))
cxl/mce: Avoid alias page retirement for corrected errors
cxl_handle_mce()offlines the aliased page of an Extended Linear Cache (ELC) region for any MCE with a usable address in the region. This includes corrected errors and non-memory errors, needlessly reducing usable memory.
Quoted 6 lines
Restrict ELC MCE handling to uncorrected memory errors. Use
mce_is_correctable()so thatMCE_AR_SEVERITYerrors continue to retire the alias when the reported page is handled elsewhere.Fixes 516e5bd0b6bf ("cxl: Add mce notifier to emit aliased address for extended linear cache")
Thanks for the detailed feedback, Alison - that's a much better structure for the commit log, and matches what you captured.
Quoted 21 lines
more below...
Quoted 2 lines
Signed-off-by Shaikh Kamaluddin <[email protected]>---
The detailed investigation and reproduction below the scissors is useful and should stay there.
snip
Quoted 9 lines
drivers/cxl/core/mce.c +0
-0@@ -18,7 +18,14 @@ static int cxl_handle_mce(struct notifier_block *nb, unsigned long val,u64 spa, spa_alias;unsigned long pfn;
Quoted 12 lines
drivers/cxl/core/mce.c static int cxl_handle_mce(struct notifier_block *nb, unsigned long val, · line 27
Quoted 8 lines
drivers/cxl/core/mce.c +7
-1-if (!mce || !mce_usable_address (mce)) +if (!mce) +return NOTIFY_DONE ; + +/* Only uncorrected memory errors warrant taking down the alias page */ +if (!mce_is_memory_error (mce) ||mce_is_correctable (mce)) +return NOTIFY_DONE ; +The comment restates what the if condition says. I think the comment can simply be dropped.
Dropped the comment from the code and sent the v2
https://lore.kernel.org/all/[email protected]/
Quoted 11 lines
Quoted 11 lines
drivers/cxl/core/mce.c +1
-0+if (!mce_usable_address (mce))return NOTIFY_DONE ; spa = mce->addr &MCI_ADDR_PHYSADDR ;base-commit:
7098e9cd98a05c0c5de2fae0c2465f9d966fdd07Signature · 1 line
2.43.0