Persist FRU memory poisons
Subsystems (8)
cover Persist FRU memory poisons
From: Muralidhara M K <[email protected]>
This patch set is based on the patches submitted
https://lore.kernel.org/linux-edac/[email protected]/T/#t
MI300A has on-die HBMv3 memory embedded on to socket. Upon reaching threshold of memory errors socket has to be replaced. Define the criteria to identify the Field Replicable Unit(FRU) based on number of poisoned pages in the socket by persisting them in a non-volatile storage.
Notifier is registered to handle the FRU memory poisons and poison count incremented based on injected MCE errors until it reaches maximum number of fru poison entries.
Sysfs entry per FRU will ease the use to look into the poison details.
During boot, Read the ERST records for identifying the poison address and retire all system physical addresses in that HBM row.
Patch 1:
Add an API to get the maximum CPER record size to be stored in NV storage
Patch 2:
Add FRU memory poison module
Patch 3:
Add sysfs entry to print the required error information from poison records
Patch 4:
Add documentation on FRU memory poisons.
Muralidhara M K (4):
ACPI/APEI: Add erst_get_size() API
RAS/fmp: Add FRU memory poison CPER support for Error persistence EDAC/amd64: Add sysfs entry to read FRU poison data RAS/fmp: Add Documentation on Persistence of FRU memory poisons
Documentation/RAS/ras.rst | 122 +++++++ MAINTAINERS | 8 + drivers/acpi/apei/erst.c | 9 + drivers/edac/amd64_edac.c | 25 ++ drivers/ras/Kconfig | 1 + drivers/ras/Makefile | 1 + drivers/ras/fmp/Kconfig | 18 + drivers/ras/fmp/Makefile | 10 + drivers/ras/fmp/fru_mem_poison.c | 595 +++++++++++++++++++++++++++++++ include/acpi/apei.h | 1 + include/linux/cper.h | 24 ++ include/linux/fru_mem_poison.h | 17 + 12 files changed, 831 insertions(+) create mode 100644 drivers/ras/fmp/Kconfig create mode 100644 drivers/ras/fmp/Makefile create mode 100644 drivers/ras/fmp/fru_mem_poison.c create mode 100644 include/linux/fru_mem_poison.h
Signature · 1 line
2.25.1
PATCH 1/4 ACPI/APEI: Add erst_get_size() API
From: Muralidhara M K <[email protected]>
Support API for size of a record in the ERST.
--- drivers/acpi/apei/erst.c | 9 +++++++++ include/acpi/apei.h | 1 + 2 files changed, 10 insertions(+)
drivers/acpi/apei/erst.c +9 -0
@@ -420,6 +420,15 @@ static int erst_get_erange(struct erst_erange *range)return 0 ; } +u64 erst_get_size (void ) +{ +if (erst_disable) +return 0 ; + +return erst_erange.size; +} +EXPORT_SYMBOL_GPL (erst_get_size); +static ssize_t __erst_get_record_count (void ) {struct apei_exec_context ctx;
include/acpi/apei.h +1 -0
@@ -41,6 +41,7 @@ static inline void acpi_hest_init(void) { }int erst_write (const struct cper_record_header *record);ssize_t erst_get_record_count (void ); +u64 erst_get_size (void );int erst_get_record_id_begin (int *pos);int erst_get_record_id_next (int *pos,u64 *record_id);void erst_get_record_id_end (void );
Signature · 1 line
2.25.1
PATCH 2/4 RAS/fmp: Add FRU memory poison CPER support for Error persistence
In reply to Muralidhara M KFrom: Muralidhara M K <[email protected]>
Large-scale Data center servers such as MI300A has on-chip stacked HBM3 (High Bandwidth Memory v3) memory embedded on to CPU socket.
Many memory errors tend to be consistent or intermittent and may recur. Upon reaching a certain threshold of these errors, the specific memory area is deemed faulty and should be replaced. In the case of on-die HBM, any returns due to these issues will likely be directed to the socket vendor.
Define a criteria to identify the Field Replicable Unit(FRU) by evaluating the count of "poisoned" pages within the socket and log these poisoned pages persistently in a non-volatile storage. This process assists in retaining information about defective memory address within the socket for potential replacement.
To achieve this, CPER structure for FRU memory poisoning is defined. For each FRU, identified by its Protected Processor Inventory Number(PPIN), one CPER is allocated. Each CPER includes a poison list offset to the corresponding PPIN, capable of holding n number of poison entries.
During boot, OS creates CPER FMP records based on number of sockets available in the system and size of each record is calculated based on available ERST non-volatile storage provided by BIOS.
In mission mode, Notifier is registered to handle the FRU memory poison events. The poison count is incremented through MCE error injections, storing MCE address, MCA_IPID and poison count in the poison list entries until it attains maximum number of FRU poison entries.
Once maximum FRU poison entries are attained, further storage of poison events is no longer possible. instead, a warning message is printed. The user can configure the number of FRU poison entries through kernel command line argument, by passing "fru_mem_poison.fru_poison_entries=N".
During next bootup, the OS reads the ERST records. The OS matches a system PPIN with MCE PPIN, enables the retirement of all the poison addresses in a column of the particular row in a HBM to prevent the usage of compromised memory address in subsequent boot.
--- MAINTAINERS | 8 + drivers/ras/Kconfig | 1 + drivers/ras/Makefile | 1 + drivers/ras/fmp/Kconfig | 18 ++ drivers/ras/fmp/Makefile | 10 + drivers/ras/fmp/fru_mem_poison.c | 522 +++++++++++++++++++++++++++++++ include/linux/cper.h | 24 ++ include/linux/fru_mem_poison.h | 15 + 8 files changed, 599 insertions(+) create mode 100644 drivers/ras/fmp/Kconfig create mode 100644 drivers/ras/fmp/Makefile create mode 100644 drivers/ras/fmp/fru_mem_poison.c create mode 100644 include/linux/fru_mem_poison.h
MAINTAINERS +8 -0
@@ -900,6 +900,14 @@ S: Supported F: drivers/ras/amd/atl F: include/linux/amd-atl.h +RAS PERSIST FRU MEMORY POISON (FMP) +M: Naveen Krishna Chatradhi <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/driver-api/persist_fru_memory_poison.rst +F: drivers/ras/fmp/fru_mem_poison.c +F: include/linux/fru_mem_poison.h + AMD CDX BUS DRIVER M: Nipun Gupta <[email protected]> M: Nikhil Agarwal <[email protected]>
drivers/ras/Kconfig +1 -0
@@ -33,5 +33,6 @@ if RAS source "arch/x86/ras/Kconfig" source "drivers/ras/amd/atl/Kconfig" +source "drivers/ras/fmp/Kconfig" endif
drivers/ras/Makefile +1 -0
@@ -3,3 +3,4 @@ obj-$(CONFIG_RAS) += ras.o obj-$(CONFIG_DEBUG_FS) += debugfs.o obj-$(CONFIG_RAS_CEC) += cec.o obj-$(CONFIG_AMD_ATL) += amd/atl/ +obj-$(CONFIG_RAS_FMP) += fmp/
drivers/ras/fmp/Kconfig new file +18 -0
@@ -0,0 +1,18 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# FRU(Field Replaceable Unit) Memory Poison Kconfig +# +# Copyright (c) 2023, Advanced Micro Devices, Inc. +# All Rights Reserved. +# +# Author: Naveen Krishna Chatradhi <[email protected]> + +config RAS_FMP + tristate "Support persisting FRU memory poisons" + default m + depends on X86_MCE && X86_LOCAL_APIC + help + Additional support for persisting FRU(Field Replaceable Unit) + memory poisons. + Include support of retiring the poison pages during boot, idea is + don't try and use the broken memory again during boot.
drivers/ras/fmp/Makefile new file +10 -0
@@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# FRU(Field Replaceable Unit) Memory Poison Makefile +# +# Copyright (c) 2023, Advanced Micro Devices, Inc. +# All Rights Reserved. +# +# Author: Naveen Krishna Chatradhi <[email protected]> + +obj-$(CONFIG_RAS_FMP) += fru_mem_poison.o
drivers/ras/fmp/fru_mem_poison.c new file +522 -0
522 lines not on this page · raw message
include/linux/cper.h +24 -0
@@ -550,6 +550,30 @@ struct cper_sec_fw_err_rec_ref {guid_t record_identifier_guid; }; +/* FRU Memory Poison Error Section data, UEFI vX.Y sec N.X.Y */ +struct cper_fru_poison_data { +u32 hw_id_type; +u32 addr_type; +u64 hw_id;/* MCA_IPID register in case of AMD processors */ +u64 addr;/* MCA_ADDR register in case of x86 processors */ +u64 timestamp;/* Timestamp is UTC format */ +}; + +/* Memory Poison Error Record Section, UEFI vX.Y sec N.X.Z */ +struct cper_sec_fru_mem_poisons { +char signature[4 ]; +u64 checksum; +u32 model_id_type; +u32 model_id; +u32 fru_id_type; +u64 fru_id; +u64 poison_count; +u64 p_list_off;/* Allocate contiguous memory for max_number_poison_entries +* based on available NV storage at the end of the +* struct cper_sec_fru_mem_poisons +*/ +}; +/* Reset to default packing */ #pragma pack()
include/linux/fru_mem_poison.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Persist FRU Memory Poison driver +* +* Copyright (c) 2023, Advanced Micro Devices, Inc. +* All Rights Reserved. +* +* Author: Naveen Krishna Chatradhi <[email protected]> +*/ +#ifndef _X86_FMP_H +#define _X86_FMP_H + +struct system_fru_poison_info **sys_fmp_info; + +#endif /* _X86_FMP_H */
Signature · 1 line
2.25.1
PATCH 3/4 EDAC/amd64: Add sysfs entry to read FRU poison data
In reply to Muralidhara M KFrom: Muralidhara M K <[email protected]>
Create sysfs file for each FRU ID with a list of DRAM MCE address and MCA IPID value stored in ERST non-volatile storage.
Read the CPER Record information at any time when the system is up using below command of particular node or FRU index of sysfs entry.
Example: cat /sys/devices/system/edac/mc/<node index>/fmpl
Data in sysfs entries is able to identify the list of poisoned addresses and FRU index to decide on the replaceble criteria instead of iterating over the kernel logs.
--- drivers/edac/amd64_edac.c | 25 +++++++++++ drivers/ras/fmp/fru_mem_poison.c | 77 +++++++++++++++++++++++++++++++- include/linux/fru_mem_poison.h | 2 + 3 files changed, 102 insertions(+), 2 deletions(-)
drivers/edac/amd64_edac.c +25 -0
@@ -2,6 +2,7 @@#include "amd64_edac.h" #include <asm/amd_nb.h> #include <linux/amd-atl.h> +#include <linux/fru_mem_poison.h> static struct edac_pci_ctl_info *pci_ctl;
@@ -574,6 +575,28 @@ static ssize_t dram_hole_show(struct device *dev, struct device_attribute *mattr hole_size); } +/* sysfs entry to read FRU(Field Repaceable Unit) memory Poisons */ +static ssize_t fmpl_show (struct device *dev,struct device_attribute *mattr, +char *data) +{ +struct mem_ctl_info *mci =to_mci (dev); +struct amd64_pvt *pvt = mci->pvt_info; +ssize_t ret_len =0 , buf_size =PAGE_SIZE ; +char *buf; + + buf =kmalloc (buf_size,GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + + ret_len =copy_fmp_data_from_cache (pvt->mc_node_id, buf, buf_size); +if (!ret_len) +return -EINVAL ; + +memcpy (data, buf, ret_len); +kfree (buf); +return ret_len; +} +/* * update NUM_DBG_ATTRS in case you add new members */
@@ -581,6 +604,7 @@ static DEVICE_ATTR(dhar, S_IRUGO, dhar_show, NULL); static DEVICE_ATTR(dbam, S_IRUGO, dbam0_show, NULL); static DEVICE_ATTR(topmem, S_IRUGO, top_mem_show, NULL); static DEVICE_ATTR(topmem2, S_IRUGO, top_mem2_show, NULL); +static DEVICE_ATTR(fmpl, S_IRUGO, fmpl_show, NULL);static DEVICE_ATTR_RO (dram_hole);static struct attribute *dbg_attrs[] = {
@@ -589,6 +613,7 @@ static struct attribute *dbg_attrs[] = { &dev_attr_topmem.attr, &dev_attr_topmem2.attr, &dev_attr_dram_hole.attr, + &dev_attr_fmpl.attr, NULL };
drivers/ras/fmp/fru_mem_poison.c +75 -2
96 lines not on this page · raw message
include/linux/fru_mem_poison.h +2 -0
@@ -12,4 +12,6 @@struct system_fru_poison_info **sys_fmp_info; +ssize_t copy_fmp_data_from_cache (int fru_idx,char *buf,ssize_t buf_size); +#endif /* _X86_FMP_H */
Signature · 1 line
2.25.1
PATCH 4/4 RAS/fmp: Add Documentation on Persistence of FRU memory poisons
In reply to Muralidhara M KFrom: Muralidhara M K <[email protected]>
On Data center servers with On chip HBM3 memory, FRU identification needs a mechanism to identify the bad page information by persisting them in non volatile storage across reboots and read them during boot helps to check the number of pages poisoned.
--- Documentation/RAS/ras.rst | 122 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 122 insertions(+)
Documentation/RAS/ras.rst +122 -0
@@ -24,3 +24,125 @@ Also, the user can pass particular family and model to decode the error string:: $ rasdaemon -p --status <STATUS> --ipid <IPID> --smca --family <CPU Family> --model <CPU Model> --bank <BANK_NUM> + +============================================================= +Persist FRU(Field Replaceable Unit) Memory Poison +============================================================= + +Large scale Data center servers such as MI300A has on-chip stacked Memory +High Bandwidth Memory v3 (HBM). + - Example: MI300A has 8 stacks of HBM/die, a total of 128Gb per socket. +Host Operating system is responsible for memory management, allocating HBM3 pages. + +Many memory errors tend to be consistent or intermittent and may reoccur. Upon +reaching a certain threshold of these errors, the specific memory area is deemed +faulty and should be replaced. In the case of on-die High Bandwidth Memory (HBM), +any returns due to these issues will likely be directed to the socket vendor. + +Define a criteria to identify the Field Replicable Unit(FRU) by evaluating the +count of "poisoned" pages within the socket and log these poisoned pages persistently +in a non-volatile storage. This process assists in retaining information about +defective memory page within the socket for potential replacement. + +Linux supports retiring pages by marking the page HW_POISON. However, it doesn't +persist these marked pages across reboots. +To address this, a potential solution is to persist bad page details in non-volatile +storage(ERST). This prevents the reuse of compromised memory region, ensuring they +are not utilized again. + +ERST to persist Bad page information +==================================== + +ERST (Error Record Serialization Table) defined by ACPI/APEI provides a mechanism for +storing and retrieve hardware error Information to and from a persistent memory. + +Platform FW(BIOS) with ERST support, reserves ERST tables usually 64KB in non-volatile +storage. Configure Linux to select ERST as backend for Pstore (read/write from NV storage). + +Upon Specific MCE errors Linux would call pstore with CPER format per FRU, platform FW +would store it in NV storage. and on next boot, Linux would query bad page information +from ERST and retire the pages again. + +FRU memory poison Common Platform Error Record (CPER) definition +================================================================ + +One CPER per FRU (Protected Processor Inventory Number (PPIN)). +1 CPER record per MI300A socket (4 X MI300As system) with the 4 CPERs in a system and +Each FRU containing poison list offset of the given PPIN. + +The FRU poison CPER record size is (BIOS ERST memory) / (Number of FRUs). +Each erst_write() or erst_read() will write/read this entire structure as one record. + +Number of poison entries that can be reached is based on the calculation below +"(size - sizeof(struct cper_poison_record)) / sizeof(struct cper_fru_poison_data)" + +FRU Poison CPER definition for storing error record as below + + struct cper_poison_record { + struct cper_record_header hdr; + struct cper_section_descriptor sec_hdr; + struct cper_sec_fru_mem_poisons fmpl; + } __packed; + +use 'struct cper_record_header' and 'struct cper_section_descriptor' as defined +in 'include/linux/cper.h' + + * Section body follows the description of a “non-standard section body” and is defined below. + + * per FRU poison section data + struct cper_sec_fru_mem_poisons { + char signature[4]; + u64 checksum; + u32 model_id_type; + u32 model_id; + u32 fru_id_type; + u64 fru_id; + u32 poison_count; + u64 p_list_off; //offset for contiguous memory to poison data structure + }; + + * FRU Poison data structure + struct cper_fru_poison_data { + u32 hw_id_type; + u32 addr_type; + u64 hw_id; + u64 addr; + }; + + +Implementation Notes on FRU Identification: +========================================== + + * HBM suppose to have total of 8 DRAM rows. + * When MCE error occurs, offline all the pages in that range in a particular row(8 columns in a row). + If all the 8 rows become bad, then entire socket has to be replaced. + * Perist the error information mentioned in "struct cper_fru_poison_data" to ERST storage. + + * Don’t delete the FMP records once they are saved in persistence store. Keep them in ERST + forever until all the poison_data entries become full. + * Once the entries full, then do not save the error information in ERST. + +At OS boot: +========== + * One CPER per FRU (Protected Processor Inventory Number (PPIN)) has been created. + * Size of each CPER will not exceed (1/4)th the available space. + * The node controller should make sure there is a CPER for each PPIN in the node. If this is a + new processor never seen before, then create a CPER with N=0. + * Read the CPERs through the Error Record Serialization Table (ERST). + * If OS matches a PPIN to a socket and identifes mce address, it will re-create the SPA for all + pages on the HBM row of the poisoned DA, retire all pages mapped to that row. + * If a CPER is found for a PPIN that isn’t in the node, OS will print a warning. + * If OS tries to persist more errors than fit in the CPER, will refuse to update the CPER + and print a message. + * OS creates sysfs file for each FRU_ID with a list of DRAM address, MCA_IPID which are retired. + $ ls /sys/devices/system/edac/mc/mc0/fmpl + * Example: mc0 for socket 0 and mc3 for socket 3. + * To read the CPER Record information at any time when the system is up follow below + $ cat /sys/devices/system/edac/mc/mc<socket_index>/fmpl + $ dmesg + +At Mission mode: +=============== + * Notifier is registered to handle the FRU memory poison errors. + * When the error is injected on particular PPIN, and If OS matches a system PPIN to a socket + with MCE PPIN, append the poison data until it reaches maximum number of poison entries.
Signature · 1 line
2.25.1