Skip to contentSkip to conversation

CXL, ACPI, APEI, EINJ: Update EINJ for CXL error types

PATCH Superseded Ben Cheatham 3 participants 8 messages Oct 10, 2023 – Oct 11, 2023

Subsystems (4)

v6 of 15 Superseded Latest is v15 →

lore
Share & tools
lore
#0 lore raw

v6 cover CXL, ACPI, APEI, EINJ: Update EINJ for CXL error types

v6 Changes:
- Reworked to have CXL error types under /sys/kernel/debug/cxl (Dan)
- Removed CXL error types from legacy EINJ interface in favor of new interface
- Removed cxl_rcrb_addr file
- Added optional patch for CXL error type #defines (patch 2/5)
- Changes to documentation updates to match rework
- Change base to cxl-fixes branch

The new CXL error types will use the Memory Address field in the SET_ERROR_TYPE_WITH_ADDRESS structure in order to target a CXL 1.1 compliant memory-mapped downstream port. The value of the memory address will be in the port's MMIO range, and it will not represent physical (normal or persistent) memory.

Add the functionality for injecting CXL 1.1 errors to the EINJ module, but not through the EINJ legacy interface under /sys/kernel/debug/apei/einj. Instead, make the error types available under /sys/kernel/debug/cxl. This allows for validating the MMIO address for a CXL 1.1 error type while also not making the user responsible for finding it.

Ben Cheatham (5):
cxl/port: Add EINJ debugfs files and callback support
ACPI: Add CXL protocol error defines
EINJ: Separate CXL errors from other EINJ errors cxl/port, EINJ: Add CXL EINJ callback functions
EINJ: Update EINJ documentation

 Documentation/ABI/testing/debugfs-cxl         |  27 ++++
 .../firmware-guide/acpi/apei/einj.rst         |  12 ++
 drivers/acpi/apei/Kconfig                     |   3 +
 drivers/acpi/apei/einj.c                      | 149 ++++++++++++++++--
 drivers/cxl/core/port.c                       |  84 ++++++++++
 drivers/cxl/cxl.h                             |  10 ++
 include/acpi/actbl1.h                         |   6 +
 7 files changed, 281 insertions(+), 10 deletions(-)

base-commit: c66650d29764e228eba40b7a59fdb70fa6567daa

Signature · 1 line
2.34.1
#1 lore raw

PATCH v6 1/5 cxl/port: Add EINJ debugfs files and callback support

Add creation of debugfs directories for ports and dports under /sys/kernel/debug/cxl when EINJ support is enabled. The dport directories will contain files for injecting CXL protocol errors. These files are only usable once the EINJ module has loaded and registered callback functions with the CXL core module, before that occurs (or if the EINJ module isn't loaded) the files will do nothing and return an ENODEV error.

---
 Documentation/ABI/testing/debugfs-cxl | 27 +++++++++
 drivers/cxl/core/port.c               | 84 +++++++++++++++++++++++++++
 drivers/cxl/cxl.h                     | 10 ++++
 3 files changed, 121 insertions(+)
Documentation/ABI/testing/debugfs-cxl +27 -0
index fe61d372e3fa..e87cb687cb1c 100644
@@ -33,3 +33,30 @@ Description:
 		device cannot clear poison from the address, -ENXIO is returned.
 		The clear_poison attribute is only visible for devices
 		supporting the capability.
+
+What:		/sys/kernel/debug/cxl/portX/dportY/einj_types
+Date:		October, 2023
+KernelVersion:	v6.7
+Contact:	[email protected]
+Description:
+		(RO) Prints the CXL protocol error types made available by
+		the platform in the format "0x<error number>	<error type>".
+		The <error number> can be written to einj_inject to inject
+		<error type> into dportY. This file is only visible if
+		CONFIG_ACPI_APEI_EINJ is enabled, and the EINJ module must
+		be able to reach one (or both) of the CXL_ACPI or CXL_PORT
+		modules to be functional.
+
+What:		/sys/kernel/debug/cxl/portX/dportY/einj_inject
+Date:		October, 2023
+KernelVersion:	v6.7
+Contact:	[email protected]
+Description:
+		(WO) Writing an integer to this file injects the corresponding
+		CXL protocol error into dportY (integer to type mapping is
+		available by reading from einj_types). If the dport was
+		enumerated in RCH mode, a CXL 1.1 error is injected, otherwise
+		a CXL 2.0 error is injected. This file is only visible if
+		CONFIG_ACPI_APEI_EINJ is enabled, and the EINJ module must
+		be able to reach one (or both) of the CXL_ACPI or CXL_PORT
+		modules to be functional.
drivers/cxl/core/port.c +84 -0

126 lines not on this page · raw message

drivers/cxl/cxl.h +10 -0
index 76d92561af29..12025ed6b64e 100644
@@ -581,6 +581,7 @@ struct cxl_dax_region {
  * @depth: How deep this port is relative to the root. depth 0 is the root.
  * @cdat: Cached CDAT data
  * @cdat_available: Should a CDAT attribute be available in sysfs
+ * @debug_dir: dentry for port in cxl debugfs (optional)
  */
 struct cxl_port {
 	struct device dev;
@@ -604,6 +605,7 @@ struct cxl_port {
 		size_t length;
 	} cdat;
 	bool cdat_available;
+	struct dentry *debug_dir;
 };
 
 static inline struct cxl_dport *
@@ -795,6 +797,14 @@ bool is_cxl_nvdimm_bridge(struct device *dev);
 int devm_cxl_add_nvdimm(struct cxl_memdev *cxlmd);
 struct cxl_nvdimm_bridge *cxl_find_nvdimm_bridge(struct cxl_memdev *cxlmd);
 
+struct cxl_einj_ops {
+	int (*einj_type)(struct seq_file *f, void *data);
+	int (*einj_inject)(struct cxl_dport *dport, u64 type);
+};
+
+void cxl_einj_set_ops_cbs(struct cxl_einj_ops *ops);
+
+
 #ifdef CONFIG_CXL_REGION
 bool is_cxl_pmem_region(struct device *dev);
 struct cxl_pmem_region *to_cxl_pmem_region(struct device *dev);
Signature · 1 line
2.34.1
#2 lore raw

PATCH v6 2/5 ACPI: Add CXL protocol error defines

In reply to Ben Cheatham

Add CXL protocol error defines to include/actbl1.h to be used in later commits.

---

I made a pull request for adding this support into the acpica project that has already been merged [1], Feel free to discard this commit and use the acpica changes instead.

[1]:

 include/acpi/actbl1.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/acpi/actbl1.h +6 -0
index 8d5572ad48cb..9741f1c40a4e 100644
@@ -1093,6 +1093,12 @@ enum acpi_einj_command_status {
 #define ACPI_EINJ_PLATFORM_CORRECTABLE      (1<<9)
 #define ACPI_EINJ_PLATFORM_UNCORRECTABLE    (1<<10)
 #define ACPI_EINJ_PLATFORM_FATAL            (1<<11)
+#define ACPI_EINJ_CXL_CACHE_CORRECTABLE     (1<<12)
+#define ACPI_EINJ_CXL_CACHE_UNCORRECTABLE   (1<<13)
+#define ACPI_EINJ_CXL_CACHE_FATAL           (1<<14)
+#define ACPI_EINJ_CXL_MEM_CORRECTABLE       (1<<15)
+#define ACPI_EINJ_CXL_MEM_UNCORRECTABLE     (1<<16)
+#define ACPI_EINJ_CXL_MEM_FATAL             (1<<17)
 #define ACPI_EINJ_VENDOR_DEFINED            (1<<31)
 
 /*******************************************************************************
Signature · 1 line
2.34.1
#3 lore raw

PATCH v6 3/5 EINJ: Separate CXL errors from other EINJ errors

In reply to Ben Cheatham

Separate CXL error types from other EINJ error types and disallow them in the legacy EINJ interface under /sys/kernel/debug/apei/einj. Support for the CXL error types will be added under /sys/kernel/debug/cxl in the next commit.

---
 drivers/acpi/apei/einj.c | 56 +++++++++++++++++++++++++++++-----------
 1 file changed, 41 insertions(+), 15 deletions(-)
drivers/acpi/apei/einj.c +41 -15
index 013eb621dc92..330329ac2f1f 100644
@@ -36,6 +36,12 @@
 #define MEM_ERROR_MASK		(ACPI_EINJ_MEMORY_CORRECTABLE | \
 				ACPI_EINJ_MEMORY_UNCORRECTABLE | \
 				ACPI_EINJ_MEMORY_FATAL)
+#define CXL_ERROR_MASK		(ACPI_EINJ_CXL_CACHE_CORRECTABLE | \
+				ACPI_EINJ_CXL_CACHE_UNCORRECTABLE | \
+				ACPI_EINJ_CXL_CACHE_FATAL | \
+				ACPI_EINJ_CXL_MEM_CORRECTABLE | \
+				ACPI_EINJ_CXL_MEM_UNCORRECTABLE | \
+				ACPI_EINJ_CXL_MEM_FATAL)
 
 /*
  * ACPI version 5 provides a SET_ERROR_TYPE_WITH_ADDRESS action.
@@ -537,8 +543,11 @@ static int einj_error_inject(u32 type, u32 flags, u64 param1, u64 param2,
 	if (type & ACPI5_VENDOR_BIT) {
 		if (vendor_flags != SETWA_FLAGS_MEM)
 			goto inject;
-	} else if (!(type & MEM_ERROR_MASK) && !(flags & SETWA_FLAGS_MEM))
+	} else if (!(type & MEM_ERROR_MASK) && !(flags & SETWA_FLAGS_MEM)) {
 		goto inject;
+	} else if ((type & CXL_ERROR_MASK) && (flags & SETWA_FLAGS_MEM)) {
+		goto inject;
+	}
 
 	/*
 	 * Disallow crazy address masks that give BIOS leeway to pick
@@ -590,6 +599,9 @@ static const char * const einj_error_type_string[] = {
 	"0x00000200\tPlatform Correctable\n",
 	"0x00000400\tPlatform Uncorrectable non-fatal\n",
 	"0x00000800\tPlatform Uncorrectable fatal\n",
+};
+
+static const char * const einj_cxl_error_type_string[] = {
 	"0x00001000\tCXL.cache Protocol Correctable\n",
 	"0x00002000\tCXL.cache Protocol Uncorrectable non-fatal\n",
 	"0x00004000\tCXL.cache Protocol Uncorrectable fatal\n",
@@ -615,29 +627,21 @@ static int available_error_type_show(struct seq_file *m, void *v)
 
 DEFINE_SHOW_ATTRIBUTE(available_error_type);
 
-static int error_type_get(void *data, u64 *val)
-{
-	*val = error_type;
-
-	return 0;
-}
-
-static int error_type_set(void *data, u64 val)
+static int validate_error_type(u64 type)
 {
+	u32 tval, vendor, available_error_type = 0;
 	int rc;
-	u32 available_error_type = 0;
-	u32 tval, vendor;
 
 	/* Only low 32 bits for error type are valid */
-	if (val & GENMASK_ULL(63, 32))
+	if (type & GENMASK_ULL(63, 32))
 		return -EINVAL;
 
 	/*
 	 * Vendor defined types have 0x80000000 bit set, and
 	 * are not enumerated by ACPI_EINJ_GET_ERROR_TYPE
 	 */
-	vendor = val & ACPI5_VENDOR_BIT;
-	tval = val & 0x7fffffff;
+	vendor = type & ACPI5_VENDOR_BIT;
+	tval = type & 0x7fffffff;
 
 	/* Only one error type can be specified */
 	if (tval & (tval - 1))
@@ -646,9 +650,31 @@ static int error_type_set(void *data, u64 val)
 		rc = einj_get_available_error_type(&available_error_type);
 		if (rc)
 			return rc;
-		if (!(val & available_error_type))
+		if (!(type & available_error_type))
 			return -EINVAL;
 	}
+
+	return 0;
+}
+
+static int error_type_get(void *data, u64 *val)
+{
+	*val = error_type;
+
+	return 0;
+}
+
+static int error_type_set(void *data, u64 val)
+{
+	int rc;
+
+	if (val & CXL_ERROR_MASK && !(val & ACPI5_VENDOR_BIT))
+		return -EINVAL;
+
+	rc = validate_error_type(val);
+	if (rc)
+		return rc;
+
 	error_type = val;
 
 	return 0;
Signature · 1 line
2.34.1
#4 lore raw

PATCH v6 4/5 cxl/port, EINJ: Add CXL EINJ callback functions

In reply to Ben Cheatham

Add functions to the EINJ module to be used in the CXL module for CXL protocol error injection. The callbacks implement the einj_types and einj_inject files under /sys/kernel/debug/cxl/portX/dportY. These two files work in the same way as the available_error_types and error_inject files under /sys/kernel/debug/apei/einj, but only for CXL error types. If the dport is enumerated in RCH (CXL 1.1) mode, a CXL 1.1 error is injected into the dport; a CXL 2.0 error is injected otherwise.

---
 drivers/acpi/apei/Kconfig |   3 ++
 drivers/acpi/apei/einj.c  | 103 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 106 insertions(+)
drivers/acpi/apei/Kconfig +3 -0
index 6b18f8bc7be3..100c27beb581 100644
@@ -11,6 +11,9 @@ config ACPI_APEI
 	select PSTORE
 	select UEFI_CPER
 	depends on HAVE_ACPI_APEI
+	imply CXL_BUS
+	imply CXL_ACPI
+	imply CXL_PORT
 	help
 	  APEI allows to report errors (for example from the chipset)
 	  to the operating system. This improves NMI handling
drivers/acpi/apei/einj.c +103 -0

154 lines not on this page · raw message

Signature · 1 line
2.34.1
#6 lore raw

PATCH v6 5/5 EINJ: Update EINJ documentation

In reply to Ben Cheatham

Update EINJ documentation with build requirements for CXL error types and how to inject CXL error types.

---
 Documentation/firmware-guide/acpi/apei/einj.rst | 12 ++++++++++++
 1 file changed, 12 insertions(+)
Documentation/firmware-guide/acpi/apei/einj.rst +12 -0
index d6b61d22f525..83afe3bac793 100644
@@ -181,6 +181,18 @@ You should see something like this in dmesg::
   [22715.834759] EDAC sbridge MC3: PROCESSOR 0:306e7 TIME 1422553404 SOCKET 0 APIC 0
   [22716.616173] EDAC MC3: 1 CE memory read error on CPU_SrcID#0_Channel#0_DIMM#0 (channel:0 slot:0 page:0x12345 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0090 socket:0 channel_mask:1 rank:0)
 
+CXL error types are supported from ACPI 6.5 onwards. These error types
+are not available in the legacy interface at /sys/kernel/debug/apei/einj,
+but are instead at /sys/kernel/debug/cxl/portX/dportY. Inside the dportY
+directory are two files, einj_types and einj_inject. These files work the
+same as the available_error_type and error_inject files (read the error
+types from einj_types and write the type to inject to einj_inject).
+
+To use these error types one of (or both) ``CONFIG_CXL_ACPI`` or
+``CONFIG_CXL_PORT`` must be reachable by the EINJ module; if
+``CONFIG_ACPI_APEI_EINJ`` == y/m, then at least one of ``CONFIG_CXL_ACPI``
+or ``CONFIG_CXL_PORT`` must also be set to y/m.
+
 Special notes for injection into SGX enclaves:
 
 There may be a separate BIOS setup option to enable SGX injection.
Signature · 1 line
2.34.1
#5 lore raw

Re: [PATCH v6 4/5] cxl/port, EINJ: Add CXL EINJ callback functions

In reply to Ben Cheatham
kernel test robot noticed the following build warnings:

Hi Ben,

kernel test robot noticed the following build warnings:

[auto build test WARNING on c66650d29764e228eba40b7a59fdb70fa6567daa]

url:    https://github.com/intel-lab-lkp/linux/commits/Ben-Cheatham/cxl-port-Add-EINJ-debugfs-files-and-callback-support/20231011-040503
base:   c66650d29764e228eba40b7a59fdb70fa6567daa
patch link:    https://lore.kernel.org/r/20231010200254.764273-5-Benjamin.Cheatham%40amd.com
patch subject: [PATCH v6 4/5] cxl/port, EINJ: Add CXL EINJ callback functions
reproduce: (https://download.01.org/0day-ci/archive/20231011/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

# many are suggestions rather than must-fix

ERROR:SPACING: space required after that ',' (ctx:VxV) #98: FILE: drivers/acpi/apei/einj.c:699:
+ devfn = (PCI_DEVFN(PCI_SLOT(pdev->devfn),PCI_FUNC(pdev->devfn)) << 8);
^

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Reviewed-by on the v6 series
#7 lore raw
In reply to Ben Cheatham
Quoted 8 lines · On Tue, 10 Oct 2023 15:02:49 -0500 Ben Cheatham <[email protected]>…

v6 Changes:
- Reworked to have CXL error types under /sys/kernel/debug/cxl (Dan)
- Removed CXL error types from legacy EINJ interface in favor of new interface
- Removed cxl_rcrb_addr file
- Added optional patch for CXL error type #defines (patch 2/5)
- Changes to documentation updates to match rework
- Change base to cxl-fixes branch

New approach looks good to me and I took a look at the implementation. Couldn't find any problems to call out in the individual patches. Feel free to add

Quoted 30 lines

The new CXL error types will use the Memory Address field in the SET_ERROR_TYPE_WITH_ADDRESS structure in order to target a CXL 1.1 compliant memory-mapped downstream port. The value of the memory address will be in the port's MMIO range, and it will not represent physical (normal or persistent) memory.

Add the functionality for injecting CXL 1.1 errors to the EINJ module, but not through the EINJ legacy interface under /sys/kernel/debug/apei/einj. Instead, make the error types available under /sys/kernel/debug/cxl. This allows for validating the MMIO address for a CXL 1.1 error type while also not making the user responsible for finding it.

Ben Cheatham (5):
cxl/port: Add EINJ debugfs files and callback support
ACPI: Add CXL protocol error defines
EINJ: Separate CXL errors from other EINJ errors cxl/port, EINJ: Add CXL EINJ callback functions
EINJ: Update EINJ documentation

 Documentation/ABI/testing/debugfs-cxl         |  27 ++++
 .../firmware-guide/acpi/apei/einj.rst         |  12 ++
 drivers/acpi/apei/Kconfig                     |   3 +
 drivers/acpi/apei/einj.c                      | 149 ++++++++++++++++--
 drivers/cxl/core/port.c                       |  84 ++++++++++
 drivers/cxl/cxl.h                             |  10 ++
 include/acpi/actbl1.h                         |   6 +
 7 files changed, 281 insertions(+), 10 deletions(-)

base-commit: c66650d29764e228eba40b7a59fdb70fa6567daa