Skip to contentSkip to conversation

CXL, ACPI, APEI, EINJ: Update EINJ for CXL 1.1 error types

PATCH Superseded Ben Cheatham 2 participants 10 messages Sep 7, 2023 – Sep 12, 2023

Subsystems (3)

v4 of 15 Superseded Latest is v15 →

lore
Share & tools
lore
#0 lore raw

v4 cover CXL, ACPI, APEI, EINJ: Update EINJ for CXL 1.1 error types

v4 Changes:
- Fix kernel test robot build errors
- Add imply rules for CONFIG_CXL_ACPI and CONFIG_CXL_BUS in EINJ Kconfig to build the CXL support correctly by default
- Change description of building CXL error type support in EINJ documentation for brevity

v3 Changes:
- Add sysfs files for finding valid CXL 1.1 downstream port MMIO addresses, along with validation of said addresses
- Update EINJ documentation to include relevant information for injecting CXL error types
- Dropped Yazen's from tag

This patch is a follow up to the discussion at [1], and builds on Tony's CXL error patch at [2].

The new CXL error types will use the Memory Address field in the SET_ERROR_TYPE_WITH_ADDRESS structure in order to target a CXL 1.1 compliant memory-mapped downstream port. The value of the memory address will be in the port's MMIO range, and it will not represent physical (normal or persistent) memory.

In v2 [3], the user supplied the MMIO address for the downstream port, but per Dan Williams' suggestion [3], the addresses are predetermined and the user only picks the error type to inject and the downstream port to inject into. In order to inject an error, the user write the error type to the error_type file under the einj debugfs directory, then writes any integer into one of the files under the cxl directory.

[1]:

[2]:

[3]:

Ben Cheatham (3):
CXL, PCIE: Add cxl_rcrb_addr file to dport_dev ACPI, APEI, EINJ: Add CXL 1.1 EINJ error type support ACPI, APEI, EINJ: Update EINJ documentation

 Documentation/ABI/testing/sysfs-bus-cxl       |  8 +++
 .../firmware-guide/acpi/apei/einj.rst         | 25 +++++++--
 drivers/acpi/apei/Kconfig                     |  2 +
 drivers/acpi/apei/einj.c                      | 24 ++++++++-
 drivers/cxl/acpi.c                            |  2 +
 drivers/cxl/core/port.c                       | 52 +++++++++++++++++++
 drivers/cxl/cxl.h                             |  3 ++
 include/linux/cxl.h                           | 18 +++++++
 8 files changed, 129 insertions(+), 5 deletions(-)
 create mode 100644 include/linux/cxl.h
Signature · 1 line
2.34.1
#1 lore raw

PATCH v4 1/3 CXL, PCIE: Add cxl_rcrb_addr file to dport_dev

Add cxl_rcrb_addr to the dport_dev (normally represented by a pcie device) for CXL RCH root ports. The file will print the RCRB base MMIO address of the root port when read and will be used by users looking to inject CXL EINJ error types for RCH hosts.

---
 Documentation/ABI/testing/sysfs-bus-cxl |  8 ++++++
 drivers/cxl/acpi.c                      |  2 ++
 drivers/cxl/core/port.c                 | 33 +++++++++++++++++++++++++
 drivers/cxl/cxl.h                       |  2 ++
 4 files changed, 45 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +8 -0
index 087f762ebfd5..a7d169235543 100644
@@ -177,6 +177,14 @@ Description:
 		integer reflects the hardware port unique-id used in the
 		hardware decoder target list.
 
+What:		/sys/bus/cxl/devices/portX/dportY/cxl_rcrb_addr
+Date:		August, 2023
+KernelVersion:	v6.6
+Contact:	[email protected]
+Description:
+		(RO) The 'cxl_rcrb_addr' device file gives the MMIO base address
+		of the RCRB of the corresponding CXL 1.1 downstream port. Only
+		present for CXL 1.1 dports.
 
 What:		/sys/bus/cxl/devices/decoderX.Y
 Date:		June, 2021
drivers/cxl/acpi.c +2 -0
index d1c559879dcc..3e2ca946bf47 100644
@@ -676,6 +676,8 @@ static int cxl_acpi_probe(struct platform_device *pdev)
 	if (IS_ERR(root_port))
 		return PTR_ERR(root_port);
 
+	set_cxl_root(root_port);
+
 	rc = bus_for_each_dev(adev->dev.bus, NULL, root_port,
 			      add_host_bridge_dport);
 	if (rc < 0)
drivers/cxl/core/port.c +33 -0
index 724be8448eb4..001ab8742e21 100644
@@ -875,6 +875,14 @@ struct cxl_port *find_cxl_root(struct cxl_port *port)
 }
 EXPORT_SYMBOL_NS_GPL(find_cxl_root, CXL);
 
+static struct cxl_port *cxl_root;
+
+void set_cxl_root(struct cxl_port *root_port)
+{
+	cxl_root = root_port;
+}
+EXPORT_SYMBOL_NS_GPL(set_cxl_root, CXL);
+
 static struct cxl_dport *find_dport(struct cxl_port *port, int id)
 {
 	struct cxl_dport *dport;
@@ -930,11 +938,30 @@ static void cond_cxl_root_unlock(struct cxl_port *port)
 		device_unlock(&port->dev);
 }
 
+static ssize_t cxl_rcrb_addr_show(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	struct cxl_dport *dport;
+
+	if (!cxl_root)
+		return -ENODEV;
+
+	dport = cxl_find_dport_by_dev(cxl_root, dev);
+	if (!dport)
+		return -ENODEV;
+
+	return sysfs_emit(buf, "0x%llx\n", (u64) dport->rcrb.base);
+}
+DEVICE_ATTR_RO(cxl_rcrb_addr);
+
 static void cxl_dport_remove(void *data)
 {
 	struct cxl_dport *dport = data;
 	struct cxl_port *port = dport->port;
 
+	if (dport->rch)
+		device_remove_file(dport->dport_dev, &dev_attr_cxl_rcrb_addr);
+
 	xa_erase(&port->dports, (unsigned long) dport->dport_dev);
 	put_device(dport->dport_dev);
 }
@@ -1021,6 +1048,12 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
 	if (rc)
 		return ERR_PTR(rc);
 
+	if (dport->rch && dport->rcrb.base != CXL_RESOURCE_NONE) {
+		rc = device_create_file(dport_dev, &dev_attr_cxl_rcrb_addr);
+		if (rc)
+			return ERR_PTR(rc);
+	}
+
 	return dport;
 }
 
drivers/cxl/cxl.h +2 -0
index 76d92561af29..4d5bce4bae7e 100644
@@ -690,6 +690,8 @@ struct cxl_port *devm_cxl_add_port(struct device *host,
 				   resource_size_t component_reg_phys,
 				   struct cxl_dport *parent_dport);
 struct cxl_port *find_cxl_root(struct cxl_port *port);
+void set_cxl_root(struct cxl_port *root_port);
+
 int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd);
 void cxl_bus_rescan(void);
 void cxl_bus_drain(void);
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH v4 1/3] CXL, PCIE: Add cxl_rcrb_addr file to dport_dev

Quoted 6 lines · On Thu, 7 Sep 2023 14:19:54 -0500 Ben Cheatham <[email protected]>…

Add cxl_rcrb_addr to the dport_dev (normally represented by a pcie device) for CXL RCH root ports. The file will print the RCRB base MMIO address of the root port when read and will be used by users looking to inject CXL EINJ error types for RCH hosts.

Can we use is_visble to control presence of the attribute rather than race condition special that is dynamic addition of a sysfs file.

You are adding the file to the linked device which is a bit odd. Why there rather than in the portX?

I'd also normally expect the docs to call out the non-link path for that device which is somewhere in the PCI topology I think.

Quoted 115 lines
---
 Documentation/ABI/testing/sysfs-bus-cxl |  8 ++++++
 drivers/cxl/acpi.c                      |  2 ++
 drivers/cxl/core/port.c                 | 33 +++++++++++++++++++++++++
 drivers/cxl/cxl.h                       |  2 ++
 4 files changed, 45 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +8 -0
index 087f762ebfd5..a7d169235543 100644
@@ -177,6 +177,14 @@ Description:
 		integer reflects the hardware port unique-id used in the
 		hardware decoder target list.
 
+What:		/sys/bus/cxl/devices/portX/dportY/cxl_rcrb_addr
+Date:		August, 2023
+KernelVersion:	v6.6
+Contact:	[email protected]
+Description:
+		(RO) The 'cxl_rcrb_addr' device file gives the MMIO base address
+		of the RCRB of the corresponding CXL 1.1 downstream port. Only
+		present for CXL 1.1 dports.
 
 What:		/sys/bus/cxl/devices/decoderX.Y
 Date:		June, 2021
drivers/cxl/acpi.c +2 -0
index d1c559879dcc..3e2ca946bf47 100644
@@ -676,6 +676,8 @@ static int cxl_acpi_probe(struct platform_device *pdev)
 	if (IS_ERR(root_port))
 		return PTR_ERR(root_port);
 
+	set_cxl_root(root_port);
+
 	rc = bus_for_each_dev(adev->dev.bus, NULL, root_port,
 			      add_host_bridge_dport);
 	if (rc < 0)
drivers/cxl/core/port.c +33 -0
index 724be8448eb4..001ab8742e21 100644
@@ -875,6 +875,14 @@ struct cxl_port *find_cxl_root(struct cxl_port *port)
 }
 EXPORT_SYMBOL_NS_GPL(find_cxl_root, CXL);
 
+static struct cxl_port *cxl_root;
+
+void set_cxl_root(struct cxl_port *root_port)
+{
+	cxl_root = root_port;
+}
+EXPORT_SYMBOL_NS_GPL(set_cxl_root, CXL);
+
 static struct cxl_dport *find_dport(struct cxl_port *port, int id)
 {
 	struct cxl_dport *dport;
@@ -930,11 +938,30 @@ static void cond_cxl_root_unlock(struct cxl_port *port)
 		device_unlock(&port->dev);
 }
 
+static ssize_t cxl_rcrb_addr_show(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	struct cxl_dport *dport;
+
+	if (!cxl_root)
+		return -ENODEV;
+
+	dport = cxl_find_dport_by_dev(cxl_root, dev);
+	if (!dport)
+		return -ENODEV;
+
+	return sysfs_emit(buf, "0x%llx\n", (u64) dport->rcrb.base);
+}
+DEVICE_ATTR_RO(cxl_rcrb_addr);
+
 static void cxl_dport_remove(void *data)
 {
 	struct cxl_dport *dport = data;
 	struct cxl_port *port = dport->port;
 
+	if (dport->rch)
+		device_remove_file(dport->dport_dev, &dev_attr_cxl_rcrb_addr);
+
 	xa_erase(&port->dports, (unsigned long) dport->dport_dev);
 	put_device(dport->dport_dev);
 }
@@ -1021,6 +1048,12 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
 	if (rc)
 		return ERR_PTR(rc);
 
+	if (dport->rch && dport->rcrb.base != CXL_RESOURCE_NONE) {
+		rc = device_create_file(dport_dev, &dev_attr_cxl_rcrb_addr);
+		if (rc)
+			return ERR_PTR(rc);
+	}
+
 	return dport;
 }
 
drivers/cxl/cxl.h +2 -0
index 76d92561af29..4d5bce4bae7e 100644
@@ -690,6 +690,8 @@ struct cxl_port *devm_cxl_add_port(struct device *host,
 				   resource_size_t component_reg_phys,
 				   struct cxl_dport *parent_dport);
 struct cxl_port *find_cxl_root(struct cxl_port *port);
+void set_cxl_root(struct cxl_port *root_port);
+
 int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd);
 void cxl_bus_rescan(void);
 void cxl_bus_drain(void);
#3 lore raw

Re: [PATCH v4 1/3] CXL, PCIE: Add cxl_rcrb_addr file to dport_dev

Hi Jonathan, thanks for the review. Responses inline.

Quoted 8 lines · Jonathan Cameron
Quoted 5 lines · On Thu, 7 Sep 2023 14:19:54 -0500 Ben Cheatham <[email protected]>…

Add cxl_rcrb_addr to the dport_dev (normally represented by a pcie device) for CXL RCH root ports. The file will print the RCRB base MMIO address of the root port when read and will be used by users looking to inject CXL EINJ error types for RCH hosts.

Quoted 5 lines
Quoted 1 line

Can we use is_visble to control presence of the attribute rather than race condition special that is dynamic addition of a sysfs file.

Yeah, I'll go ahead and change it. Not sure why I did it that way to be honest.

Quoted 3 lines

You are adding the file to the linked device which is a bit odd. Why there rather than in the portX?

I agree it's a bit odd. I went with adding the file to the linked device because the ACPI spec specifies using downstream ports for EINJ. The alternative was to have a file for each dport under portX (i.e. dportY_rcrb_addr) which seemed messier to me. Now that I think about it though, I could just have a single file under portX that you write the dport name/number to and it returns the rcrb address. Let me know if you think that would be better.

Quoted 3 lines

I'd also normally expect the docs to call out the non-link path for that device which is somewhere in the PCI topology I think.

Yeah that makes sense, I'll go ahead and change that.

Quoted 116 lines
Quoted 115 lines
---
 Documentation/ABI/testing/sysfs-bus-cxl |  8 ++++++
 drivers/cxl/acpi.c                      |  2 ++
 drivers/cxl/core/port.c                 | 33 +++++++++++++++++++++++++
 drivers/cxl/cxl.h                       |  2 ++
 4 files changed, 45 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +8 -0
index 087f762ebfd5..a7d169235543 100644
@@ -177,6 +177,14 @@ Description:
 		integer reflects the hardware port unique-id used in the
 		hardware decoder target list.
 
+What:		/sys/bus/cxl/devices/portX/dportY/cxl_rcrb_addr
+Date:		August, 2023
+KernelVersion:	v6.6
+Contact:	[email protected]
+Description:
+		(RO) The 'cxl_rcrb_addr' device file gives the MMIO base address
+		of the RCRB of the corresponding CXL 1.1 downstream port. Only
+		present for CXL 1.1 dports.
 
 What:		/sys/bus/cxl/devices/decoderX.Y
 Date:		June, 2021
drivers/cxl/acpi.c +2 -0
index d1c559879dcc..3e2ca946bf47 100644
@@ -676,6 +676,8 @@ static int cxl_acpi_probe(struct platform_device *pdev)
 	if (IS_ERR(root_port))
 		return PTR_ERR(root_port);
 
+	set_cxl_root(root_port);
+
 	rc = bus_for_each_dev(adev->dev.bus, NULL, root_port,
 			      add_host_bridge_dport);
 	if (rc < 0)
drivers/cxl/core/port.c +33 -0
index 724be8448eb4..001ab8742e21 100644
@@ -875,6 +875,14 @@ struct cxl_port *find_cxl_root(struct cxl_port *port)
 }
 EXPORT_SYMBOL_NS_GPL(find_cxl_root, CXL);
 
+static struct cxl_port *cxl_root;
+
+void set_cxl_root(struct cxl_port *root_port)
+{
+	cxl_root = root_port;
+}
+EXPORT_SYMBOL_NS_GPL(set_cxl_root, CXL);
+
 static struct cxl_dport *find_dport(struct cxl_port *port, int id)
 {
 	struct cxl_dport *dport;
@@ -930,11 +938,30 @@ static void cond_cxl_root_unlock(struct cxl_port *port)
 		device_unlock(&port->dev);
 }
 
+static ssize_t cxl_rcrb_addr_show(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	struct cxl_dport *dport;
+
+	if (!cxl_root)
+		return -ENODEV;
+
+	dport = cxl_find_dport_by_dev(cxl_root, dev);
+	if (!dport)
+		return -ENODEV;
+
+	return sysfs_emit(buf, "0x%llx\n", (u64) dport->rcrb.base);
+}
+DEVICE_ATTR_RO(cxl_rcrb_addr);
+
 static void cxl_dport_remove(void *data)
 {
 	struct cxl_dport *dport = data;
 	struct cxl_port *port = dport->port;
 
+	if (dport->rch)
+		device_remove_file(dport->dport_dev, &dev_attr_cxl_rcrb_addr);
+
 	xa_erase(&port->dports, (unsigned long) dport->dport_dev);
 	put_device(dport->dport_dev);
 }
@@ -1021,6 +1048,12 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
 	if (rc)
 		return ERR_PTR(rc);
 
+	if (dport->rch && dport->rcrb.base != CXL_RESOURCE_NONE) {
+		rc = device_create_file(dport_dev, &dev_attr_cxl_rcrb_addr);
+		if (rc)
+			return ERR_PTR(rc);
+	}
+
 	return dport;
 }
 
drivers/cxl/cxl.h +2 -0
index 76d92561af29..4d5bce4bae7e 100644
@@ -690,6 +690,8 @@ struct cxl_port *devm_cxl_add_port(struct device *host,
 				   resource_size_t component_reg_phys,
 				   struct cxl_dport *parent_dport);
 struct cxl_port *find_cxl_root(struct cxl_port *port);
+void set_cxl_root(struct cxl_port *root_port);
+
 int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd);
 void cxl_bus_rescan(void);
 void cxl_bus_drain(void);
#4 lore raw

Re: [PATCH v4 1/3] CXL, PCIE: Add cxl_rcrb_addr file to dport_dev

Quoted 24 lines · On Tue, 12 Sep 2023 09:49:00 -0500 Ben Cheatham <[email protected]>…

Hi Jonathan, thanks for the review. Responses inline.

Quoted 8 lines · Jonathan Cameron
Quoted 5 lines · On Thu, 7 Sep 2023 14:19:54 -0500 Ben Cheatham <[email protected]>…

Add cxl_rcrb_addr to the dport_dev (normally represented by a pcie device) for CXL RCH root ports. The file will print the RCRB base MMIO address of the root port when read and will be used by users looking to inject CXL EINJ error types for RCH hosts.

Quoted 5 lines
Quoted 1 line

Can we use is_visble to control presence of the attribute rather than race condition special that is dynamic addition of a sysfs file.

Yeah, I'll go ahead and change it. Not sure why I did it that way to be honest.

Quoted 3 lines

You are adding the file to the linked device which is a bit odd. Why there rather than in the portX?

Quoted 6 lines

I agree it's a bit odd. I went with adding the file to the linked device because the ACPI spec specifies using downstream ports for EINJ. The alternative was to have a file for each dport under portX (i.e. dportY_rcrb_addr) which seemed messier to me. Now that I think about it though, I could just have a single file under portX that you write the dport name/number to and it returns the rcrb address. Let me know if you think that would be better.

Ah. I was being half asleep. I'd missed the point that a portX can have multiple dportY (which is obvious given the naming :)

What makes here probably one for Dan or Bjorn to comment on..

Jonathan

Quoted 123 lines
Quoted 4 lines

I'd also normally expect the docs to call out the non-link path for that device which is somewhere in the PCI topology I think.

Yeah that makes sense, I'll go ahead and change that.

Quoted 116 lines
Quoted 115 lines
---
 Documentation/ABI/testing/sysfs-bus-cxl |  8 ++++++
 drivers/cxl/acpi.c                      |  2 ++
 drivers/cxl/core/port.c                 | 33 +++++++++++++++++++++++++
 drivers/cxl/cxl.h                       |  2 ++
 4 files changed, 45 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +8 -0
index 087f762ebfd5..a7d169235543 100644
@@ -177,6 +177,14 @@ Description:
 		integer reflects the hardware port unique-id used in the
 		hardware decoder target list.
 
+What:		/sys/bus/cxl/devices/portX/dportY/cxl_rcrb_addr
+Date:		August, 2023
+KernelVersion:	v6.6
+Contact:	[email protected]
+Description:
+		(RO) The 'cxl_rcrb_addr' device file gives the MMIO base address
+		of the RCRB of the corresponding CXL 1.1 downstream port. Only
+		present for CXL 1.1 dports.
 
 What:		/sys/bus/cxl/devices/decoderX.Y
 Date:		June, 2021
drivers/cxl/acpi.c +2 -0
index d1c559879dcc..3e2ca946bf47 100644
@@ -676,6 +676,8 @@ static int cxl_acpi_probe(struct platform_device *pdev)
 	if (IS_ERR(root_port))
 		return PTR_ERR(root_port);
 
+	set_cxl_root(root_port);
+
 	rc = bus_for_each_dev(adev->dev.bus, NULL, root_port,
 			      add_host_bridge_dport);
 	if (rc < 0)
drivers/cxl/core/port.c +33 -0
index 724be8448eb4..001ab8742e21 100644
@@ -875,6 +875,14 @@ struct cxl_port *find_cxl_root(struct cxl_port *port)
 }
 EXPORT_SYMBOL_NS_GPL(find_cxl_root, CXL);
 
+static struct cxl_port *cxl_root;
+
+void set_cxl_root(struct cxl_port *root_port)
+{
+	cxl_root = root_port;
+}
+EXPORT_SYMBOL_NS_GPL(set_cxl_root, CXL);
+
 static struct cxl_dport *find_dport(struct cxl_port *port, int id)
 {
 	struct cxl_dport *dport;
@@ -930,11 +938,30 @@ static void cond_cxl_root_unlock(struct cxl_port *port)
 		device_unlock(&port->dev);
 }
 
+static ssize_t cxl_rcrb_addr_show(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	struct cxl_dport *dport;
+
+	if (!cxl_root)
+		return -ENODEV;
+
+	dport = cxl_find_dport_by_dev(cxl_root, dev);
+	if (!dport)
+		return -ENODEV;
+
+	return sysfs_emit(buf, "0x%llx\n", (u64) dport->rcrb.base);
+}
+DEVICE_ATTR_RO(cxl_rcrb_addr);
+
 static void cxl_dport_remove(void *data)
 {
 	struct cxl_dport *dport = data;
 	struct cxl_port *port = dport->port;
 
+	if (dport->rch)
+		device_remove_file(dport->dport_dev, &dev_attr_cxl_rcrb_addr);
+
 	xa_erase(&port->dports, (unsigned long) dport->dport_dev);
 	put_device(dport->dport_dev);
 }
@@ -1021,6 +1048,12 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
 	if (rc)
 		return ERR_PTR(rc);
 
+	if (dport->rch && dport->rcrb.base != CXL_RESOURCE_NONE) {
+		rc = device_create_file(dport_dev, &dev_attr_cxl_rcrb_addr);
+		if (rc)
+			return ERR_PTR(rc);
+	}
+
 	return dport;
 }
 
drivers/cxl/cxl.h +2 -0
index 76d92561af29..4d5bce4bae7e 100644
@@ -690,6 +690,8 @@ struct cxl_port *devm_cxl_add_port(struct device *host,
 				   resource_size_t component_reg_phys,
 				   struct cxl_dport *parent_dport);
 struct cxl_port *find_cxl_root(struct cxl_port *port);
+void set_cxl_root(struct cxl_port *root_port);
+
 int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd);
 void cxl_bus_rescan(void);
 void cxl_bus_drain(void);  
#5 lore raw

PATCH v4 2/3 ACPI, APEI, EINJ: Add CXL 1.1 EINJ error type support

In reply to Ben Cheatham

Add support for CXL EINJ error types for CXL 1.1 hosts added in ACPI v6.5. Because these error types target memory-mapped CXL 1.1 compliant downstream ports and not physical (normal/persistent) memory, these error types are not currently allowed through the memory range validation done by the EINJ driver.

The MMIO address of a CXL 1.1 downstream port can be found in the cxl_rcrb_addr file in the corresponding dport directory under /sys/bus/cxl/devices/portX. CXL 1.1 error types follow the same procedure as a memory error type, but with param1 set to the downstream port MMIO address.

Example usage:

$ cd /sys/kernel/debug/apei/einj
$ cat available_error_type
    0x00000008      Memory Correctable
    0x00000010      Memory Uncorrectable non-fatal
    0x00000020      Memory Uncorrectable fatal
    0x00000040      PCI Express Correctable
    0x00000080      PCI Express Uncorrectable non-fatal
    0x00000100      PCI Express Uncorrectable fatal
    0x00008000      CXL.mem Protocol Correctable
    0x00020000      CXL.mem Protocol Uncorrectable fatal
$ echo 0x8000 > error_type
$ echo 0xfffffffffffff000 > param2
$ echo 0x3 > flags
$ cat /sys/bus/cxl/devices/portX/dportY/cxl_rcrb_addr
0xb2f00000
$ echo 0xb2f00000 > param1
$ echo 1 > error_inject
---
 drivers/acpi/apei/Kconfig |  2 ++
 drivers/acpi/apei/einj.c  | 24 +++++++++++++++++++++++-
 drivers/cxl/core/port.c   | 19 +++++++++++++++++++
 drivers/cxl/cxl.h         |  1 +
 include/linux/cxl.h       | 18 ++++++++++++++++++
 5 files changed, 63 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/cxl.h
drivers/acpi/apei/Kconfig +2 -0
index 6b18f8bc7be3..eb9cc7157342 100644
@@ -55,6 +55,8 @@ config ACPI_APEI_MEMORY_FAILURE
 config ACPI_APEI_EINJ
 	tristate "APEI Error INJection (EINJ)"
 	depends on ACPI_APEI && DEBUG_FS
+	imply CXL_BUS
+	imply CXL_ACPI
 	help
 	  EINJ provides a hardware error injection mechanism, it is
 	  mainly used for debugging and testing the other parts of
drivers/acpi/apei/einj.c +23 -1
index 013eb621dc92..8000417a5f26 100644
@@ -21,6 +21,7 @@
 #include <linux/nmi.h>
 #include <linux/delay.h>
 #include <linux/mm.h>
+#include <linux/cxl.h>
 #include <asm/unaligned.h>
 
 #include "apei-internal.h"
@@ -36,6 +37,7 @@
 #define MEM_ERROR_MASK		(ACPI_EINJ_MEMORY_CORRECTABLE | \
 				ACPI_EINJ_MEMORY_UNCORRECTABLE | \
 				ACPI_EINJ_MEMORY_FATAL)
+#define CXL_ERROR_MASK		GENMASK(17, 12)
 
 /*
  * ACPI version 5 provides a SET_ERROR_TYPE_WITH_ADDRESS action.
@@ -512,6 +514,22 @@ static int __einj_error_inject(u32 type, u32 flags, u64 param1, u64 param2,
 	return rc;
 }
 
+static int is_valid_cxl_addr(u64 addr)
+{
+	struct cxl_dport *dport;
+
+	if (IS_REACHABLE(CONFIG_CXL_ACPI))
+		dport = cxl_find_rch_dport_by_rcrb((resource_size_t) addr);
+	else
+		return 0;
+
+	if (!IS_ERR_OR_NULL(dport))
+		return 1;
+
+	pr_info("Could not find dport with rcrb 0x%llx\n", addr);
+	return 0;
+}
+
 /* Inject the specified hardware error */
 static int einj_error_inject(u32 type, u32 flags, u64 param1, u64 param2,
 			     u64 param3, u64 param4)
@@ -537,8 +555,11 @@ static int einj_error_inject(u32 type, u32 flags, u64 param1, u64 param2,
 	if (type & ACPI5_VENDOR_BIT) {
 		if (vendor_flags != SETWA_FLAGS_MEM)
 			goto inject;
-	} else if (!(type & MEM_ERROR_MASK) && !(flags & SETWA_FLAGS_MEM))
+	} else if (!(type & MEM_ERROR_MASK) && !(flags & SETWA_FLAGS_MEM)) {
 		goto inject;
+	} else if (type & CXL_ERROR_MASK && is_valid_cxl_addr(param1)) {
+		goto inject;
+	}
 
 	/*
 	 * Disallow crazy address masks that give BIOS leeway to pick
@@ -807,3 +828,4 @@ module_exit(einj_exit);
 MODULE_AUTHOR("Huang Ying");
 MODULE_DESCRIPTION("APEI Error INJection support");
 MODULE_LICENSE("GPL");
+MODULE_IMPORT_NS(CXL);
drivers/cxl/core/port.c +19 -0
index 001ab8742e21..f8f300496140 100644
@@ -1122,6 +1122,25 @@ struct cxl_dport *devm_cxl_add_rch_dport(struct cxl_port *port,
 }
 EXPORT_SYMBOL_NS_GPL(devm_cxl_add_rch_dport, CXL);
 
+#if IS_ENABLED(CONFIG_CXL_ACPI)
+struct cxl_dport *cxl_find_rch_dport_by_rcrb(resource_size_t rcrb_base)
+{
+	struct cxl_dport *dport;
+	unsigned long index;
+
+	if (!cxl_root)
+		return ERR_PTR(-ENODEV);
+
+	xa_for_each(&cxl_root->dports, index, dport)
+		if ((dport->rch && dport->rcrb.base != CXL_RESOURCE_NONE)
+		    && dport->rcrb.base == rcrb_base)
+			return dport;
+
+	return NULL;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_find_rch_dport_by_rcrb, CXL);
+#endif
+
 static int add_ep(struct cxl_ep *new)
 {
 	struct cxl_port *port = new->dport->port;
drivers/cxl/cxl.h +1 -0
index 4d5bce4bae7e..3e6779dbcd23 100644
@@ -8,6 +8,7 @@
 #include <linux/bitfield.h>
 #include <linux/bitops.h>
 #include <linux/log2.h>
+#include <linux/cxl.h>
 #include <linux/io.h>
 
 /**
include/linux/cxl.h new file +18 -0
new file mode 100644
index 000000000000..b57a4cc85005
@@ -0,0 +1,18 @@
+#ifndef _LINUX_CXL_H
+#define _LINUX_CXL_H
+
+#include <linux/xarray.h>
+#include <linux/errno.h>
+
+struct cxl_dport;
+
+#if IS_ENABLED(CONFIG_CXL_ACPI)
+struct cxl_dport *cxl_find_rch_dport_by_rcrb(resource_size_t rcrb_base);
+#else
+static inline struct cxl_dport *cxl_find_rch_dport_by_rcrb(resource_size_t rcrb_base)
+{
+	return NULL;
+}
+#endif
+
+#endif
Signature · 1 line
2.34.1
#6 lore raw

Re: [PATCH v4 2/3] ACPI, APEI, EINJ: Add CXL 1.1 EINJ error type support

Quoted 13 lines · On Thu, 7 Sep 2023 14:19:55 -0500 Ben Cheatham <[email protected]>…

Add support for CXL EINJ error types for CXL 1.1 hosts added in ACPI v6.5. Because these error types target memory-mapped CXL 1.1 compliant downstream ports and not physical (normal/persistent) memory, these error types are not currently allowed through the memory range validation done by the EINJ driver.

The MMIO address of a CXL 1.1 downstream port can be found in the cxl_rcrb_addr file in the corresponding dport directory under /sys/bus/cxl/devices/portX. CXL 1.1 error types follow the same procedure as a memory error type, but with param1 set to the downstream port MMIO address.

Example usage:

Quoted 13 lines
$ cd /sys/kernel/debug/apei/einj
$ cat available_error_type
    0x00000008      Memory Correctable
    0x00000010      Memory Uncorrectable non-fatal
    0x00000020      Memory Uncorrectable fatal
    0x00000040      PCI Express Correctable
    0x00000080      PCI Express Uncorrectable non-fatal
    0x00000100      PCI Express Uncorrectable fatal
    0x00008000      CXL.mem Protocol Correctable
    0x00020000      CXL.mem Protocol Uncorrectable fatal
$ echo 0x8000 > error_type
$ echo 0xfffffffffffff000 > param2
$ echo 0x3 > flags

From einj.rst (and the ACPI spec) bit 0 here is Processor APIC field valid. Why is that relevant here? If it were you'd be writing param3 I think. So probably harmless, but I think this should be 0x2 > flags

Quoted 6 lines
$ cat /sys/bus/cxl/devices/portX/dportY/cxl_rcrb_addr
0xb2f00000
$ echo 0xb2f00000 > param1
$ echo 1 > error_inject

Hi Ben,

Why bother with the config dependent build?
It doesn't save that much in code built and right now there are no non ACPI CXL systems so in reality it's always built anyway.

Otherwise this LGTM

drivers/cxl/core/port.c struct cxl_dport *devm_cxl_add_rch_dport(struct cxl_port *port, · line 1125

Quoted 9 lines
drivers/cxl/core/port.c +1 -0
index 001ab8742e21..f8f300496140 100644
@@ -1122,6 +1122,25 @@ struct cxl_dport *devm_cxl_add_rch_dport(struct cxl_port *port,
 }
 EXPORT_SYMBOL_NS_GPL(devm_cxl_add_rch_dport, CXL);
 
+#if IS_ENABLED(CONFIG_CXL_ACPI)

I'm not a particular fan processor magic down in c files.

Do we really care about the saving of not having this in builds where CONFIG_CXL_ACPI isn't set? I'm thinking those don't really exist.

Quoted 21 lines
drivers/cxl/core/port.c +18 -0
+struct cxl_dport *cxl_find_rch_dport_by_rcrb(resource_size_t rcrb_base)
+{
+	struct cxl_dport *dport;
+	unsigned long index;
+
+	if (!cxl_root)
+		return ERR_PTR(-ENODEV);
+
+	xa_for_each(&cxl_root->dports, index, dport)
+		if ((dport->rch && dport->rcrb.base != CXL_RESOURCE_NONE)
+		    && dport->rcrb.base == rcrb_base)
+			return dport;
+
+	return NULL;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_find_rch_dport_by_rcrb, CXL);
+#endif
+
 static int add_ep(struct cxl_ep *new)
 {
 	struct cxl_port *port = new->dport->port;
#7 lore raw

Re: [PATCH v4 2/3] ACPI, APEI, EINJ: Add CXL 1.1 EINJ error type support

Quoted 30 lines · Jonathan Cameron
Quoted 13 lines · On Thu, 7 Sep 2023 14:19:55 -0500 Ben Cheatham <[email protected]>…

Add support for CXL EINJ error types for CXL 1.1 hosts added in ACPI v6.5. Because these error types target memory-mapped CXL 1.1 compliant downstream ports and not physical (normal/persistent) memory, these error types are not currently allowed through the memory range validation done by the EINJ driver.

The MMIO address of a CXL 1.1 downstream port can be found in the cxl_rcrb_addr file in the corresponding dport directory under /sys/bus/cxl/devices/portX. CXL 1.1 error types follow the same procedure as a memory error type, but with param1 set to the downstream port MMIO address.

Example usage:

Quoted 13 lines
$ cd /sys/kernel/debug/apei/einj
$ cat available_error_type
    0x00000008      Memory Correctable
    0x00000010      Memory Uncorrectable non-fatal
    0x00000020      Memory Uncorrectable fatal
    0x00000040      PCI Express Correctable
    0x00000080      PCI Express Uncorrectable non-fatal
    0x00000100      PCI Express Uncorrectable fatal
    0x00008000      CXL.mem Protocol Correctable
    0x00020000      CXL.mem Protocol Uncorrectable fatal
$ echo 0x8000 > error_type
$ echo 0xfffffffffffff000 > param2
$ echo 0x3 > flags
Quoted 5 lines

From einj.rst (and the ACPI spec) bit 0 here is Processor APIC field valid. Why is that relevant here? If it were you'd be writing param3 I think. So probably harmless, but I think this should be 0x2 > flags

I don't think it matters in this case, but I agree. I'll change it to use 0x2 instead.

Quoted 5 lines
Quoted 5 lines
$ cat /sys/bus/cxl/devices/portX/dportY/cxl_rcrb_addr
0xb2f00000
$ echo 0xb2f00000 > param1
$ echo 1 > error_inject
Quoted 8 lines
Quoted 1 line

Hi Ben,

Why bother with the config dependent build?
It doesn't save that much in code built and right now there are no non ACPI CXL systems so in reality it's always built anyway.

Ok, I'll remove the config dependencies. It'll make my life a lot easier :).

Quoted 11 lines

Otherwise this LGTM

Quoted 8 lines
drivers/cxl/core/port.c +0 -0
index 001ab8742e21..f8f300496140 100644
@@ -1122,6 +1122,25 @@ struct cxl_dport *devm_cxl_add_rch_dport(struct cxl_port *port,
 }
 EXPORT_SYMBOL_NS_GPL(devm_cxl_add_rch_dport, CXL);
 
Quoted 6 lines

drivers/cxl/core/port.c struct cxl_dport *devm_cxl_add_rch_dport(struct cxl_port *port, · line 1125

Quoted 1 line
drivers/cxl/core/port.c +1 -0
+#if IS_ENABLED(CONFIG_CXL_ACPI)

I'm not a particular fan processor magic down in c files.

Do we really care about the saving of not having this in builds where CONFIG_CXL_ACPI isn't set? I'm thinking those don't really exist.

I don't like it much either, I did it as a quick fix for the build errors from the kernel test bot.

Thanks,
Ben

Quoted 22 lines
Quoted 21 lines
drivers/cxl/core/port.c +18 -0
+struct cxl_dport *cxl_find_rch_dport_by_rcrb(resource_size_t rcrb_base)
+{
+	struct cxl_dport *dport;
+	unsigned long index;
+
+	if (!cxl_root)
+		return ERR_PTR(-ENODEV);
+
+	xa_for_each(&cxl_root->dports, index, dport)
+		if ((dport->rch && dport->rcrb.base != CXL_RESOURCE_NONE)
+		    && dport->rcrb.base == rcrb_base)
+			return dport;
+
+	return NULL;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_find_rch_dport_by_rcrb, CXL);
+#endif
+
 static int add_ep(struct cxl_ep *new)
 {
 	struct cxl_port *port = new->dport->port;
#8 lore raw

PATCH v4 3/3 ACPI, APEI, EINJ: Update EINJ documentation

In reply to Ben Cheatham

Update EINJ documentation to include CXL errors in available_error_types table and usage of the types.

Also fix a formatting error in the param4 file description that caused the description to be on the same line as the bullet point.

---
 .../firmware-guide/acpi/apei/einj.rst         | 25 ++++++++++++++++---
 1 file changed, 21 insertions(+), 4 deletions(-)
Documentation/firmware-guide/acpi/apei/einj.rst +21 -4
index d6b61d22f525..c6f28118c48b 100644
@@ -32,6 +32,9 @@ configuration::
   CONFIG_ACPI_APEI
   CONFIG_ACPI_APEI_EINJ
 
+To use CXL error types ``CONFIG_CXL_ACPI`` needs to be set to the same
+value as ``CONFIG_ACPI_APEI_EINJ`` (either "y" or "m").
+
 The EINJ user interface is in <debugfs mount point>/apei/einj.
 
 The following files belong to it:
@@ -40,9 +43,9 @@ The following files belong to it:
 
   This file shows which error types are supported:
 
-  ================  ===================================
+  ================  =========================================
   Error Type Value	Error Description
-  ================  ===================================
+  ================  =========================================
   0x00000001        Processor Correctable
   0x00000002        Processor Uncorrectable non-fatal
   0x00000004        Processor Uncorrectable fatal
@@ -55,7 +58,13 @@ The following files belong to it:
   0x00000200        Platform Correctable
   0x00000400        Platform Uncorrectable non-fatal
   0x00000800        Platform Uncorrectable fatal
-  ================  ===================================
+  0x00001000        CXL.cache Protocol Correctable
+  0x00002000        CXL.cache Protocol Uncorrectable non-fatal
+  0x00004000        CXL.cache Protocol Uncorrectable fatal
+  0x00008000        CXL.mem Protocol Correctable
+  0x00010000        CXL.mem Protocol Uncorrectable non-fatal
+  0x00020000        CXL.mem Protocol Uncorrectable fatal
+  ================  =========================================
 
   The format of the file contents are as above, except present are only
   the available error types.
@@ -106,6 +115,7 @@ The following files belong to it:
   Used when the 0x1 bit is set in "flags" to specify the APIC id
 
 - param4
+
   Used when the 0x4 bit is set in "flags" to specify target PCIe device
 
 - notrigger
@@ -159,6 +169,13 @@ and param2 (1 = PROCESSOR, 2 = MEMORY, 4 = PCI). See your BIOS vendor
 documentation for details (and expect changes to this API if vendors
 creativity in using this feature expands beyond our expectations).
 
+CXL error types are supported from ACPI 6.5 onwards. To use these error
+types you need the MMIO address of a CXL 1.1 downstream port. You can
+find the address of dportY in /sys/bus/cxl/devices/portX/dportY/cxl_rcrb_addr
+(it's possible that the dport is under the CXL root, in that case the
+path would be /sys/us/cxl/devices/rootX/dportY/cxl_rcrb_addr).
+From there, write the address to param1 and continue as you would for a
+memory error type.
 
 An error injection example::
 
@@ -201,4 +218,4 @@ The following sequence can be used:
   7) Read from the virtual address. This will trigger the error
 
 For more information about EINJ, please refer to ACPI specification
-version 4.0, section 17.5 and ACPI 5.0, section 18.6.
+version 4.0, section 17.5 and ACPI 6.5, section 18.6.
Signature · 1 line
2.34.1
Reviewed-by on v4 3/3
#9 lore raw

Re: [PATCH v4 3/3] ACPI, APEI, EINJ: Update EINJ documentation

Quoted 7 lines · On Thu, 7 Sep 2023 14:19:56 -0500 Ben Cheatham <[email protected]>…

Update EINJ documentation to include CXL errors in available_error_types table and usage of the types.

Also fix a formatting error in the param4 file description that caused the description to be on the same line as the bullet point.

Matches what the spec says, so

Quoted 73 lines
---
 .../firmware-guide/acpi/apei/einj.rst         | 25 ++++++++++++++++---
 1 file changed, 21 insertions(+), 4 deletions(-)
Documentation/firmware-guide/acpi/apei/einj.rst +21 -4
index d6b61d22f525..c6f28118c48b 100644
@@ -32,6 +32,9 @@ configuration::
   CONFIG_ACPI_APEI
   CONFIG_ACPI_APEI_EINJ
 
+To use CXL error types ``CONFIG_CXL_ACPI`` needs to be set to the same
+value as ``CONFIG_ACPI_APEI_EINJ`` (either "y" or "m").
+
 The EINJ user interface is in <debugfs mount point>/apei/einj.
 
 The following files belong to it:
@@ -40,9 +43,9 @@ The following files belong to it:
 
   This file shows which error types are supported:
 
-  ================  ===================================
+  ================  =========================================
   Error Type Value	Error Description
-  ================  ===================================
+  ================  =========================================
   0x00000001        Processor Correctable
   0x00000002        Processor Uncorrectable non-fatal
   0x00000004        Processor Uncorrectable fatal
@@ -55,7 +58,13 @@ The following files belong to it:
   0x00000200        Platform Correctable
   0x00000400        Platform Uncorrectable non-fatal
   0x00000800        Platform Uncorrectable fatal
-  ================  ===================================
+  0x00001000        CXL.cache Protocol Correctable
+  0x00002000        CXL.cache Protocol Uncorrectable non-fatal
+  0x00004000        CXL.cache Protocol Uncorrectable fatal
+  0x00008000        CXL.mem Protocol Correctable
+  0x00010000        CXL.mem Protocol Uncorrectable non-fatal
+  0x00020000        CXL.mem Protocol Uncorrectable fatal
+  ================  =========================================
 
   The format of the file contents are as above, except present are only
   the available error types.
@@ -106,6 +115,7 @@ The following files belong to it:
   Used when the 0x1 bit is set in "flags" to specify the APIC id
 
 - param4
+
   Used when the 0x4 bit is set in "flags" to specify target PCIe device
 
 - notrigger
@@ -159,6 +169,13 @@ and param2 (1 = PROCESSOR, 2 = MEMORY, 4 = PCI). See your BIOS vendor
 documentation for details (and expect changes to this API if vendors
 creativity in using this feature expands beyond our expectations).
 
+CXL error types are supported from ACPI 6.5 onwards. To use these error
+types you need the MMIO address of a CXL 1.1 downstream port. You can
+find the address of dportY in /sys/bus/cxl/devices/portX/dportY/cxl_rcrb_addr
+(it's possible that the dport is under the CXL root, in that case the
+path would be /sys/us/cxl/devices/rootX/dportY/cxl_rcrb_addr).
+From there, write the address to param1 and continue as you would for a
+memory error type.
 
 An error injection example::
 
@@ -201,4 +218,4 @@ The following sequence can be used:
   7) Read from the virtual address. This will trigger the error
 
 For more information about EINJ, please refer to ACPI specification
-version 4.0, section 17.5 and ACPI 5.0, section 18.6.
+version 4.0, section 17.5 and ACPI 6.5, section 18.6.