Skip to contentSkip to conversation

Enable CXL protocol testing

PATCH In review Terry Bowman 4 participants 10 messages Apr 8 – Apr 21 libnvdimm: non-volatile memory device

lore
Share & tools
lore
#0 lore raw

cover Enable CXL protocol testing

Current CXL error injection (EINJ) only supports Root Port protocol error injection but a method to test all CXL devices is needed. This series outlines methods to update both the kernel and the 'aer-inject' tool-without relying on EINJ-to enable CXL RAS protocol error handling across all CXL devices.

The first patch provides the scripts to enable and trigger AER protocol errors. This patch also includes the README.md with install details.

The second patch enables correctable and uncorrectable AER internal error handling in the aer-inject tool.

The third patch is a kernel patch to set the RAS status in the handler.

Terry Bowman (3):
test/cxl: Enable CXL protocol error testing using aer-inject test/aer-inject: Add aer-inject correctable and uncorrectable interanl
error support
test/cxl: Force RAS status in cxl_handle_cor_ras() and
cxl_handle_ras()

 test/contrib/cxl-aer-einj/README.md           | 80 ++++++++++++++++
 ...Add-internal-error-injection-support.patch | 91 +++++++++++++++++++
 ...AS-status-in-cxl_handle_cor_ras-and-.patch | 51 +++++++++++
 .../cxl-aer-einj/scripts/ds-ce-inject.sh      |  4 +
 .../cxl-aer-einj/scripts/ds-uce-inject.sh     |  4 +
 .../cxl-aer-einj/scripts/enable-trace.sh      |  5 +
 .../cxl-aer-einj/scripts/ep-ce-inject.sh      |  4 +
 .../cxl-aer-einj/scripts/ep-uce-inject.sh     |  4 +
 .../cxl-aer-einj/scripts/root-ce-inject.sh    |  4 +
 .../cxl-aer-einj/scripts/root-uce-inject.sh   |  4 +
 .../cxl-aer-einj/scripts/us-ce-inject.sh      |  4 +
 .../cxl-aer-einj/scripts/us-uce-inject.sh     |  4 +
 12 files changed, 259 insertions(+)
 create mode 100644 test/contrib/cxl-aer-einj/README.md
 create mode 100644 test/contrib/cxl-aer-einj/patches/0001-aer-inject-Add-internal-error-injection-support.patch
 create mode 100644 test/contrib/cxl-aer-einj/patches/0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-uce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/enable-trace.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-uce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/root-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/root-uce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/us-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/us-uce-inject.sh

base-commit: 8ad90e54f0ff4f7291e7f21d44d769d10f24e2b6

Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/3 test/cxl: Enable CXL protocol error testing using aer-inject

CXL protocol errors are signaled to the kernel's CXL drivers via PCIe Advanced Error Reporting (AER) internal error types: Uncorrectable Internal Errors (UIE) and Correctable Internal Errors (CIE). These errors in-turn trigger RAS handling paths in the CXL core. The aer-inject tool has lacked the ability to generate AER UIE and CIE events, making it difficult to verify kernel handling without actual hardware protocol error conditions. To address this testing gap, this patch introduces tooling and scripts that allow for injected CXL protocol errors to be delivered to the CXL core.

This change adds a new test/contrib/cxl-aer-einj directory containing:

- A README with instructions, prerequisites, and caveats for simulating CXL protocol errors using UIE/CIE AER injection.
- Example sed commands for hardcoding ECC cache bits in CXL RAS handlers as a debug workaround for testing with zero hardware status.
- Scripts to enable CXL tracing and to invoke CE/UCE injections for root ports, upstream switches, downstream ports, and endpoints.

The below patches are required to complete support. These patches will follow:

- Patch (0001-aer-inject-Add-internal-error-injection-support.patch) for aer-inject to support UIE and CIE injection by defining new constants and updating parser rules.
- Kernel test patch ('0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch') to force setting RAS status.

With internal error injection support in aer-inject, developers can trigger RAS paths reliably in the CXL core and validate their protocol error handling logic without relying on physical fault conditions.

---
 test/contrib/cxl-aer-einj/README.md           | 80 +++++++++++++++++++
 .../cxl-aer-einj/scripts/ds-ce-inject.sh      |  4 +
 .../cxl-aer-einj/scripts/ds-uce-inject.sh     |  4 +
 .../cxl-aer-einj/scripts/enable-trace.sh      |  5 ++
 .../cxl-aer-einj/scripts/ep-ce-inject.sh      |  4 +
 .../cxl-aer-einj/scripts/ep-uce-inject.sh     |  4 +
 .../cxl-aer-einj/scripts/root-ce-inject.sh    |  4 +
 .../cxl-aer-einj/scripts/root-uce-inject.sh   |  4 +
 .../cxl-aer-einj/scripts/us-ce-inject.sh      |  4 +
 .../cxl-aer-einj/scripts/us-uce-inject.sh     |  4 +
 10 files changed, 117 insertions(+)
 create mode 100644 test/contrib/cxl-aer-einj/README.md
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-uce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/enable-trace.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-uce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/root-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/root-uce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/us-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/us-uce-inject.sh
test/contrib/cxl-aer-einj/README.md new file +80 -0
new file mode 100644
index 0000000..d31b572
@@ -0,0 +1,80 @@
+**Testing CXL Protocol Errors Using AER Injection**
+
+The `aer-inject` tool currently does not support injecting internal errors such as Correctable Internal Errors (CIE) and Uncorrectable Internal Errors (UIE). By default, internal errors are masked according to the PCI specification and are rarely used. However, these internal errors are now leveraged to notify the PCI and CXL subsystems of CXL protocol errors. The attached patches enable support for CE and UCE internal errors in `aer-inject`, allowing you to test CXL RAS functionality.
+
+**Important Caveats:**
+- `aer-inject` will only inject AER errors and does not inject CXL RAS-specific errors directly.
+- As a result, functions like `cxl_handle_ras()` and `cxl_handle_cor_ras()` will detect a status of 0 and exit early, which hampers testing.
+- To work around this, a debug patch must be added (example included) to hardcode the last RAS error status in `cxl_handle_ras()` and `cxl_handle_cor_ras()`. While not ideal, this workaround facilitates testing of the software paths involved. This is addressed below in 'Patch'.
+
+---
+
+### Prerequisites
+- `aer-einj` tool from: https://github.com/intel/aer-inject
+- Kernel configuration options:
+  ```
+  CONFIG_PCIEAER=y
+  CONFIG_PCIEAER_INJECT=y
+  CONFIG_PCIEPORTBUS=y
+  CONFIG_DEBUG_FS=y
+  CONFIG_CXL_PCI
+  CONFIG_CXL_RAS
+  CONFIG_CXL_PORT
+  CONFIG_CXL_BUS
+  ```
+  
+---
+
+### aer-inject Patch Details
+- The patch adds support for injecting both correctable (CE) and uncorrectable (UCE) internal errors.
+- The patch is located in `./patches` and should be applied to the `aer-inject` repository, based on the master branch (commit `81701cb`). The patch is '0001-aer-inject-Add-internal-error-injection-support.patch'.
+- Additionally, you'll need to apply a kernel-side workaround by hardcoding the RAS error status in the relevant handler, as described earlier.
+
+### Kernel patch Details
+Below is patch to set the RAS for testing. 'sed' scripts are also included 
+
+#### Kernel Patch to set CXL RAS status for testing
+Setting CXL protocol RAS status, is based on v7.0-rc6 (7aaa8047eafd). Patch is:
+0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch
+
+#### Script to set the Kernel's CXL RAS status
+##### 1: Correctable Errors (CE)
+```bash
+sed -i '
+/void cxl_handle_cor_ras/,/}/ {
+	/status = readl(addr);/ {
+		i #define CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC 0x1
+		a\    status |= CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC;
+	}
+}' drivers/cxl/core/ras.c
+```
+##### 2: Uncorrectable Errors (UCE)
+```bash
+sed -i '
+/bool cxl_handle_ras/,/}/ {
+	/status = readl(addr);/ {
+		i #define CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC 0x1
+		a\    status |= CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC;
+	}
+}' drivers/cxl/core/ras.c
+```
+---
+
+### Testing Procedure
+- The provided scripts illustrate how I ran the tests. You'll need to modify the scripts to use the correct BDFs for your system.
+- Alternatively, you can run the tests manually using commands like:
+
+```bash
+aer-inject -s ${bdf} examples/correctable.internal
+```
+
+and
+
+```bash
+aer-inject -s ${bdf} examples/fatal.internal
+```
+
+*Ensure you replace `${bdf}` with the appropriate PCI BDF for your device.*
+
+---
+
test/contrib/cxl-aer-einj/scripts/ds-ce-inject.sh new file +4 -0
new file mode 100755
index 0000000..c0e3417
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0e:00.0"
+
+aer-inject -s ${bdf} examples/correctable.internal
test/contrib/cxl-aer-einj/scripts/ds-uce-inject.sh new file +4 -0
new file mode 100755
index 0000000..e238f63
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0e:00.0"
+
+aer-inject -s ${bdf} examples/fatal.internal
test/contrib/cxl-aer-einj/scripts/enable-trace.sh new file +5 -0
new file mode 100755
index 0000000..753419f
@@ -0,0 +1,5 @@
+#!/bin/bash
+
+echo 1 >  /sys/kernel/debug/tracing/events/cxl/enable
+echo 1 > /sys/kernel/debug/tracing/events/cxl/cxl_aer_correctable_error/enable
+echo 1 > /sys/kernel/debug/tracing/events/cxl/cxl_aer_uncorrectable_error/enable
test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh new file +4 -0
new file mode 100755
index 0000000..3077c3c
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0f:00.0"
+
+aer-inject -s ${bdf} examples/correctable.internal
test/contrib/cxl-aer-einj/scripts/ep-uce-inject.sh new file +4 -0
new file mode 100755
index 0000000..9dad325
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0000:0f:00.0"
+
+aer-inject -s ${bdf} examples/fatal.internal
test/contrib/cxl-aer-einj/scripts/root-ce-inject.sh new file +4 -0
new file mode 100755
index 0000000..768522e
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0c:00.0"
+
+aer-inject -s ${bdf} examples/correctable.internal
test/contrib/cxl-aer-einj/scripts/root-uce-inject.sh new file +4 -0
new file mode 100755
index 0000000..7238983
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0c:00.0"
+
+aer-inject -s ${bdf} examples/fatal.internal
test/contrib/cxl-aer-einj/scripts/us-ce-inject.sh new file +4 -0
new file mode 100755
index 0000000..12ac104
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0d:00.0"
+
+aer-inject -s ${bdf} examples/correctable.internal
test/contrib/cxl-aer-einj/scripts/us-uce-inject.sh new file +4 -0
new file mode 100755
index 0000000..bcd130e
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0d:00.0"
+
+aer-inject -s ${bdf} examples/fatal.internal
Signature · 1 line
2.34.1
#2 lore raw

Re: [ndctl PATCH 1/3] test/cxl: Enable CXL protocol error testing using aer-inject

Quoted 18 lines · Terry Bowman

CXL protocol errors are signaled to the kernel's CXL drivers via PCIe Advanced Error Reporting (AER) internal error types: Uncorrectable Internal Errors (UIE) and Correctable Internal Errors (CIE). These errors in-turn trigger RAS handling paths in the CXL core. The aer-inject tool has lacked the ability to generate AER UIE and CIE events, making it difficult to verify kernel handling without actual hardware protocol error conditions. To address this testing gap, this patch introduces tooling and scripts that allow for injected CXL protocol errors to be delivered to the CXL core.

This change adds a new test/contrib/cxl-aer-einj directory containing:

- A README with instructions, prerequisites, and caveats for simulating CXL protocol errors using UIE/CIE AER injection.
- Example sed commands for hardcoding ECC cache bits in CXL RAS handlers as a debug workaround for testing with zero hardware status.
- Scripts to enable CXL tracing and to invoke CE/UCE injections for root ports, upstream switches, downstream ports, and endpoints.

Quoted 5 lines

The below patches are required to complete support. These patches will follow:

- Patch (0001-aer-inject-Add-internal-error-injection-support.patch) for aer-inject to support UIE and CIE injection by defining new constants and updating parser rules.

I have no clue what the aer-inject PR process is, but I'd expect that to be added in that project instead of this one. If it isn't maintained or private then I guess this can work, but it's probably out of scope for ndctl (if I had to guess).

Quoted 2 lines

- Kernel test patch ('0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch') to force setting RAS status.

Same thing for this, but I'll save comments for patch 3/3.

Quoted 6 lines

With internal error injection support in aer-inject, developers can trigger RAS paths reliably in the CXL core and validate their protocol error handling logic without relying on physical fault conditions.

Quoted 11 lines
---
 test/contrib/cxl-aer-einj/README.md           | 80 +++++++++++++++++++
 .../cxl-aer-einj/scripts/ds-ce-inject.sh      |  4 +
 .../cxl-aer-einj/scripts/ds-uce-inject.sh     |  4 +
 .../cxl-aer-einj/scripts/enable-trace.sh      |  5 ++
 .../cxl-aer-einj/scripts/ep-ce-inject.sh      |  4 +
 .../cxl-aer-einj/scripts/ep-uce-inject.sh     |  4 +
 .../cxl-aer-einj/scripts/root-ce-inject.sh    |  4 +
 .../cxl-aer-einj/scripts/root-uce-inject.sh   |  4 +
 .../cxl-aer-einj/scripts/us-ce-inject.sh      |  4 +
 .../cxl-aer-einj/scripts/us-uce-inject.sh     |  4 +

These paths are somewhat redundant. There's already a scripts directory, so I'd put all of the .sh scripts in a cxl-aer-einj subdir in the scripts directory (so ndctl/scripts/cxl-aer-einj/*.sh).

As for the README, the contents would be merged into the inject-protocol-error documentation and the file removed (see my comments on the cover letter).

Quoted 67 lines
 10 files changed, 117 insertions(+)
 create mode 100644 test/contrib/cxl-aer-einj/README.md
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-uce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/enable-trace.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-uce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/root-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/root-uce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/us-ce-inject.sh
 create mode 100755 test/contrib/cxl-aer-einj/scripts/us-uce-inject.sh
test/contrib/cxl-aer-einj/README.md new file +49 -0
new file mode 100644
index 0000000..d31b572
@@ -0,0 +1,80 @@
+**Testing CXL Protocol Errors Using AER Injection**
+
+The `aer-inject` tool currently does not support injecting internal errors such as Correctable Internal Errors (CIE) and Uncorrectable Internal Errors (UIE). By default, internal errors are masked according to the PCI specification and are rarely used. However, these internal errors are now leveraged to notify the PCI and CXL subsystems of CXL protocol errors. The attached patches enable support for CE and UCE internal errors in `aer-inject`, allowing you to test CXL RAS functionality.
+
+**Important Caveats:**
+- `aer-inject` will only inject AER errors and does not inject CXL RAS-specific errors directly.
+- As a result, functions like `cxl_handle_ras()` and `cxl_handle_cor_ras()` will detect a status of 0 and exit early, which hampers testing.
+- To work around this, a debug patch must be added (example included) to hardcode the last RAS error status in `cxl_handle_ras()` and `cxl_handle_cor_ras()`. While not ideal, this workaround facilitates testing of the software paths involved. This is addressed below in 'Patch'.
+
+---
+
+### Prerequisites
+- `aer-einj` tool from: https://github.com/intel/aer-inject
+- Kernel configuration options:
+  ```
+  CONFIG_PCIEAER=y
+  CONFIG_PCIEAER_INJECT=y
+  CONFIG_PCIEPORTBUS=y
+  CONFIG_DEBUG_FS=y
+  CONFIG_CXL_PCI
+  CONFIG_CXL_RAS
+  CONFIG_CXL_PORT
+  CONFIG_CXL_BUS
+  ```
+  
+---
+
+### aer-inject Patch Details
+- The patch adds support for injecting both correctable (CE) and uncorrectable (UCE) internal errors.
+- The patch is located in `./patches` and should be applied to the `aer-inject` repository, based on the master branch (commit `81701cb`). The patch is '0001-aer-inject-Add-internal-error-injection-support.patch'.
+- Additionally, you'll need to apply a kernel-side workaround by hardcoding the RAS error status in the relevant handler, as described earlier.
+
+### Kernel patch Details
+Below is patch to set the RAS for testing. 'sed' scripts are also included 
+
+#### Kernel Patch to set CXL RAS status for testing
+Setting CXL protocol RAS status, is based on v7.0-rc6 (7aaa8047eafd). Patch is:
+0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch
+
+#### Script to set the Kernel's CXL RAS status
+##### 1: Correctable Errors (CE)
+```bash
+sed -i '
+/void cxl_handle_cor_ras/,/}/ {
+	/status = readl(addr);/ {
+		i #define CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC 0x1
+		a\    status |= CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC;
+	}
+}' drivers/cxl/core/ras.c

test/contrib/cxl-aer-einj/README.md line 59

Quoted 10 lines
test/contrib/cxl-aer-einj/README.md new file +10 -0
+```
+##### 2: Uncorrectable Errors (UCE)
+```bash
+sed -i '
+/bool cxl_handle_ras/,/}/ {
+	/status = readl(addr);/ {
+		i #define CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC 0x1
+		a\    status |= CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC;
+	}
+}' drivers/cxl/core/ras.c

This section probably isn't needed, see comments on patch 3/3.

test/contrib/cxl-aer-einj/README.md line 65

Quoted 6 lines
test/contrib/cxl-aer-einj/README.md new file +6 -0
+```
+---
+
+### Testing Procedure
+- The provided scripts illustrate how I ran the tests. You'll need to modify the scripts to use the correct BDFs for your system.
+- Alternatively, you can run the tests manually using commands like:

I think you should drop the scripts. If the user has to manually change half the script to make them work, it's probably better to just give an example or two and let them make the script(s).

test/contrib/cxl-aer-einj/README.md line 77

Quoted 12 lines
test/contrib/cxl-aer-einj/README.md new file +12 -0
+
+```bash
+aer-inject -s ${bdf} examples/correctable.internal
+```
+
+and
+
+```bash
+aer-inject -s ${bdf} examples/fatal.internal
+```
+
+*Ensure you replace `${bdf}` with the appropriate PCI BDF for your device.*

Should probably be renamed SBDF instead of BDF. Most systems don't use the S part, but I'm sure ones exist that do. It may be helpful to also mention how to find the SBDF (i.e. an example 'lspci' command), but I think anyone who needs that probably shouldn't be running these scripts...

Quoted 40 lines
test/contrib/cxl-aer-einj/README.md new file +3 -0
+
+---
+
test/contrib/cxl-aer-einj/scripts/ds-ce-inject.sh new file +4 -0
new file mode 100755
index 0000000..c0e3417
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0e:00.0"
+
+aer-inject -s ${bdf} examples/correctable.internal
test/contrib/cxl-aer-einj/scripts/ds-uce-inject.sh new file +4 -0
new file mode 100755
index 0000000..e238f63
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0e:00.0"
+
+aer-inject -s ${bdf} examples/fatal.internal
test/contrib/cxl-aer-einj/scripts/enable-trace.sh new file +5 -0
new file mode 100755
index 0000000..753419f
@@ -0,0 +1,5 @@
+#!/bin/bash
+
+echo 1 >  /sys/kernel/debug/tracing/events/cxl/enable
+echo 1 > /sys/kernel/debug/tracing/events/cxl/cxl_aer_correctable_error/enable
+echo 1 > /sys/kernel/debug/tracing/events/cxl/cxl_aer_uncorrectable_error/enable
test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh new file +0 -0
new file mode 100755
index 0000000..3077c3c

test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh line 4

Quoted 4 lines
test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh new file +4 -0
@@ -0,0 +1,4 @@
+#!/bin/bash
+bdf="0f:00.0"
+
+aer-inject -s ${bdf} examples/correctable.internal

If you don't want to remove the scripts as per my suggestion above, I would at least consolidate them based on the examples/* file they use and make the user give the bdf as an argument. For example: ep-uce-inject.sh and root-uce-inject.sh would get turned into something like "uce-inject.sh".

#3 lore raw

PATCH 2/3 test/aer-inject: Add aer-inject correctable and uncorrectable interanl error support

In reply to Terry Bowman

The aer-inject tool currently does not support injecting Correctable Internal Errors (CIE) or Uncorrectable Internal Errors (UIE). By default, internal errors are masked according to the PCI specification and are generally not used. However, these internal errors are now leveraged to notify the PCI and CXL subsystems of CXL protocol errors. The attached patches enable support for CIE and UIE internal errors in aer-inject, allowing for injected CXL protocol errors to be delivered to the CXL core.

---
 ...Add-internal-error-injection-support.patch | 91 +++++++++++++++++++
 1 file changed, 91 insertions(+)
 create mode 100644 test/contrib/cxl-aer-einj/patches/0001-aer-inject-Add-internal-error-injection-support.patch
test/contrib/cxl-aer-einj/patches/0001-aer-inject-Add-internal-error-injection-support.patch new file +91 -0
new file mode 100644
index 0000000..e5675ee
@@ -0,0 +1,91 @@
+From 9d273a798950122059e9428a698d1d9d2520362b Mon Sep 17 00:00:00 2001
+From: Terry Bowman <[email protected]>
+Date: Thu, 17 Oct 2024 12:12:58 -0500
+Subject: [PATCH] aer-inject: Add internal error injection support
+
+The `aer-inject` tool currently does not support injecting internal errors
+such as Correctable Errors (CE) and Uncorrectable Errors (UCE). By default,
+internal errors are masked according to the PCI specification and are
+generally not used. However, these internal errors are now leveraged to
+notify the PCI and CXL subsystems of CXL protocol errors. The attached
+patches enable support for CE and UCE internal errors in `aer-inject`,
+allowing for injected CXL protocol errors to be delivered to the CXL core.
+
+Signed-off-by: Terry Bowman <[email protected]>
+---
+ aer.h   | 2 ++
+ aer.lex | 2 ++
+ aer.y   | 8 ++++----
+ 3 files changed, 8 insertions(+), 4 deletions(-)
+
+diff --git a/aer.h b/aer.h
+index a0ad152..e55a731 100644
+--- a/aer.h
++++ b/aer.h
+@@ -30,11 +30,13 @@ struct aer_error_inj
+ #define  PCI_ERR_UNC_MALF_TLP	0x00040000	/* Malformed TLP */
+ #define  PCI_ERR_UNC_ECRC	0x00080000	/* ECRC Error Status */
+ #define  PCI_ERR_UNC_UNSUP	0x00100000	/* Unsupported Request */
++#define  PCI_ERR_UNC_INTERNAL   0x00400000      /* Internal error */
+ #define  PCI_ERR_COR_RCVR	0x00000001	/* Receiver Error Status */
+ #define  PCI_ERR_COR_BAD_TLP	0x00000040	/* Bad TLP Status */
+ #define  PCI_ERR_COR_BAD_DLLP	0x00000080	/* Bad DLLP Status */
+ #define  PCI_ERR_COR_REP_ROLL	0x00000100	/* REPLAY_NUM Rollover */
+ #define  PCI_ERR_COR_REP_TIMER	0x00001000	/* Replay Timer Timeout */
++#define  PCI_ERR_COR_CINTERNAL	0x00004000	/* Internal error */
+ 
+ extern void init_aer(struct aer_error_inj *err);
+ extern void submit_aer(struct aer_error_inj *err);
+diff --git a/aer.lex b/aer.lex
+index 6121e4e..4fadd0e 100644
+--- a/aer.lex
++++ b/aer.lex
+@@ -82,11 +82,13 @@ static struct key {
+ 	KEYVAL(MALF_TLP, PCI_ERR_UNC_MALF_TLP),
+ 	KEYVAL(ECRC, PCI_ERR_UNC_ECRC),
+ 	KEYVAL(UNSUP, PCI_ERR_UNC_UNSUP),
++	KEYVAL(INTERNAL, PCI_ERR_UNC_INTERNAL),
+ 	KEYVAL(RCVR, PCI_ERR_COR_RCVR),
+ 	KEYVAL(BAD_TLP, PCI_ERR_COR_BAD_TLP),
+ 	KEYVAL(BAD_DLLP, PCI_ERR_COR_BAD_DLLP),
+ 	KEYVAL(REP_ROLL, PCI_ERR_COR_REP_ROLL),
+ 	KEYVAL(REP_TIMER, PCI_ERR_COR_REP_TIMER),
++	KEYVAL(CINTERNAL, PCI_ERR_COR_CINTERNAL),
+ };
+ 
+ static int cmp_key(const void *av, const void *bv)
+diff --git a/aer.y b/aer.y
+index e5ecc7d..500dc97 100644
+--- a/aer.y
++++ b/aer.y
+@@ -34,8 +34,8 @@ static void init(void);
+ 
+ %token AER DOMAIN BUS DEV FN PCI_ID UNCOR_STATUS COR_STATUS HEADER_LOG
+ %token <num> TRAIN DLP POISON_TLP FCP COMP_TIME COMP_ABORT UNX_COMP RX_OVER
+-%token <num> MALF_TLP ECRC UNSUP
+-%token <num> RCVR BAD_TLP BAD_DLLP REP_ROLL REP_TIMER
++%token <num> MALF_TLP ECRC UNSUP INTERNAL
++%token <num> RCVR BAD_TLP BAD_DLLP REP_ROLL REP_TIMER CINTERNAL
+ %token <num> SYMBOL NUMBER
+ %token <str> PCI_ID_STR
+ 
+@@ -77,14 +77,14 @@ uncor_status_list: /* empty */			{ $$ = 0; }
+ 	;
+ 
+ uncor_status: TRAIN | DLP | POISON_TLP | FCP | COMP_TIME | COMP_ABORT
+-	| UNX_COMP | RX_OVER | MALF_TLP | ECRC | UNSUP | NUMBER
++	| UNX_COMP | RX_OVER | MALF_TLP | ECRC | UNSUP | INTERNAL | NUMBER
+ 	;
+ 
+ cor_status_list: /* empty */			{ $$ = 0; }
+ 	| cor_status_list cor_status		{ $$ = $1 | $2; }
+ 	;
+ 
+-cor_status: RCVR | BAD_TLP | BAD_DLLP | REP_ROLL | REP_TIMER | NUMBER
++cor_status: RCVR | BAD_TLP | BAD_DLLP | REP_ROLL | REP_TIMER | CINTERNAL | NUMBER
+ 	;
+ 
+ %% 
+-- 
+2.34.1
+
Signature · 1 line
2.34.1
#4 lore raw

PATCH 3/3 test/cxl: Force RAS status in cxl_handle_cor_ras() and cxl_handle_ras()

In reply to Terry Bowman

CXL RAS error injection (EINJ) is present for Root Ports but not for other CXL devices. Provide the means to test protocol errors for all CXL devices in a testing environment. Use 'aer-inject' userspace tool to deliver AER internal error notification which will trigger the CXL driver's RAS handling. Hardcode the status for CE and UCE errors in cxl_handle_ras() and cxl_handle_cor_ras().

---
 ...AS-status-in-cxl_handle_cor_ras-and-.patch | 51 +++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 test/contrib/cxl-aer-einj/patches/0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch
test/contrib/cxl-aer-einj/patches/0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch new file +51 -0
new file mode 100644
index 0000000..d8562cc
@@ -0,0 +1,51 @@
+From 1b4054d82a1834e211ef3f284b9f51926db8f060 Mon Sep 17 00:00:00 2001
+From: Terry Bowman <[email protected]>
+Date: Tue, 7 Apr 2026 16:47:45 -0500
+Subject: [PATCH] test/cxl: Force RAS status in cxl_handle_cor_ras() and
+ cxl_handle_ras()
+
+CXL RAS error injection is present for Root Ports but not for other CXL
+devices. Provide the means to test protocol errors for all CXL devices
+in a testing environment. Use 'aer-inject' userspace tool to deliver AER
+internal error notification which will trigger the CXL driver's RAS
+handling. Hardcode the status for CE and UCE errors in cxl_handle_ras()
+and cxl_handle_cor_ras().
+
+Signed-off-by: Terry Bowman <[email protected]>
+---
+ drivers/cxl/core/ras.c | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/drivers/cxl/core/ras.c b/drivers/cxl/core/ras.c
+index 006c6ffc2f56..09ff82973c70 100644
+--- a/drivers/cxl/core/ras.c
++++ b/drivers/cxl/core/ras.c
+@@ -183,6 +183,9 @@ void devm_cxl_port_ras_setup(struct cxl_port *port)
+ }
+ EXPORT_SYMBOL_NS_GPL(devm_cxl_port_ras_setup, "CXL");
+ 
++#define CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC 0x1
++#define CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC 0x1
++
+ void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base)
+ {
+ 	void __iomem *addr;
+@@ -193,6 +196,7 @@ void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base)
+ 
+ 	addr = ras_base + CXL_RAS_CORRECTABLE_STATUS_OFFSET;
+ 	status = readl(addr);
++	status |= CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC;
+ 	if (status & CXL_RAS_CORRECTABLE_STATUS_MASK) {
+ 		writel(status & CXL_RAS_CORRECTABLE_STATUS_MASK, addr);
+ 		trace_cxl_aer_correctable_error(to_cxl_memdev(dev), status);
+@@ -232,6 +236,7 @@ bool cxl_handle_ras(struct device *dev, void __iomem *ras_base)
+ 
+ 	addr = ras_base + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET;
+ 	status = readl(addr);
++	status |= CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC;
+ 	if (!(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK))
+ 		return false;
+ 
+-- 
+2.34.1
+
Signature · 1 line
2.34.1
#5 lore raw

Re: [ndctl PATCH 3/3] test/cxl: Force RAS status in cxl_handle_cor_ras() and cxl_handle_ras()

Quoted 7 lines · Terry Bowman

CXL RAS error injection (EINJ) is present for Root Ports but not for other CXL devices. Provide the means to test protocol errors for all CXL devices in a testing environment. Use 'aer-inject' userspace tool to deliver AER internal error notification which will trigger the CXL driver's RAS handling. Hardcode the status for CE and UCE errors in cxl_handle_ras() and cxl_handle_cor_ras().

Quoted 6 lines
---
 ...AS-status-in-cxl_handle_cor_ras-and-.patch | 51 +++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 test/contrib/cxl-aer-einj/patches/0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch

The below patch should just be sent to the kernel list. I'm not sure you can create a cxl_test wrapper for cxl_handle_cor_ras()/cxl_handle_ras() since they're in the core, but that would be the ideal situation. Otherwise, you could create a Kconfig symbol that's explicitly for testing and create stub functions to force the values as needed. Basically something like:

cxl/core/core.h:
#ifdef CONFIG_CXL_RAS_TEST
#define CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC 0x1 #define CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC 0x1

u32 cxl_ras_uncor_readl(void *addr) {
	u32 status = readl(addr);
	return status | CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC;
}
u32 cxl_ras_cor_readl(void *addr) {
	u32 status = readl(addr);
	return status | CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC;
}

#else
#define cxl_ras_uncor_readl readl
#define cxl_ras_cor_readl readl
#endif

You can obviously consolidate the uncor/cor versions, but I'll let you figure out the naming there ;). Can also be a bit safer and create static inline functions in the #else, but I just went the easy route.

Quoted 57 lines
test/contrib/cxl-aer-einj/patches/0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch new file +51 -0
new file mode 100644
index 0000000..d8562cc
@@ -0,0 +1,51 @@
+From 1b4054d82a1834e211ef3f284b9f51926db8f060 Mon Sep 17 00:00:00 2001
+From: Terry Bowman <[email protected]>
+Date: Tue, 7 Apr 2026 16:47:45 -0500
+Subject: [PATCH] test/cxl: Force RAS status in cxl_handle_cor_ras() and
+ cxl_handle_ras()
+
+CXL RAS error injection is present for Root Ports but not for other CXL
+devices. Provide the means to test protocol errors for all CXL devices
+in a testing environment. Use 'aer-inject' userspace tool to deliver AER
+internal error notification which will trigger the CXL driver's RAS
+handling. Hardcode the status for CE and UCE errors in cxl_handle_ras()
+and cxl_handle_cor_ras().
+
+Signed-off-by: Terry Bowman <[email protected]>
+---
+ drivers/cxl/core/ras.c | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/drivers/cxl/core/ras.c b/drivers/cxl/core/ras.c
+index 006c6ffc2f56..09ff82973c70 100644
+--- a/drivers/cxl/core/ras.c
++++ b/drivers/cxl/core/ras.c
+@@ -183,6 +183,9 @@ void devm_cxl_port_ras_setup(struct cxl_port *port)
+ }
+ EXPORT_SYMBOL_NS_GPL(devm_cxl_port_ras_setup, "CXL");
+ 
++#define CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC 0x1
++#define CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC 0x1
++
+ void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base)
+ {
+ 	void __iomem *addr;
+@@ -193,6 +196,7 @@ void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base)
+ 
+ 	addr = ras_base + CXL_RAS_CORRECTABLE_STATUS_OFFSET;
+ 	status = readl(addr);
++	status |= CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC;
+ 	if (status & CXL_RAS_CORRECTABLE_STATUS_MASK) {
+ 		writel(status & CXL_RAS_CORRECTABLE_STATUS_MASK, addr);
+ 		trace_cxl_aer_correctable_error(to_cxl_memdev(dev), status);
+@@ -232,6 +236,7 @@ bool cxl_handle_ras(struct device *dev, void __iomem *ras_base)
+ 
+ 	addr = ras_base + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET;
+ 	status = readl(addr);
++	status |= CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC;
+ 	if (!(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK))
+ 		return false;
+ 
+-- 
+2.34.1
+
#6 lore raw
In reply to Terry Bowman
Quoted 6 lines · Terry Bowman

Current CXL error injection (EINJ) only supports Root Port protocol error injection but a method to test all CXL devices is needed. This series outlines methods to update both the kernel and the 'aer-inject' tool-without relying on EINJ-to enable CXL RAS protocol error handling across all CXL devices.

This functionality should probably be added to the inject-protocol-error subcommand instead of spread out across the directory as a bunch of scripts + patches. The command is only set up for protocol error injection, but I don't think it would be *too* hard to extend.

I think the first thing you have to do is expand the accepted device types to include ports and memdevs instead of just dports. That should be simple enough, there are already helpers to find both based on sbdf, name, etc. Then, you'd need to change the interface used to inject the error based on device type (is it a root port? then use EINJ, otherwise use aer-inject). All that's left at that point is to actually call the aer-inject command with the correct options (and update the documentation/help messages).

I would be happy to help with any of the above if you agree with the direction, just let me know!

Thanks,
Ben

maintainer
#7 lore raw
Quoted 20 lines · Benjamin
Quoted 6 lines · Terry Bowman

Current CXL error injection (EINJ) only supports Root Port protocol error injection but a method to test all CXL devices is needed. This series outlines methods to update both the kernel and the 'aer-inject' tool-without relying on EINJ-to enable CXL RAS protocol error handling across all CXL devices.

This functionality should probably be added to the inject-protocol-error subcommand instead of spread out across the directory as a bunch of scripts + patches. The command is only set up for protocol error injection, but I don't think it would be *too* hard to extend.

I think the first thing you have to do is expand the accepted device types to include ports and memdevs instead of just dports. That should be simple enough, there are already helpers to find both based on sbdf, name, etc. Then, you'd need to change the interface used to inject the error based on device type (is it a root port? then use EINJ, otherwise use aer-inject). All that's left at that point is to actually call the aer-inject command with the correct options (and update the documentation/help messages).

Quoted 2 lines

I would be happy to help with any of the above if you agree with the direction, just let me know!

Thanks for the Review Ben! Initially Dan and I asked Terry to just provide his test scripts in the CXL CLI test/contrib directory as a place holder until we can figure out how to integrate it. But if you have better ideas of how to improve the usability, help is much appreciated!

Quoted 3 lines

Thanks,
Ben

maintainer
#8 lore raw
In reply to Cheatham, Benjamin
Quoted 20 lines · Benjamin
Quoted 6 lines · Terry Bowman

Current CXL error injection (EINJ) only supports Root Port protocol error injection but a method to test all CXL devices is needed. This series outlines methods to update both the kernel and the 'aer-inject' tool-without relying on EINJ-to enable CXL RAS protocol error handling across all CXL devices.

This functionality should probably be added to the inject-protocol-error subcommand instead of spread out across the directory as a bunch of scripts + patches. The command is only set up for protocol error injection, but I don't think it would be *too* hard to extend.

I think the first thing you have to do is expand the accepted device types to include ports and memdevs instead of just dports. That should be simple enough, there are already helpers to find both based on sbdf, name, etc. Then, you'd need to change the interface used to inject the error based on device type (is it a root port? then use EINJ, otherwise use aer-inject). All that's left at that point is to actually call the aer-inject command with the correct options (and update the documentation/help messages).

Quoted 2 lines

I would be happy to help with any of the above if you agree with the direction, just let me know!

Terry & Ben,

Terry thanks for sharing all this!

Reading patches and Bens comments and trying to understand the pieces needed to make this reusable. Help me out here:

1) expand our (Bens recently added) cxl-cli inject-protocol-error cmds
- to handle new types
- to route to aer-inject tool

2) make aer-inject tool handle corr/uncor internal errors
https://github.com/intel/aer-inject

3) make cxl/test capable of do special RAS status
- maybe a wrapper

4) then we write scripts that run under cxl/test, require aer-inject tool AND what else do they require? Does this require CXL QEMU devices? Are they doing something special in this recipe? Is this something our CXL Test mock devices can fake?

Above is my quick attempt to understand. Please advise me.

--Alison

Quoted 3 lines

Thanks,
Ben

#9 lore raw
Quoted 41 lines · Alison Schofield
Quoted 20 lines · Benjamin
Quoted 6 lines · Terry Bowman

Current CXL error injection (EINJ) only supports Root Port protocol error injection but a method to test all CXL devices is needed. This series outlines methods to update both the kernel and the 'aer-inject' tool-without relying on EINJ-to enable CXL RAS protocol error handling across all CXL devices.

This functionality should probably be added to the inject-protocol-error subcommand instead of spread out across the directory as a bunch of scripts + patches. The command is only set up for protocol error injection, but I don't think it would be *too* hard to extend.

I think the first thing you have to do is expand the accepted device types to include ports and memdevs instead of just dports. That should be simple enough, there are already helpers to find both based on sbdf, name, etc. Then, you'd need to change the interface used to inject the error based on device type (is it a root port? then use EINJ, otherwise use aer-inject). All that's left at that point is to actually call the aer-inject command with the correct options (and update the documentation/help messages).

Quoted 2 lines

I would be happy to help with any of the above if you agree with the direction, just let me know!

Terry & Ben,

Terry thanks for sharing all this!

Reading patches and Bens comments and trying to understand the pieces needed to make this reusable. Help me out here:

1) expand our (Bens recently added) cxl-cli inject-protocol-error cmds
- to handle new types
- to route to aer-inject tool

2) make aer-inject tool handle corr/uncor internal errors
https://github.com/intel/aer-inject

3) make cxl/test capable of do special RAS status
- maybe a wrapper

Quoted 9 lines

4) then we write scripts that run under cxl/test, require aer-inject tool AND what else do they require? Does this require CXL QEMU devices? Are they doing something special in this recipe? Is this something our CXL Test mock devices can fake?

Above is my quick attempt to understand. Please advise me.

--Alison

Hi Allison,

The test patches I sent would not be reusable for a cxl-cli–based solution.

For the cxl-cli approach, item #2 is not required. cxl-cli will handle CXL RAS injection directly, which makes aer-inject unnecessary.

Item #3 could be useful for managing or coordinating test pass/fail results, but it is not strictly required otherwise.

Item #4 could be implemented using either QEMU or hardware. Updates to QEMU’s CXL RAS support may be required to properly emulate CXL RAS messaging from USP and DSP devices to the Root Port. This includes QEMU detecting CXL RAS conditions and then generating an AER message to the Root Port. I’m not certain whether this QEMU support already exists.

I believe the CXL Test mock could drive this, as it effectively serves as another pass/fail test.

- Terry

Quoted 4 lines
Quoted 4 lines

Thanks,
Ben