Enable CXL protocol testing
cover Enable CXL protocol testing
Current CXL error injection (EINJ) only supports Root Port protocol error injection but a method to test all CXL devices is needed. This series outlines methods to update both the kernel and the 'aer-inject' tool-without relying on EINJ-to enable CXL RAS protocol error handling across all CXL devices.
The first patch provides the scripts to enable and trigger AER protocol errors. This patch also includes the README.md with install details.
The second patch enables correctable and uncorrectable AER internal error handling in the aer-inject tool.
The third patch is a kernel patch to set the RAS status in the handler.
Terry Bowman (3):
test/cxl: Enable CXL protocol error testing using aer-inject test/aer-inject: Add aer-inject correctable and uncorrectable interanl
error support
test/cxl: Force RAS status in cxl_handle_cor_ras() and
cxl_handle_ras()
test/contrib/cxl-aer-einj/README.md | 80 ++++++++++++++++ ...Add-internal-error-injection-support.patch | 91 +++++++++++++++++++ ...AS-status-in-cxl_handle_cor_ras-and-.patch | 51 +++++++++++ .../cxl-aer-einj/scripts/ds-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/ds-uce-inject.sh | 4 + .../cxl-aer-einj/scripts/enable-trace.sh | 5 + .../cxl-aer-einj/scripts/ep-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/ep-uce-inject.sh | 4 + .../cxl-aer-einj/scripts/root-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/root-uce-inject.sh | 4 + .../cxl-aer-einj/scripts/us-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/us-uce-inject.sh | 4 + 12 files changed, 259 insertions(+) create mode 100644 test/contrib/cxl-aer-einj/README.md create mode 100644 test/contrib/cxl-aer-einj/patches/0001-aer-inject-Add-internal-error-injection-support.patch create mode 100644 test/contrib/cxl-aer-einj/patches/0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-uce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/enable-trace.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-uce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/root-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/root-uce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/us-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/us-uce-inject.sh
base-commit: 8ad90e54f0ff4f7291e7f21d44d769d10f24e2b6
Signature · 1 line
2.34.1
PATCH 1/3 test/cxl: Enable CXL protocol error testing using aer-inject
CXL protocol errors are signaled to the kernel's CXL drivers via PCIe Advanced Error Reporting (AER) internal error types: Uncorrectable Internal Errors (UIE) and Correctable Internal Errors (CIE). These errors in-turn trigger RAS handling paths in the CXL core. The aer-inject tool has lacked the ability to generate AER UIE and CIE events, making it difficult to verify kernel handling without actual hardware protocol error conditions. To address this testing gap, this patch introduces tooling and scripts that allow for injected CXL protocol errors to be delivered to the CXL core.
This change adds a new test/contrib/cxl-aer-einj directory containing:
- A README with instructions, prerequisites, and caveats for simulating CXL protocol errors using UIE/CIE AER injection.
- Example sed commands for hardcoding ECC cache bits in CXL RAS handlers as a debug workaround for testing with zero hardware status.
- Scripts to enable CXL tracing and to invoke CE/UCE injections for root ports, upstream switches, downstream ports, and endpoints.
The below patches are required to complete support. These patches will follow:
- Patch (0001-aer-inject-Add-internal-error-injection-support.patch) for aer-inject to support UIE and CIE injection by defining new constants and updating parser rules.
- Kernel test patch ('0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch') to force setting RAS status.
With internal error injection support in aer-inject, developers can trigger RAS paths reliably in the CXL core and validate their protocol error handling logic without relying on physical fault conditions.
--- test/contrib/cxl-aer-einj/README.md | 80 +++++++++++++++++++ .../cxl-aer-einj/scripts/ds-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/ds-uce-inject.sh | 4 + .../cxl-aer-einj/scripts/enable-trace.sh | 5 ++ .../cxl-aer-einj/scripts/ep-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/ep-uce-inject.sh | 4 + .../cxl-aer-einj/scripts/root-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/root-uce-inject.sh | 4 + .../cxl-aer-einj/scripts/us-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/us-uce-inject.sh | 4 + 10 files changed, 117 insertions(+) create mode 100644 test/contrib/cxl-aer-einj/README.md create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-uce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/enable-trace.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-uce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/root-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/root-uce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/us-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/us-uce-inject.sh
test/contrib/cxl-aer-einj/README.md new file +80 -0
@@ -0,0 +1,80 @@ +**Testing CXL Protocol Errors Using AER Injection** + +The `aer-inject` tool currently does not support injecting internal errors such as Correctable Internal Errors (CIE) and Uncorrectable Internal Errors (UIE). By default, internal errors are masked according to the PCI specification and are rarely used. However, these internal errors are now leveraged to notify the PCI and CXL subsystems of CXL protocol errors. The attached patches enable support for CE and UCE internal errors in `aer-inject`, allowing you to test CXL RAS functionality. + +**Important Caveats:** +- `aer-inject` will only inject AER errors and does not inject CXL RAS-specific errors directly. +- As a result, functions like `cxl_handle_ras()` and `cxl_handle_cor_ras()` will detect a status of 0 and exit early, which hampers testing. +- To work around this, a debug patch must be added (example included) to hardcode the last RAS error status in `cxl_handle_ras()` and `cxl_handle_cor_ras()`. While not ideal, this workaround facilitates testing of the software paths involved. This is addressed below in 'Patch'. + +--- + +### Prerequisites +- `aer-einj` tool from: https://github.com/intel/aer-inject +- Kernel configuration options: + ``` + CONFIG_PCIEAER=y + CONFIG_PCIEAER_INJECT=y + CONFIG_PCIEPORTBUS=y + CONFIG_DEBUG_FS=y + CONFIG_CXL_PCI + CONFIG_CXL_RAS + CONFIG_CXL_PORT + CONFIG_CXL_BUS + ``` + +--- + +### aer-inject Patch Details +- The patch adds support for injecting both correctable (CE) and uncorrectable (UCE) internal errors. +- The patch is located in `./patches` and should be applied to the `aer-inject` repository, based on the master branch (commit `81701cb`). The patch is '0001-aer-inject-Add-internal-error-injection-support.patch'. +- Additionally, you'll need to apply a kernel-side workaround by hardcoding the RAS error status in the relevant handler, as described earlier. + +### Kernel patch Details +Below is patch to set the RAS for testing. 'sed' scripts are also included + +#### Kernel Patch to set CXL RAS status for testing +Setting CXL protocol RAS status, is based on v7.0-rc6 (7aaa8047eafd). Patch is: +0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch + +#### Script to set the Kernel's CXL RAS status +##### 1: Correctable Errors (CE) +```bash +sed -i ' +/void cxl_handle_cor_ras/,/}/ { + /status = readl(addr);/ { + i #define CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC 0x1 + a\ status |= CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC; + } +}' drivers/cxl/core/ras.c +``` +##### 2: Uncorrectable Errors (UCE) +```bash +sed -i ' +/bool cxl_handle_ras/,/}/ { + /status = readl(addr);/ { + i #define CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC 0x1 + a\ status |= CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC; + } +}' drivers/cxl/core/ras.c +``` +--- + +### Testing Procedure +- The provided scripts illustrate how I ran the tests. You'll need to modify the scripts to use the correct BDFs for your system. +- Alternatively, you can run the tests manually using commands like: + +```bash +aer-inject -s ${bdf} examples/correctable.internal +``` + +and + +```bash +aer-inject -s ${bdf} examples/fatal.internal +``` + +*Ensure you replace `${bdf}` with the appropriate PCI BDF for your device.* + +--- +
test/contrib/cxl-aer-einj/scripts/ds-ce-inject.sh new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0e:00.0" + +aer-inject -s ${bdf} examples/correctable.internal
test/contrib/cxl-aer-einj/scripts/ds-uce-inject.sh new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0e:00.0" + +aer-inject -s ${bdf} examples/fatal.internal
test/contrib/cxl-aer-einj/scripts/enable-trace.sh new file +5 -0
@@ -0,0 +1,5 @@ +#!/bin/bash + +echo 1 > /sys/kernel/debug/tracing/events/cxl/enable +echo 1 > /sys/kernel/debug/tracing/events/cxl/cxl_aer_correctable_error/enable +echo 1 > /sys/kernel/debug/tracing/events/cxl/cxl_aer_uncorrectable_error/enable
test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0f:00.0" + +aer-inject -s ${bdf} examples/correctable.internal
test/contrib/cxl-aer-einj/scripts/ep-uce-inject.sh new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0000:0f:00.0" + +aer-inject -s ${bdf} examples/fatal.internal
test/contrib/cxl-aer-einj/scripts/root-ce-inject.sh new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0c:00.0" + +aer-inject -s ${bdf} examples/correctable.internal
test/contrib/cxl-aer-einj/scripts/root-uce-inject.sh new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0c:00.0" + +aer-inject -s ${bdf} examples/fatal.internal
test/contrib/cxl-aer-einj/scripts/us-ce-inject.sh new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0d:00.0" + +aer-inject -s ${bdf} examples/correctable.internal
test/contrib/cxl-aer-einj/scripts/us-uce-inject.sh new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0d:00.0" + +aer-inject -s ${bdf} examples/fatal.internal
Signature · 1 line
2.34.1
Re: [ndctl PATCH 1/3] test/cxl: Enable CXL protocol error testing using aer-inject
Quoted 18 lines · Terry Bowman
CXL protocol errors are signaled to the kernel's CXL drivers via PCIe Advanced Error Reporting (AER) internal error types: Uncorrectable Internal Errors (UIE) and Correctable Internal Errors (CIE). These errors in-turn trigger RAS handling paths in the CXL core. The
aer-injecttool has lacked the ability to generate AER UIE and CIE events, making it difficult to verify kernel handling without actual hardware protocol error conditions. To address this testing gap, this patch introduces tooling and scripts that allow for injected CXL protocol errors to be delivered to the CXL core.This change adds a new
test/contrib/cxl-aer-einjdirectory containing:- A README with instructions, prerequisites, and caveats for simulating CXL protocol errors using UIE/CIE AER injection.
- Example sed commands for hardcoding ECC cache bits in CXL RAS handlers as a debug workaround for testing with zero hardware status.
- Scripts to enable CXL tracing and to invoke CE/UCE injections for root ports, upstream switches, downstream ports, and endpoints.
Quoted 5 lines
The below patches are required to complete support. These patches will follow:
- Patch (
0001-aer-inject-Add-internal-error-injection-support.patch) foraer-injectto support UIE and CIE injection by defining new constants and updating parser rules.
I have no clue what the aer-inject PR process is, but I'd expect that to be added in that project instead of this one. If it isn't maintained or private then I guess this can work, but it's probably out of scope for ndctl (if I had to guess).
Quoted 2 lines
- Kernel test patch ('0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch') to force setting RAS status.
Same thing for this, but I'll save comments for patch 3/3.
Quoted 6 lines
With internal error injection support in
aer-inject, developers can trigger RAS paths reliably in the CXL core and validate their protocol error handling logic without relying on physical fault conditions.Signed-off-by Terry Bowman <[email protected]>
Quoted 11 lines
--- test/contrib/cxl-aer-einj/README.md | 80 +++++++++++++++++++ .../cxl-aer-einj/scripts/ds-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/ds-uce-inject.sh | 4 + .../cxl-aer-einj/scripts/enable-trace.sh | 5 ++ .../cxl-aer-einj/scripts/ep-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/ep-uce-inject.sh | 4 + .../cxl-aer-einj/scripts/root-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/root-uce-inject.sh | 4 + .../cxl-aer-einj/scripts/us-ce-inject.sh | 4 + .../cxl-aer-einj/scripts/us-uce-inject.sh | 4 +
These paths are somewhat redundant. There's already a scripts directory, so I'd put all of the .sh scripts in a cxl-aer-einj subdir in the scripts directory (so ndctl/scripts/cxl-aer-einj/*.sh).
As for the README, the contents would be merged into the inject-protocol-error documentation and the file removed (see my comments on the cover letter).
Quoted 67 lines
10 files changed, 117 insertions(+) create mode 100644 test/contrib/cxl-aer-einj/README.md create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/ds-uce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/enable-trace.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/ep-uce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/root-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/root-uce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/us-ce-inject.sh create mode 100755 test/contrib/cxl-aer-einj/scripts/us-uce-inject.shtest/contrib/cxl-aer-einj/README.md new file +49
-0@@ -0,0 +1,80 @@ +**Testing CXL Protocol Errors Using AER Injection** + +The `aer-inject` tool currently does not support injecting internal errors such as Correctable Internal Errors (CIE) and Uncorrectable Internal Errors (UIE). By default, internal errors are masked according to the PCI specification and are rarely used. However, these internal errors are now leveraged to notify the PCI and CXL subsystems of CXL protocol errors. The attached patches enable support for CE and UCE internal errors in `aer-inject`, allowing you to test CXL RAS functionality. + +**Important Caveats:** +- `aer-inject` will only inject AER errors and does not inject CXL RAS-specific errors directly. +- As a result, functions like `cxl_handle_ras()` and `cxl_handle_cor_ras()` will detect a status of 0 and exit early, which hampers testing. +- To work around this, a debug patch must be added (example included) to hardcode the last RAS error status in `cxl_handle_ras()` and `cxl_handle_cor_ras()`. While not ideal, this workaround facilitates testing of the software paths involved. This is addressed below in 'Patch'. + +--- + +### Prerequisites +- `aer-einj` tool from: https://github.com/intel/aer-inject +- Kernel configuration options: + ``` + CONFIG_PCIEAER=y + CONFIG_PCIEAER_INJECT=y + CONFIG_PCIEPORTBUS=y + CONFIG_DEBUG_FS=y + CONFIG_CXL_PCI + CONFIG_CXL_RAS + CONFIG_CXL_PORT + CONFIG_CXL_BUS + ``` + +--- + +### aer-inject Patch Details +- The patch adds support for injecting both correctable (CE) and uncorrectable (UCE) internal errors. +- The patch is located in `./patches` and should be applied to the `aer-inject` repository, based on the master branch (commit `81701cb`). The patch is '0001-aer-inject-Add-internal-error-injection-support.patch'. +- Additionally, you'll need to apply a kernel-side workaround by hardcoding the RAS error status in the relevant handler, as described earlier. + +### Kernel patch Details +Below is patch to set the RAS for testing. 'sed' scripts are also included + +#### Kernel Patch to set CXL RAS status for testing +Setting CXL protocol RAS status, is based on v7.0-rc6 (7aaa8047eafd). Patch is: +0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch + +#### Script to set the Kernel's CXL RAS status +##### 1: Correctable Errors (CE) +```bash +sed -i ' +/void cxl_handle_cor_ras/,/}/ { + /status = readl(addr);/ { + i #define CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC 0x1 + a\ status |= CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC; + } +}' drivers/cxl/core/ras.c
test/contrib/cxl-aer-einj/README.md line 59
Quoted 10 lines
test/contrib/cxl-aer-einj/README.md new file +10
-0+``` +##### 2: Uncorrectable Errors (UCE) +```bash +sed -i ' +/bool cxl_handle_ras/,/}/ { + /status = readl(addr);/ { + i #define CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC 0x1 + a\ status |= CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC; + } +}' drivers/cxl/core/ras.c
This section probably isn't needed, see comments on patch 3/3.
test/contrib/cxl-aer-einj/README.md line 65
Quoted 6 lines
test/contrib/cxl-aer-einj/README.md new file +6
-0+``` +--- + +### Testing Procedure +- The provided scripts illustrate how I ran the tests. You'll need to modify the scripts to use the correct BDFs for your system. +- Alternatively, you can run the tests manually using commands like:
I think you should drop the scripts. If the user has to manually change half the script to make them work, it's probably better to just give an example or two and let them make the script(s).
test/contrib/cxl-aer-einj/README.md line 77
Quoted 12 lines
test/contrib/cxl-aer-einj/README.md new file +12
-0+ +```bash +aer-inject -s ${bdf} examples/correctable.internal +``` + +and + +```bash +aer-inject -s ${bdf} examples/fatal.internal +``` + +*Ensure you replace `${bdf}` with the appropriate PCI BDF for your device.*
Should probably be renamed SBDF instead of BDF. Most systems don't use the S part, but I'm sure ones exist that do. It may be helpful to also mention how to find the SBDF (i.e. an example 'lspci' command), but I think anyone who needs that probably shouldn't be running these scripts...
Quoted 40 lines
test/contrib/cxl-aer-einj/README.md new file +3
-0+ +--- +test/contrib/cxl-aer-einj/scripts/ds-ce-inject.sh new file +4
-0@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0e:00.0" + +aer-inject -s ${bdf} examples/correctable.internaltest/contrib/cxl-aer-einj/scripts/ds-uce-inject.sh new file +4
-0@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0e:00.0" + +aer-inject -s ${bdf} examples/fatal.internaltest/contrib/cxl-aer-einj/scripts/enable-trace.sh new file +5
-0@@ -0,0 +1,5 @@ +#!/bin/bash + +echo 1 > /sys/kernel/debug/tracing/events/cxl/enable +echo 1 > /sys/kernel/debug/tracing/events/cxl/cxl_aer_correctable_error/enable +echo 1 > /sys/kernel/debug/tracing/events/cxl/cxl_aer_uncorrectable_error/enabletest/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh new file +0
-0
test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh line 4
Quoted 4 lines
test/contrib/cxl-aer-einj/scripts/ep-ce-inject.sh new file +4
-0@@ -0,0 +1,4 @@ +#!/bin/bash +bdf="0f:00.0" + +aer-inject -s ${bdf} examples/correctable.internal
If you don't want to remove the scripts as per my suggestion above, I would at least consolidate them based on the examples/* file they use and make the user give the bdf as an argument. For example: ep-uce-inject.sh and root-uce-inject.sh would get turned into something like "uce-inject.sh".
PATCH 2/3 test/aer-inject: Add aer-inject correctable and uncorrectable interanl error support
In reply to Terry BowmanThe aer-inject tool currently does not support injecting Correctable Internal Errors (CIE) or Uncorrectable Internal Errors (UIE). By default, internal errors are masked according to the PCI specification and are generally not used. However, these internal errors are now leveraged to notify the PCI and CXL subsystems of CXL protocol errors. The attached patches enable support for CIE and UIE internal errors in aer-inject, allowing for injected CXL protocol errors to be delivered to the CXL core.
--- ...Add-internal-error-injection-support.patch | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 test/contrib/cxl-aer-einj/patches/0001-aer-inject-Add-internal-error-injection-support.patch
test/contrib/cxl-aer-einj/patches/0001-aer-inject-Add-internal-error-injection-support.patch new file +91 -0
@@ -0,0 +1,91 @@ +From 9d273a798950122059e9428a698d1d9d2520362b Mon Sep 17 00:00:00 2001 +From: Terry Bowman <[email protected]> +Date: Thu, 17 Oct 2024 12:12:58 -0500 +Subject: [PATCH] aer-inject: Add internal error injection support + +The `aer-inject` tool currently does not support injecting internal errors +such as Correctable Errors (CE) and Uncorrectable Errors (UCE). By default, +internal errors are masked according to the PCI specification and are +generally not used. However, these internal errors are now leveraged to +notify the PCI and CXL subsystems of CXL protocol errors. The attached +patches enable support for CE and UCE internal errors in `aer-inject`, +allowing for injected CXL protocol errors to be delivered to the CXL core. + +Signed-off-by: Terry Bowman <[email protected]> +--- + aer.h | 2 ++ + aer.lex | 2 ++ + aer.y | 8 ++++---- + 3 files changed, 8 insertions(+), 4 deletions(-) + +diff --git a/aer.h b/aer.h +index a0ad152..e55a731 100644 +--- a/aer.h ++++ b/aer.h +@@ -30,11 +30,13 @@ struct aer_error_inj + #define PCI_ERR_UNC_MALF_TLP 0x00040000 /* Malformed TLP */ + #define PCI_ERR_UNC_ECRC 0x00080000 /* ECRC Error Status */ + #define PCI_ERR_UNC_UNSUP 0x00100000 /* Unsupported Request */ ++#define PCI_ERR_UNC_INTERNAL 0x00400000 /* Internal error */ + #define PCI_ERR_COR_RCVR 0x00000001 /* Receiver Error Status */ + #define PCI_ERR_COR_BAD_TLP 0x00000040 /* Bad TLP Status */ + #define PCI_ERR_COR_BAD_DLLP 0x00000080 /* Bad DLLP Status */ + #define PCI_ERR_COR_REP_ROLL 0x00000100 /* REPLAY_NUM Rollover */ + #define PCI_ERR_COR_REP_TIMER 0x00001000 /* Replay Timer Timeout */ ++#define PCI_ERR_COR_CINTERNAL 0x00004000 /* Internal error */ + + extern void init_aer(struct aer_error_inj *err); + extern void submit_aer(struct aer_error_inj *err); +diff --git a/aer.lex b/aer.lex +index 6121e4e..4fadd0e 100644 +--- a/aer.lex ++++ b/aer.lex +@@ -82,11 +82,13 @@ static struct key { + KEYVAL(MALF_TLP, PCI_ERR_UNC_MALF_TLP), + KEYVAL(ECRC, PCI_ERR_UNC_ECRC), + KEYVAL(UNSUP, PCI_ERR_UNC_UNSUP), ++ KEYVAL(INTERNAL, PCI_ERR_UNC_INTERNAL), + KEYVAL(RCVR, PCI_ERR_COR_RCVR), + KEYVAL(BAD_TLP, PCI_ERR_COR_BAD_TLP), + KEYVAL(BAD_DLLP, PCI_ERR_COR_BAD_DLLP), + KEYVAL(REP_ROLL, PCI_ERR_COR_REP_ROLL), + KEYVAL(REP_TIMER, PCI_ERR_COR_REP_TIMER), ++ KEYVAL(CINTERNAL, PCI_ERR_COR_CINTERNAL), + }; + + static int cmp_key(const void *av, const void *bv) +diff --git a/aer.y b/aer.y +index e5ecc7d..500dc97 100644 +--- a/aer.y ++++ b/aer.y +@@ -34,8 +34,8 @@ static void init(void); + + %token AER DOMAIN BUS DEV FN PCI_ID UNCOR_STATUS COR_STATUS HEADER_LOG + %token <num> TRAIN DLP POISON_TLP FCP COMP_TIME COMP_ABORT UNX_COMP RX_OVER +-%token <num> MALF_TLP ECRC UNSUP +-%token <num> RCVR BAD_TLP BAD_DLLP REP_ROLL REP_TIMER ++%token <num> MALF_TLP ECRC UNSUP INTERNAL ++%token <num> RCVR BAD_TLP BAD_DLLP REP_ROLL REP_TIMER CINTERNAL + %token <num> SYMBOL NUMBER + %token <str> PCI_ID_STR + +@@ -77,14 +77,14 @@ uncor_status_list: /* empty */ { $$ = 0; } + ; + + uncor_status: TRAIN | DLP | POISON_TLP | FCP | COMP_TIME | COMP_ABORT +- | UNX_COMP | RX_OVER | MALF_TLP | ECRC | UNSUP | NUMBER ++ | UNX_COMP | RX_OVER | MALF_TLP | ECRC | UNSUP | INTERNAL | NUMBER + ; + + cor_status_list: /* empty */ { $$ = 0; } + | cor_status_list cor_status { $$ = $1 | $2; } + ; + +-cor_status: RCVR | BAD_TLP | BAD_DLLP | REP_ROLL | REP_TIMER | NUMBER ++cor_status: RCVR | BAD_TLP | BAD_DLLP | REP_ROLL | REP_TIMER | CINTERNAL | NUMBER + ; + + %% +-- +2.34.1 +
Signature · 1 line
2.34.1
PATCH 3/3 test/cxl: Force RAS status in cxl_handle_cor_ras() and cxl_handle_ras()
In reply to Terry BowmanCXL RAS error injection (EINJ) is present for Root Ports but not for other CXL devices. Provide the means to test protocol errors for all CXL devices in a testing environment. Use 'aer-inject' userspace tool to deliver AER internal error notification which will trigger the CXL driver's RAS handling. Hardcode the status for CE and UCE errors in cxl_handle_ras() and cxl_handle_cor_ras().
--- ...AS-status-in-cxl_handle_cor_ras-and-.patch | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 test/contrib/cxl-aer-einj/patches/0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch
test/contrib/cxl-aer-einj/patches/0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch new file +51 -0
@@ -0,0 +1,51 @@ +From 1b4054d82a1834e211ef3f284b9f51926db8f060 Mon Sep 17 00:00:00 2001 +From: Terry Bowman <[email protected]> +Date: Tue, 7 Apr 2026 16:47:45 -0500 +Subject: [PATCH] test/cxl: Force RAS status in cxl_handle_cor_ras() and + cxl_handle_ras() + +CXL RAS error injection is present for Root Ports but not for other CXL +devices. Provide the means to test protocol errors for all CXL devices +in a testing environment. Use 'aer-inject' userspace tool to deliver AER +internal error notification which will trigger the CXL driver's RAS +handling. Hardcode the status for CE and UCE errors in cxl_handle_ras() +and cxl_handle_cor_ras(). + +Signed-off-by: Terry Bowman <[email protected]> +--- + drivers/cxl/core/ras.c | 5 +++++ + 1 file changed, 5 insertions(+) + +diff --git a/drivers/cxl/core/ras.c b/drivers/cxl/core/ras.c +index 006c6ffc2f56..09ff82973c70 100644 +--- a/drivers/cxl/core/ras.c ++++ b/drivers/cxl/core/ras.c +@@ -183,6 +183,9 @@ void devm_cxl_port_ras_setup(struct cxl_port *port) + } + EXPORT_SYMBOL_NS_GPL(devm_cxl_port_ras_setup, "CXL"); + ++#define CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC 0x1 ++#define CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC 0x1 ++ + void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base) + { + void __iomem *addr; +@@ -193,6 +196,7 @@ void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base) + + addr = ras_base + CXL_RAS_CORRECTABLE_STATUS_OFFSET; + status = readl(addr); ++ status |= CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC; + if (status & CXL_RAS_CORRECTABLE_STATUS_MASK) { + writel(status & CXL_RAS_CORRECTABLE_STATUS_MASK, addr); + trace_cxl_aer_correctable_error(to_cxl_memdev(dev), status); +@@ -232,6 +236,7 @@ bool cxl_handle_ras(struct device *dev, void __iomem *ras_base) + + addr = ras_base + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET; + status = readl(addr); ++ status |= CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC; + if (!(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK)) + return false; + +-- +2.34.1 +
Signature · 1 line
2.34.1
Re: [ndctl PATCH 3/3] test/cxl: Force RAS status in cxl_handle_cor_ras() and cxl_handle_ras()
Quoted 7 lines · Terry Bowman
CXL RAS error injection (EINJ) is present for Root Ports but not for other CXL devices. Provide the means to test protocol errors for all CXL devices in a testing environment. Use 'aer-inject' userspace tool to deliver AER internal error notification which will trigger the CXL driver's RAS handling. Hardcode the status for CE and UCE errors in
cxl_handle_ras()andcxl_handle_cor_ras().
Quoted 6 lines
Signed-off-by Terry Bowman <[email protected]>--- ...AS-status-in-cxl_handle_cor_ras-and-.patch | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 test/contrib/cxl-aer-einj/patches/0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch
The below patch should just be sent to the kernel list. I'm not sure you can create a cxl_test wrapper for cxl_handle_cor_ras()/cxl_handle_ras() since they're in the core, but that would be the ideal situation. Otherwise, you could create a Kconfig symbol that's explicitly for testing and create stub functions to force the values as needed. Basically something like:
cxl/core/core.h:
#ifdef CONFIG_CXL_RAS_TEST
#define CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC 0x1 #define CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC 0x1
u32 cxl_ras_uncor_readl(void *addr) {
u32 status = readl(addr);
return status | CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC;
}u32 cxl_ras_cor_readl(void *addr) {
u32 status = readl(addr);
return status | CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC;
}#else
#define cxl_ras_uncor_readl readl
#define cxl_ras_cor_readl readl
#endif
You can obviously consolidate the uncor/cor versions, but I'll let you figure out the naming there ;). Can also be a bit safer and create static inline functions in the #else, but I just went the easy route.
Quoted 57 lines
test/contrib/cxl-aer-einj/patches/0001-test-cxl-Force-RAS-status-in-cxl_handle_cor_ras-and-.patch new file +51
-0@@ -0,0 +1,51 @@ +From 1b4054d82a1834e211ef3f284b9f51926db8f060 Mon Sep 17 00:00:00 2001 +From: Terry Bowman <[email protected]> +Date: Tue, 7 Apr 2026 16:47:45 -0500 +Subject: [PATCH] test/cxl: Force RAS status in cxl_handle_cor_ras() and + cxl_handle_ras() + +CXL RAS error injection is present for Root Ports but not for other CXL +devices. Provide the means to test protocol errors for all CXL devices +in a testing environment. Use 'aer-inject' userspace tool to deliver AER +internal error notification which will trigger the CXL driver's RAS +handling. Hardcode the status for CE and UCE errors in cxl_handle_ras() +and cxl_handle_cor_ras(). + +Signed-off-by: Terry Bowman <[email protected]> +--- + drivers/cxl/core/ras.c | 5 +++++ + 1 file changed, 5 insertions(+) + +diff --git a/drivers/cxl/core/ras.c b/drivers/cxl/core/ras.c +index 006c6ffc2f56..09ff82973c70 100644 +--- a/drivers/cxl/core/ras.c ++++ b/drivers/cxl/core/ras.c +@@ -183,6 +183,9 @@ void devm_cxl_port_ras_setup(struct cxl_port *port) + } + EXPORT_SYMBOL_NS_GPL(devm_cxl_port_ras_setup, "CXL"); + ++#define CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC 0x1 ++#define CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC 0x1 ++ + void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base) + { + void __iomem *addr; +@@ -193,6 +196,7 @@ void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base) + + addr = ras_base + CXL_RAS_CORRECTABLE_STATUS_OFFSET; + status = readl(addr); ++ status |= CXL_RAS_CORRECTABLE_STATUS_CACHE_ECC; + if (status & CXL_RAS_CORRECTABLE_STATUS_MASK) { + writel(status & CXL_RAS_CORRECTABLE_STATUS_MASK, addr); + trace_cxl_aer_correctable_error(to_cxl_memdev(dev), status); +@@ -232,6 +236,7 @@ bool cxl_handle_ras(struct device *dev, void __iomem *ras_base) + + addr = ras_base + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET; + status = readl(addr); ++ status |= CXL_RAS_UNCORRECTABLE_STATUS_CACHE_ECC; + if (!(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK)) + return false; + +-- +2.34.1 +
Quoted 6 lines · Terry Bowman
Current CXL error injection (EINJ) only supports Root Port protocol error injection but a method to test all CXL devices is needed. This series outlines methods to update both the kernel and the 'aer-inject' tool-without relying on EINJ-to enable CXL RAS protocol error handling across all CXL devices.
This functionality should probably be added to the inject-protocol-error subcommand instead of spread out across the directory as a bunch of scripts + patches. The command is only set up for protocol error injection, but I don't think it would be *too* hard to extend.
I think the first thing you have to do is expand the accepted device types to include ports and memdevs instead of just dports. That should be simple enough, there are already helpers to find both based on sbdf, name, etc. Then, you'd need to change the interface used to inject the error based on device type (is it a root port? then use EINJ, otherwise use aer-inject). All that's left at that point is to actually call the aer-inject command with the correct options (and update the documentation/help messages).
I would be happy to help with any of the above if you agree with the direction, just let me know!
Thanks,
Ben
Dave Jiang
maintainerQuoted 20 lines · Benjamin
Quoted 6 lines · Terry Bowman
Current CXL error injection (EINJ) only supports Root Port protocol error injection but a method to test all CXL devices is needed. This series outlines methods to update both the kernel and the 'aer-inject' tool-without relying on EINJ-to enable CXL RAS protocol error handling across all CXL devices.
This functionality should probably be added to the inject-protocol-error subcommand instead of spread out across the directory as a bunch of scripts + patches. The command is only set up for protocol error injection, but I don't think it would be *too* hard to extend.
I think the first thing you have to do is expand the accepted device types to include ports and memdevs instead of just dports. That should be simple enough, there are already helpers to find both based on sbdf, name, etc. Then, you'd need to change the interface used to inject the error based on device type (is it a root port? then use EINJ, otherwise use aer-inject). All that's left at that point is to actually call the aer-inject command with the correct options (and update the documentation/help messages).
Quoted 2 lines
I would be happy to help with any of the above if you agree with the direction, just let me know!
Thanks for the Review Ben! Initially Dan and I asked Terry to just provide his test scripts in the CXL CLI test/contrib directory as a place holder until we can figure out how to integrate it. But if you have better ideas of how to improve the usability, help is much appreciated!
Quoted 3 lines
Thanks,
Ben
Alison Schofield
maintainerQuoted 20 lines · Benjamin
Quoted 6 lines · Terry Bowman
Current CXL error injection (EINJ) only supports Root Port protocol error injection but a method to test all CXL devices is needed. This series outlines methods to update both the kernel and the 'aer-inject' tool-without relying on EINJ-to enable CXL RAS protocol error handling across all CXL devices.
This functionality should probably be added to the inject-protocol-error subcommand instead of spread out across the directory as a bunch of scripts + patches. The command is only set up for protocol error injection, but I don't think it would be *too* hard to extend.
I think the first thing you have to do is expand the accepted device types to include ports and memdevs instead of just dports. That should be simple enough, there are already helpers to find both based on sbdf, name, etc. Then, you'd need to change the interface used to inject the error based on device type (is it a root port? then use EINJ, otherwise use aer-inject). All that's left at that point is to actually call the aer-inject command with the correct options (and update the documentation/help messages).
Quoted 2 lines
I would be happy to help with any of the above if you agree with the direction, just let me know!
Terry & Ben,
Terry thanks for sharing all this!
Reading patches and Bens comments and trying to understand the pieces needed to make this reusable. Help me out here:
1) expand our (Bens recently added) cxl-cli inject-protocol-error cmds
- to handle new types
- to route to aer-inject tool
2) make aer-inject tool handle corr/uncor internal errors
https://github.com/intel/aer-inject
3) make cxl/test capable of do special RAS status
- maybe a wrapper
4) then we write scripts that run under cxl/test, require aer-inject tool AND what else do they require? Does this require CXL QEMU devices? Are they doing something special in this recipe? Is this something our CXL Test mock devices can fake?
Above is my quick attempt to understand. Please advise me.
--Alison
Quoted 3 lines
Thanks,
Ben
Quoted 41 lines · Alison Schofield
Quoted 20 lines · Benjamin
Quoted 6 lines · Terry Bowman
Current CXL error injection (EINJ) only supports Root Port protocol error injection but a method to test all CXL devices is needed. This series outlines methods to update both the kernel and the 'aer-inject' tool-without relying on EINJ-to enable CXL RAS protocol error handling across all CXL devices.
This functionality should probably be added to the inject-protocol-error subcommand instead of spread out across the directory as a bunch of scripts + patches. The command is only set up for protocol error injection, but I don't think it would be *too* hard to extend.
I think the first thing you have to do is expand the accepted device types to include ports and memdevs instead of just dports. That should be simple enough, there are already helpers to find both based on sbdf, name, etc. Then, you'd need to change the interface used to inject the error based on device type (is it a root port? then use EINJ, otherwise use aer-inject). All that's left at that point is to actually call the aer-inject command with the correct options (and update the documentation/help messages).
Quoted 2 lines
I would be happy to help with any of the above if you agree with the direction, just let me know!
Terry & Ben,
Terry thanks for sharing all this!
Reading patches and Bens comments and trying to understand the pieces needed to make this reusable. Help me out here:
1) expand our (Bens recently added) cxl-cli inject-protocol-error cmds
- to handle new types
- to route to aer-inject tool2) make aer-inject tool handle corr/uncor internal errors
https://github.com/intel/aer-inject3) make cxl/test capable of do special RAS status
- maybe a wrapper
Quoted 9 lines
4) then we write scripts that run under cxl/test, require aer-inject tool AND what else do they require? Does this require CXL QEMU devices? Are they doing something special in this recipe? Is this something our CXL Test mock devices can fake?
Above is my quick attempt to understand. Please advise me.
--Alison
Hi Allison,
The test patches I sent would not be reusable for a cxl-cli–based solution.
For the cxl-cli approach, item #2 is not required. cxl-cli will handle CXL RAS injection directly, which makes aer-inject unnecessary.
Item #3 could be useful for managing or coordinating test pass/fail results, but it is not strictly required otherwise.
Item #4 could be implemented using either QEMU or hardware. Updates to QEMU’s CXL RAS support may be required to properly emulate CXL RAS messaging from USP and DSP devices to the Root Port. This includes QEMU detecting CXL RAS conditions and then generating an AER message to the Root Port. I’m not certain whether this QEMU support already exists.
I believe the CXL Test mock could drive this, as it effectively serves as another pass/fail test.
- Terry
Quoted 4 lines
Quoted 4 lines
Thanks,
Ben