Skip to content

RFC Add RAS support for CXL root ports, CXL downstream switch ports, and CXL upstream switch ports

Terry Bowman linux-cxl linux-pci lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest9 patches 49 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/9PCI/AER: Update AER driver to call root port and downstream port UCE handlersdrivers/pci/pcie/err.c+20 -08 replies
2/9PCI/AER: Call AER CE handler before clearing AER CE status registerdrivers/pci/pcie/aer.c+4 -34 replies
3/9PCI/portdrv: Update portdrv with an atomic notifier for reporting AER internal errorsdrivers/pci/pcie/portdrv.c drivers/pci/pcie/portdrv.h+34 -08 replies
4/9cxl/pci: Map CXL PCIe ports' RAS registersdrivers/cxl/core/pci.c drivers/cxl/cxl.h drivers/cxl/mem.c+60 -74 replies
5/9cxl/pci: Update RAS handler interfaces to support CXL PCIe portsdrivers/cxl/core/pci.c+11 -92 replies
6/9cxl/pci: Add trace logging for CXL PCIe port RAS errorsdrivers/cxl/core/trace.h+34 -03 replies
7/9cxl/pci: Add atomic notifier callback for CXL PCIe port AER internal errorsdrivers/cxl/core/core.h drivers/cxl/core/pci.c drivers/cxl/core/port.c +3 more+123 -105 replies
8/9PCI/AER: Export pci_aer_unmask_internal_errors()drivers/pci/pcie/aer.c include/linux/aer.h+8 -15 replies
9/9cxl/pci: Enable interrupts for CXL PCIe ports' AER internal errorsdrivers/cxl/core/pci.c+12 -03 replies

Reviews trailers per patch

2 of 9 patches have a Reviewed-by or Acked-by.

PatchJonathan CameronDan Williams
1/9
2/9R-b A-b
3/9
4/9
5/9R-b
6/9
7/9
8/9
9/9

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [RFC PATCH 0/9] Add RAS support for CXL root ports, CXL downstream switch ports, and CXL upstream switch ports

This patchset provides RAS logging for CXL root ports, CXL downstream switch ports, and CXL upstream switch ports. This includes changes to use a portdrv notifier chain to communicate CXL AER/RAS errors to a cxl_pci callback.

The first 3 patches prepare for and add an atomic notifier chain to the portdrv driver. The portdrv's notifier chain reports the port device's AER internal errors to the registered callback(s). The preparation changes include a portdrv update to call the uncorrectable handler for PCIe root ports and PCIe downstream switch ports. Also, the AER correctable error (CE) status is made available to the AER CE handler.

The next 4 patches are in preparation for adding an atomic notification callback in the cxl_pci driver. This is for receiving AER internal error events from the portdrv notifier chain. Preparation includes adding RAS register block mapping, adding trace functions for logging, and refactoring cxl_pci RAS functions for reuse.

The final 2 patches enable the AER internal error interrupts.

Testing RAS CE/UCE:
QEMU was used for testing CXL root port, CXL downstream switch port, and CXL upstream switch port. The aer-inject tool was used to inject AER and a test patch was used to set the AER CIE/UIE and RAS CE/UCE status during testing. Testing passed with no issues.

An AMD platform with the AMD RAS error injection tool was used for testing CXL root port injection. Testing passed with no issues.

TODO - regression test CXL1.1 RCH handling.

Solutions Considered (1-4):
Below are solutions that were considered. Solution #4 is implemented in this patchset.

1.) Reassigning portdrv error handler for CXL port devices

This solution was based on reassigning the portdrv's CE/UCE err_handler to be CXL cxl_pci driver functions.

I started with this solution and once the flow was working I realized the endpoint removal would have to be addressed as well. While this could be resolved it does highlight the odd coupling and dependency between the CXL port devices error handling with cxl_pci endpoint's handlers. Also, the err_handler re-assignment at runtime required ignoring the 'const' definition. I don't believe this should be considered as a possible solution.

2.) Update the AER driver to call cxl_pci driver's error handler before calling pci_aer_handle_error()

This is similar to the existing RCH port error approach in aer.c. In this solution the AER driver searches for a downstream CXL endpoint to 'handle' detected CXL port protocol errors.

This is a good solution to consider if the one presented in this patchset is not acceptable. I was initially reluctant to this approach because it adds more CXL coupling to the AER driver. But, I think this solution would technically work. I believe Ming was working towards this solution.

3.) Refactor portdrv
The portdrv refactoring solution is to change the portdrv service drivers into PCIe auxiliary drivers. With this change the facility drivers can be associated with a PCIe driver instead fixed bound to the portdrv driver.

In this case the CXL port functionality would be added either as a CXL auxiliary driver or as a CXL specific port driver (PCI_CLASS_BRIDGE_PCI_NORMAL).

This solution has challenges in the interrupt allocation by separate auxiliary drivers and in binding of a specific driver. Binding is currently based on PCIe class and would require extending the binding logic to support multiple drivers for the same class.

Jonathan Cameron is working towards this solution by initially solving for the PMU service driver.[1] It is using the auxiliary bus to associate what were service drivers with the portdrv driver. Using a CXL auxiliary for handling CXL port RAS errors would result in RAS logic called from the cxl_pci and CXL auxiliary drivers. This may need a library driver.

4.) Using a portdrv notifier chain/callback for CIE/UIE (Implemented in this patchset)

This solution uses a portdrv atomic chain notifier and a cxl_pci callback to handle and log CXL port RAS errors.

I chose this after trying solution#1 above. I see a couple advantages to this solution are:
- Is general port implementation for CIE/UIE specific handling mentioned in the PCIe spec.[2]
- Notifier is used in RAS MCE driver as an existing example.
- Does not introduce further CXL dependencies into the AER driver.
- The notifier chain provides registration/unregistration and synchronization.

A disadvantage of this approach is coupling still exists between the CXL port's driver (portdrv) and the cxl_pci driver. The CXL port device's RAS is handled by a notifier callback in the cxl_pci endpoint driver.

Most of the patches in this patchset could be reused to work with solution#3 or solution#2. The atomic notifier could be dropped and instead use an auxiliary device or AER driver awareness. The other changes in this patchset could possibly be reused.

[1] Kernel.org -
https://lore.kernel.org/all/[email protected]
[2] PCI6.0 - 6.2.10 Internal errors

 drivers/cxl/core/core.h    |   4 +
 drivers/cxl/core/pci.c     | 153 ++++++++++++++++++++++++++++++++-----
 drivers/cxl/core/port.c    |   6 +-
 drivers/cxl/core/trace.h   |  34 +++++++++
 drivers/cxl/cxl.h          |  10 +++
 drivers/cxl/cxlpci.h       |   2 +
 drivers/cxl/mem.c          |  32 +++++++-
 drivers/cxl/pci.c          |  19 ++++-
 drivers/pci/pcie/aer.c     |  10 ++-
 drivers/pci/pcie/err.c     |  20 +++++
 drivers/pci/pcie/portdrv.c |  32 ++++++++
 drivers/pci/pcie/portdrv.h |   2 +
 include/linux/aer.h        |   6 ++
 13 files changed, 303 insertions(+), 27 deletions(-)

base-commit: ca3d4767c8054447ac2a58356080e299a59e05b8

Signature · 1 line
2.34.1