Skip to content

CXL.mem error isolation support

Ben Cheatham linux-acpi linux-cxl linux-pci 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest16 patches 20 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/16cxl/regs: Add cxl_unmap_component_regs()drivers/cxl/core/regs.c drivers/cxl/cxl.h+62 -182 replies
2/16cxl/regs: Add CXL Isolation capability mappingdrivers/cxl/core/regs.c drivers/cxl/cxl.h+15 -01 reply
3/16PCI: PCIe portdrv: Add CXL Isolation service driverdrivers/cxl/Kconfig drivers/cxl/cxl.h drivers/pci/pcie/Makefile +3 more+124 -12 replies
4/16PCI: PCIe portdrv: Allocate CXL isolation MSI/-X vectordrivers/pci/pcie/portdrv.c+16 -42 replies
5/16PCI: PCIe portdrv: Add interface for getting CXL isolation IRQdrivers/pci/pcie/cxl_isolation.c include/cxl/isolation.h+51 -12 replies
6/16cxl/core: Enable CXL.mem isolationdrivers/cxl/core/pci.c drivers/cxl/core/port.c drivers/cxl/cxl.h +1 more+96 -02 replies
7/16cxl/core: Set up isolation interruptsdrivers/cxl/core/port.c drivers/cxl/cxl.h drivers/cxl/cxlmem.h +1 more+125 -32 replies
8/16cxl/core: Enable CXL isolation interruptsdrivers/cxl/core/core.h drivers/cxl/core/pci.c drivers/cxl/core/port.c +1 more+34 -10 replies
9/16cxl/core: Prevent onlining CXL memory behind isolated portsdrivers/cxl/core/port.c drivers/cxl/core/region.c drivers/cxl/cxl.h+33 -00 replies
10/16cxl/core: Enable CXL.mem timeoutdrivers/cxl/core/pci.c drivers/cxl/core/port.c drivers/cxl/cxl.h +1 more+40 -50 replies
11/16cxl/pci: Add isolation handlerdrivers/cxl/pci.c+9 -00 replies
12/16PCI: PCIe portdrv: Add cxl_isolation sysfs attributesdrivers/cxl/core/port.c drivers/pci/pci-sysfs.c drivers/pci/pci.h +2 more+202 -02 replies
13/16cxl/core, PCI: PCIe portdrv: Add CXL timeout range programmingdrivers/cxl/core/pci.c drivers/pci/pcie/cxl_isolation.c include/cxl/isolation.h+154 -04 replies
14/16ACPI: Add CXL isolation _OSC fieldsdrivers/acpi/pci_root.c include/linux/acpi.h+12 -01 reply
15/16cxl/core, cxl/acpi: Enable CXL isolation based on _OSC handshakedrivers/cxl/acpi.c drivers/cxl/core/port.c drivers/cxl/cxl.h +1 more+48 -20 replies
16/16cxl/core, cxl/acpi: Add CXL isolation notify handlerdrivers/cxl/acpi.c drivers/cxl/core/port.c drivers/cxl/cxl.h+86 -90 replies

Reviews trailers per patch

2 of 16 patches have a Reviewed-by or Acked-by.

PatchRafael J. Wysocki (Intel)Jonathan Cameron
1/16
2/16R-b
3/16
4/16
5/16
6/16
7/16
8/16
9/16
10/16
11/16
12/16
13/16
14/16A-b
15/16
16/16

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 00/16] CXL.mem error isolation support

Overview
========
This series adds support for the CXL.mem Timeout & Isolation Capability as defined by the CXL 3.2 spec (section 8.2.4.24), with some extras (explained below). This is an optional capability implemented by CXL-capable PCIe Root Ports to prevent the host system from resetting when the CXL.mem link times out or goes down (i.e. CXL memory device is suprise removed or dies). Without this capability, the system is expected to immediately reset or power off when either of these conditions occurs.

When CXL.mem isolation is triggered, the CXL memory below the port is no longer accessible. Writes to the memory from the host are expected to silently drop, while a synchronous response is expected for reads. This response is implemntation specific, but an example response would be poisoned data.

The specific features enabled by this series are:
- Enabling CXL.mem isolation on link down conditions and transaction timeout
- Setting up, enabling, and handling CXL isolation interrupts
- Preventing onlining/enabling of isolated CXL memory
- Sysfs attributes for system administrators to tune isolation capabilities

The Extras
==========
The last 3 commits provide support for an ECN [1] submitted by AMD that allows platform firmware to modify how the OS enables and handles CXL isolation. The ECN contents are expected to land in revision 4 of the CXL spec. The link at [1] is only accessible to CXL SSWG members, but I've done my best to explain the changes in the relevant commits.

The changes in these commits could probably be moved to earlier commits, but I've opted to leave them tacked on the end just in case anyone has a problem with their inclusion.

Intended Behavior
=================
Due to how CXL memory is currently handled by Linux, this feature isn't all that useful for type 3 cards. The intended behavior for type 3 cards is to panic when isolation is triggered, which defeats the purpose of the feature.

The reason I'm sending this out anyway is twofold:
1) I've seen rumblings that CXL memory will be part of it's own opt-in allocator in the future and the memory may be safely removable at that point.
2) CXL memory provided by a Type 2 card may be safely removable, though it's left up to the type 2 endpoint driver to handle isolation recovery.

I've also not included a flow for isolation recovery. This is because I
a) don't have a system that supports it, and b) it's not applicable to the type 3 driver.

Building the Set
================
This series is based on both Terry's port error handling patch set (v10) and Dave's deferred downstream port probe set (v7). Terry's set was needed since it introduces the uncorrectable CXL error = system panic paradigm, as well as the routines for logging the AER info from the CXL subsystem.

I included Dave's set due to a timing issue I saw where the PCIe portdrv code would run after the CXL ports that have the isolation capability were probed. This caused the isolation set up to fail because the PCIe portdrv provides the information to allocate the CXL isolation interrupt. I tried deferring the probe, but the deferral caused the cxl_mem driver to break because the port wasn't probed yet. I could have introduced a scheme to get around this, but it seemed easier to just use Dave's set to fix it.

The isolation support is gated behind the CXL core being built-in because the CXL isolation PCIe service needs the mapping code in cxl/core/regs.c. I realize a rework is planned for the PCIe portdrv to (hopefully) not make this the case, so I've kept the code as minimal as possible.

To build the set I applied Terry's set to the base commit below, Dave's on top of that, then my patches.

Patch Breakdown
===============
Patches 3-5 & 12-13 will need eyes from PCIe folks. Patch 14 needs an ACPI reviewer.

- Patches 1-2: Register mapping updates needed for isolation support
- Patches 3-5: CXL isolation service driver & MSI/-X vector allocation
- Patch 6: Enable CXL.mem isolation
- Patches 7-8: Set up and enable CXL isolation interrupts
- Patch 9: Preventing onlining isolated memory
- Patch 10: Enable CXL.mem transaction timeout
- Patch 11: cxl_pci isolation handler
- Patches 12-13: CXL isolation sysfs attributes
- Patch 14: ECN changes to CXL _OSC method
- Patches 15-16: ECN additions

[1]:

Ben Cheatham (16):
cxl/regs: Add cxl_unmap_component_regs()
cxl/regs: Add CXL Isolation capability mapping
PCI: PCIe portdrv: Add CXL Isolation service driver
PCI: PCIe portdrv: Allocate CXL isolation MSI/-X vector
PCI: PCIe portdrv: Add interface for getting CXL isolation IRQ cxl/core: Enable CXL.mem isolation
cxl/core: Set up isolation interrupts
cxl/core: Enable CXL isolation interrupts
cxl/core: Prevent onlining CXL memory behind isolated ports cxl/core: Enable CXL.mem timeout
cxl/pci: Add isolation handler
PCI: PCIe portdrv: Add cxl_isolation sysfs attributes cxl/core, PCI: PCIe portdrv: Add CXL timeout range programming
ACPI: Add CXL isolation _OSC fields
cxl/core, cxl/acpi: Enable CXL isolation based on _OSC handshake cxl/core, cxl/acpi: Add CXL isolation notify handler

 drivers/acpi/pci_root.c          |   9 +
 drivers/cxl/Kconfig              |  14 ++
 drivers/cxl/acpi.c               |  75 +++++++
 drivers/cxl/core/core.h          |   2 +
 drivers/cxl/core/pci.c           | 138 ++++++++++++
 drivers/cxl/core/port.c          | 248 +++++++++++++++++++++
 drivers/cxl/core/region.c        |   3 +
 drivers/cxl/core/regs.c          |  85 +++++--
 drivers/cxl/cxl.h                |  35 +++
 drivers/cxl/cxlmem.h             |   4 +
 drivers/cxl/pci.c                |   9 +
 drivers/pci/pci-sysfs.c          |   3 +
 drivers/pci/pci.h                |   4 +
 drivers/pci/pcie/Makefile        |   1 +
 drivers/pci/pcie/cxl_isolation.c | 371 +++++++++++++++++++++++++++++++
 drivers/pci/pcie/portdrv.c       |  21 +-
 drivers/pci/pcie/portdrv.h       |  18 +-
 include/cxl/isolation.h          |  66 ++++++
 include/linux/acpi.h             |   3 +
 19 files changed, 1086 insertions(+), 23 deletions(-)
 create mode 100644 drivers/pci/pcie/cxl_isolation.c
 create mode 100644 include/cxl/isolation.h

base-commit: a403fe6c0b17f472e01246eb350f5eef105243ac

Signature · 1 line
2.34.1