Skip to content

cxl: Add cxl_reset sysfs attribute for memdevs

Srirangan Madhavan linux-cxl linux-pci linux-tegra lkml 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v6 Latest9 patches 22 repliesIn review
  2. v57 patches 13 repliesSuperseded

Changes in v6 from the cover letter

- Rebased on the current CXL tree used for v7.1-rc4 development.
- Move the ABI from /sys/bus/pci/devices/.../cxl_reset to /sys/bus/cxl/devices/memX/cxl_reset.
- Use the memdev as the userspace handle while keeping the reset orchestration scoped to the CXL device reset scope.
- Reduce the earlier PCI/CXL save/restore series [3] to a single CXL HDM decoder restore/commit helper patch, included here as patch 1.
- Do not offline or hot-remove memory as part of reset. Return -EBUSY if an affected CXL region is online as System RAM or has an active region driver bound.
- Add reset-idle validation and CPU cache invalidation for affected CXL regions.
- Add CXL sibling PCI function discovery using the Non-CXL Function Map DVSEC and CXL.cache/CXL.mem capability bits.
- Coordinate PCI save/disable/restore and IOMMU reset prepare/done for the target and affected sibling functions.
- Add CXL DVSEC reset sequencing, including CXL.cache disable, writeback-invalidate, a minimum 100ms quiet period, reset-complete polling, and Reset Error reporting.
- Track affected memdevs, lock active memdevs across reset, restore and commit decoder state, re-enable CXL.mem, and wait for media ready after reset.
- Cache reset capability at memdev registration time for sysfs visibility.
- Document reset scope, Memory Clear not being requested, and -EBUSY behavior for active CXL regions.

Patches in v6 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/9cxl/hdm: Add helpers to restore and commit memdev decodersdrivers/cxl/core/hdm.c drivers/cxl/cxl.h+317 -35 replies
2/9PCI: Export pci_dev_save_and_disable() and pci_dev_restore()drivers/pci/pci.c include/linux/pci.h+22 -22 replies
3/9cxl: Add reset-idle and cache flush helpersdrivers/cxl/core/pci.c+170 -03 replies
4/9PCI/CXL: Add sibling function coordination for resetdrivers/cxl/core/pci.c include/uapi/linux/pci_regs.h+209 -03 replies
5/9cxl/pci: Add CXL DVSEC reset helperdrivers/cxl/core/pci.c include/uapi/linux/pci_regs.h+198 -01 reply
6/9cxl/pci: Track memdevs affected by CXL resetdrivers/cxl/core/pci.c+176 -01 reply
7/9cxl/pci: Orchestrate CXL reset for affected memdevsdrivers/cxl/core/pci.c+358 -562 replies
8/9cxl/memdev: Add cxl_reset sysfs attributedrivers/cxl/core/memdev.c drivers/cxl/core/pci.c drivers/cxl/cxl.h +1 more+136 -12 replies
9/9Documentation/ABI: Document CXL memdev cxl_resetDocumentation/ABI/testing/sysfs-bus-cxl+28 -01 reply

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v6 yet.

Read the v6 thread

Cover letter [PATCH v6 0/9] cxl: Add cxl_reset sysfs attribute for memdevs

Hi folks!

This patch series introduces support for the CXL Reset method for CXL Type 2 devices, implementing the reset procedure outlined in the CXL Specification r3.2 [1], Sections 8.1.3, 9.6, and 9.7.

The userspace ABI is a write-only cxl_reset attribute under the CXL memdev device:

/sys/bus/cxl/devices/memX/cxl_reset

The memdev is the userspace handle, while the implementation coordinates the target PCI function, affected sibling PCI functions, active CXL memdevs, and any CXL regions reachable through those memdevs.

v6 changes (from v5 [2]):
- Rebased on the current CXL tree used for v7.1-rc4 development.
- Move the ABI from /sys/bus/pci/devices/.../cxl_reset to /sys/bus/cxl/devices/memX/cxl_reset.
- Use the memdev as the userspace handle while keeping the reset orchestration scoped to the CXL device reset scope.
- Reduce the earlier PCI/CXL save/restore series [3] to a single CXL HDM decoder restore/commit helper patch, included here as patch 1.
- Do not offline or hot-remove memory as part of reset. Return -EBUSY if an affected CXL region is online as System RAM or has an active region driver bound.
- Add reset-idle validation and CPU cache invalidation for affected CXL regions.
- Add CXL sibling PCI function discovery using the Non-CXL Function Map DVSEC and CXL.cache/CXL.mem capability bits.
- Coordinate PCI save/disable/restore and IOMMU reset prepare/done for the target and affected sibling functions.
- Add CXL DVSEC reset sequencing, including CXL.cache disable, writeback-invalidate, a minimum 100ms quiet period, reset-complete polling, and Reset Error reporting.
- Track affected memdevs, lock active memdevs across reset, restore and commit decoder state, re-enable CXL.mem, and wait for media ready after reset.
- Cache reset capability at memdev registration time for sysfs visibility.
- Document reset scope, Memory Clear not being requested, and -EBUSY behavior for active CXL regions.

Motivation:
-----------
- As support for Type 2 devices is being introduced, more devices need a CXL-specific reset mechanism beyond bus-wide PCI reset methods.

- FLR does not affect CXL.cache or CXL.mem protocol state, making CXL Reset the appropriate mechanism for cases where those protocols must be reset.

- The CXL specification highlights use cases such as function rebinding and error recovery where CXL Reset is explicitly required.

Change Description:
-------------------

Patch 1: cxl/hdm: Add helpers to restore and commit memdev decoders
- Restore endpoint decoder programming from CXL core's cached decoder objects while keeping CXL.mem disabled.
- Commit restored HDM decoders as a separate step so reset orchestration can re-enable CXL.mem only after safety checks complete.

Patch 2: PCI: Export pci_dev_save_and_disable() and pci_dev_restore()
- Export PCI reset lifecycle helpers so CXL reset orchestration can save, disable, restore, and invoke reset callbacks for affected functions.

Patch 3: cxl: Add reset-idle and cache flush helpers
- Collect CXL regions affected by a memdev reset.
- Fail reset if affected regions are not idle.
- Invalidate CPU caches for each affected region once.

Patch 4: PCI/CXL: Add sibling function coordination for reset
- Identify CXL.cache/CXL.mem sibling functions in the reset scope.
- Use the Non-CXL Function Map DVSEC to exclude non-CXL functions.
- Save, disable, restore, and unlock affected PCI sibling functions.

Patch 5: cxl/pci: Add CXL DVSEC reset helper
- Execute CXL Reset through the CXL Device DVSEC.
- Disable CXL.cache and request writeback-invalidate where supported.
- Enforce the post-reset quiet period and poll for reset completion.
- Block and restore IOMMU traffic while reset is active.

Patch 6: cxl/pci: Track memdevs affected by CXL reset
- Track the target memdev and any sibling-function memdevs affected by reset.
- Revalidate and lock active memdevs before reset proceeds.

Patch 7: cxl/pci: Orchestrate CXL reset for affected memdevs
- Coordinate region validation, CPU cache invalidation, PCI function preparation, DVSEC reset, decoder restore and commit, CXL.mem enable, and media-ready wait.

Patch 8: cxl/memdev: Add cxl_reset sysfs attribute
- Expose /sys/bus/cxl/devices/memX/cxl_reset.
- Only make the attribute visible when the underlying PCI function is Type 2 and reset capable.
- Write a boolean true value, such as "1" or "true", to trigger reset.

Patch 9: Documentation/ABI: Document CXL memdev cxl_reset
- Document the new memdev sysfs ABI, reset scope, Memory Clear behavior, and idle-region requirement.

The CPU cache invalidation step depends on cpu_cache_invalidate_memregion() support for the affected address ranges. If no provider is available, reset fails before hardware reset is requested.

Command line to test CXL reset on a capable memdev:

echo 1 > /sys/bus/cxl/devices/memX/cxl_reset

Basic CXL DVSEC reset testing was done on a CXL Type 2 device. The reset sequence completed successfully and ResetComplete was observed. Full memdev/region integration testing is still in progress.

References:
[1] https://computeexpresslink.org/wp-content/uploads/2024/12/CXL_3.2-Spec-Announcement_FINAL-1.pdf
[2] https://lore.kernel.org/linux-cxl/[email protected]/
[3] https://lore.kernel.org/linux-cxl/[email protected]/

Srirangan Madhavan (9):
cxl/hdm: Add helpers to restore and commit memdev decoders
PCI: Export pci_dev_save_and_disable() and pci_dev_restore()
cxl: Add reset-idle and cache flush helpers PCI/CXL: Add sibling function coordination for reset cxl/pci: Add CXL DVSEC reset helper
cxl/pci: Track memdevs affected by CXL reset cxl/pci: Orchestrate CXL reset for affected memdevs cxl/memdev: Add cxl_reset sysfs attribute Documentation/ABI: Document CXL memdev cxl_reset

 Documentation/ABI/testing/sysfs-bus-cxl |   28 +
 drivers/cxl/core/hdm.c                  |  318 ++++++-
 drivers/cxl/core/memdev.c               |   30 +
 drivers/cxl/core/pci.c                  | 1140 +++++++++++++++++++++++
 drivers/cxl/cxl.h                       |    5 +
 drivers/cxl/cxlmem.h                    |    2 +
 drivers/pci/pci.c                       |   22 +-
 include/linux/pci.h                     |    2 +
 include/uapi/linux/pci_regs.h           |   15 +
 9 files changed, 1557 insertions(+), 5 deletions(-)

base-commit: abb3c0de119032f4c0c81177884a3bb0a133e6ca

Signature · 1 line
2.43.0