Skip to contentSkip to conversation

PCI/CXL: Add CXL reset support for Type 2 devices

PATCH Superseded Srirangan Madhavan 5 participants 32 messages Jun 23 – Jun 26

Subsystems (3)

v7 of 17 Superseded Latest is v17 →

lore
Share & tools
lore
#0 lore raw

v7 cover PCI/CXL: Add CXL reset support for Type 2 devices

Hi folks!

This series adds CXL Reset support for CXL Type 2 devices through the existing PCI reset_method ABI. The reset sequence follows the CXL 4.0 specification [1], including CXL.cache disable, optional cache writeback, CXL Reset initiation, ResetComplete polling, and ResetError reporting.

The userspace ABI is the existing PCI reset interface:

    /sys/bus/pci/devices/.../reset_method
    /sys/bus/pci/devices/.../reset

Userspace can select "cxl_reset" in reset_method and then trigger reset through the existing reset attribute.

Following Dan's v6 feedback, this replaces the proposed memdev sysfs ABI with the existing PCI reset_method interface.

v7 changes from v6 [2]:
- Move the ABI from a CXL memdev attribute to PCI reset_method.
- Drop the memdev dependency from reset entry; advertise cxl_reset for Type 2 functions that report CXL Reset support in the CXL Device DVSEC.
- Incorporate Dan's HDM reset refactor: shared decoder settings, pci_dev->hdm cached state, and built-in CONFIG_CXL_HDM helpers.
- Cache endpoint HDM settings during PCI enumeration when MMIO decoding is already enabled, and let CXL core refresh the same cache later.
- Reduce the earlier PCI/CXL save/restore series [3] to the HDM state cache and restore infrastructure needed by this reset flow.
- Use cached HDM ranges to reject reset while affected ranges are busy and to invalidate CPU caches before reset.
- Discover the CXL reset scope with the Non-CXL Function Map and CXL cache/mem capability bits.
- Quiesce affected sibling functions with PCI save/disable and IOMMU reset prepare/done before executing reset.
- Restore cached HDM decoder state after reset before completing PCI reset recovery.
- Keep CXL Reset Memory Clear disabled.

Motivation:
-----------
- Type 2 devices need a CXL-specific reset mechanism beyond existing PCI reset methods.

- FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is the architectural reset mechanism for those protocols.

- The PCI reset_method ABI lets userspace select this narrower CXL reset before falling back to broader bus reset methods.

Change Description:
-------------------

Patch 1: cxl/hdm: Split decoder programming into a reusable helper
- Move shared decoder settings to include/cxl/cxl.h.
- Factor low-level HDM register programming into cxl_commit().

Patch 2: cxl/hdm: Cache decoder settings on PCI devices
- Cache CXL core HDM decoder settings in pci_dev->hdm.
- Refresh the cache as decoders are enumerated, committed, or reset.

Patch 3: cxl/hdm: Cache endpoint decoder settings during PCI enumeration
- Snapshot endpoint HDM state during PCI capability initialization when memory decoding is already enabled.
- Reuse the same cache when CXL core later enumerates the device.

Patch 4: PCI: Export pci_dev_save_and_disable() and pci_dev_restore()
- Export PCI reset lifecycle helpers for CXL reset orchestration.

Patch 5: PCI/CXL: Add CXL Device Reset helper
- Add the internal DVSEC reset sequence.
- Disable CXL.cache, perform cache writeback where supported, initiate CXL Reset, and wait for completion.

Patch 6: PCI/CXL: Validate HDM ranges before CXL reset
- Collect enabled cached HDM ranges.
- Reject reset if affected ranges are busy and invalidate CPU caches.

Patch 7: PCI/CXL: Discover the CXL reset scope
- Discover same-scope CXL functions with the Non-CXL Function Map and CXL cache/mem capability bits.

Patch 8: PCI/CXL: Coordinate sibling functions for CXL reset
- Lock, save, disable, and IOMMU-block affected sibling functions.
- Include mem-capable siblings in HDM range validation and cache flush.

Patch 9: cxl/pci: Restore CXL HDM state after PCI reset
- Restore cached global and per-decoder HDM state after reset.
- Keep IOMMU reset blocks active until HDM restore completes.

Patch 10: PCI/CXL: Expose CXL Reset as a PCI reset method
- Add "cxl_reset" to the PCI reset_method table for Type 2 reset-capable CXL devices.

Patch 11: Documentation/ABI: Document CXL Reset PCI reset method
- Document the new reset_method value and reset behavior.

The CPU cache invalidation step depends on cpu_cache_invalidate_memregion() support for the affected address ranges. If no provider is available, reset fails before hardware reset is requested.

Example:

    echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method
    echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset

Basic CXL DVSEC reset testing was done on a CXL Type 2 device. The reset sequence completed successfully and ResetComplete was observed.

References:
[1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
[2] https://lore.kernel.org/linux-cxl/[email protected]/
[3] https://lore.kernel.org/linux-cxl/[email protected]/

Srirangan Madhavan (11):
cxl/hdm: Split decoder programming into a reusable helper cxl/hdm: Cache decoder settings on PCI devices cxl/hdm: Cache endpoint decoder settings during PCI enumeration
PCI: Export pci_dev_save_and_disable() and pci_dev_restore() PCI/CXL: Add CXL Device Reset helper
PCI/CXL: Validate HDM ranges before CXL reset PCI/CXL: Discover the CXL reset scope
PCI/CXL: Coordinate sibling functions for CXL reset cxl/pci: Restore CXL HDM state after PCI reset PCI/CXL: Expose CXL Reset as a PCI reset method Documentation/ABI: Document CXL Reset PCI reset method

 Documentation/ABI/testing/sysfs-bus-pci |   14 +
 drivers/cxl/Kconfig                     |    4 +
 drivers/cxl/core/Makefile               |    2 +-
 drivers/cxl/core/hdm.c                  |  234 ++---
 drivers/cxl/core/region.c               |    6 +-
 drivers/cxl/core/reset.c                | 1276 +++++++++++++++++++++++
 drivers/cxl/cxl.h                       |   43 -
 drivers/pci/pci.c                       |   25 +-
 drivers/pci/probe.c                     |    2 +
 include/cxl/cxl.h                       |   85 +-
 include/linux/pci.h                     |   10 +-
 include/uapi/linux/pci_regs.h           |   15 +
 tools/testing/cxl/test/cxl.c            |   10 +-
 13 files changed, 1554 insertions(+), 172 deletions(-)
 create mode 100644 drivers/cxl/core/reset.c

base-commit: 72afdd8181219f459142e571999b3b44ef7b85fb

Signature · 1 line
2.43.0
#1 lore raw

PATCH v7 1/11 cxl: Split decoder programming into a reusable helper

Move common HDM decoder settings to include/cxl/cxl.h and route the register programming sequence through cxl_commit(). This lets reset code restore cached HDM state without depending on private cxl_core types while keeping hdm.c in charge of the existing commit policy checks.

Build the low-level HDM helper under CONFIG_CXL_HDM so it is available even when cxl_core is modular.

---
 drivers/cxl/Kconfig          |   4 ++
 drivers/cxl/core/Makefile    |   1 +
 drivers/cxl/core/hdm.c       | 122 ++++-------------------------------
 drivers/cxl/core/region.c    |   6 +-
 drivers/cxl/core/reset.c     | 118 +++++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h            |  43 ------------
 include/cxl/cxl.h            |  55 +++++++++++++++-
 tools/testing/cxl/test/cxl.c |  10 +--
 8 files changed, 197 insertions(+), 162 deletions(-)
 create mode 100644 drivers/cxl/core/reset.c
drivers/cxl/Kconfig +4 -0
index 80aeb0d556bd..87d719ea1e14 100644
@@ -6,6 +6,7 @@ menuconfig CXL_BUS
 	select FW_UPLOAD
 	select PCI_DOE
 	select FIRMWARE_TABLE
+	select CXL_HDM
 	select NUMA_KEEP_MEMINFO if NUMA_MEMBLKS
 	select FWCTL if CXL_FEATURES
 	help
@@ -243,4 +244,7 @@ config CXL_ATL
 	depends on CXL_REGION
 	depends on ACPI_PRMT && AMD_NB
 
+config CXL_HDM
+	bool
+
 endif
drivers/cxl/core/Makefile +1 -0
index ce7213818d3c..dc075cee0450 100644
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_BUS) += cxl_core.o
+obj-$(CONFIG_CXL_HDM) += reset.o
 obj-$(CONFIG_CXL_SUSPEND) += suspend.o
 
 ccflags-y += -I$(srctree)/drivers/cxl
drivers/cxl/core/hdm.c +12 -110

166 lines not on this page · raw message

drivers/cxl/core/region.c +3 -3
index f5cd20f48d2b..adaf583ccbab 100644
@@ -2912,9 +2912,9 @@ static int poison_by_decoder(struct device *dev, void *arg)
 	cxlds = cxlmd->cxlds;
 	mode = cxlds->part[cxled->part].mode;
 
-	if (cxled->skip) {
-		offset = cxled->dpa_res->start - cxled->skip;
-		length = cxled->skip;
+	if (cxled->cxld.skip) {
+		offset = cxled->dpa_res->start - cxled->cxld.skip;
+		length = cxled->cxld.skip;
 		rc = cxl_mem_get_poison(cxlmd, offset, length, NULL);
 		if (rc == -EFAULT && mode == CXL_PARTMODE_RAM)
 			rc = 0;
drivers/cxl/core/reset.c new file +118 -0

118 lines not on this page · raw message

drivers/cxl/cxl.h +0 -43

63 lines not on this page · raw message

include/cxl/cxl.h +54 -1

75 lines not on this page · raw message

tools/testing/cxl/test/cxl.c +5 -5
index 4281d34cd0e7..244e8ec28769 100644
@@ -843,11 +843,11 @@ static int cxld_registry_restore(struct cxl_decoder *cxld,
 		dbg_cxld(port, "restore", &td->cxled.cxld);
 		cxld_copy(cxld, &td->cxled.cxld);
 		cxled->state = td->cxled.state;
-		cxled->skip = td->cxled.skip;
+		cxld->skip = td->cxled.cxld.skip;
 		if (range_len(&td->dpa_range)) {
 			rc = devm_cxl_dpa_reserve(cxled, td->dpa_range.start,
 						  range_len(&td->dpa_range),
-						  td->cxled.skip);
+						  td->cxled.cxld.skip);
 			if (rc) {
 				init_disabled_mock_decoder(cxld);
 				return rc;
@@ -885,7 +885,7 @@ static void __cxld_registry_save(struct cxl_test_decoder *td,
 
 		cxld_copy(&td->cxled.cxld, cxld);
 		td->cxled.state = cxled->state;
-		td->cxled.skip = cxled->skip;
+		td->cxled.cxld.skip = cxled->cxld.skip;
 
 		if (!(cxld->flags & CXL_DECODER_F_ENABLE)) {
 			td->dpa_range.start = 0;
@@ -973,7 +973,7 @@ static void mock_decoder_reset(struct cxl_decoder *cxld)
 			to_cxl_endpoint_decoder(&cxld->dev);
 
 		cxled->state = CXL_DECODER_STATE_MANUAL;
-		cxled->skip = 0;
+		cxled->cxld.skip = 0;
 	}
 	if (decoder_reset_preserve_registry)
 		dev_dbg(port->uport_dev, "decoder%d: skip registry update\n",
@@ -1024,7 +1024,7 @@ static void init_disabled_mock_decoder(struct cxl_decoder *cxld)
 			to_cxl_endpoint_decoder(&cxld->dev);
 
 		cxled->state = CXL_DECODER_STATE_MANUAL;
-		cxled->skip = 0;
+		cxled->cxld.skip = 0;
 	}
 }
 
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH v7 01/11] cxl: Split decoder programming into a reusable helper

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines

Move common HDM decoder settings to include/cxl/cxl.h and route the register programming sequence through cxl_commit(). This lets reset code restore cached HDM state without depending on private cxl_core types while keeping hdm.c in charge of the existing commit policy checks.

Build the low-level HDM helper under CONFIG_CXL_HDM so it is available even when cxl_core is modular.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

PATCH v7 2/11 cxl: Cache decoder settings on PCI devices

In reply to Srirangan Madhavan

Cache CXL core's HDM decoder settings in pci_dev->hdm as decoders are enumerated, committed, or reset. PCI reset paths can use this snapshot to restore HDM programming without walking CXL topology during reset recovery.

---
 drivers/cxl/core/hdm.c | 81 +++++++++++++++++++++++++++++++++++++++++-
 include/cxl/cxl.h      | 12 +++++++
 include/linux/pci.h    |  6 ++++
 3 files changed, 98 insertions(+), 1 deletion(-)
drivers/cxl/core/hdm.c +80 -1
index fa978c297546..83cda63f76a5 100644
@@ -84,6 +84,76 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm)
 		cxlhdm->iw_cap_mask |= BIT(16);
 }
 
+static void clear_hdm_info(void *data)
+{
+	struct pci_dev *pdev = data;
+
+	WRITE_ONCE(pdev->hdm, NULL);
+}
+
+static int devm_cxl_pci_setup_hdm_info(struct cxl_hdm *cxlhdm)
+{
+	struct cxl_port *port = cxlhdm->port;
+	struct cxl_hdm_info *info;
+	struct pci_dev *pdev;
+	struct device *uport;
+
+	if (is_cxl_endpoint(port)) {
+		struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport_dev);
+
+		uport = cxlmd->dev.parent;
+	} else {
+		uport = port->uport_dev;
+	}
+
+	if (!dev_is_pci(uport))
+		return 0;
+
+	pdev = to_pci_dev(uport);
+	info = devm_kzalloc(&pdev->dev,
+			    struct_size(info, settings, cxlhdm->decoder_count),
+			    GFP_KERNEL);
+	if (!info)
+		return -ENOMEM;
+
+	info->decoder_count = cxlhdm->decoder_count;
+	WRITE_ONCE(pdev->hdm, info);
+
+	return devm_add_action_or_reset(&pdev->dev, clear_hdm_info, pdev);
+}
+
+static void cxl_hdm_info_set_decoder(struct cxl_hdm *cxlhdm,
+				     struct cxl_decoder *cxld)
+{
+	struct cxl_port *port = cxlhdm->port;
+	struct cxl_hdm_info *info;
+	struct pci_dev *pdev;
+	struct device *uport;
+
+	if (is_cxl_endpoint(port)) {
+		struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport_dev);
+
+		uport = cxlmd->dev.parent;
+	} else {
+		uport = port->uport_dev;
+	}
+
+	if (!dev_is_pci(uport))
+		return;
+
+	pdev = to_pci_dev(uport);
+	info = READ_ONCE(pdev->hdm);
+	if (!info || cxld->id >= info->decoder_count)
+		return;
+
+	if (cxld->flags & CXL_DECODER_F_ENABLE)
+		info->settings[cxld->id] = cxld->settings;
+	else
+		info->settings[cxld->id] = (struct cxl_decoder_settings) {
+			.id = cxld->id,
+		};
+}
+
 static bool should_emulate_decoders(struct cxl_endpoint_dvsec_info *info)
 {
 	struct cxl_hdm *cxlhdm;
@@ -747,6 +817,7 @@ static int cxl_decoder_commit(struct cxl_decoder *cxld)
 		return rc;
 	}
 	port->commit_end++;
+	cxl_hdm_info_set_decoder(cxlhdm, cxld);
 
 	return 0;
 }
@@ -819,6 +890,7 @@ static void cxl_decoder_reset(struct cxl_decoder *cxld)
 	writel(0, hdm + CXL_HDM_DECODER0_BASE_LOW_OFFSET(id));
 
 	cxld->flags &= ~CXL_DECODER_F_ENABLE;
+	cxl_hdm_info_set_decoder(cxlhdm, cxld);
 
 	/* Userspace is now responsible for reconfiguring this decoder */
 	if (is_endpoint_decoder(&cxld->dev)) {
@@ -989,6 +1061,7 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
 		lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(which));
 		hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(which));
 		target_list.value = (hi << 32) + lo;
+		cxld->targets = target_list.value;
 		for (i = 0; i < cxld->interleave_ways; i++)
 			cxld->target_map[i] = target_list.target_id[i];
 
@@ -1062,11 +1135,16 @@ static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,
 	struct cxl_port *port = cxlhdm->port;
 	int i;
 	u64 dpa_base = 0;
+	int rc;
 
 	cxl_settle_decoders(cxlhdm);
 
+	rc = devm_cxl_pci_setup_hdm_info(cxlhdm);
+	if (rc)
+		return rc;
+
 	for (i = 0; i < cxlhdm->decoder_count; i++) {
-		int rc, target_count = cxlhdm->target_count;
+		int target_count = cxlhdm->target_count;
 		struct cxl_decoder *cxld;
 
 		if (is_cxl_endpoint(port)) {
@@ -1101,6 +1179,7 @@ static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,
 			put_device(&cxld->dev);
 			return rc;
 		}
+		cxl_hdm_info_set_decoder(cxlhdm, cxld);
 		rc = add_hdm_decoder(port, cxld);
 		if (rc) {
 			dev_warn(&port->dev,
include/cxl/cxl.h +12 -0
index 93d4eba6947a..cc933379f67b 100644
@@ -121,6 +121,18 @@ struct cxl_regs {
 	);
 };
 
+/**
+ * struct cxl_hdm_info - PCI device HDM decoder programming cache
+ * @decoder_count: number of decoder settings entries
+ * @regs: mapped CXL component registers for this HDM decoder block
+ * @settings: cached per-decoder programming state
+ */
+struct cxl_hdm_info {
+	int decoder_count;
+	struct cxl_component_regs regs;
+	struct cxl_decoder_settings settings[] __counted_by(decoder_count);
+};
+
 int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm);
 
 struct cxl_reg_map {
include/linux/pci.h +6 -0
index 2c4454583c11..7db2daf8597c 100644
@@ -334,6 +334,9 @@ struct pcie_link_state;
 struct pci_sriov;
 struct pci_p2pdma;
 struct rcec_ea;
+#ifdef CONFIG_CXL_HDM
+struct cxl_hdm_info;
+#endif
 
 /* struct pci_dev - describes a PCI device
  *
@@ -563,6 +566,9 @@ struct pci_dev {
 #ifdef CONFIG_PCI_DOE
 	struct xarray	doe_mbs;	/* Data Object Exchange mailboxes */
 #endif
+#ifdef CONFIG_CXL_HDM
+	struct cxl_hdm_info *hdm;	/* CXL HDM decoder reset state */
+#endif
 #ifdef CONFIG_PCI_NPEM
 	struct npem	*npem;		/* Native PCIe Enclosure Management */
 #endif
Signature · 1 line
2.43.0
#4 lore raw

Re: [PATCH v7 02/11] cxl: Cache decoder settings on PCI devices

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] Cached target list is truncated to 8 targets, breaking reset recovery for 16-way interleaving.

Pre-existing issues:
- [High] HDM target list parsing uses an endian-unsafe union, reversing target IDs on big-endian architectures.
- [High] Stack out-of-bounds read occurs when processing decoders with more than 8 interleave ways. --

commit 1a9ec0491bde62ac73e5c79d561f966396fbb2b8
Author: Srirangan Madhavan <[email protected]>

cxl: Cache decoder settings on PCI devices

Cache CXL core's HDM decoder settings in pci_dev->hdm as decoders are enumerated, committed, or reset. PCI reset paths can use this snapshot to restore HDM programming without walking CXL topology during reset recovery.

Quoted 4 lines
drivers/cxl/core/hdm.c +0 -0
index fa978c297546f..83cda63f76a5a 100644

[ ... ]

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1064

Quoted 5 lines
drivers/cxl/core/hdm.c +1 -0
@@ -989,6 +1061,7 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
 		lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(which));
 		hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(which));
 		target_list.value = (hi << 32) + lo;
+		cxld->targets = target_list.value;

[Severity: High]
Does this truncate the target list to 8 targets?

Since target_list.value is a u64, if a decoder is configured for 16-way interleaving, it looks like the extended target list registers are ignored and the upper 8 targets will be permanently lost during reset recovery.

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1066

Quoted 2 lines
drivers/cxl/core/hdm.c +0 -0
 		for (i = 0; i < cxld->interleave_ways; i++)
 			cxld->target_map[i] = target_list.target_id[i];

[Severity: High]
This isn't a bug introduced by this patch, but doesn't the target_list union overlay reverse the target list on big-endian architectures?

The target_list union maps an 8-byte array over a u64, which accesses the value in memory order.

[Severity: High]
This is also a pre-existing issue, but since target_id is strictly bounded to 8 bytes in the union, does this loop perform a stack out-of-bounds read into cxld->target_map if cxld->interleave_ways is greater than 8?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#5 lore raw

Re: [PATCH v7 02/11] cxl: Cache decoder settings on PCI devices

In reply to Srirangan Madhavan
Quoted 45 lines · Srirangan Madhavan

Cache CXL core's HDM decoder settings in pci_dev->hdm as decoders are enumerated, committed, or reset. PCI reset paths can use this snapshot to restore HDM programming without walking CXL topology during reset recovery.

---
 drivers/cxl/core/hdm.c | 81 +++++++++++++++++++++++++++++++++++++++++-
 include/cxl/cxl.h      | 12 +++++++
 include/linux/pci.h    |  6 ++++
 3 files changed, 98 insertions(+), 1 deletion(-)
drivers/cxl/core/hdm.c +26 -0
index fa978c297546..83cda63f76a5 100644
@@ -84,6 +84,76 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm)
 		cxlhdm->iw_cap_mask |= BIT(16);
 }
 
+static void clear_hdm_info(void *data)
+{
+	struct pci_dev *pdev = data;
+
+	WRITE_ONCE(pdev->hdm, NULL);
+}
+
+static int devm_cxl_pci_setup_hdm_info(struct cxl_hdm *cxlhdm)
+{
+	struct cxl_port *port = cxlhdm->port;
+	struct cxl_hdm_info *info;
+	struct pci_dev *pdev;
+	struct device *uport;
+
+	if (is_cxl_endpoint(port)) {
+		struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport_dev);
+
+		uport = cxlmd->dev.parent;
+	} else {
+		uport = port->uport_dev;
+	}
+
+	if (!dev_is_pci(uport))
+		return 0;
+
+	pdev = to_pci_dev(uport);

drivers/cxl/core/hdm.c static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) · line 122

Quoted 10 lines
drivers/cxl/core/hdm.c +10 -0
+	info = devm_kzalloc(&pdev->dev,
+			    struct_size(info, settings, cxlhdm->decoder_count),
+			    GFP_KERNEL);
+	if (!info)
+		return -ENOMEM;
+
+	info->decoder_count = cxlhdm->decoder_count;
+	WRITE_ONCE(pdev->hdm, info);
+
+	return devm_add_action_or_reset(&pdev->dev, clear_hdm_info, pdev);

The CXL core can update the PCI cached HDM settings under the device lock, but it should not be doing its own allocation. It also should not clear the cached settings on shutdown. That would defeat the purpose of having the HDM settings available while the device is disabled.

#6 lore raw

PATCH v7 3/11 cxl: Cache endpoint decoder settings during PCI enumeration

In reply to Srirangan Madhavan

Populate pci_dev->hdm from PCI capability initialization when a CXL.mem function already has memory decoding enabled. This gives driver-free reset paths an early HDM snapshot without enabling BAR decoding during enumeration.

CXL core later reuses and refreshes the same cache. Move the register helpers into the built-in CONFIG_CXL_HDM set so the early cache path is available without cxl_core.

---
 drivers/cxl/core/Makefile |   3 +-
 drivers/cxl/core/hdm.c    |  59 +++++++----
 drivers/cxl/core/reset.c  | 202 ++++++++++++++++++++++++++++++++++++++
 drivers/pci/probe.c       |   2 +
 include/cxl/cxl.h         |   9 ++
 5 files changed, 252 insertions(+), 23 deletions(-)
drivers/cxl/core/Makefile +1 -2
index dc075cee0450..69cf2ea7ee74 100644
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_BUS) += cxl_core.o
-obj-$(CONFIG_CXL_HDM) += reset.o
+obj-$(CONFIG_CXL_HDM) += regs.o reset.o
 obj-$(CONFIG_CXL_SUSPEND) += suspend.o
 
 ccflags-y += -I$(srctree)/drivers/cxl
@@ -8,7 +8,6 @@ CFLAGS_trace.o = -DTRACE_INCLUDE_PATH=. -I$(src)
 
 cxl_core-y := port.o
 cxl_core-y += pmem.o
-cxl_core-y += regs.o
 cxl_core-y += memdev.o
 cxl_core-y += mbox.o
 cxl_core-y += pci.o
drivers/cxl/core/hdm.c +38 -21

98 lines not on this page · raw message

drivers/cxl/core/reset.c +202 -0

214 lines not on this page · raw message

drivers/pci/probe.c +2 -0
index b63cd0c310bc..9e214446fd42 100644
@@ -24,6 +24,7 @@
 #include <linux/pm_runtime.h>
 #include <linux/bitfield.h>
 #include <trace/events/pci.h>
+#include <cxl/cxl.h>
 #include "pci.h"
 
 static struct resource busn_resource = {
@@ -2679,6 +2680,7 @@ static void pci_init_capabilities(struct pci_dev *dev)
 	pci_rebar_init(dev);		/* Resizable BAR */
 	pci_dev3_init(dev);		/* Device 3 capabilities */
 	pci_ide_init(dev);		/* Link Integrity and Data Encryption */
+	pci_cxl_hdm_init(dev);		/* CXL HDM Decoder Capability */
 
 	pcie_report_downtraining(dev);
 	pci_init_reset_methods(dev);
include/cxl/cxl.h +9 -0
index cc933379f67b..e3087b7517e8 100644
@@ -26,6 +26,7 @@ enum cxl_devtype {
 };
 
 struct cxl_region;
+struct pci_dev;
 
 enum cxl_decoder_type {
 	CXL_DECODER_DEVMEM = 2,
@@ -134,6 +135,14 @@ struct cxl_hdm_info {
 };
 
 int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm);
+#ifdef CONFIG_CXL_HDM
+int pci_cxl_hdm_init(struct pci_dev *pdev);
+#else
+static inline int pci_cxl_hdm_init(struct pci_dev *pdev)
+{
+	return -ENOTTY;
+}
+#endif
 
 struct cxl_reg_map {
 	bool valid;
Signature · 1 line
2.43.0
#7 lore raw

Re: [PATCH v7 03/11] cxl: Cache endpoint decoder settings during PCI enumeration

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] Allocating devres resources during PCI device enumeration causes driver probing to fail…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] Allocating devres resources during PCI device enumeration causes driver probing to fail permanently with -EBUSY.
- [High] devm_cxl_pci_setup_hdm_info() falsely rejects devices using DVSEC emulation when their DVSEC range count differs from their HDM decoder count.
- [Medium] Missing release/acquire memory barriers when publishing and consuming the pdev->hdm pointer allows concurrent readers to see uninitialized memory. --

commit e3058b1fc7aea013a57ebe41db76dc923a188403
Author: Srirangan Madhavan <[email protected]>

cxl: Cache endpoint decoder settings during PCI enumeration

Populate pci_dev->hdm from PCI capability initialization when a CXL.mem function already has memory decoding enabled. This gives driver-free reset paths an early HDM snapshot without enabling BAR decoding during enumeration.

Quoted 4 lines
drivers/cxl/core/hdm.c +0 -0
index 83cda63f76a5a..0230ebfada42b 100644

[ ... ]

Quoted 12 lines
drivers/cxl/core/hdm.c +8 -0
@@ -107,9 +105,27 @@ static int devm_cxl_pci_setup_hdm_info(struct cxl_hdm *cxlhdm)
 	}
 
 	if (!dev_is_pci(uport))
+		return NULL;
+
+	return to_pci_dev(uport);
+}
+
+static int devm_cxl_pci_setup_hdm_info(struct cxl_hdm *cxlhdm)
+{
+	struct cxl_hdm_info *info;

drivers/cxl/core/hdm.c static int devm_cxl_pci_setup_hdm_info(struct cxl_hdm *cxlhdm) · line 125

Quoted 10 lines
drivers/cxl/core/hdm.c +10 -0
+	struct pci_dev *pdev;
+
+	pdev = cxl_hdm_to_pci_dev(cxlhdm);
+	if (!pdev)
+		return 0;
+
+	info = READ_ONCE(pdev->hdm);
+	if (info) {
+		if (info->decoder_count != cxlhdm->decoder_count)
+			return -ENXIO;

[Severity: High]
Will this strict equality check reject valid devices that fall back to DVSEC emulation?

When the cxl_port driver probes, devm_cxl_setup_hdm() might fall back to DVSEC emulation, which overwrites the local cxlhdm->decoder_count with the number of DVSEC ranges. If that happens, this check against the raw hardware HDM count will fail and return -ENXIO, breaking the probe.

Quoted 7 lines
drivers/cxl/core/hdm.c +1 -1
 		return 0;
+	}
 
-	pdev = to_pci_dev(uport);
 	info = devm_kzalloc(&pdev->dev,
 			    struct_size(info, settings, cxlhdm->decoder_count),
 			    GFP_KERNEL);

[ ... ]

Quoted 4 lines
drivers/cxl/core/reset.c +0 -0
index 14f024098e826..fc52d3abdb5b4 100644

[ ... ]

Quoted 2 lines
drivers/cxl/core/reset.c +2 -0
+int pci_cxl_hdm_init(struct pci_dev *pdev)
+{

[ ... ]

drivers/cxl/core/reset.c int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm) · line 256

Quoted 5 lines
drivers/cxl/core/reset.c +5 -0
+	if (!info) {
+		info = devm_kzalloc(&pdev->dev,
+				    struct_size(info, settings,
+						CXL_HDM_DECODER_MAX_COUNT),
+				    GFP_KERNEL);

[Severity: Critical]
Does allocating devres memory during PCI enumeration prevent the device from being probed later?

The driver core requires a device's devres_head to be empty before a driver can bind to it. Since pci_cxl_hdm_init() is called during the early pci_init_capabilities() enumeration phase, this devm_kzalloc() will attach resources before probing.

Will really_probe() then abort with a resources present before probing error and permanently prevent the cxl_pci driver from binding to the device?

Quoted 4 lines
drivers/cxl/core/reset.c +4 -0
+		if (!info)
+			return -ENOMEM;
+		allocated_info = true;
+	}

[ ... ]

drivers/cxl/core/reset.c int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm) · line 307

Quoted 9 lines
drivers/cxl/core/reset.c +9 -0
+	settings = info->settings;
+	for (int i = 0; i < info->decoder_count; i++) {
+		rc = cxl_pci_hdm_read_decoder(pdev, &settings[i],
+					      regs.hdm_decoder, i);
+		if (rc)
+			goto out_unmap;
+	}
+
+	WRITE_ONCE(pdev->hdm, info);

[Severity: Medium]
Are memory barriers needed here to ensure the initialized memory is visible before the pointer is published?

Since WRITE_ONCE() and READ_ONCE() do not provide SMP memory barriers, can the CPU reorder the devm_kzalloc() memory zeroing after the pointer publication? If a concurrent sysfs reader accesses the pointer, it could potentially see a non-zero garbage value for info->regs.hdm_decoder.

Quoted 8 lines
drivers/cxl/core/reset.c +8 -0
+	if (allocated_info) {
+		rc = devm_add_action(&pdev->dev, cxl_pci_hdm_clear, pdev);
+		if (rc) {
+			WRITE_ONCE(pdev->hdm, NULL);
+			goto out_unmap;
+		}
+	}
+	return 0;

[ ... ]

Quoted 4 lines
drivers/pci/probe.c +0 -0
index b63cd0c310bc0..9e214446fd42f 100644

[ ... ]

Quoted 8 lines
drivers/pci/probe.c +1 -0
@@ -2679,6 +2680,7 @@ static void pci_init_capabilities(struct pci_dev *dev)
 	pci_rebar_init(dev);		/* Resizable BAR */
 	pci_dev3_init(dev);		/* Device 3 capabilities */
 	pci_ide_init(dev);		/* Link Integrity and Data Encryption */
+	pci_cxl_hdm_init(dev);		/* CXL HDM Decoder Capability */
 
 	pcie_report_downtraining(dev);
 	pci_init_reset_methods(dev);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#8 lore raw

Re: [PATCH v7 03/11] cxl: Cache endpoint decoder settings during PCI enumeration

In reply to Srirangan Madhavan
Quoted 123 lines · Srirangan Madhavan

Populate pci_dev->hdm from PCI capability initialization when a CXL.mem function already has memory decoding enabled. This gives driver-free reset paths an early HDM snapshot without enabling BAR decoding during enumeration.

CXL core later reuses and refreshes the same cache. Move the register helpers into the built-in CONFIG_CXL_HDM set so the early cache path is available without cxl_core.

---
 drivers/cxl/core/Makefile |   3 +-
 drivers/cxl/core/hdm.c    |  59 +++++++----
 drivers/cxl/core/reset.c  | 202 ++++++++++++++++++++++++++++++++++++++
 drivers/pci/probe.c       |   2 +
 include/cxl/cxl.h         |   9 ++
 5 files changed, 252 insertions(+), 23 deletions(-)
drivers/cxl/core/Makefile +1 -2
index dc075cee0450..69cf2ea7ee74 100644
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_BUS) += cxl_core.o
-obj-$(CONFIG_CXL_HDM) += reset.o
+obj-$(CONFIG_CXL_HDM) += regs.o reset.o
 obj-$(CONFIG_CXL_SUSPEND) += suspend.o
 
 ccflags-y += -I$(srctree)/drivers/cxl
@@ -8,7 +8,6 @@ CFLAGS_trace.o = -DTRACE_INCLUDE_PATH=. -I$(src)
 
 cxl_core-y := port.o
 cxl_core-y += pmem.o
-cxl_core-y += regs.o
 cxl_core-y += memdev.o
 cxl_core-y += mbox.o
 cxl_core-y += pci.o
drivers/cxl/core/hdm.c +23 -17
index 83cda63f76a5..0230ebfada42 100644
@@ -91,11 +91,9 @@ static void clear_hdm_info(void *data)
 	WRITE_ONCE(pdev->hdm, NULL);
 }
 
-static int devm_cxl_pci_setup_hdm_info(struct cxl_hdm *cxlhdm)
+static struct pci_dev *cxl_hdm_to_pci_dev(struct cxl_hdm *cxlhdm)
 {
 	struct cxl_port *port = cxlhdm->port;
-	struct cxl_hdm_info *info;
-	struct pci_dev *pdev;
 	struct device *uport;
 
 	if (is_cxl_endpoint(port)) {
@@ -107,9 +105,27 @@ static int devm_cxl_pci_setup_hdm_info(struct cxl_hdm *cxlhdm)
 	}
 
 	if (!dev_is_pci(uport))
+		return NULL;
+
+	return to_pci_dev(uport);
+}
+
+static int devm_cxl_pci_setup_hdm_info(struct cxl_hdm *cxlhdm)
+{
+	struct cxl_hdm_info *info;
+	struct pci_dev *pdev;
+
+	pdev = cxl_hdm_to_pci_dev(cxlhdm);
+	if (!pdev)
+		return 0;
+
+	info = READ_ONCE(pdev->hdm);
+	if (info) {
+		if (info->decoder_count != cxlhdm->decoder_count)
+			return -ENXIO;
 		return 0;
+	}
 
-	pdev = to_pci_dev(uport);
 	info = devm_kzalloc(&pdev->dev,
 			    struct_size(info, settings, cxlhdm->decoder_count),
 			    GFP_KERNEL);
@@ -125,23 +141,13 @@ static int devm_cxl_pci_setup_hdm_info(struct cxl_hdm *cxlhdm)
 static void cxl_hdm_info_set_decoder(struct cxl_hdm *cxlhdm,
 				     struct cxl_decoder *cxld)
 {
-	struct cxl_port *port = cxlhdm->port;
 	struct cxl_hdm_info *info;
 	struct pci_dev *pdev;
-	struct device *uport;
-
-	if (is_cxl_endpoint(port)) {
-		struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport_dev);
-
-		uport = cxlmd->dev.parent;
-	} else {
-		uport = port->uport_dev;
-	}
 
-	if (!dev_is_pci(uport))
+	pdev = cxl_hdm_to_pci_dev(cxlhdm);
+	if (!pdev)
 		return;
 
-	pdev = to_pci_dev(uport);
 	info = READ_ONCE(pdev->hdm);
 	if (!info || cxld->id >= info->decoder_count)
 		return;
@@ -202,6 +208,7 @@ static struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port,
 	struct cxl_register_map *reg_map = &port->reg_map;
 	struct device *dev = &port->dev;
 	struct cxl_hdm *cxlhdm;
+	struct pci_dev *pdev;
 	int rc;
 
 	cxlhdm = devm_kzalloc(dev, sizeof(*cxlhdm), GFP_KERNEL);
@@ -227,11 +234,21 @@ static struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port,
 		return ERR_PTR(-ENODEV);
 	}
 
-	rc = cxl_map_component_regs(reg_map, &cxlhdm->regs,

drivers/cxl/core/hdm.c static struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port, · line 242

Quoted 10 lines
drivers/cxl/core/hdm.c +6 -4
-				    BIT(CXL_CM_CAP_CAP_ID_HDM));
-	if (rc) {
-		dev_err(dev, "Failed to map HDM capability.\n");
-		return ERR_PTR(rc);
+	pdev = cxl_hdm_to_pci_dev(cxlhdm);
+	if (pdev) {
+		struct cxl_hdm_info *info = READ_ONCE(pdev->hdm);
+
+		if (info && info->regs.hdm_decoder)
+			cxlhdm->regs = info->regs;

The driver probably should not reuse the core mapping. The driver's only role is to update the cached settings if it changes them.

Quoted 80 lines
drivers/cxl/core/hdm.c +9 -0
+	}
+
+	if (!cxlhdm->regs.hdm_decoder) {
+		rc = cxl_map_component_regs(reg_map, &cxlhdm->regs,
+					    BIT(CXL_CM_CAP_CAP_ID_HDM));
+		if (rc) {
+			dev_err(dev, "Failed to map HDM capability.\n");
+			return ERR_PTR(rc);
+		}
 	}
 
 	parse_hdm_decoder_caps(cxlhdm);
drivers/cxl/core/reset.c +50 -0
index 14f024098e82..fc52d3abdb5b 100644
@@ -2,9 +2,16 @@
 /* Copyright(c) 2026 NVIDIA Corporation. All rights reserved. */
 #include <linux/delay.h>
 #include <linux/bug.h>
+#include <linux/bitfield.h>
 #include <linux/errno.h>
 #include <linux/export.h>
+#include <linux/io.h>
+#include <linux/ioport.h>
 #include <linux/kernel.h>
+#include <linux/pci.h>
+#include <linux/slab.h>
+
+#include <cxlpci.h>
 
 #include "cxl.h"
 #include "core.h"
@@ -116,3 +123,198 @@ int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm)
 	return 0;
 }
 EXPORT_SYMBOL_FOR_MODULES(cxl_commit, "cxl_core");
+
+#define CXL_HDM_DECODER_MAX_COUNT 32
+
+static void cxl_pci_hdm_clear(void *data)
+{
+	struct pci_dev *pdev = data;
+
+	WRITE_ONCE(pdev->hdm, NULL);
+}
+
+static void cxl_pci_hdm_unmap(struct pci_dev *pdev,
+			      struct cxl_component_regs *regs,
+			      struct cxl_register_map *map)
+{
+	struct cxl_reg_map *hdm_map = &map->component_map.hdm_decoder;
+
+	if (!regs->hdm_decoder)
+		return;
+
+	devm_iounmap(&pdev->dev, regs->hdm_decoder);
+	devm_release_mem_region(&pdev->dev, map->resource + hdm_map->offset,
+				hdm_map->size);
+}
+
+static int cxl_pci_hdm_read_decoder(struct pci_dev *pdev,
+				    struct cxl_decoder_settings *settings,
+				    void __iomem *hdm, int id)
+{
+	u64 target_or_skip, base, size;
+	u32 ctrl, lo, hi;
+	int rc;
+
+	*settings = (struct cxl_decoder_settings) {
+		.id = id,
+	};
+
+	ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(id));
+	if (!(ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED))
+		return 0;
+
+	lo = readl(hdm + CXL_HDM_DECODER0_BASE_LOW_OFFSET(id));
+	hi = readl(hdm + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(id));
+	base = ((u64)hi << 32) | lo;

drivers/cxl/core/reset.c int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm) · line 178

Quoted 10 lines
drivers/cxl/core/reset.c +10 -0
+
+	lo = readl(hdm + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(id));
+	hi = readl(hdm + CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(id));
+	size = ((u64)hi << 32) | lo;
+
+	if (!size || base == U64_MAX || size == U64_MAX ||
+	    base > U64_MAX - (size - 1)) {
+		pci_err(pdev, "CXL HDM decoder %d has invalid range\n", id);
+		return -ENXIO;
+	}

Zero size is ok now (see series from Richard), and the U64_MAX checks in the main driver are more about catching common MMIO read failure cases. In any event I think the core of this function and the validation should be shared with init_hdm_decoder() so these kinds of disconnects are prevented.

Quoted 38 lines
drivers/cxl/core/reset.c +38 -0
+
+	lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(id));
+	hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(id));
+	target_or_skip = ((u64)hi << 32) | lo;
+
+	settings->hpa_range = (struct range) {
+		.start = base,
+		.end = base + size - 1,
+	};
+	settings->targets = target_or_skip;
+	settings->target_type = FIELD_GET(CXL_HDM_DECODER0_CTRL_HOSTONLY, ctrl) ?
+				CXL_DECODER_HOSTONLYMEM : CXL_DECODER_DEVMEM;
+	settings->flags = CXL_DECODER_F_ENABLE;
+	if (ctrl & CXL_HDM_DECODER0_CTRL_LOCK)
+		settings->flags |= CXL_DECODER_F_LOCK;
+
+	rc = eiw_to_ways(FIELD_GET(CXL_HDM_DECODER0_CTRL_IW_MASK, ctrl),
+			 &settings->interleave_ways);
+	if (rc)
+		return rc;
+
+	return eig_to_granularity(FIELD_GET(CXL_HDM_DECODER0_CTRL_IG_MASK,
+					    ctrl),
+				  &settings->interleave_granularity);
+}
+
+static int cxl_pci_hdm_capable(struct pci_dev *pdev)
+{
+	u16 cap;
+	int dvsec;
+	int rc;
+
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		return -ENOTTY;
+
+	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);

drivers/cxl/core/reset.c int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm) · line 226

Quoted 10 lines
drivers/cxl/core/reset.c +10 -0
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	if (!(cap & PCI_DVSEC_CXL_MEM_CAPABLE))
+		return -ENOTTY;
+
+	return 0;
+}
+
+int pci_cxl_hdm_init(struct pci_dev *pdev)

Capability init does not fail, so this can be void.

If you want you can make an inner function to report the init failure via an info message, but otherwise PCI core is not looking for failures.

Quoted 14 lines
drivers/cxl/core/reset.c +14 -0
+{
+	struct cxl_decoder_settings *settings;
+	struct cxl_component_regs regs = { 0 };
+	struct cxl_register_map map = { 0 };
+	struct cxl_hdm_info *info;
+	bool allocated_info = false;
+	int decoder_count;
+	u16 command;
+	int rc;
+
+	info = READ_ONCE(pdev->hdm);
+	if (info && info->regs.hdm_decoder)
+		return 0;
+

drivers/cxl/core/reset.c int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm) · line 250

Quoted 10 lines
drivers/cxl/core/reset.c +10 -0
+	rc = cxl_pci_hdm_capable(pdev);
+	if (rc)
+		return rc;
+
+	rc = pci_read_config_word(pdev, PCI_COMMAND, &command);
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	if (!(command & PCI_COMMAND_MEMORY))
+		return -ENOTTY;

This needs some commentary about expectations and likely some formal recommendation to, and agreement with firmware implementers.

The problem is that CXL memory may be, and usually often is, handed off actively mapped to the OS. When that happens this needs to be able to assume that PCI_COMMAND_MEMORY is also enabled for access to the HDM decoders. It would be broken for firmware to say "here is a device actively mapping CXL.mem and by the way you can not find out what those settings are without enabling the device MMIO memory space."

So the expectation is that if a device arrives without PCI_COMMAND_MEMORY then the OS is free to disable CXL reset when a driver is not loaded. Conversely, if a device arrives actively providing system resources, it needs to also arrange for active management.

That said, is it safe to assume that in your testing the memory space is indeed arriving enabled?

drivers/cxl/core/reset.c int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm) · line 253

Quoted 3 lines
drivers/cxl/core/reset.c +3 -0
+
+	if (!info) {
+		info = devm_kzalloc(&pdev->dev,

Devres is not suitable for use outside of a driver.

Quoted 12 lines
drivers/cxl/core/reset.c +12 -0
+				    struct_size(info, settings,
+						CXL_HDM_DECODER_MAX_COUNT),
+				    GFP_KERNEL);
+		if (!info)
+			return -ENOMEM;
+		allocated_info = true;
+	}
+
+	rc = cxl_find_regblock(pdev, CXL_REGLOC_RBI_COMPONENT, &map);
+	if (rc)
+		return rc;
+

drivers/cxl/core/reset.c int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm) · line 275

Quoted 10 lines
drivers/cxl/core/reset.c +10 -0
+	rc = cxl_setup_regs(&map);
+	if (rc)
+		return rc;
+
+	if (!map.component_map.hdm_decoder.valid) {
+		rc = -ENODEV;
+		return rc;
+	}
+
+	rc = cxl_map_component_regs(&map, &regs, BIT(CXL_CM_CAP_CAP_ID_HDM));

This uses devres internally so needs a flavor that does typical allocation. One way to differentiate would be to set "map.host = NULL" for this case.

Quoted 22 lines
drivers/cxl/core/reset.c +22 -0
+	if (rc)
+		return rc;
+
+	decoder_count = cxl_hdm_decoder_count(readl(regs.hdm_decoder +
+						    CXL_HDM_DECODER_CAP_OFFSET));
+	if (decoder_count < 0) {
+		rc = decoder_count;
+		goto out_unmap;
+	}
+
+	if (decoder_count > CXL_HDM_DECODER_MAX_COUNT) {
+		rc = -ENXIO;
+		goto out_unmap;
+	}
+
+	if (info->decoder_count && info->decoder_count != decoder_count) {
+		rc = -ENXIO;
+		goto out_unmap;
+	}
+
+	info->decoder_count = decoder_count;
+	info->regs = regs;

drivers/cxl/core/reset.c int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm) · line 307

Quoted 10 lines
drivers/cxl/core/reset.c +10 -0
+
+	settings = info->settings;
+	for (int i = 0; i < info->decoder_count; i++) {
+		rc = cxl_pci_hdm_read_decoder(pdev, &settings[i],
+					      regs.hdm_decoder, i);
+		if (rc)
+			goto out_unmap;
+	}
+
+	WRITE_ONCE(pdev->hdm, info);

The HDM info is accessed either by the reset path or the driver and should be protected by cxl_rwsem.dpa. So WRITE_ONCE() becomes unnecessary if info update happens under that synchronization.

#9 lore raw

PATCH v7 4/11 PCI: Export pci_dev_save_and_disable() and pci_dev_restore()

In reply to Srirangan Madhavan

Export the standard PCI reset save/disable and restore helpers so CXL reset can split that lifecycle around CXL-specific sequencing while preserving driver reset_prepare/reset_done callbacks and PCI config-state handling.

---
 drivers/pci/pci.c   | 23 +++++++++++++++++++++--
 include/linux/pci.h |  2 ++
 2 files changed, 23 insertions(+), 2 deletions(-)
drivers/pci/pci.c +21 -2
index d34266651ad0..360f2aaee10c 100644
@@ -5003,7 +5003,15 @@ void pci_dev_unlock(struct pci_dev *dev)
 }
 EXPORT_SYMBOL_GPL(pci_dev_unlock);
 
-static void pci_dev_save_and_disable(struct pci_dev *dev)
+/**
+ * pci_dev_save_and_disable - Save device state and disable it
+ * @dev: PCI device to save and disable
+ *
+ * Save the PCI configuration state, invoke the driver's reset_prepare()
+ * callback if present, and disable the device by clearing the Command
+ * register. The device lock must be held by the caller.
+ */
+void pci_dev_save_and_disable(struct pci_dev *dev)
 {
 	const struct pci_error_handlers *err_handler =
 			dev->driver ? dev->driver->err_handler : NULL;
@@ -5036,12 +5044,22 @@ static void pci_dev_save_and_disable(struct pci_dev *dev)
 	 */
 	pci_write_config_word(dev, PCI_COMMAND, PCI_COMMAND_INTX_DISABLE);
 }
+EXPORT_SYMBOL_GPL(pci_dev_save_and_disable);
 
-static void pci_dev_restore(struct pci_dev *dev)
+/**
+ * pci_dev_restore - Restore device state after reset
+ * @dev: PCI device to restore
+ *
+ * Restore the saved PCI configuration state and invoke the driver's
+ * reset_done() callback if present. The device lock must be held by the
+ * caller.
+ */
+void pci_dev_restore(struct pci_dev *dev)
 {
 	const struct pci_error_handlers *err_handler =
 			dev->driver ? dev->driver->err_handler : NULL;
 
+	device_lock_assert(&dev->dev);
 	pci_restore_state(dev);
 
 	/*
@@ -5054,6 +5072,7 @@ static void pci_dev_restore(struct pci_dev *dev)
 	else if (dev->driver)
 		pci_warn(dev, "reset done");
 }
+EXPORT_SYMBOL_GPL(pci_dev_restore);
 
 /* dev->reset_methods[] is a 0-terminated list of indices into this array */
 const struct pci_reset_fn_method pci_reset_fn_methods[] = {
include/linux/pci.h +2 -0
index 7db2daf8597c..4df030837a3a 100644
@@ -2018,6 +2018,8 @@ void pci_dev_lock(struct pci_dev *dev);
 int pci_dev_trylock(struct pci_dev *dev);
 void pci_dev_unlock(struct pci_dev *dev);
 DEFINE_GUARD(pci_dev, struct pci_dev *, pci_dev_lock(_T), pci_dev_unlock(_T))
+void pci_dev_save_and_disable(struct pci_dev *dev);
+void pci_dev_restore(struct pci_dev *dev);
 
 /*
  * PCI domain support.  Sometimes called PCI segment (eg by ACPI),
Signature · 1 line
2.43.0
#10 lore raw

Re: [PATCH v7 04/11] PCI: Export pci_dev_save_and_disable() and pci_dev_restore()

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 5 lines

Export the standard PCI reset save/disable and restore helpers so CXL reset can split that lifecycle around CXL-specific sequencing while preserving driver reset_prepare/reset_done callbacks and PCI config-state handling.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#11 lore raw

Re: [PATCH v7 04/11] PCI: Export pci_dev_save_and_disable() and pci_dev_restore()

In reply to Srirangan Madhavan
Quoted 3 lines · Srirangan Madhavan

Export the standard PCI reset save/disable and restore helpers so CXL reset can split that lifecycle around CXL-specific sequencing while preserving driver reset_prepare/reset_done callbacks and PCI config-state handling.

Part of the motivation for creating drivers/cxl/core/reset.c as a builtin was to avoid needing to export PCI core functions. So no need for these exports given no modular consumers.

#12 lore raw

PATCH v7 5/11 cxl: Add CXL Device Reset helper

In reply to Srirangan Madhavan

Add an internal CXL Device Reset helper for Type 2 functions that advertise CXL Reset in the CXL Device DVSEC. The helper disables CXL.cache, performs cache writeback when supported, initiates reset with Memory Clear disabled, waits for completion, and re-enables CXL.cache on exit.

Leave the helper unregistered until range validation and reset-scope coordination are in place.

---
 drivers/cxl/core/reset.c      | 221 ++++++++++++++++++++++++++++++++++
 include/cxl/cxl.h             |   7 ++
 include/uapi/linux/pci_regs.h |  14 +++
 3 files changed, 242 insertions(+)
drivers/cxl/core/reset.c +221 -0
index fc52d3abdb5b..fdfcc9e825e0 100644
@@ -7,6 +7,8 @@
 #include <linux/export.h>
 #include <linux/io.h>
 #include <linux/ioport.h>
+#include <linux/iommu.h>
+#include <linux/jiffies.h>
 #include <linux/kernel.h>
 #include <linux/pci.h>
 #include <linux/slab.h>
@@ -318,3 +320,222 @@ int pci_cxl_hdm_init(struct pci_dev *pdev)
 	cxl_pci_hdm_unmap(pdev, &regs, &map);
 	return rc;
 }
+
+/*
+ * CXL r4.0 sec 9.7.2 defines the reset completion timeout encodings.
+ * Sec 9.7.3 leaves config-space access behavior undefined for 100 ms after
+ * initiating CXL Reset, then limits software to CXL Status2 access until
+ * reset completion, timeout, or error.
+ */
+#define CXL_RESET_RRS_WAIT_MS 100
+#define CXL_RESET_STATUS_POLL_MS 20
+static const u32 cxl_reset_timeout_ms[] = {
+	10, 100, 1000, 10000, 100000,
+};
+
+#define CXL_CACHE_WBI_TIMEOUT_US 100000
+#define CXL_CACHE_WBI_POLL_US 100
+
+static int cxl_reset_dvsec(struct pci_dev *pdev)
+{
+	int dvsec, rc;
+	u16 cap;
+
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		return -ENOTTY;
+
+	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	if ((cap & (PCI_DVSEC_CXL_CACHE_CAPABLE |
+		    PCI_DVSEC_CXL_MEM_CAPABLE)) !=
+	    (PCI_DVSEC_CXL_CACHE_CAPABLE | PCI_DVSEC_CXL_MEM_CAPABLE))
+		return -ENOTTY;
+
+	if (!(cap & PCI_DVSEC_CXL_RST_CAPABLE))
+		return -ENOTTY;
+
+	return dvsec;
+}
+
+static int cxl_reset_update_ctrl2(struct pci_dev *pdev, int dvsec, u16 set,
+				  u16 clear)
+{
+	u16 ctrl2;
+	int rc;
+
+	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, &ctrl2);
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	ctrl2 |= set;
+	ctrl2 &= ~clear;
+
+	rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2);
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	return 0;
+}
+
+static int cxl_reset_enable_cache(struct pci_dev *pdev, int dvsec)
+{
+	return cxl_reset_update_ctrl2(pdev, dvsec, 0,
+				      PCI_DVSEC_CXL_DISABLE_CACHING);
+}
+
+static int cxl_reset_disable_cache(struct pci_dev *pdev, int dvsec, u16 cap)
+{
+	int remaining_us = CXL_CACHE_WBI_TIMEOUT_US;
+	u16 status2;
+	int rc, rc2;
+
+	rc = cxl_reset_update_ctrl2(pdev, dvsec,
+				    PCI_DVSEC_CXL_DISABLE_CACHING, 0);
+	if (rc)
+		return rc;
+
+	if (!(cap & PCI_DVSEC_CXL_CACHE_WBI_CAPABLE))
+		return 0;
+
+	rc = cxl_reset_update_ctrl2(pdev, dvsec,
+				    PCI_DVSEC_CXL_INIT_CACHE_WBI, 0);
+	if (rc)
+		goto err_enable_cache;
+
+	do {
+		usleep_range(CXL_CACHE_WBI_POLL_US, CXL_CACHE_WBI_POLL_US + 1);
+		remaining_us -= CXL_CACHE_WBI_POLL_US;
+
+		rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2,
+					  &status2);
+		if (rc) {
+			rc = pcibios_err_to_errno(rc);
+			goto err_enable_cache;
+		}
+	} while (!(status2 & PCI_DVSEC_CXL_CACHE_INV) && remaining_us > 0);
+
+	if (!(status2 & PCI_DVSEC_CXL_CACHE_INV)) {
+		rc = -ETIMEDOUT;
+		goto err_enable_cache;
+	}
+
+	return 0;
+
+err_enable_cache:
+	/*
+	 * DISABLE_CACHING can be rolled back here. INIT_CACHE_WBI is
+	 * self-clearing on completion, so leave any in-flight writeback alone.
+	 */
+	rc2 = cxl_reset_enable_cache(pdev, dvsec);
+	if (rc2)
+		pci_warn(pdev, "failed to re-enable CXL caching: %d\n", rc2);
+	return rc;
+}
+
+static int cxl_reset_wait_done(struct pci_dev *pdev, int dvsec, u16 cap)
+{
+	unsigned long deadline;
+	u32 timeout_ms;
+	u16 status2;
+	int idx, rc;
+
+	idx = FIELD_GET(PCI_DVSEC_CXL_RST_TIMEOUT, cap);
+	if (idx >= ARRAY_SIZE(cxl_reset_timeout_ms)) {
+		int last = ARRAY_SIZE(cxl_reset_timeout_ms) - 1;
+
+		pci_warn(pdev,
+			 "unknown CXL reset timeout encoding %d; using %u ms\n",
+			 idx, cxl_reset_timeout_ms[last]);
+		idx = last;
+	}
+
+	timeout_ms = max_t(u32, cxl_reset_timeout_ms[idx],
+			   CXL_RESET_RRS_WAIT_MS);
+	deadline = jiffies + msecs_to_jiffies(timeout_ms);
+	msleep(CXL_RESET_RRS_WAIT_MS);
+
+	do {
+		rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2,
+					  &status2);
+		if (rc)
+			return pcibios_err_to_errno(rc);
+
+		if (status2 & PCI_DVSEC_CXL_RST_ERR)
+			return -EIO;
+
+		if (status2 & PCI_DVSEC_CXL_RST_DONE)
+			return 0;
+
+		if (time_after_eq(jiffies, deadline))
+			return -ETIMEDOUT;
+
+		msleep(CXL_RESET_STATUS_POLL_MS);
+	} while (true);
+}
+
+static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)
+{
+	bool cache_disabled = false;
+	u16 cap;
+	int rc;
+
+	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	if (!pci_wait_for_pending_transaction(pdev))
+		pci_err(pdev, "timed out waiting for pending transactions\n");
+
+	rc = pci_dev_reset_iommu_prepare(pdev);
+	if (rc) {
+		pci_err(pdev, "failed to stop IOMMU for CXL reset: %d\n", rc);
+		return rc;
+	}
+
+	rc = cxl_reset_disable_cache(pdev, dvsec, cap);
+	if (rc)
+		goto out;
+	cache_disabled = true;
+
+	rc = cxl_reset_update_ctrl2(pdev, dvsec, PCI_DVSEC_CXL_INIT_CXL_RST,
+				    PCI_DVSEC_CXL_RST_MEM_CLR_EN);
+	if (rc)
+		goto out;
+
+	rc = cxl_reset_wait_done(pdev, dvsec, cap);
+	if (rc)
+		goto out;
+
+out:
+	if (cache_disabled) {
+		int rc2;
+
+		rc2 = cxl_reset_enable_cache(pdev, dvsec);
+		if (rc2 && rc)
+			pci_warn(pdev, "failed to re-enable CXL caching: %d\n",
+				 rc2);
+		else if (rc2)
+			rc = rc2;
+	}
+
+	pci_dev_reset_iommu_done(pdev);
+	return rc;
+}
+
+int cxl_reset_function(struct pci_dev *pdev, bool probe)
+{
+	int dvsec;
+
+	dvsec = cxl_reset_dvsec(pdev);
+	if (dvsec < 0)
+		return dvsec;
+
+	if (probe)
+		return 0;
+
+	return cxl_reset_execute(pdev, dvsec);
+}
include/cxl/cxl.h +7 -0
index e3087b7517e8..1fe606f15733 100644
@@ -9,6 +9,7 @@
 #include <linux/node.h>
 #include <linux/ioport.h>
 #include <linux/range.h>
+#include <linux/errno.h>
 #include <cxl/mailbox.h>
 
 /**
@@ -137,11 +138,17 @@ struct cxl_hdm_info {
 int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm);
 #ifdef CONFIG_CXL_HDM
 int pci_cxl_hdm_init(struct pci_dev *pdev);
+int cxl_reset_function(struct pci_dev *pdev, bool probe);
 #else
 static inline int pci_cxl_hdm_init(struct pci_dev *pdev)
 {
 	return -ENOTTY;
 }
+
+static inline int cxl_reset_function(struct pci_dev *pdev, bool probe)
+{
+	return -ENOTTY;
+}
 #endif
 
 struct cxl_reg_map {
include/uapi/linux/pci_regs.h +14 -0
index 14f634ab9350..194ae56b4404 100644
@@ -1349,10 +1349,24 @@
 /* CXL r4.0, 8.1.3: PCIe DVSEC for CXL Device */
 #define PCI_DVSEC_CXL_DEVICE				0
 #define  PCI_DVSEC_CXL_CAP				0xA
+#define   PCI_DVSEC_CXL_CACHE_CAPABLE			_BITUL(0)
 #define   PCI_DVSEC_CXL_MEM_CAPABLE			_BITUL(2)
 #define   PCI_DVSEC_CXL_HDM_COUNT			__GENMASK(5, 4)
+#define   PCI_DVSEC_CXL_CACHE_WBI_CAPABLE		_BITUL(6)
+#define   PCI_DVSEC_CXL_RST_CAPABLE			_BITUL(7)
+#define   PCI_DVSEC_CXL_RST_TIMEOUT			__GENMASK(10, 8)
+#define   PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE		_BITUL(11)
 #define  PCI_DVSEC_CXL_CTRL				0xC
 #define   PCI_DVSEC_CXL_MEM_ENABLE			_BITUL(2)
+#define  PCI_DVSEC_CXL_CTRL2				0x10
+#define   PCI_DVSEC_CXL_DISABLE_CACHING			_BITUL(0)
+#define   PCI_DVSEC_CXL_INIT_CACHE_WBI			_BITUL(1)
+#define   PCI_DVSEC_CXL_INIT_CXL_RST			_BITUL(2)
+#define   PCI_DVSEC_CXL_RST_MEM_CLR_EN			_BITUL(3)
+#define  PCI_DVSEC_CXL_STATUS2				0x12
+#define   PCI_DVSEC_CXL_CACHE_INV			_BITUL(0)
+#define   PCI_DVSEC_CXL_RST_DONE			_BITUL(1)
+#define   PCI_DVSEC_CXL_RST_ERR			_BITUL(2)
 #define  PCI_DVSEC_CXL_RANGE_SIZE_HIGH(i)		(0x18 + (i * 0x10))
 #define  PCI_DVSEC_CXL_RANGE_SIZE_LOW(i)		(0x1C + (i * 0x10))
 #define   PCI_DVSEC_CXL_MEM_INFO_VALID			_BITUL(0)
Signature · 1 line
2.43.0
#13 lore raw

Re: [PATCH v7 05/11] cxl: Add CXL Device Reset helper

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Read-modify-write of CXL_CTRL2 register accidentally re-triggers self-clearing commands on error…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Read-modify-write of CXL_CTRL2 register accidentally re-triggers self-clearing commands on error paths. --

commit 2cd253f3fe48a32fcac990882cdcc0aac716f026
Author: Srirangan Madhavan <[email protected]>

cxl: Add CXL Device Reset helper

This commit adds an internal helper for Type 2 CXL functions that support CXL Reset. The helper disables caching, initiates writeback and reset, waits for completion, and re-enables caching.

Quoted 4 lines
drivers/cxl/core/reset.c +0 -0
index fc52d3abdb5b4..fdfcc9e825e03 100644

[ ... ]

Quoted 4 lines
drivers/cxl/core/reset.c +4 -0
+static int cxl_reset_update_ctrl2(struct pci_dev *pdev, int dvsec, u16 set,
+				  u16 clear)
+{
+	u16 ctrl2;

drivers/cxl/core/reset.c int pci_cxl_hdm_init(struct pci_dev *pdev) · line 377

Quoted 10 lines
drivers/cxl/core/reset.c +10 -0
+	int rc;
+
+	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, &ctrl2);
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	ctrl2 |= set;
+	ctrl2 &= ~clear;
+
+	rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2);

[Severity: High]
If a timeout occurs in cxl_reset_disable_cache() while the PCI_DVSEC_CXL_INIT_CACHE_WBI command is still in flight, the error path calls cxl_reset_enable_cache() which relies on this read-modify-write operation.

Since the hardware operation timed out, the self-clearing command bit might still be read as 1. Does writing 1 back to this bit here unintentionally re-trigger the hung hardware operation?

Should self-clearing command bits like PCI_DVSEC_CXL_INIT_CACHE_WBI and PCI_DVSEC_CXL_INIT_CXL_RST be explicitly masked out from ctrl2 before the pci_write_config_word() call to prevent this?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#14 lore raw

Re: [PATCH v7 05/11] cxl: Add CXL Device Reset helper

In reply to Srirangan Madhavan
Quoted 169 lines · Srirangan Madhavan

Add an internal CXL Device Reset helper for Type 2 functions that advertise CXL Reset in the CXL Device DVSEC. The helper disables CXL.cache, performs cache writeback when supported, initiates reset with Memory Clear disabled, waits for completion, and re-enables CXL.cache on exit.

Leave the helper unregistered until range validation and reset-scope coordination are in place.

---
 drivers/cxl/core/reset.c      | 221 ++++++++++++++++++++++++++++++++++
 include/cxl/cxl.h             |   7 ++
 include/uapi/linux/pci_regs.h |  14 +++
 3 files changed, 242 insertions(+)
drivers/cxl/core/reset.c +139 -0
index fc52d3abdb5b..fdfcc9e825e0 100644
@@ -7,6 +7,8 @@
 #include <linux/export.h>
 #include <linux/io.h>
 #include <linux/ioport.h>
+#include <linux/iommu.h>
+#include <linux/jiffies.h>
 #include <linux/kernel.h>
 #include <linux/pci.h>
 #include <linux/slab.h>
@@ -318,3 +320,222 @@ int pci_cxl_hdm_init(struct pci_dev *pdev)
 	cxl_pci_hdm_unmap(pdev, &regs, &map);
 	return rc;
 }
+
+/*
+ * CXL r4.0 sec 9.7.2 defines the reset completion timeout encodings.
+ * Sec 9.7.3 leaves config-space access behavior undefined for 100 ms after
+ * initiating CXL Reset, then limits software to CXL Status2 access until
+ * reset completion, timeout, or error.
+ */
+#define CXL_RESET_RRS_WAIT_MS 100
+#define CXL_RESET_STATUS_POLL_MS 20
+static const u32 cxl_reset_timeout_ms[] = {
+	10, 100, 1000, 10000, 100000,
+};
+
+#define CXL_CACHE_WBI_TIMEOUT_US 100000
+#define CXL_CACHE_WBI_POLL_US 100
+
+static int cxl_reset_dvsec(struct pci_dev *pdev)
+{
+	int dvsec, rc;
+	u16 cap;
+
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		return -ENOTTY;
+
+	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	if ((cap & (PCI_DVSEC_CXL_CACHE_CAPABLE |
+		    PCI_DVSEC_CXL_MEM_CAPABLE)) !=
+	    (PCI_DVSEC_CXL_CACHE_CAPABLE | PCI_DVSEC_CXL_MEM_CAPABLE))
+		return -ENOTTY;
+
+	if (!(cap & PCI_DVSEC_CXL_RST_CAPABLE))
+		return -ENOTTY;
+
+	return dvsec;
+}
+
+static int cxl_reset_update_ctrl2(struct pci_dev *pdev, int dvsec, u16 set,
+				  u16 clear)
+{
+	u16 ctrl2;
+	int rc;
+
+	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, &ctrl2);
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	ctrl2 |= set;
+	ctrl2 &= ~clear;
+
+	rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2);
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	return 0;
+}
+
+static int cxl_reset_enable_cache(struct pci_dev *pdev, int dvsec)
+{
+	return cxl_reset_update_ctrl2(pdev, dvsec, 0,
+				      PCI_DVSEC_CXL_DISABLE_CACHING);
+}
+
+static int cxl_reset_disable_cache(struct pci_dev *pdev, int dvsec, u16 cap)
+{
+	int remaining_us = CXL_CACHE_WBI_TIMEOUT_US;
+	u16 status2;
+	int rc, rc2;
+
+	rc = cxl_reset_update_ctrl2(pdev, dvsec,
+				    PCI_DVSEC_CXL_DISABLE_CACHING, 0);
+	if (rc)
+		return rc;
+
+	if (!(cap & PCI_DVSEC_CXL_CACHE_WBI_CAPABLE))
+		return 0;
+
+	rc = cxl_reset_update_ctrl2(pdev, dvsec,
+				    PCI_DVSEC_CXL_INIT_CACHE_WBI, 0);
+	if (rc)
+		goto err_enable_cache;
+
+	do {
+		usleep_range(CXL_CACHE_WBI_POLL_US, CXL_CACHE_WBI_POLL_US + 1);
+		remaining_us -= CXL_CACHE_WBI_POLL_US;
+
+		rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2,
+					  &status2);
+		if (rc) {
+			rc = pcibios_err_to_errno(rc);
+			goto err_enable_cache;
+		}
+	} while (!(status2 & PCI_DVSEC_CXL_CACHE_INV) && remaining_us > 0);
+
+	if (!(status2 & PCI_DVSEC_CXL_CACHE_INV)) {
+		rc = -ETIMEDOUT;
+		goto err_enable_cache;
+	}
+
+	return 0;
+
+err_enable_cache:
+	/*
+	 * DISABLE_CACHING can be rolled back here. INIT_CACHE_WBI is
+	 * self-clearing on completion, so leave any in-flight writeback alone.
+	 */
+	rc2 = cxl_reset_enable_cache(pdev, dvsec);
+	if (rc2)
+		pci_warn(pdev, "failed to re-enable CXL caching: %d\n", rc2);
+	return rc;
+}
+
+static int cxl_reset_wait_done(struct pci_dev *pdev, int dvsec, u16 cap)
+{
+	unsigned long deadline;
+	u32 timeout_ms;
+	u16 status2;
+	int idx, rc;
+
+	idx = FIELD_GET(PCI_DVSEC_CXL_RST_TIMEOUT, cap);
+	if (idx >= ARRAY_SIZE(cxl_reset_timeout_ms)) {
+		int last = ARRAY_SIZE(cxl_reset_timeout_ms) - 1;
+
+		pci_warn(pdev,
+			 "unknown CXL reset timeout encoding %d; using %u ms\n",
+			 idx, cxl_reset_timeout_ms[last]);
+		idx = last;
+	}
+
+	timeout_ms = max_t(u32, cxl_reset_timeout_ms[idx],
+			   CXL_RESET_RRS_WAIT_MS);
+	deadline = jiffies + msecs_to_jiffies(timeout_ms);
+	msleep(CXL_RESET_RRS_WAIT_MS);

drivers/cxl/core/reset.c int pci_cxl_hdm_init(struct pci_dev *pdev) · line 469

Quoted 10 lines
drivers/cxl/core/reset.c +10 -0
+
+	do {
+		rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2,
+					  &status2);
+		if (rc)
+			return pcibios_err_to_errno(rc);
+
+		if (status2 & PCI_DVSEC_CXL_RST_ERR)
+			return -EIO;
+

I think this returns -EIO for a reset that actually succeeds.

During the post-reset RRS window the Status2 read comes back 0xffff. 0xffff has bit 2 set, so it's taken as RST_ERR and we bail on the very first poll. A transient config-read error is likewise treated as fatal instead of being retried.

I would suggest to skip all-ones reads as "not ready" and keep polling until the deadline, only act on RST_ERR and RST_DONE from a valid read.

Best regards,
Richard Cheng

Quoted 134 lines
drivers/cxl/core/reset.c +72 -0
+		if (status2 & PCI_DVSEC_CXL_RST_DONE)
+			return 0;
+
+		if (time_after_eq(jiffies, deadline))
+			return -ETIMEDOUT;
+
+		msleep(CXL_RESET_STATUS_POLL_MS);
+	} while (true);
+}
+
+static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)
+{
+	bool cache_disabled = false;
+	u16 cap;
+	int rc;
+
+	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);
+	if (rc)
+		return pcibios_err_to_errno(rc);
+
+	if (!pci_wait_for_pending_transaction(pdev))
+		pci_err(pdev, "timed out waiting for pending transactions\n");
+
+	rc = pci_dev_reset_iommu_prepare(pdev);
+	if (rc) {
+		pci_err(pdev, "failed to stop IOMMU for CXL reset: %d\n", rc);
+		return rc;
+	}
+
+	rc = cxl_reset_disable_cache(pdev, dvsec, cap);
+	if (rc)
+		goto out;
+	cache_disabled = true;
+
+	rc = cxl_reset_update_ctrl2(pdev, dvsec, PCI_DVSEC_CXL_INIT_CXL_RST,
+				    PCI_DVSEC_CXL_RST_MEM_CLR_EN);
+	if (rc)
+		goto out;
+
+	rc = cxl_reset_wait_done(pdev, dvsec, cap);
+	if (rc)
+		goto out;
+
+out:
+	if (cache_disabled) {
+		int rc2;
+
+		rc2 = cxl_reset_enable_cache(pdev, dvsec);
+		if (rc2 && rc)
+			pci_warn(pdev, "failed to re-enable CXL caching: %d\n",
+				 rc2);
+		else if (rc2)
+			rc = rc2;
+	}
+
+	pci_dev_reset_iommu_done(pdev);
+	return rc;
+}
+
+int cxl_reset_function(struct pci_dev *pdev, bool probe)
+{
+	int dvsec;
+
+	dvsec = cxl_reset_dvsec(pdev);
+	if (dvsec < 0)
+		return dvsec;
+
+	if (probe)
+		return 0;
+
+	return cxl_reset_execute(pdev, dvsec);
+}
include/cxl/cxl.h +7 -0
index e3087b7517e8..1fe606f15733 100644
@@ -9,6 +9,7 @@
 #include <linux/node.h>
 #include <linux/ioport.h>
 #include <linux/range.h>
+#include <linux/errno.h>
 #include <cxl/mailbox.h>
 
 /**
@@ -137,11 +138,17 @@ struct cxl_hdm_info {
 int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm);
 #ifdef CONFIG_CXL_HDM
 int pci_cxl_hdm_init(struct pci_dev *pdev);
+int cxl_reset_function(struct pci_dev *pdev, bool probe);
 #else
 static inline int pci_cxl_hdm_init(struct pci_dev *pdev)
 {
 	return -ENOTTY;
 }
+
+static inline int cxl_reset_function(struct pci_dev *pdev, bool probe)
+{
+	return -ENOTTY;
+}
 #endif
 
 struct cxl_reg_map {
include/uapi/linux/pci_regs.h +14 -0
index 14f634ab9350..194ae56b4404 100644
@@ -1349,10 +1349,24 @@
 /* CXL r4.0, 8.1.3: PCIe DVSEC for CXL Device */
 #define PCI_DVSEC_CXL_DEVICE				0
 #define  PCI_DVSEC_CXL_CAP				0xA
+#define   PCI_DVSEC_CXL_CACHE_CAPABLE			_BITUL(0)
 #define   PCI_DVSEC_CXL_MEM_CAPABLE			_BITUL(2)
 #define   PCI_DVSEC_CXL_HDM_COUNT			__GENMASK(5, 4)
+#define   PCI_DVSEC_CXL_CACHE_WBI_CAPABLE		_BITUL(6)
+#define   PCI_DVSEC_CXL_RST_CAPABLE			_BITUL(7)
+#define   PCI_DVSEC_CXL_RST_TIMEOUT			__GENMASK(10, 8)
+#define   PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE		_BITUL(11)
 #define  PCI_DVSEC_CXL_CTRL				0xC
 #define   PCI_DVSEC_CXL_MEM_ENABLE			_BITUL(2)
+#define  PCI_DVSEC_CXL_CTRL2				0x10
+#define   PCI_DVSEC_CXL_DISABLE_CACHING			_BITUL(0)
+#define   PCI_DVSEC_CXL_INIT_CACHE_WBI			_BITUL(1)
+#define   PCI_DVSEC_CXL_INIT_CXL_RST			_BITUL(2)
+#define   PCI_DVSEC_CXL_RST_MEM_CLR_EN			_BITUL(3)
+#define  PCI_DVSEC_CXL_STATUS2				0x12
+#define   PCI_DVSEC_CXL_CACHE_INV			_BITUL(0)
+#define   PCI_DVSEC_CXL_RST_DONE			_BITUL(1)
+#define   PCI_DVSEC_CXL_RST_ERR			_BITUL(2)
 #define  PCI_DVSEC_CXL_RANGE_SIZE_HIGH(i)		(0x18 + (i * 0x10))
 #define  PCI_DVSEC_CXL_RANGE_SIZE_LOW(i)		(0x1C + (i * 0x10))
 #define   PCI_DVSEC_CXL_MEM_INFO_VALID			_BITUL(0)
Signature · 1 line
2.43.0
#15 lore raw

PATCH v7 6/11 cxl: Validate HDM ranges before CXL reset

In reply to Srirangan Madhavan

Before reset, collect enabled cached HDM decoder ranges, reserve them with request_mem_region(), and invalidate CPU caches. This rejects reset while affected CXL memory is busy and keeps the validation stable through reset.

---
 drivers/cxl/core/reset.c | 239 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 238 insertions(+), 1 deletion(-)
drivers/cxl/core/reset.c +238 -1
index fdfcc9e825e0..786d1060e40d 100644
@@ -10,6 +10,8 @@
 #include <linux/iommu.h>
 #include <linux/jiffies.h>
 #include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/memregion.h>
 #include <linux/pci.h>
 #include <linux/slab.h>
 
@@ -336,6 +338,230 @@ static const u32 cxl_reset_timeout_ms[] = {
 #define CXL_CACHE_WBI_TIMEOUT_US 100000
 #define CXL_CACHE_WBI_POLL_US 100
 
+struct cxl_hdm_range {
+	struct list_head list;
+	struct pci_dev *pdev;
+	struct range hpa_range;
+	struct resource *res;
+};
+
+struct cxl_hdm_range_context {
+	struct list_head ranges;
+};
+
+static void cxl_hdm_range_context_init(struct cxl_hdm_range_context *ctx)
+{
+	INIT_LIST_HEAD(&ctx->ranges);
+}
+
+static void cxl_hdm_range_context_destroy(struct cxl_hdm_range_context *ctx)
+{
+	struct cxl_hdm_range *range, *next;
+
+	list_for_each_entry_safe(range, next, &ctx->ranges, list) {
+		list_del(&range->list);
+		if (range->res)
+			release_mem_region(range->hpa_range.start,
+					   resource_size(range->res));
+		kfree(range);
+	}
+}
+
+static int cxl_hdm_range_add(struct cxl_hdm_range_context *ctx,
+			     struct pci_dev *pdev, const struct range *hpa_range)
+{
+	struct cxl_hdm_range *range;
+
+	if (hpa_range->end < hpa_range->start)
+		return -EINVAL;
+
+	list_for_each_entry(range, &ctx->ranges, list)
+		if (range->hpa_range.start == hpa_range->start &&
+		    range->hpa_range.end == hpa_range->end)
+			return 0;
+
+	range = kzalloc_obj(*range);
+	if (!range)
+		return -ENOMEM;
+
+	range->pdev = pdev;
+	range->hpa_range = *hpa_range;
+	list_add_tail(&range->list, &ctx->ranges);
+
+	return 0;
+}
+
+static int cxl_hdm_ranges_collect(struct cxl_hdm_range_context *ctx,
+				  struct pci_dev *pdev)
+{
+	struct cxl_hdm_info *info = READ_ONCE(pdev->hdm);
+	int rc;
+
+	if (!info) {
+		pci_err(pdev, "CXL HDM decoder state unavailable\n");
+		return -ENXIO;
+	}
+
+	for (int i = 0; i < info->decoder_count; i++) {
+		struct cxl_decoder_settings *settings = &info->settings[i];
+
+		if (!(settings->flags & CXL_DECODER_F_ENABLE))
+			continue;
+
+		if (settings->flags & CXL_DECODER_F_NORMALIZED_ADDRESSING) {
+			pci_err(pdev,
+				"CXL reset does not support normalized address decoders\n");
+			return -EOPNOTSUPP;
+		}
+
+		rc = cxl_hdm_range_add(ctx, pdev, &settings->hpa_range);
+		if (rc)
+			return rc;
+	}
+
+	return 0;
+}
+
+static int cxl_hdm_range_len(struct pci_dev *pdev,
+			     const struct range *hpa_range, u64 *len)
+{
+	if (sizeof(resource_size_t) < sizeof(hpa_range->start) &&
+	    (hpa_range->start > (resource_size_t)~0ULL ||
+	     hpa_range->end > (resource_size_t)~0ULL)) {
+		pci_err(pdev,
+			"CXL reset range [%#llx-%#llx] exceeds resource address size\n",
+			hpa_range->start, hpa_range->end);
+		return -EOVERFLOW;
+	}
+
+	if (hpa_range->end < hpa_range->start)
+		return -EINVAL;
+
+	if (!hpa_range->start && hpa_range->end == U64_MAX) {
+		pci_err(pdev,
+			"CXL reset range [%#llx-%#llx] exceeds resource size\n",
+			hpa_range->start, hpa_range->end);
+		return -EOVERFLOW;
+	}
+
+	*len = range_len(hpa_range);
+	if (sizeof(resource_size_t) < sizeof(*len) &&
+	    *len > (resource_size_t)~0ULL) {
+		pci_err(pdev,
+			"CXL reset range [%#llx-%#llx] exceeds resource size\n",
+			hpa_range->start, hpa_range->end);
+		return -EOVERFLOW;
+	}
+
+	if (sizeof(size_t) < sizeof(*len) && *len > SIZE_MAX) {
+		pci_err(pdev,
+			"CXL reset range [%#llx-%#llx] exceeds cache flush size\n",
+			hpa_range->start, hpa_range->end);
+		return -EOVERFLOW;
+	}
+
+	return 0;
+}
+
+static int cxl_hdm_range_request(struct cxl_hdm_range *range)
+{
+	struct pci_dev *pdev = range->pdev;
+	const struct range *hpa_range = &range->hpa_range;
+	u64 len;
+	int rc;
+
+	rc = cxl_hdm_range_len(pdev, hpa_range, &len);
+	if (rc)
+		return rc;
+
+	range->res = request_mem_region(hpa_range->start, len, "cxl_reset");
+	if (!range->res) {
+		pci_err(pdev,
+			"cannot reset while CXL memory range is busy [%#llx-%#llx]\n",
+			hpa_range->start, hpa_range->end);
+		return -EBUSY;
+	}
+
+	return 0;
+}
+
+static int cxl_hdm_ranges_request(struct cxl_hdm_range_context *ctx)
+{
+	struct cxl_hdm_range *range;
+	int rc;
+
+	lockdep_assert_held_write(&cxl_rwsem.region);
+
+	list_for_each_entry(range, &ctx->ranges, list) {
+		rc = cxl_hdm_range_request(range);
+		if (rc)
+			return rc;
+	}
+
+	return 0;
+}
+
+static int cxl_hdm_range_flush_cache(struct cxl_hdm_range *range)
+{
+	struct pci_dev *pdev = range->pdev;
+	const struct range *hpa_range = &range->hpa_range;
+	u64 len;
+	int rc;
+
+	rc = cxl_hdm_range_len(pdev, hpa_range, &len);
+	if (rc)
+		return rc;
+
+	rc = cpu_cache_invalidate_memregion(hpa_range->start, len);
+	if (rc)
+		pci_err(pdev,
+			"failed to invalidate CPU cache [%#llx-%#llx]: %d\n",
+			hpa_range->start, hpa_range->end, rc);
+
+	return rc;
+}
+
+static int cxl_hdm_ranges_flush_cpu_caches(struct cxl_hdm_range_context *ctx,
+					   struct pci_dev *pdev)
+{
+	struct cxl_hdm_range *range;
+	int rc;
+
+	if (list_empty(&ctx->ranges))
+		return 0;
+
+	if (!cpu_cache_has_invalidate_memregion()) {
+		pci_err(pdev, "failed to synchronize CPU cache state\n");
+		return -ENXIO;
+	}
+
+	list_for_each_entry(range, &ctx->ranges, list) {
+		rc = cxl_hdm_range_flush_cache(range);
+		if (rc)
+			return rc;
+	}
+
+	return 0;
+}
+
+static int cxl_hdm_ranges_prepare(struct cxl_hdm_range_context *ctx,
+				  struct pci_dev *pdev)
+{
+	int rc;
+
+	lockdep_assert_held_write(&cxl_rwsem.region);
+
+	rc = cxl_hdm_ranges_collect(ctx, pdev);
+	if (rc)
+		return rc;
+
+	rc = cxl_hdm_ranges_request(ctx);
+	if (rc)
+		return rc;
+
+	return cxl_hdm_ranges_flush_cpu_caches(ctx, pdev);
+}
+
 static int cxl_reset_dvsec(struct pci_dev *pdev)
 {
 	int dvsec, rc;
@@ -528,7 +754,9 @@ static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)
 
 int cxl_reset_function(struct pci_dev *pdev, bool probe)
 {
+	struct cxl_hdm_range_context range_ctx;
 	int dvsec;
+	int rc;
 
 	dvsec = cxl_reset_dvsec(pdev);
 	if (dvsec < 0)
@@ -537,5 +765,14 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	if (probe)
 		return 0;
 
-	return cxl_reset_execute(pdev, dvsec);
+	cxl_hdm_range_context_init(&range_ctx);
+
+	scoped_guard(rwsem_write, &cxl_rwsem.region) {
+		rc = cxl_hdm_ranges_prepare(&range_ctx, pdev);
+		if (!rc)
+			rc = cxl_reset_execute(pdev, dvsec);
+	}
+
+	cxl_hdm_range_context_destroy(&range_ctx);
+	return rc;
 }
Signature · 1 line
2.43.0
#16 lore raw

Re: [PATCH v7 06/11] cxl: Validate HDM ranges before CXL reset

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 5 lines

Before reset, collect enabled cached HDM decoder ranges, reserve them with request_mem_region(), and invalidate CPU caches. This rejects reset while affected CXL memory is busy and keeps the validation stable through reset.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#17 lore raw

PATCH v7 7/11 PCI/cxl: Discover the CXL reset scope

In reply to Srirangan Madhavan

Add reset context support to discover same-scope CXL functions before reset. Use the Non-CXL Function Map, ARI/devfn rules, and CXL.cache/mem capability bits to identify participating siblings, then hold references until the reset context is destroyed.

If the Function Map cannot be read, warn and treat all candidate siblings as CXL functions.

---
 drivers/cxl/core/reset.c      | 180 +++++++++++++++++++++++++++++++++-
 include/uapi/linux/pci_regs.h |   1 +
 2 files changed, 180 insertions(+), 1 deletion(-)
drivers/cxl/core/reset.c +179 -1
index 786d1060e40d..1ae714a3595c 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-only
 /* Copyright(c) 2026 NVIDIA Corporation. All rights reserved. */
+#include <linux/bitmap.h>
 #include <linux/delay.h>
 #include <linux/bug.h>
 #include <linux/bitfield.h>
@@ -338,6 +339,25 @@ static const u32 cxl_reset_timeout_ms[] = {
 #define CXL_CACHE_WBI_TIMEOUT_US 100000
 #define CXL_CACHE_WBI_POLL_US 100
 
+/* CXL r4.0 sec 8.1.4 defines 256 bits of Non-CXL Function Map. */
+#define CXL_RESET_MAX_FUNCTIONS 256
+#define CXL_RESET_FUNCTION_MAP_REGS (CXL_RESET_MAX_FUNCTIONS / 32)
+#define CXL_RESET_SIBLINGS_INIT 8
+
+struct cxl_reset_context {
+	struct pci_dev *target;
+	struct pci_dev **siblings;
+	int nr_siblings;
+	int sibling_capacity;
+};
+
+struct cxl_reset_walk_context {
+	struct cxl_reset_context *ctx;
+	DECLARE_BITMAP(non_cxl_func_map, CXL_RESET_MAX_FUNCTIONS);
+	bool ari;
+	int rc;
+};
+
 struct cxl_hdm_range {
 	struct list_head list;
 	struct pci_dev *pdev;
@@ -349,6 +369,157 @@ struct cxl_hdm_range_context {
 	struct list_head ranges;
 };
 
+static void cxl_reset_context_init(struct cxl_reset_context *ctx,
+				   struct pci_dev *pdev)
+{
+	*ctx = (struct cxl_reset_context) {
+		.target = pdev,
+	};
+}
+
+static void cxl_reset_context_destroy(struct cxl_reset_context *ctx)
+{
+	for (int i = 0; i < ctx->nr_siblings; i++)
+		pci_dev_put(ctx->siblings[i]);
+	kfree(ctx->siblings);
+}
+
+static void cxl_reset_read_non_cxl_func_map(struct pci_dev *pdev,
+					    unsigned long *map)
+{
+	u32 words[CXL_RESET_FUNCTION_MAP_REGS];
+	int dvsec, reg;
+
+	bitmap_zero(map, CXL_RESET_MAX_FUNCTIONS);
+
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_FUNCTION_MAP);
+	if (!dvsec)
+		return;
+
+	for (reg = 0; reg < CXL_RESET_FUNCTION_MAP_REGS; reg++) {
+		int rc;
+
+		rc = pci_read_config_dword(pdev,
+					   dvsec + PCI_DVSEC_CXL_FUNCTION_MAP_REG +
+					   reg * sizeof(u32), &words[reg]);
+		if (rc) {
+			pci_warn(pdev,
+				 "failed to read Non-CXL Function Map; treating all siblings as CXL\n");
+			bitmap_zero(map, CXL_RESET_MAX_FUNCTIONS);
+			return;
+		}
+	}
+
+	bitmap_from_arr32(map, words, CXL_RESET_MAX_FUNCTIONS);
+}
+
+static int cxl_reset_func_map_bit(struct pci_dev *sibling, bool ari)
+{
+	if (ari)
+		return sibling->devfn;
+
+	/*
+	 * Without ARI, the Function Map is organized as 32 device slots per
+	 * conventional 3-bit function number.
+	 */
+	return PCI_FUNC(sibling->devfn) * 32 + PCI_SLOT(sibling->devfn);
+}
+
+static int cxl_reset_has_cache_or_mem(struct pci_dev *pdev)
+{
+	int dvsec, rc;
+	u16 cap;
+
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		return 0;
+
+	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);
+	if (rc) {
+		rc = pcibios_err_to_errno(rc);
+		pci_warn(pdev,
+			 "failed to read CXL capability; cannot determine reset scope: %d\n",
+			 rc);
+		return rc;
+	}
+
+	return !!(cap & (PCI_DVSEC_CXL_CACHE_CAPABLE |
+			 PCI_DVSEC_CXL_MEM_CAPABLE));
+}
+
+static int cxl_reset_add_sibling(struct cxl_reset_context *ctx,
+				 struct pci_dev *sibling)
+{
+	if (ctx->nr_siblings >= ctx->sibling_capacity) {
+		int capacity = ctx->sibling_capacity ?: CXL_RESET_SIBLINGS_INIT;
+		struct pci_dev **siblings;
+
+		if (capacity > INT_MAX / 2)
+			return -ENOMEM;
+		if (ctx->sibling_capacity)
+			capacity *= 2;
+
+		siblings = krealloc_array(ctx->siblings, capacity,
+					  sizeof(*siblings), GFP_KERNEL);
+		if (!siblings)
+			return -ENOMEM;
+
+		ctx->siblings = siblings;
+		ctx->sibling_capacity = capacity;
+	}
+
+	ctx->siblings[ctx->nr_siblings++] = pci_dev_get(sibling);
+	return 0;
+}
+
+static int cxl_reset_collect_sibling(struct pci_dev *sibling, void *data)
+{
+	struct cxl_reset_walk_context *wctx = data;
+	struct cxl_reset_context *ctx = wctx->ctx;
+	struct pci_dev *pdev = ctx->target;
+	int fn, rc;
+
+	if (sibling == pdev)
+		return 0;
+
+	if (sibling->bus != pdev->bus)
+		return 0;
+
+	if (!wctx->ari && PCI_SLOT(sibling->devfn) != PCI_SLOT(pdev->devfn))
+		return 0;
+
+	fn = cxl_reset_func_map_bit(sibling, wctx->ari);
+	if (test_bit(fn, wctx->non_cxl_func_map))
+		return 0;
+
+	rc = cxl_reset_has_cache_or_mem(sibling);
+	if (rc < 0) {
+		wctx->rc = rc;
+		return rc;
+	}
+	if (!rc)
+		return 0;
+
+	wctx->rc = cxl_reset_add_sibling(ctx, sibling);
+	return wctx->rc;
+}
+
+static int cxl_reset_collect_siblings(struct cxl_reset_context *ctx)
+{
+	struct pci_dev *pdev = ctx->target;
+	struct cxl_reset_walk_context wctx = {
+		.ctx = ctx,
+		.ari = pci_ari_enabled(pdev->bus),
+	};
+
+	cxl_reset_read_non_cxl_func_map(pdev, wctx.non_cxl_func_map);
+	pci_walk_bus(pdev->bus, cxl_reset_collect_sibling, &wctx);
+
+	return wctx.rc;
+}
+
 static void cxl_hdm_range_context_init(struct cxl_hdm_range_context *ctx)
 {
 	INIT_LIST_HEAD(&ctx->ranges);
@@ -755,6 +926,7 @@ static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)
 int cxl_reset_function(struct pci_dev *pdev, bool probe)
 {
 	struct cxl_hdm_range_context range_ctx;
+	struct cxl_reset_context ctx;
 	int dvsec;
 	int rc;
 
@@ -765,14 +937,20 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	if (probe)
 		return 0;
 
+	cxl_reset_context_init(&ctx, pdev);
 	cxl_hdm_range_context_init(&range_ctx);
 
+	rc = cxl_reset_collect_siblings(&ctx);
+	if (rc)
+		goto out;
+
 	scoped_guard(rwsem_write, &cxl_rwsem.region) {
 		rc = cxl_hdm_ranges_prepare(&range_ctx, pdev);
 		if (!rc)
 			rc = cxl_reset_execute(pdev, dvsec);
 	}
-
+out:
 	cxl_hdm_range_context_destroy(&range_ctx);
+	cxl_reset_context_destroy(&ctx);
 	return rc;
 }
include/uapi/linux/pci_regs.h +1 -0
index 194ae56b4404..7fc1d34fcce7 100644
@@ -1380,6 +1380,7 @@
 
 /* CXL r4.0, 8.1.4: Non-CXL Function Map DVSEC */
 #define PCI_DVSEC_CXL_FUNCTION_MAP			2
+#define  PCI_DVSEC_CXL_FUNCTION_MAP_REG			0x0C
 
 /* CXL r4.0, 8.1.5: Extensions DVSEC for Ports */
 #define PCI_DVSEC_CXL_PORT				3
Signature · 1 line
2.43.0
#18 lore raw

Re: [PATCH v7 07/11] PCI/cxl: Discover the CXL reset scope

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Mixing `goto`-based error handling with scope-based cleanup helpers (`scoped_guard`) in…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Mixing goto-based error handling with scope-based cleanup helpers (scoped_guard) in cxl_reset_function(), violating cleanup guidelines. --

commit 0809a08ef80af3b49b6dba78ed81070b5efc26c5
Author: Srirangan Madhavan <[email protected]>

PCI/cxl: Discover the CXL reset scope

This commit adds support to discover the CXL reset context for siblings sharing a reset scope. It uses the Non-CXL Function Map and device capabilities to identify participating siblings and holds references to them until the reset context is destroyed.

Quoted 4 lines
drivers/cxl/core/reset.c +0 -0
index 786d1060e40dd..1ae714a3595cb 100644

[ ... ]

drivers/cxl/core/reset.c int cxl_reset_function(struct pci_dev *pdev, bool probe) · line 947

Quoted 12 lines
drivers/cxl/core/reset.c +5 -0
@@ -765,14 +937,20 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	if (probe)
 		return 0;
 
+	cxl_reset_context_init(&ctx, pdev);
 	cxl_hdm_range_context_init(&range_ctx);
 
+	rc = cxl_reset_collect_siblings(&ctx);
+	if (rc)
+		goto out;
+
 	scoped_guard(rwsem_write, &cxl_rwsem.region) {

[Severity: Medium]
Does this mix goto-based error handling with scope-based cleanup helpers?

The cleanup subsystem guidelines state that the usage of goto and scope-based cleanup helpers should never be mixed in the same function. Since cxl_reset_function() already uses scoped_guard(), should the initializations and cleanups for ctx and range_ctx be converted to use scope-based cleanups (such as __free) instead of introducing a goto label?

Quoted 10 lines
drivers/cxl/core/reset.c +2 -1
 		rc = cxl_hdm_ranges_prepare(&range_ctx, pdev);
 		if (!rc)
 			rc = cxl_reset_execute(pdev, dvsec);
 	}
-
+out:
 	cxl_hdm_range_context_destroy(&range_ctx);
+	cxl_reset_context_destroy(&ctx);
 	return rc;
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
#19 lore raw

PATCH v7 8/11 cxl: Coordinate sibling functions for CXL reset

In reply to Srirangan Madhavan

CXL Device Reset affects all CXL.cache and CXL.mem functions in the reset scope. Lock same-scope siblings with pci_dev_trylock(), save/disable them, drain pending transactions, and hold IOMMU reset blocks until recovery.

Also include mem-capable siblings in HDM range validation and CPU cache invalidation. Cache-only siblings are quiesced, but skipped for HDM range handling.

---
 drivers/cxl/core/reset.c | 146 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 130 insertions(+), 16 deletions(-)
drivers/cxl/core/reset.c +130 -16
index 1ae714a3595c..69bcfab89858 100644
@@ -344,10 +344,17 @@ static const u32 cxl_reset_timeout_ms[] = {
 #define CXL_RESET_FUNCTION_MAP_REGS (CXL_RESET_MAX_FUNCTIONS / 32)
 #define CXL_RESET_SIBLINGS_INIT 8
 
+struct cxl_reset_sibling {
+	struct pci_dev *pdev;
+	bool has_mem;
+};
+
 struct cxl_reset_context {
 	struct pci_dev *target;
-	struct pci_dev **siblings;
+	struct cxl_reset_sibling *siblings;
 	int nr_siblings;
+	int nr_siblings_locked;
+	int nr_siblings_prepared;
 	int sibling_capacity;
 };
 
@@ -380,7 +387,7 @@ static void cxl_reset_context_init(struct cxl_reset_context *ctx,
 static void cxl_reset_context_destroy(struct cxl_reset_context *ctx)
 {
 	for (int i = 0; i < ctx->nr_siblings; i++)
-		pci_dev_put(ctx->siblings[i]);
+		pci_dev_put(ctx->siblings[i].pdev);
 	kfree(ctx->siblings);
 }
 
@@ -426,35 +433,49 @@ static int cxl_reset_func_map_bit(struct pci_dev *sibling, bool ari)
 	return PCI_FUNC(sibling->devfn) * 32 + PCI_SLOT(sibling->devfn);
 }
 
-static int cxl_reset_has_cache_or_mem(struct pci_dev *pdev)
+static int cxl_reset_read_cxl_cap(struct pci_dev *pdev, u16 *cap)
 {
 	int dvsec, rc;
-	u16 cap;
 
 	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
 					  PCI_DVSEC_CXL_DEVICE);
 	if (!dvsec)
-		return 0;
+		return -ENODEV;
 
-	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);
+	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, cap);
 	if (rc) {
 		rc = pcibios_err_to_errno(rc);
-		pci_warn(pdev,
-			 "failed to read CXL capability; cannot determine reset scope: %d\n",
-			 rc);
+		pci_warn(pdev, "failed to read CXL capability: %d\n", rc);
 		return rc;
 	}
 
+	return 0;
+}
+
+static int cxl_reset_has_cache_or_mem(struct pci_dev *pdev, bool *has_mem)
+{
+	u16 cap;
+	int rc;
+
+	*has_mem = false;
+
+	rc = cxl_reset_read_cxl_cap(pdev, &cap);
+	if (rc == -ENODEV)
+		return 0;
+	if (rc)
+		return rc;
+
+	*has_mem = cap & PCI_DVSEC_CXL_MEM_CAPABLE;
 	return !!(cap & (PCI_DVSEC_CXL_CACHE_CAPABLE |
 			 PCI_DVSEC_CXL_MEM_CAPABLE));
 }
 
 static int cxl_reset_add_sibling(struct cxl_reset_context *ctx,
-				 struct pci_dev *sibling)
+				 struct pci_dev *sibling, bool has_mem)
 {
 	if (ctx->nr_siblings >= ctx->sibling_capacity) {
 		int capacity = ctx->sibling_capacity ?: CXL_RESET_SIBLINGS_INIT;
-		struct pci_dev **siblings;
+		struct cxl_reset_sibling *siblings;
 
 		if (capacity > INT_MAX / 2)
 			return -ENOMEM;
@@ -470,7 +491,11 @@ static int cxl_reset_add_sibling(struct cxl_reset_context *ctx,
 		ctx->sibling_capacity = capacity;
 	}
 
-	ctx->siblings[ctx->nr_siblings++] = pci_dev_get(sibling);
+	ctx->siblings[ctx->nr_siblings] = (struct cxl_reset_sibling) {
+		.pdev = pci_dev_get(sibling),
+		.has_mem = has_mem,
+	};
+	ctx->nr_siblings++;
 	return 0;
 }
 
@@ -479,6 +504,7 @@ static int cxl_reset_collect_sibling(struct pci_dev *sibling, void *data)
 	struct cxl_reset_walk_context *wctx = data;
 	struct cxl_reset_context *ctx = wctx->ctx;
 	struct pci_dev *pdev = ctx->target;
+	bool has_mem;
 	int fn, rc;
 
 	if (sibling == pdev)
@@ -494,7 +520,7 @@ static int cxl_reset_collect_sibling(struct pci_dev *sibling, void *data)
 	if (test_bit(fn, wctx->non_cxl_func_map))
 		return 0;
 
-	rc = cxl_reset_has_cache_or_mem(sibling);
+	rc = cxl_reset_has_cache_or_mem(sibling, &has_mem);
 	if (rc < 0) {
 		wctx->rc = rc;
 		return rc;
@@ -502,7 +528,7 @@ static int cxl_reset_collect_sibling(struct pci_dev *sibling, void *data)
 	if (!rc)
 		return 0;
 
-	wctx->rc = cxl_reset_add_sibling(ctx, sibling);
+	wctx->rc = cxl_reset_add_sibling(ctx, sibling, has_mem);
 	return wctx->rc;
 }
 
@@ -520,6 +546,69 @@ static int cxl_reset_collect_siblings(struct cxl_reset_context *ctx)
 	return wctx.rc;
 }
 
+static void cxl_pci_functions_unlock(struct cxl_reset_context *ctx)
+{
+	while (ctx->nr_siblings_locked) {
+		struct pci_dev *sibling;
+
+		sibling = ctx->siblings[--ctx->nr_siblings_locked].pdev;
+		pci_dev_unlock(sibling);
+	}
+}
+
+static int cxl_pci_functions_lock(struct cxl_reset_context *ctx)
+{
+	for (int i = 0; i < ctx->nr_siblings; i++) {
+		struct pci_dev *sibling = ctx->siblings[i].pdev;
+
+		if (!pci_dev_trylock(sibling)) {
+			cxl_pci_functions_unlock(ctx);
+			return -EAGAIN;
+		}
+
+		ctx->nr_siblings_locked++;
+	}
+
+	return 0;
+}
+
+static void cxl_pci_functions_reset_done(struct cxl_reset_context *ctx)
+{
+	while (ctx->nr_siblings_prepared) {
+		struct pci_dev *sibling;
+
+		sibling = ctx->siblings[--ctx->nr_siblings_prepared].pdev;
+		pci_dev_reset_iommu_done(sibling);
+		pci_dev_restore(sibling);
+	}
+}
+
+static int cxl_pci_functions_reset_prepare(struct cxl_reset_context *ctx)
+{
+	for (int i = 0; i < ctx->nr_siblings_locked; i++) {
+		struct pci_dev *sibling = ctx->siblings[i].pdev;
+		int rc;
+
+		pci_dev_save_and_disable(sibling);
+		if (!pci_wait_for_pending_transaction(sibling))
+			pci_err(sibling,
+				"timed out waiting for pending transactions\n");
+
+		rc = pci_dev_reset_iommu_prepare(sibling);
+		if (rc) {
+			pci_err(sibling,
+				"failed to stop IOMMU for CXL reset: %d\n",
+				rc);
+			pci_dev_restore(sibling);
+			return rc;
+		}
+
+		ctx->nr_siblings_prepared++;
+	}
+
+	return 0;
+}
+
 static void cxl_hdm_range_context_init(struct cxl_hdm_range_context *ctx)
 {
 	INIT_LIST_HEAD(&ctx->ranges);
@@ -716,8 +805,9 @@ static int cxl_hdm_ranges_flush_cpu_caches(struct cxl_hdm_range_context *ctx,
 }
 
 static int cxl_hdm_ranges_prepare(struct cxl_hdm_range_context *ctx,
-				  struct pci_dev *pdev)
+				  struct cxl_reset_context *reset_ctx)
 {
+	struct pci_dev *pdev = reset_ctx->target;
 	int rc;
 
 	lockdep_assert_held_write(&cxl_rwsem.region);
@@ -726,6 +816,17 @@ static int cxl_hdm_ranges_prepare(struct cxl_hdm_range_context *ctx,
 	if (rc)
 		return rc;
 
+	for (int i = 0; i < reset_ctx->nr_siblings; i++) {
+		struct cxl_reset_sibling *sibling = &reset_ctx->siblings[i];
+
+		if (!sibling->has_mem)
+			continue;
+
+		rc = cxl_hdm_ranges_collect(ctx, sibling->pdev);
+		if (rc)
+			return rc;
+	}
+
 	rc = cxl_hdm_ranges_request(ctx);
 	if (rc)
 		return rc;
@@ -944,11 +1045,24 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	if (rc)
 		goto out;
 
+	rc = cxl_pci_functions_lock(&ctx);
+	if (rc)
+		goto out_unlock;
+
+	rc = cxl_pci_functions_reset_prepare(&ctx);
+	if (rc)
+		goto out_functions_done;
+
 	scoped_guard(rwsem_write, &cxl_rwsem.region) {
-		rc = cxl_hdm_ranges_prepare(&range_ctx, pdev);
+		rc = cxl_hdm_ranges_prepare(&range_ctx, &ctx);
 		if (!rc)
 			rc = cxl_reset_execute(pdev, dvsec);
 	}
+
+out_functions_done:
+	cxl_pci_functions_reset_done(&ctx);
+out_unlock:
+	cxl_pci_functions_unlock(&ctx);
 out:
 	cxl_hdm_range_context_destroy(&range_ctx);
 	cxl_reset_context_destroy(&ctx);
Signature · 1 line
2.43.0
#20 lore raw

Re: [PATCH v7 08/11] cxl: Coordinate sibling functions for CXL reset

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] AB-BA deadlock between `device_lock` and `pci_bus_sem` during CXL reset - [Low] Mixed usage of…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] AB-BA deadlock between device_lock and pci_bus_sem during CXL reset
- [Low] Mixed usage of goto-based error handling and scope-based cleanup helpers (scoped_guard) --

commit e204e091df9c0890781be70b3733c1aeae326d91
Author: Srirangan Madhavan <[email protected]>

cxl: Coordinate sibling functions for CXL reset

CXL Device Reset affects all CXL.cache and CXL.mem functions in the reset scope. Lock same-scope siblings with pci_dev_trylock(), save/disable them, drain pending transactions, and hold IOMMU reset blocks until recovery.

Quoted 4 lines
drivers/cxl/core/reset.c +0 -0
index 1ae714a3595cb..69bcfab898589 100644

[ ... ]

drivers/cxl/core/reset.c int cxl_reset_function(struct pci_dev *pdev, bool probe)

Quoted 1 line
drivers/cxl/core/reset.c +0 -0
@@ -944,11 +1045,24 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)

[Severity: High]
Could this lead to an AB-BA deadlock between the device lock and pci_bus_sem?

When the PCI core invokes a device-specific reset method, it holds the target device's lock (device_lock(&pdev->dev)). If cxl_reset_function() subsequently calls cxl_reset_collect_siblings() which uses pci_walk_bus(), that will attempt to acquire a read lock on pci_bus_sem.

The standard PCI lock acquisition order is pci_bus_sem followed by device_lock (for example, in pcie_do_recovery()). This inversion could create a deadlock if another thread (like hotplug) requests a write lock on pci_bus_sem, queuing the read lock request and blocking all involved threads.

Quoted 9 lines
drivers/cxl/core/reset.c +6 -0
 	if (rc)
 		goto out;
 
+	rc = cxl_pci_functions_lock(&ctx);
+	if (rc)
+		goto out_unlock;
+
+	rc = cxl_pci_functions_reset_prepare(&ctx);
+	if (rc)

drivers/cxl/core/reset.c int cxl_reset_function(struct pci_dev *pdev, bool probe) · line 1062

Quoted 10 lines
drivers/cxl/core/reset.c +5 -1
+		goto out_functions_done;
+
 	scoped_guard(rwsem_write, &cxl_rwsem.region) {
-		rc = cxl_hdm_ranges_prepare(&range_ctx, pdev);
+		rc = cxl_hdm_ranges_prepare(&range_ctx, &ctx);
 		if (!rc)
 			rc = cxl_reset_execute(pdev, dvsec);
 	}
+
+out_functions_done:

[Severity: Low]
Does this mix goto-based error handling with scope-based cleanup helpers?

According to the cleanup guidelines, functions should not mix goto-based cleanup labels (like out_functions_done) with scope-based helpers like scoped_guard(). All resources needing a cleanup should either be converted to scope-based cleanup, or none of them should use it.

Quoted 6 lines
drivers/cxl/core/reset.c +3 -0
+	cxl_pci_functions_reset_done(&ctx);
+out_unlock:
+	cxl_pci_functions_unlock(&ctx);
 out:
 	cxl_hdm_range_context_destroy(&range_ctx);
 	cxl_reset_context_destroy(&ctx);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
#21 lore raw

Re: [PATCH v7 08/11] cxl: Coordinate sibling functions for CXL reset

In reply to Srirangan Madhavan
Quoted 7 lines · Srirangan Madhavan

CXL Device Reset affects all CXL.cache and CXL.mem functions in the reset scope. Lock same-scope siblings with pci_dev_trylock(), save/disable them, drain pending transactions, and hold IOMMU reset blocks until recovery.

Also include mem-capable siblings in HDM range validation and CPU cache invalidation. Cache-only siblings are quiesced, but skipped for HDM range handling.

PCI reset locking and ordering is already a source of some burden without adding this new sibling model to consider.

Is there evidence that multi-function CXL devices, where most of the functions are non-CXL, is going to be a common occurrence?

In other words if CXL reset borrowed the bus reset locking model:

    if (pci_bus_trylock(bus)) {
        pci_bus_save_and_disable_locked(bus);
        might_sleep();
	rc = cxl_request_and_flush_hdm(bus);
	if (rc == 0) {
		rc = cxl_reset_execute(pdev);
		cxl_release_and_flush_hdm(bus);
	}
        pci_bus_restore_locked(bus);
        pci_bus_unlock(bus);
    }

The cost is disturbing some non-CXL functions, the benefit is reusing an existing reset order / locking model.

#22 lore raw

Re: [PATCH v7 08/11] cxl: Coordinate sibling functions for CXL reset

Quoted 29 lines · On Tue, 23 Jun 2026 16:00:23 -0700 "Dan Williams (nvidia)" <[email protected]>…
Quoted 7 lines · Srirangan Madhavan

CXL Device Reset affects all CXL.cache and CXL.mem functions in the reset scope. Lock same-scope siblings with pci_dev_trylock(), save/disable them, drain pending transactions, and hold IOMMU reset blocks until recovery.

Also include mem-capable siblings in HDM range validation and CPU cache invalidation. Cache-only siblings are quiesced, but skipped for HDM range handling.

PCI reset locking and ordering is already a source of some burden without adding this new sibling model to consider.

Is there evidence that multi-function CXL devices, where most of the functions are non-CXL, is going to be a common occurrence?

In other words if CXL reset borrowed the bus reset locking model:

    if (pci_bus_trylock(bus)) {
        pci_bus_save_and_disable_locked(bus);
        might_sleep();
	rc = cxl_request_and_flush_hdm(bus);
	if (rc == 0) {
		rc = cxl_reset_execute(pdev);
		cxl_release_and_flush_hdm(bus);
	}
        pci_bus_restore_locked(bus);
        pci_bus_unlock(bus);
    }
Quoted 2 lines

The cost is disturbing some non-CXL functions, the benefit is reusing an existing reset order / locking model.

I'd say further that this exceeds the boundaries of what pci_reset_function(), or the @reset sysfs attribute per pci_dev, is scoped to do. pci_reset_function() must limit the scope to the pci_dev (and in this case the CXL state associated with only that pci_dev). See for instance how bus and slot use cases through pci_reset_function() are limited to non-multifunction devices.

For multiple functions, the precedent is something more like pci_reset_bus(), where the caller is responsible for coordinating the set of affected devices. The locking is still complicated, but at least it's managed in vfio-pci-core, with a variant driver that actually owns the device, rather than pci-core.

Also note that there's currently no mechanism for performing a multi-function scoped reset through sysfs (excluding raw access to the parent bridge that bypasses all save/restore mechanics). I'd suggest that cxl_reset can only be available as a function scoped reset when only function 0 supports cxl.mem or cxl.cache, but that may also lead to the question of whether the reset sysfs attribute should be exposed at all if it only resets the cxl.io state, for example via FLR. Thanks,

Alex

#23 lore raw

PATCH v7 9/11 cxl: Restore CXL HDM state after PCI reset

In reply to Srirangan Madhavan

After CXL reset, restore PCI config state enough to reach HDM MMIO, restore cached global and per-decoder HDM state, and then run the normal PCI restore callbacks.

Keep target and sibling IOMMU reset blocks active until HDM restore completes so Bus Master Enable cannot reopen DMA before decoder state is valid.

---
 drivers/cxl/core/hdm.c   |   4 +
 drivers/cxl/core/reset.c | 195 ++++++++++++++++++++++++++++++++++++---
 include/cxl/cxl.h        |   2 +
 3 files changed, 190 insertions(+), 11 deletions(-)
drivers/cxl/core/hdm.c +4 -0
index 0230ebfada42..095cc13e5d00 100644
@@ -152,6 +152,10 @@ static void cxl_hdm_info_set_decoder(struct cxl_hdm *cxlhdm,
 	if (!info || cxld->id >= info->decoder_count)
 		return;
 
+	if (cxlhdm->regs.hdm_decoder)
+		info->global_ctrl = readl(cxlhdm->regs.hdm_decoder +
+					  CXL_HDM_DECODER_CTRL_OFFSET);
+
 	if (cxld->flags & CXL_DECODER_F_ENABLE)
 		info->settings[cxld->id] = cxld->settings;
 	else
drivers/cxl/core/reset.c +184 -11

257 lines not on this page · raw message

include/cxl/cxl.h +2 -0
index 1fe606f15733..eddc48f1fa49 100644
@@ -127,11 +127,13 @@ struct cxl_regs {
  * struct cxl_hdm_info - PCI device HDM decoder programming cache
  * @decoder_count: number of decoder settings entries
  * @regs: mapped CXL component registers for this HDM decoder block
+ * @global_ctrl: cached HDM decoder global control register
  * @settings: cached per-decoder programming state
  */
 struct cxl_hdm_info {
 	int decoder_count;
 	struct cxl_component_regs regs;
+	u32 global_ctrl;
 	struct cxl_decoder_settings settings[] __counted_by(decoder_count);
 };
 
Signature · 1 line
2.43.0
#24 lore raw

Re: [PATCH v7 09/11] cxl: Restore CXL HDM state after PCI reset

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] The function `cxl_reset_function` mixes `goto`-based error handling with scope-based cleanup…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The function cxl_reset_function mixes goto-based error handling with scope-based cleanup (scoped_guard). --

commit 632a408ad919cb36ef2b58d6bfde50d19355ae4d
Author: Srirangan Madhavan <[email protected]>

cxl: Restore CXL HDM state after PCI reset

After CXL reset, restore PCI config state enough to reach HDM MMIO, restore cached global and per-decoder HDM state, and then run the normal PCI restore callbacks.

Quoted 4 lines
drivers/cxl/core/reset.c +0 -0
index 69bcfab898589..d801c91a5cbf8 100644

[ ... ]

Quoted 10 lines
drivers/cxl/core/reset.c +4 -0
@@ -1053,12 +1219,19 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	if (rc)
 		goto out_functions_done;
 
+	rc = cxl_pci_target_reset_prepare(&ctx);
+	if (rc)
+		goto out_functions_done;
+
 	scoped_guard(rwsem_write, &cxl_rwsem.region) {
 		rc = cxl_hdm_ranges_prepare(&range_ctx, &ctx);

drivers/cxl/core/reset.c int cxl_reset_function(struct pci_dev *pdev, bool probe) · line 1237

Quoted 10 lines
drivers/cxl/core/reset.c +3 -0
 		if (!rc)
 			rc = cxl_reset_execute(pdev, dvsec);
+		if (!rc)
+			rc = cxl_restore_hdm_decoders(&ctx);
 	}
 
+	cxl_pci_target_reset_done(&ctx);
 out_functions_done:
 	cxl_pci_functions_reset_done(&ctx);
 out_unlock:

[Severity: Low]
Does this code violate the guideline against mixing goto-based error handling with scope-based cleanups in the same function?

The cleanup documentation specifies that for a given routine, either all resources that need a goto cleanup should be converted to scope-based cleanup, or none of them. Could cxl_reset_function() in drivers/cxl/core/reset.c be refactored to use a consistent error handling approach rather than mixing goto labels with scoped_guard blocks?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
#25 lore raw

Re: [PATCH v7 09/11] cxl: Restore CXL HDM state after PCI reset

In reply to Srirangan Madhavan
Quoted 286 lines · Srirangan Madhavan

After CXL reset, restore PCI config state enough to reach HDM MMIO, restore cached global and per-decoder HDM state, and then run the normal PCI restore callbacks.

Keep target and sibling IOMMU reset blocks active until HDM restore completes so Bus Master Enable cannot reopen DMA before decoder state is valid.

---
 drivers/cxl/core/hdm.c   |   4 +
 drivers/cxl/core/reset.c | 195 ++++++++++++++++++++++++++++++++++++---
 include/cxl/cxl.h        |   2 +
 3 files changed, 190 insertions(+), 11 deletions(-)
drivers/cxl/core/hdm.c +4 -0
index 0230ebfada42..095cc13e5d00 100644
@@ -152,6 +152,10 @@ static void cxl_hdm_info_set_decoder(struct cxl_hdm *cxlhdm,
 	if (!info || cxld->id >= info->decoder_count)
 		return;
 
+	if (cxlhdm->regs.hdm_decoder)
+		info->global_ctrl = readl(cxlhdm->regs.hdm_decoder +
+					  CXL_HDM_DECODER_CTRL_OFFSET);
+
 	if (cxld->flags & CXL_DECODER_F_ENABLE)
 		info->settings[cxld->id] = cxld->settings;
 	else
drivers/cxl/core/reset.c +179 -11
index 69bcfab89858..d801c91a5cbf 100644
@@ -83,6 +83,21 @@ static int cxld_await_commit(void __iomem *hdm, int id)
 	return -ETIMEDOUT;
 }
 
+static int cxld_await_uncommit(void __iomem *hdm, int id)
+{
+	u32 ctrl;
+	int i;
+
+	for (i = 0; i < COMMIT_TIMEOUT_MS; i++) {
+		ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(id));
+		if (!FIELD_GET(CXL_HDM_DECODER0_CTRL_COMMITTED, ctrl))
+			return 0;
+		fsleep(1000);
+	}
+
+	return -ETIMEDOUT;
+}
+
 static void setup_hw_decoder(struct cxl_decoder_settings *settings,
 			     void __iomem *hdm)
 {
@@ -92,6 +107,8 @@ static void setup_hw_decoder(struct cxl_decoder_settings *settings,
 	u32 ctrl;
 
 	ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(id));
+	ctrl &= ~(CXL_HDM_DECODER0_CTRL_COMMIT |
+		  CXL_HDM_DECODER0_CTRL_COMMIT_ERROR);
 	cxld_set_interleave(settings, &ctrl);
 	cxld_set_type(settings, &ctrl);
 	base = settings->hpa_range.start;
@@ -300,6 +317,8 @@ int pci_cxl_hdm_init(struct pci_dev *pdev)
 
 	info->decoder_count = decoder_count;
 	info->regs = regs;
+	info->global_ctrl = readl(regs.hdm_decoder +
+				  CXL_HDM_DECODER_CTRL_OFFSET);
 
 	settings = info->settings;
 	for (int i = 0; i < info->decoder_count; i++) {
@@ -324,6 +343,100 @@ int pci_cxl_hdm_init(struct pci_dev *pdev)
 	return rc;
 }
 
+static int cxl_hdm_decoder_uncommit(struct pci_dev *pdev, void __iomem *hdm,
+				    int id, bool *locked_committed)
+{
+	u32 ctrl;
+	int rc;
+
+	*locked_committed = false;
+	ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(id));
+	if (ctrl & CXL_HDM_DECODER0_CTRL_LOCK) {
+		if (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED) {
+			pci_dbg(pdev,
+				"CXL HDM decoder %d retained locked committed state\n",
+				id);
+			*locked_committed = true;
+			return 0;
+		}
+
+		pci_err(pdev, "CXL HDM decoder %d is locked\n", id);
+		return -EBUSY;
+	}
+
+	if (!(ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED))
+		return 0;
+
+	ctrl &= ~CXL_HDM_DECODER0_CTRL_COMMIT;
+	writel(ctrl, hdm + CXL_HDM_DECODER0_CTRL_OFFSET(id));
+
+	rc = cxld_await_uncommit(hdm, id);
+	if (rc)
+		pci_err(pdev, "CXL HDM decoder %d uncommit failed: %d\n",
+			id, rc);
+
+	return rc;
+}
+
+static int cxl_restore_hdm_decoder(struct pci_dev *pdev,
+				   struct cxl_decoder_settings *settings,
+				   void __iomem *hdm)
+{
+	bool locked_committed;
+	int rc;
+
+	if (!(settings->flags & CXL_DECODER_F_ENABLE))
+		return 0;
+
+	rc = cxl_hdm_decoder_uncommit(pdev, hdm, settings->id,
+				      &locked_committed);
+	if (rc)
+		return rc;
+	if (locked_committed)
+		return 0;
+
+	rc = cxl_commit(settings, hdm);
+	if (rc)
+		pci_err(pdev, "CXL HDM decoder %d restore failed: %d\n",
+			settings->id, rc);
+
+	return rc;
+}
+
+static int cxl_restore_hdm(struct pci_dev *pdev)
+{
+	struct cxl_hdm_info *info = READ_ONCE(pdev->hdm);
+	void __iomem *hdm;
+	int first_rc = 0;
+
+	if (!info)
+		return 0;
+
+	hdm = info->regs.hdm_decoder;
+	if (!hdm) {
+		pci_err(pdev, "CXL HDM decoder registers unavailable\n");
+		return -ENXIO;
+	}
+
+	/*
+	 * Restore global HDM control before per-decoder commit. PCI config
+	 * state has been restored for MMIO access, but IOMMU reset blocks
+	 * remain active until HDM restore completes.
+	 */
+	writel(info->global_ctrl, hdm + CXL_HDM_DECODER_CTRL_OFFSET);
+
+	for (int i = 0; i < info->decoder_count; i++) {
+		struct cxl_decoder_settings *settings = &info->settings[i];
+		int rc;
+
+		rc = cxl_restore_hdm_decoder(pdev, settings, hdm);
+		if (rc && !first_rc)
+			first_rc = rc;
+	}
+
+	return first_rc;
+}
+
 /*
  * CXL r4.0 sec 9.7.2 defines the reset completion timeout encodings.
  * Sec 9.7.3 leaves config-space access behavior undefined for 100 ms after
@@ -355,6 +468,7 @@ struct cxl_reset_context {
 	int nr_siblings;
 	int nr_siblings_locked;
 	int nr_siblings_prepared;
+	bool target_prepared;
 	int sibling_capacity;
 };
 
@@ -609,6 +723,68 @@ static int cxl_pci_functions_reset_prepare(struct cxl_reset_context *ctx)
 	return 0;
 }
 
+static void cxl_pci_target_reset_done(struct cxl_reset_context *ctx)
+{
+	if (!ctx->target_prepared)
+		return;
+
+	pci_dev_reset_iommu_done(ctx->target);
+	ctx->target_prepared = false;
+}
+
+static int cxl_pci_target_reset_prepare(struct cxl_reset_context *ctx)
+{
+	struct pci_dev *pdev = ctx->target;
+	int rc;
+
+	if (!pci_wait_for_pending_transaction(pdev))
+		pci_err(pdev, "timed out waiting for pending transactions\n");
+
+	rc = pci_dev_reset_iommu_prepare(pdev);
+	if (rc) {
+		pci_err(pdev, "failed to stop IOMMU for CXL reset: %d\n", rc);
+		return rc;
+	}
+
+	ctx->target_prepared = true;
+	return 0;
+}
+
+static void cxl_pci_functions_restore_state(struct cxl_reset_context *ctx)
+{
+	/*
+	 * Restore PCI config state first so HDM MMIO is reachable. The final
+	 * pci_dev_restore() pass deliberately replays pci_restore_state()
+	 * before invoking driver reset_done() callbacks.
+	 */
+	pci_restore_state(ctx->target);
+
+	for (int i = 0; i < ctx->nr_siblings_prepared; i++)
+		pci_restore_state(ctx->siblings[i].pdev);
+}
+
+static int cxl_restore_hdm_decoders(struct cxl_reset_context *ctx)
+{
+	int first_rc = 0;
+	int rc;
+
+	cxl_pci_functions_restore_state(ctx);
+
+	rc = cxl_restore_hdm(ctx->target);
+	if (rc && !first_rc)
+		first_rc = rc;
+
+	for (int i = 0; i < ctx->nr_siblings_prepared; i++) {
+		struct pci_dev *sibling = ctx->siblings[i].pdev;
+
+		rc = cxl_restore_hdm(sibling);
+		if (rc && !first_rc)
+			first_rc = rc;
+	}
+
+	return first_rc;
+}
+
 static void cxl_hdm_range_context_init(struct cxl_hdm_range_context *ctx)
 {
 	INIT_LIST_HEAD(&ctx->ranges);
@@ -985,18 +1161,9 @@ static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)
 	if (rc)
 		return pcibios_err_to_errno(rc);
 
-	if (!pci_wait_for_pending_transaction(pdev))
-		pci_err(pdev, "timed out waiting for pending transactions\n");
-
-	rc = pci_dev_reset_iommu_prepare(pdev);
-	if (rc) {
-		pci_err(pdev, "failed to stop IOMMU for CXL reset: %d\n", rc);
-		return rc;
-	}
-
 	rc = cxl_reset_disable_cache(pdev, dvsec, cap);
 	if (rc)
-		goto out;
+		return rc;
 	cache_disabled = true;
 
 	rc = cxl_reset_update_ctrl2(pdev, dvsec, PCI_DVSEC_CXL_INIT_CXL_RST,
@@ -1020,7 +1187,6 @@ static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)
 			rc = rc2;
 	}
 
-	pci_dev_reset_iommu_done(pdev);
 	return rc;
 }
 
@@ -1053,12 +1219,19 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	if (rc)
 		goto out_functions_done;
 
+	rc = cxl_pci_target_reset_prepare(&ctx);
+	if (rc)

drivers/cxl/core/reset.c int cxl_reset_function(struct pci_dev *pdev, bool probe) · line 1233

Quoted 10 lines
drivers/cxl/core/reset.c +4 -0
+		goto out_functions_done;
+
 	scoped_guard(rwsem_write, &cxl_rwsem.region) {
 		rc = cxl_hdm_ranges_prepare(&range_ctx, &ctx);
 		if (!rc)
 			rc = cxl_reset_execute(pdev, dvsec);
+		if (!rc)
+			rc = cxl_restore_hdm_decoders(&ctx);
 	}
 

HDM restore only runs when cxl_reset_execute() returns 0. Combine with the wait_done false -EIO at patch 5, the reset succeeds in HW but execute returns -EIO, so restore is skipped entirely.

Maybe we should detect an unreachable device (all-ones config) and chain a link-level recovery instead of leaving it wedged.

--Richard

Quoted 25 lines
drivers/cxl/core/reset.c +1 -0
+	cxl_pci_target_reset_done(&ctx);
 out_functions_done:
 	cxl_pci_functions_reset_done(&ctx);
 out_unlock:
include/cxl/cxl.h +2 -0
index 1fe606f15733..eddc48f1fa49 100644
@@ -127,11 +127,13 @@ struct cxl_regs {
  * struct cxl_hdm_info - PCI device HDM decoder programming cache
  * @decoder_count: number of decoder settings entries
  * @regs: mapped CXL component registers for this HDM decoder block
+ * @global_ctrl: cached HDM decoder global control register
  * @settings: cached per-decoder programming state
  */
 struct cxl_hdm_info {
 	int decoder_count;
 	struct cxl_component_regs regs;
+	u32 global_ctrl;
 	struct cxl_decoder_settings settings[] __counted_by(decoder_count);
 };
 
Signature · 1 line
2.43.0
#26 lore raw

PATCH v7 10/11 PCI/cxl: Expose CXL Reset as a PCI reset method

In reply to Srirangan Madhavan

Add the CXL Reset helper to the PCI reset-method table so userspace can select it through the existing reset_method ABI.

Advertise the method for Type 2 CXL devices that report CXL Reset support in the CXL Device DVSEC. Reset execution still requires cached HDM decoder state for the target and mem-capable siblings so that affected ranges can be validated and HDM programming can be restored. If that state is unavailable at reset time, return -ENOTTY so PCI can try the next reset method.

---
 drivers/cxl/core/reset.c | 33 +++++++++++++++++++++++++++++++++
 drivers/pci/pci.c        |  2 ++
 include/linux/pci.h      |  2 +-
 3 files changed, 36 insertions(+), 1 deletion(-)
drivers/cxl/core/reset.c +33 -0
index d801c91a5cbf..694d8a3789a4 100644
@@ -1035,6 +1035,34 @@ static int cxl_reset_dvsec(struct pci_dev *pdev)
 	return dvsec;
 }
 
+static bool cxl_reset_hdm_available(struct pci_dev *pdev)
+{
+	struct cxl_hdm_info *info = READ_ONCE(pdev->hdm);
+
+	/*
+	 * pdev->hdm is allocated with PCI-device devres. Reset requests
+	 * operate on a live pci_dev, so the devres allocation remains valid
+	 * for this check.
+	 */
+	return info && info->regs.hdm_decoder;
+}
+
+static bool cxl_reset_scope_hdm_available(struct cxl_reset_context *ctx)
+{
+	if (!cxl_reset_hdm_available(ctx->target))
+		return false;
+
+	for (int i = 0; i < ctx->nr_siblings; i++) {
+		struct cxl_reset_sibling *sibling = &ctx->siblings[i];
+
+		if (sibling->has_mem &&
+		    !cxl_reset_hdm_available(sibling->pdev))
+			return false;
+	}
+
+	return true;
+}
+
 static int cxl_reset_update_ctrl2(struct pci_dev *pdev, int dvsec, u16 set,
 				  u16 clear)
 {
@@ -1211,6 +1239,11 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	if (rc)
 		goto out;
 
+	if (!cxl_reset_scope_hdm_available(&ctx)) {
+		rc = -ENOTTY;
+		goto out;
+	}
+
 	rc = cxl_pci_functions_lock(&ctx);
 	if (rc)
 		goto out_unlock;
drivers/pci/pci.c +2 -0
index 360f2aaee10c..b1ec20126390 100644
@@ -33,6 +33,7 @@
 #include <asm/dma.h>
 #include <linux/aer.h>
 #include <linux/bitfield.h>
+#include <cxl/cxl.h>
 #include "pci.h"
 
 DEFINE_MUTEX(pci_slot_mutex);
@@ -5081,6 +5082,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = {
 	{ pci_dev_acpi_reset, .name = "acpi" },
 	{ pcie_reset_flr, .name = "flr" },
 	{ pci_af_flr, .name = "af_flr" },
+	{ cxl_reset_function, .name = "cxl_reset" },
 	{ pci_pm_reset, .name = "pm" },
 	{ pci_reset_bus_function, .name = "bus" },
 	{ cxl_reset_bus_function, .name = "cxl_bus" },
include/linux/pci.h +1 -1
index 4df030837a3a..05b5feac5a49 100644
@@ -51,7 +51,7 @@
 			       PCI_STATUS_PARITY)
 
 /* Number of reset methods used in pci_reset_fn_methods array in pci.c */
-#define PCI_NUM_RESET_METHODS 8
+#define PCI_NUM_RESET_METHODS 9
 
 #define PCI_RESET_PROBE		true
 #define PCI_RESET_DO_RESET	false
Signature · 1 line
2.43.0
#27 lore raw

Re: [PATCH v7 10/11] PCI/cxl: Expose CXL Reset as a PCI reset method

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Accessing devres-managed `pdev->hdm` before holding sibling device locks leads to a…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Accessing devres-managed pdev->hdm before holding sibling device locks leads to a Time-of-Check-to-Time-of-Use (TOCTOU) Use-After-Free during concurrent driver unbind or hot-unplug. --

commit c1746289413d4e364b4a471be081d1821258af44
Author: Srirangan Madhavan <[email protected]>

PCI/cxl: Expose CXL Reset as a PCI reset method

This commit adds the CXL Reset helper to the PCI reset-method table, allowing userspace to select it via the reset_method ABI. It introduces checks to verify that cached HDM decoder state is available for the target and mem-capable siblings before executing the reset.

Quoted 8 lines
drivers/cxl/core/reset.c +0 -0
index d801c91a5cbf8..694d8a3789a42 100644
@@ -1035,6 +1035,34 @@ static int cxl_reset_dvsec(struct pci_dev *pdev)
 	return dvsec;
 }
 

drivers/cxl/core/reset.c static int cxl_reset_dvsec(struct pci_dev *pdev) · line 1047

Quoted 10 lines
drivers/cxl/core/reset.c +10 -0
+static bool cxl_reset_hdm_available(struct pci_dev *pdev)
+{
+	struct cxl_hdm_info *info = READ_ONCE(pdev->hdm);
+
+	/*
+	 * pdev->hdm is allocated with PCI-device devres. Reset requests
+	 * operate on a live pci_dev, so the devres allocation remains valid
+	 * for this check.
+	 */
+	return info && info->regs.hdm_decoder;

[Severity: High]
Does this assumption hold true during a concurrent driver unbind or hot-unplug event?

Devres is tied to the driver binding lifecycle, which is protected by the device lock, not just a reference to the live pci_dev.

If a sibling device's driver is unbound concurrently, devres_release_all() will free the info structure. Since READ_ONCE(pdev->hdm) could read a valid pointer right before it is freed, couldn't the subsequent access to info->regs.hdm_decoder result in a use-after-free?

[ ... ]

drivers/cxl/core/reset.c int cxl_reset_function(struct pci_dev *pdev, bool probe) · line 1246

Quoted 9 lines
drivers/cxl/core/reset.c +5 -0
@@ -1211,6 +1239,11 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	if (rc)
 		goto out;
 
+	if (!cxl_reset_scope_hdm_available(&ctx)) {
+		rc = -ENOTTY;
+		goto out;
+	}
+

[Severity: High]
Is it safe to call cxl_reset_scope_hdm_available() before holding the sibling device locks?

Because the check iterates through siblings and accesses their devres-managed pdev->hdm pointers without the device locks held, there appears to be a race condition against concurrent unbinds.

Could moving cxl_reset_scope_hdm_available() to after cxl_pci_functions_lock() protect the devres structures from being freed during this check?

Quoted 3 lines
drivers/cxl/core/reset.c +0 -0
 	rc = cxl_pci_functions_lock(&ctx);
 	if (rc)
 		goto out_unlock;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
#28 lore raw

Re: [PATCH v7 10/11] PCI/cxl: Expose CXL Reset as a PCI reset method

In reply to Srirangan Madhavan
Quoted 13 lines · On Tue, 23 Jun 2026 03:24:52 +0000 Srirangan Madhavan <[email protected]>…
drivers/pci/pci.c +1 -0
index 360f2aaee10c..b1ec20126390 100644
@@ -33,6 +33,7 @@
 #include <asm/dma.h>
 #include <linux/aer.h>
 #include <linux/bitfield.h>
+#include <cxl/cxl.h>
 #include "pci.h"
 
 DEFINE_MUTEX(pci_slot_mutex);

drivers/pci/pci.c const struct pci_reset_fn_method pci_reset_fn_methods[] = { · line 5088

Quoted 7 lines
drivers/pci/pci.c +1 -0
@@ -5081,6 +5082,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = {
 	{ pci_dev_acpi_reset, .name = "acpi" },
 	{ pcie_reset_flr, .name = "flr" },
 	{ pci_af_flr, .name = "af_flr" },
+	{ cxl_reset_function, .name = "cxl_reset" },
 	{ pci_pm_reset, .name = "pm" },
 	{ pci_reset_bus_function, .name = "bus" },
 	{ cxl_reset_bus_function, .name = "cxl_bus" },

Follow-on to the previous comments, for a device supporting cxl.mem and/or cxl.cache, cxl_reset is the best option, perhaps with the exception of device specific or acpi. Why isn't this placed above FLR?

If we decide cxl.io scoped function resets should not be exposed for cxl.cache/mem devices this might be moot as we'll filter through to cxl_reset anyway. Thanks,

Alex

#29 lore raw

PATCH v7 11/11 Documentation/ABI: Document CXL Reset PCI reset method

In reply to Srirangan Madhavan

Document the "cxl_reset" PCI reset_method value for Type 2 CXL devices. CXL Reset is device scoped, requires affected memory to be idle, invalidates CPU caches, restores cached HDM decoder state, and does not request Memory Clear.

---
 Documentation/ABI/testing/sysfs-bus-pci | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +14 -0
index b767db2c52cb..dd8de5c7eb77 100644
@@ -153,6 +153,20 @@ Description:
 		"default" enables all supported reset methods in the
 		default ordering.
 
+		If present, "cxl_reset" selects CXL Reset for CXL Type 2
+		devices that advertise CXL Reset support.  CXL Reset is device
+		scoped and affects all CXL.cache and CXL.mem functions.
+
+		Before issuing CXL Reset, the kernel quiesces affected PCI
+		functions, rejects the operation if affected CXL memory is busy,
+		invalidates CPU caches for enabled HDM ranges, disables
+		CXL.cache, and initiates cache write-back where supported.  After
+		reset, the kernel restores PCI config state to access HDM MMIO,
+		restores cached HDM decoder state, and then completes reset
+		recovery for the affected functions.  If cached HDM decoder
+		state is unavailable at reset time, the kernel skips this reset
+		method.  "cxl_reset" does not request CXL Reset Memory Clear.
+
 What:		/sys/bus/pci/devices/.../reset
 Date:		July 2009
 Contact:	Michael S. Tsirkin <[email protected]>
Signature · 1 line
2.43.0
#30 lore raw

Re: [PATCH v7 11/11] Documentation/ABI: Document CXL Reset PCI reset method

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines

Document the "cxl_reset" PCI reset_method value for Type 2 CXL devices. CXL Reset is device scoped, requires affected memory to be idle, invalidates CPU caches, restores cached HDM decoder state, and does not request Memory Clear.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
#31 lore raw
In reply to Srirangan Madhavan
Quoted 8 lines · Srirangan Madhavan

Hi folks!

This series adds CXL Reset support for CXL Type 2 devices through the existing PCI reset_method ABI. The reset sequence follows the CXL 4.0 specification [1], including CXL.cache disable, optional cache writeback, CXL Reset initiation, ResetComplete polling, and ResetError reporting.

Quoted 8 lines

The userspace ABI is the existing PCI reset interface:

    /sys/bus/pci/devices/.../reset_method
    /sys/bus/pci/devices/.../reset

Userspace can select "cxl_reset" in reset_method and then trigger reset through the existing reset attribute.

Hi Srirangan,

Thanks for the work, I applied your series and run some tests on a CXL type-2 capable GPU, seems like something is wrong.

Device's BDF is 0002:81:00.0 and DVSEC base is at 0x10c CAP=0x116, CTRL2=0x11c, STATUS2=0x11e
STATUS2 bits: bit0 CACHE_INV, bit1 RST_DONE, bit2 RST_ERR CTRL2 bits: bit0 = DISABLE_CACHING

I run all the following command as root.

"""
# b=0002:81:00.0; dev=/sys/bus/pci/devices/$b

# echo cxl_reset > $dev/reset_method
# echo "PRE  CAP=0x$(setpci -s $b 0x116.w) CTRL2=0x$(setpci -s $b 0x11c.w) STATUS2=0x$(setpci -s $b 0x11e.w)"
# dmesg -C
# time echo 1 > $dev/reset
# echo "POST CAP=0x$(setpci -s $b 0x116.w) CTRL2=0x$(setpci -s $b 0x11c.w) STATUS2=0x$(setpci -s $b 0x11e.w)"
"""

So we know,
PRE CAP=0x8bd7 CTRL2=0x0000 STATUS2=0x8000
==> RESET rc=1 elapsed_ms=114 err=[bash: line 9: echo: write error: Input/output error] POST CAP=0x8bd7 CTRL2=0x0001 STATUS2=0x8003
device-present=1 reset_method=cxl_reset leaked_cxl_reset_iomem=0

with dmesg no output.

The write() to reset failed with -EIO after ~114 ms, but STATUS2 went 0x8000 -> 0x8003 .
The device completed the reset, so the kernel returned failure for a reset the HW did successfully. CTRL2 went 0x0000 -> 0x0001, the device's CXL.cache is disabled after the "failed" reset.
~114ms is almost equal to msleep(100) in cxl_reset_wait_done() + the first poll. On the first poll the Status2 read returns 0xffff, 0xffff has bit 2 set, which the code reads as RST_ERR -> return -EIO with no retry.

After that I ran the same "echo 1 > $dev/reset" in a 25x loop, logging rc, elapsed ms, CTRL2, STATUS2 at each iteration, then dmesg. """
PRE CAP=0x8bd7 CTRL2=0x0001 STATUS2=0x8003

iter  1 rc=1 ms= 114 CTRL2=0x0001 STATUS2=0x8003 present=1
iter  2 rc=1 ms= 171 CTRL2=0xffff STATUS2=0xffff present=1
iter  3 rc=1 ms=   6 CTRL2=0xffff STATUS2=0xffff present=1

[snip]
iter 25 rc=1 ms= 5 CTRL2=0xffff STATUS2=0xffff present=1
### rc histogram: rc=1 : 25x
### leaked cxl_reset iomem regions: 0
"""

The complete dmesg shows
"""

  [ 1892.870193] ------------[ cut here ]------------
  [ 1892.870215] index 7 is out of range for type 'resource_size_t [6]'
  [ 1892.870218] CPU: 121 UID: 0 PID: 19436 Comm: bash Not tainted 7.1.0-rc7+ #1 PREEMPT(full)
  [ 1892.870221] Hardware name: NVIDIA VR NVL72/P3809-BMC, BIOS NV_SBIOS: 06.02.00.00, OEM_SBIOS: 06.02.00.00 Mon
  Jun  8 08:22:03 PM UTC 2026
  [ 1892.870222] Call trace:
  [ 1892.870223]  show_stack+0x24/0x50 (C)
  [ 1892.870229]  dump_stack_lvl+0x80/0x140
  [ 1892.870236]  dump_stack+0x1c/0x38
  [ 1892.870237]  __ubsan_handle_out_of_bounds+0xd0/0x128
  [ 1892.870242]  pci_restore_iov_state+0x250/0x270
  [ 1892.870249]  pci_restore_state+0x10c/0x2c0
  [ 1892.870251]  pci_dev_restore+0x6c/0xb0
  [ 1892.870252]  pci_reset_function+0x94/0x160
  [ 1892.870254]  reset_store+0x78/0xf0
  [ 1892.870258]  dev_attr_store+0x24/0x78
  [ 1892.870263]  sysfs_kf_write+0x88/0xc8
  [ 1892.870268]  kernfs_fop_write_iter+0x170/0x228
  [ 1892.870272]  vfs_write+0x270/0x3a8
  [ 1892.870276]  ksys_write+0x7c/0x138
  [ 1892.870277]  __arm64_sys_write+0x28/0x50
  [ 1892.870279]  invoke_syscall.constprop.0+0xac/0x100
  [ 1892.870282]  do_el0_svc+0x4c/0x100
  [ 1892.870283]  el0_svc+0x50/0x2b0
  [ 1892.870285]  el0t_64_sync_handler+0xc0/0x108
  [ 1892.870286]  el0t_64_sync+0x1b8/0x1c0
  [ 1892.870291] ---[ end trace ]---
  [ 1892.870293] ------------[ cut here ]------------
  [ 1892.870293] UBSAN: shift-out-of-bounds in ./include/linux/log2.h:57:13
  [ 1892.870295] shift exponent 64 is too large for 64-bit type 'long unsigned int'
  [ 1892.870296] CPU: 121 UID: 0 PID: 19436 Comm: bash Not tainted 7.1.0-rc7+ #1 PREEMPT(full)
  [ 1892.870297] Hardware name: NVIDIA VR NVL72/P3809-BMC, BIOS NV_SBIOS: 06.02.00.00, OEM_SBIOS: 06.02.00.00 Mon
  Jun  8 08:22:03 PM UTC 2026
  [ 1892.870298] Call trace:
  [ 1892.870298]  show_stack+0x24/0x50 (C)
  [ 1892.870299]  dump_stack_lvl+0x80/0x140
  [ 1892.870300]  dump_stack+0x1c/0x38
  [ 1892.870301]  __ubsan_handle_shift_out_of_bounds+0x154/0x260
  [ 1892.870302]  pci_rebar_bytes_to_size+0x98/0xc8
  [ 1892.870305]  pci_restore_iov_state+0x1f0/0x270
  [ 1892.870306]  pci_restore_state+0x10c/0x2c0
  [ 1892.870307]  pci_dev_restore+0x6c/0xb0
  [ 1892.870308]  pci_reset_function+0x94/0x160
  [ 1892.870308]  reset_store+0x78/0xf0
  [ 1892.870309]  dev_attr_store+0x24/0x78
  [ 1892.870310]  sysfs_kf_write+0x88/0xc8
  [ 1892.870311]  kernfs_fop_write_iter+0x170/0x228
  [ 1892.870312]  vfs_write+0x270/0x3a8
  [ 1892.870313]  ksys_write+0x7c/0x138
  [ 1892.870314]  __arm64_sys_write+0x28/0x50
  [ 1892.870315]  invoke_syscall.constprop.0+0xac/0x100
  [ 1892.870315]  do_el0_svc+0x4c/0x100
  [ 1892.870316]  el0_svc+0x50/0x2b0
  [ 1892.870317]  el0t_64_sync_handler+0xc0/0x108
  [ 1892.870318]  el0t_64_sync+0x1b8/0x1c0
  [ 1892.870319] ---[ end trace ]---

"""

After iter 1, iter 2 took the device to CTRL2=0xffff STATUS2=0xffff, the config space returns all-ones, which means the function stopped responding to config cycles and didn't self-recover.
During that iteration's pci_dev_restore(), the PCI core read the device's VF-ReBAR config, which is now 0xffffffff, and used the garbage as array indices: bar_idx = 7 to dev->sriov->barsz[] which is sized 6. Two UBSAN reports.

Best regards,
Richard Cheng.

Quoted 131 lines

Following Dan's v6 feedback, this replaces the proposed memdev sysfs ABI with the existing PCI reset_method interface.

v7 changes from v6 [2]:
- Move the ABI from a CXL memdev attribute to PCI reset_method.
- Drop the memdev dependency from reset entry; advertise cxl_reset for Type 2 functions that report CXL Reset support in the CXL Device DVSEC.
- Incorporate Dan's HDM reset refactor: shared decoder settings, pci_dev->hdm cached state, and built-in CONFIG_CXL_HDM helpers.
- Cache endpoint HDM settings during PCI enumeration when MMIO decoding is already enabled, and let CXL core refresh the same cache later.
- Reduce the earlier PCI/CXL save/restore series [3] to the HDM state cache and restore infrastructure needed by this reset flow.
- Use cached HDM ranges to reject reset while affected ranges are busy and to invalidate CPU caches before reset.
- Discover the CXL reset scope with the Non-CXL Function Map and CXL cache/mem capability bits.
- Quiesce affected sibling functions with PCI save/disable and IOMMU reset prepare/done before executing reset.
- Restore cached HDM decoder state after reset before completing PCI reset recovery.
- Keep CXL Reset Memory Clear disabled.

Motivation:
-----------
- Type 2 devices need a CXL-specific reset mechanism beyond existing PCI reset methods.

- FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is the architectural reset mechanism for those protocols.

- The PCI reset_method ABI lets userspace select this narrower CXL reset before falling back to broader bus reset methods.

Change Description:
-------------------

Patch 1: cxl/hdm: Split decoder programming into a reusable helper
- Move shared decoder settings to include/cxl/cxl.h.
- Factor low-level HDM register programming into cxl_commit().

Patch 2: cxl/hdm: Cache decoder settings on PCI devices
- Cache CXL core HDM decoder settings in pci_dev->hdm.
- Refresh the cache as decoders are enumerated, committed, or reset.

Patch 3: cxl/hdm: Cache endpoint decoder settings during PCI enumeration
- Snapshot endpoint HDM state during PCI capability initialization when memory decoding is already enabled.
- Reuse the same cache when CXL core later enumerates the device.

Patch 4: PCI: Export pci_dev_save_and_disable() and pci_dev_restore()
- Export PCI reset lifecycle helpers for CXL reset orchestration.

Patch 5: PCI/CXL: Add CXL Device Reset helper
- Add the internal DVSEC reset sequence.
- Disable CXL.cache, perform cache writeback where supported, initiate CXL Reset, and wait for completion.

Patch 6: PCI/CXL: Validate HDM ranges before CXL reset
- Collect enabled cached HDM ranges.
- Reject reset if affected ranges are busy and invalidate CPU caches.

Patch 7: PCI/CXL: Discover the CXL reset scope
- Discover same-scope CXL functions with the Non-CXL Function Map and CXL cache/mem capability bits.

Patch 8: PCI/CXL: Coordinate sibling functions for CXL reset
- Lock, save, disable, and IOMMU-block affected sibling functions.
- Include mem-capable siblings in HDM range validation and cache flush.

Patch 9: cxl/pci: Restore CXL HDM state after PCI reset
- Restore cached global and per-decoder HDM state after reset.
- Keep IOMMU reset blocks active until HDM restore completes.

Patch 10: PCI/CXL: Expose CXL Reset as a PCI reset method
- Add "cxl_reset" to the PCI reset_method table for Type 2 reset-capable CXL devices.

Patch 11: Documentation/ABI: Document CXL Reset PCI reset method
- Document the new reset_method value and reset behavior.

The CPU cache invalidation step depends on cpu_cache_invalidate_memregion() support for the affected address ranges. If no provider is available, reset fails before hardware reset is requested.

Example:

    echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method
    echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset

Basic CXL DVSEC reset testing was done on a CXL Type 2 device. The reset sequence completed successfully and ResetComplete was observed.

References:
[1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
[2] https://lore.kernel.org/linux-cxl/[email protected]/
[3] https://lore.kernel.org/linux-cxl/[email protected]/

Srirangan Madhavan (11):
cxl/hdm: Split decoder programming into a reusable helper cxl/hdm: Cache decoder settings on PCI devices cxl/hdm: Cache endpoint decoder settings during PCI enumeration
PCI: Export pci_dev_save_and_disable() and pci_dev_restore() PCI/CXL: Add CXL Device Reset helper
PCI/CXL: Validate HDM ranges before CXL reset PCI/CXL: Discover the CXL reset scope
PCI/CXL: Coordinate sibling functions for CXL reset cxl/pci: Restore CXL HDM state after PCI reset PCI/CXL: Expose CXL Reset as a PCI reset method Documentation/ABI: Document CXL Reset PCI reset method

 Documentation/ABI/testing/sysfs-bus-pci |   14 +
 drivers/cxl/Kconfig                     |    4 +
 drivers/cxl/core/Makefile               |    2 +-
 drivers/cxl/core/hdm.c                  |  234 ++---
 drivers/cxl/core/region.c               |    6 +-
 drivers/cxl/core/reset.c                | 1276 +++++++++++++++++++++++
 drivers/cxl/cxl.h                       |   43 -
 drivers/pci/pci.c                       |   25 +-
 drivers/pci/probe.c                     |    2 +
 include/cxl/cxl.h                       |   85 +-
 include/linux/pci.h                     |   10 +-
 include/uapi/linux/pci_regs.h           |   15 +
 tools/testing/cxl/test/cxl.c            |   10 +-
 13 files changed, 1554 insertions(+), 172 deletions(-)
 create mode 100644 drivers/cxl/core/reset.c

base-commit: 72afdd8181219f459142e571999b3b44ef7b85fb

Signature · 1 line
2.43.0