Skip to contentSkip to conversation

CXL Reset support for Type 2 devices

PATCH Superseded [email protected] 12 participants 48 messages Jan 20 – Mar 12 1 message not in the archive

Subsystems (2)

v4 of 17 Superseded Latest is v17 →

lore
Share & tools
lore
#0 lore raw

v4 cover CXL Reset support for Type 2 devices

From: Srirangan Madhavan <[email protected]>

Hi folks!

This patch series introduces support for the CXL Reset method for CXL devices, implementing the reset procedure outlined in CXL Spec [1] v3.2, Sections 9.6 and 9.7.

v4 changes:
- Fix CXL reset capability check parentheses warning
- Gate CXL reset path on CONFIG_CXL_PCI reachability

v3 changes:
- Restrict CXL reset to Type 2 devices only
- Add host and device cache flushing for
* all sibling functions on multi-function devices
* all sibling devices in a given region
- Add region teardown and memory online detection before reset
- Add configuration state save/restore (DVSEC, HDM, IDE)
- Split the series by subsystem and functional blocks

v2 changes:
- De-duplicate CXL DVSEC register defines under include/cxl/pci.h
- Fix style-related issues

v1 changes:
- Added cover letter and dropped the RFC

The RFC patches can be found here [2]
v2 patches can be found here [3]

Motivation:
-----------
This change is broadly useful for reasons including but not limited to the following:

- As support for Type 2 devices [4] is being introduced, more devices will require finer-grained reset mechanisms beyond bus-wide reset methods.

- FLR does not affect CXL.cache or CXL.mem protocols, making CXL Reset the preferred method in some cases.

- The CXL spec (Sections 7.2.3 Binding and Unbinding, 9.5 FLR) highlights use cases like function rebinding and error recovery, where CXL Reset is explicitly mentioned.

Change Description:
-------------------

Patch 1: Move CXL DVSEC defines to the CXL PCI header
- Consolidate DVSEC register definitions under include/cxl/pci.h

Patch 2: Switch PCI CXL port DVSEC defines
- Use the shared CXL PCI header in the PCI core

Patch 3: Add Type 2 helper and reset DVSEC bits
- Add helper to identify Type 2 devices
- Define DVSEC reset/cache control bits

Patch 4: Add the CXL reset method in the PCI core
- Implement cxl_reset() method with capability checks and reset sequence
- Restrict to Type 2 devices

Patch 5: Add reset preparation and region teardown
- Implement region validation and teardown before reset
- Add device cache flush for all sibling devices in a given region

Patch 6: Wire CXL reset prepare/cleanup in PCI
- Call CXL reset prepare/cleanup around the core reset flow

Patch 7: Add host CPU cache flush and multi-function support
- Add host CPU cache flush (x86: wbinvd, arm64: dcache_clean_inval_poc)
- Add device cache flush for all sibling functions on multi-function devices

Patch 8: Add DVSEC configuration state save/restore
- Save/restore DVSEC registers (DEVCTL, DEVCTL2) with CONFIG_LOCK handling

Patch 9: Save/restore CXL config around reset
- Save PCI and CXL config before reset and restore afterwards

Patch 10: Add HDM decoder and IDE state save/restore
- Save/restore HDM decoder and IDE register state

The reset sequence: validate device type, check memory offline, tear down regions, flush host CPU caches, flush device caches (all functions), save config state, initiate reset, wait for completion, restore config state.

Command line to test the CXL reset on a capable device:
echo cxl_reset > /sys/bus/pci/devices/<pci_device>/reset_method echo 1 > /sys/bus/pci/devices/<pci_device>/reset

[1] https://computeexpresslink.org/cxl-specification/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/linux-cxl/[email protected]/

Srirangan Madhavan (10):
[PATCH v4 1/10] cxl: move DVSEC defines to cxl pci header [PATCH v4 2/10] PCI: switch CXL port DVSEC defines [PATCH v4 3/10] cxl: add type 2 helper and reset DVSEC bits [PATCH v4 4/10] PCI: add CXL reset method
[PATCH v4 5/10] cxl: add reset prepare and region teardown [PATCH v4 6/10] PCI: wire CXL reset prepare/cleanup [PATCH v4 7/10] cxl: add host cache flush and multi-function reset [PATCH v4 8/10] cxl: add DVSEC config save/restore [PATCH v4 9/10] PCI: save/restore CXL config around reset [PATCH v4 10/10] cxl: add HDM decoder and IDE save/restore

 drivers/cxl/core/pci.c        |   1 +
 drivers/cxl/core/regs.c       |   8 +
 drivers/cxl/cxl.h             |   4 +
 drivers/cxl/cxlpci.h          |  53 ---
 drivers/cxl/pci.c             | 621 +++++++++++++++++++++++++++++++++-
 drivers/pci/pci.c             | 150 +++++++-
 include/cxl/pci.h             | 134 ++++++++
 include/linux/pci.h           |  21 +-
 include/uapi/linux/pci_regs.h |   5 -
 9 files changed, 929 insertions(+), 68 deletions(-)
 create mode 100644 include/cxl/pci.h

--
2.34.1

#1 lore raw

PATCH v4 1/10 cxl: move DVSEC defines to cxl pci header

From: Srirangan Madhavan <[email protected]>

CXL DVSEC definitions are shared across PCI core and CXL drivers, so move the register macros into the common CXL PCI header. This keeps the DVSEC surface in one place and avoids duplication as the reset and config helpers build on these offsets and bitfields.

---
 drivers/cxl/core/pci.c  |  1 +
 drivers/cxl/core/regs.c |  1 +
 drivers/cxl/cxlpci.h    | 53 -----------------------------------
 drivers/cxl/pci.c       |  1 +
 include/cxl/pci.h       | 62 +++++++++++++++++++++++++++++++++++++++++
 5 files changed, 65 insertions(+), 53 deletions(-)
 create mode 100644 include/cxl/pci.h
drivers/cxl/core/pci.c +1 -0
index 5b023a0178a4..968babcc09a2 100644
@@ -7,6 +7,7 @@
 #include <linux/pci.h>
 #include <linux/pci-doe.h>
 #include <linux/aer.h>
+#include <cxl/pci.h>
 #include <cxlpci.h>
 #include <cxlmem.h>
 #include <cxl.h>
drivers/cxl/core/regs.c +1 -0
index 5ca7b0eed568..ecdb22ae6952 100644
@@ -4,6 +4,7 @@
 #include <linux/device.h>
 #include <linux/slab.h>
 #include <linux/pci.h>
+#include <cxl/pci.h>
 #include <cxlmem.h>
 #include <cxlpci.h>
 #include <pmu.h>
drivers/cxl/cxlpci.h +0 -53

59 lines not on this page · raw message

drivers/cxl/pci.c +1 -0
index 3b2293dffb3f..55c767df4543 100644
@@ -12,6 +12,7 @@
 #include <linux/aer.h>
 #include <linux/io.h>
 #include <cxl/mailbox.h>
+#include <cxl/pci.h>
 #include "cxlmem.h"
 #include "cxlpci.h"
 #include "cxl.h"
include/cxl/pci.h new file +62 -0

62 lines not on this page · raw message

--
2.34.1

#2 lore raw

Re: [PATCH v4 01/10] cxl: move DVSEC defines to cxl pci header

Quoted 8 lines · On Tue, 20 Jan 2026 22:26:01 +0000 [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

CXL DVSEC definitions are shared across PCI core and CXL drivers, so move the register macros into the common CXL PCI header. This keeps the DVSEC surface in one place and avoids duplication as the reset and config helpers build on these offsets and bitfields.

These are moving in:
https://lore.kernel.org/all/[email protected]/
However it is to the main uapi/linux/pci_regs.h file.

*fingers crossed* that should land this cycle and simplify your set a little.

Jonathan

Quoted 181 lines
---
 drivers/cxl/core/pci.c  |  1 +
 drivers/cxl/core/regs.c |  1 +
 drivers/cxl/cxlpci.h    | 53 -----------------------------------
 drivers/cxl/pci.c       |  1 +
 include/cxl/pci.h       | 62 +++++++++++++++++++++++++++++++++++++++++
 5 files changed, 65 insertions(+), 53 deletions(-)
 create mode 100644 include/cxl/pci.h
drivers/cxl/core/pci.c +1 -0
index 5b023a0178a4..968babcc09a2 100644
@@ -7,6 +7,7 @@
 #include <linux/pci.h>
 #include <linux/pci-doe.h>
 #include <linux/aer.h>
+#include <cxl/pci.h>
 #include <cxlpci.h>
 #include <cxlmem.h>
 #include <cxl.h>
drivers/cxl/core/regs.c +1 -0
index 5ca7b0eed568..ecdb22ae6952 100644
@@ -4,6 +4,7 @@
 #include <linux/device.h>
 #include <linux/slab.h>
 #include <linux/pci.h>
+#include <cxl/pci.h>
 #include <cxlmem.h>
 #include <cxlpci.h>
 #include <pmu.h>
drivers/cxl/cxlpci.h +0 -53
index 1d526bea8431..cdb7cf3dbcb4 100644
@@ -7,59 +7,6 @@
 
 #define CXL_MEMORY_PROGIF	0x10
 
-/*
- * See section 8.1 Configuration Space Registers in the CXL 2.0
- * Specification. Names are taken straight from the specification with "CXL" and
- * "DVSEC" redundancies removed. When obvious, abbreviations may be used.
- */
-#define PCI_DVSEC_HEADER1_LENGTH_MASK	GENMASK(31, 20)
-
-/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */
-#define CXL_DVSEC_PCIE_DEVICE					0
-#define   CXL_DVSEC_CAP_OFFSET		0xA
-#define     CXL_DVSEC_MEM_CAPABLE	BIT(2)
-#define     CXL_DVSEC_HDM_COUNT_MASK	GENMASK(5, 4)
-#define   CXL_DVSEC_CTRL_OFFSET		0xC
-#define     CXL_DVSEC_MEM_ENABLE	BIT(2)
-#define   CXL_DVSEC_RANGE_SIZE_HIGH(i)	(0x18 + (i * 0x10))
-#define   CXL_DVSEC_RANGE_SIZE_LOW(i)	(0x1C + (i * 0x10))
-#define     CXL_DVSEC_MEM_INFO_VALID	BIT(0)
-#define     CXL_DVSEC_MEM_ACTIVE	BIT(1)
-#define     CXL_DVSEC_MEM_SIZE_LOW_MASK	GENMASK(31, 28)
-#define   CXL_DVSEC_RANGE_BASE_HIGH(i)	(0x20 + (i * 0x10))
-#define   CXL_DVSEC_RANGE_BASE_LOW(i)	(0x24 + (i * 0x10))
-#define     CXL_DVSEC_MEM_BASE_LOW_MASK	GENMASK(31, 28)
-
-#define CXL_DVSEC_RANGE_MAX		2
-
-/* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */
-#define CXL_DVSEC_FUNCTION_MAP					2
-
-/* CXL 2.0 8.1.5: CXL 2.0 Extensions DVSEC for Ports */
-#define CXL_DVSEC_PORT_EXTENSIONS				3
-
-/* CXL 2.0 8.1.6: GPF DVSEC for CXL Port */
-#define CXL_DVSEC_PORT_GPF					4
-#define   CXL_DVSEC_PORT_GPF_PHASE_1_CONTROL_OFFSET		0x0C
-#define     CXL_DVSEC_PORT_GPF_PHASE_1_TMO_BASE_MASK		GENMASK(3, 0)
-#define     CXL_DVSEC_PORT_GPF_PHASE_1_TMO_SCALE_MASK		GENMASK(11, 8)
-#define   CXL_DVSEC_PORT_GPF_PHASE_2_CONTROL_OFFSET		0xE
-#define     CXL_DVSEC_PORT_GPF_PHASE_2_TMO_BASE_MASK		GENMASK(3, 0)
-#define     CXL_DVSEC_PORT_GPF_PHASE_2_TMO_SCALE_MASK		GENMASK(11, 8)
-
-/* CXL 2.0 8.1.7: GPF DVSEC for CXL Device */
-#define CXL_DVSEC_DEVICE_GPF					5
-
-/* CXL 2.0 8.1.8: PCIe DVSEC for Flex Bus Port */
-#define CXL_DVSEC_PCIE_FLEXBUS_PORT				7
-
-/* CXL 2.0 8.1.9: Register Locator DVSEC */
-#define CXL_DVSEC_REG_LOCATOR					8
-#define   CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET			0xC
-#define     CXL_DVSEC_REG_LOCATOR_BIR_MASK			GENMASK(2, 0)
-#define	    CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK			GENMASK(15, 8)
-#define     CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK		GENMASK(31, 16)
-
 /*
  * NOTE: Currently all the functions which are enabled for CXL require their
  * vectors to be in the first 16.  Use this as the default max.
drivers/cxl/pci.c +1 -0
index 3b2293dffb3f..55c767df4543 100644
@@ -12,6 +12,7 @@
 #include <linux/aer.h>
 #include <linux/io.h>
 #include <cxl/mailbox.h>
+#include <cxl/pci.h>
 #include "cxlmem.h"
 #include "cxlpci.h"
 #include "cxl.h"
include/cxl/pci.h new file +62 -1
new file mode 100644
index 000000000000..728ba0cdd289
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+
+#ifndef __CXL_ACCEL_PCI_H
+#define __CXL_ACCEL_PCI_H
+
+/*
+ * See section 8.1 Configuration Space Registers in the CXL 2.0
+ * Specification. Names are taken straight from the specification with "CXL" and
+ * "DVSEC" redundancies removed. When obvious, abbreviations may be used.
+ */
+#define PCI_DVSEC_HEADER1_LENGTH_MASK  GENMASK(31, 20)
+
+/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */
+#define CXL_DVSEC_PCIE_DEVICE					0
+#define   CXL_DVSEC_CAP_OFFSET		0xA
+#define     CXL_DVSEC_MEM_CAPABLE	BIT(2)
+#define     CXL_DVSEC_HDM_COUNT_MASK	GENMASK(5, 4)
+#define   CXL_DVSEC_CTRL_OFFSET		0xC
+#define     CXL_DVSEC_MEM_ENABLE	BIT(2)
+#define   CXL_DVSEC_RANGE_SIZE_HIGH(i)	(0x18 + ((i) * 0x10))
+#define   CXL_DVSEC_RANGE_SIZE_LOW(i)	(0x1C + ((i) * 0x10))
+#define     CXL_DVSEC_MEM_INFO_VALID	BIT(0)
+#define     CXL_DVSEC_MEM_ACTIVE	BIT(1)
+#define     CXL_DVSEC_MEM_SIZE_LOW_MASK	GENMASK(31, 28)
+#define   CXL_DVSEC_RANGE_BASE_HIGH(i)	(0x20 + ((i) * 0x10))
+#define   CXL_DVSEC_RANGE_BASE_LOW(i)	(0x24 + ((i) * 0x10))
+#define     CXL_DVSEC_MEM_BASE_LOW_MASK	GENMASK(31, 28)
+
+#define CXL_DVSEC_RANGE_MAX		2
+
+/* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */
+#define CXL_DVSEC_FUNCTION_MAP					2
+
+/* CXL 2.0 8.1.5: CXL 2.0 Extensions DVSEC for Ports */
+#define CXL_DVSEC_PORT_EXTENSIONS				3
+#define   CXL_DVSEC_PORT_CTL		0xC
+#define     CXL_DVSEC_UNMASK_SBR		BIT(0)
+
+/* CXL 2.0 8.1.6: GPF DVSEC for CXL Port */
+#define CXL_DVSEC_PORT_GPF					4
+#define   CXL_DVSEC_PORT_GPF_PHASE_1_CONTROL_OFFSET		0x0C
+#define     CXL_DVSEC_PORT_GPF_PHASE_1_TMO_BASE_MASK		GENMASK(3, 0)
+#define     CXL_DVSEC_PORT_GPF_PHASE_1_TMO_SCALE_MASK		GENMASK(11, 8)
+#define   CXL_DVSEC_PORT_GPF_PHASE_2_CONTROL_OFFSET		0xE
+#define     CXL_DVSEC_PORT_GPF_PHASE_2_TMO_BASE_MASK		GENMASK(3, 0)
+#define     CXL_DVSEC_PORT_GPF_PHASE_2_TMO_SCALE_MASK		GENMASK(11, 8)
+
+/* CXL 2.0 8.1.7: GPF DVSEC for CXL Device */
+#define CXL_DVSEC_DEVICE_GPF					5
+
+/* CXL 2.0 8.1.8: PCIe DVSEC for Flex Bus Port */
+#define CXL_DVSEC_PCIE_FLEXBUS_PORT				7
+
+/* CXL 2.0 8.1.9: Register Locator DVSEC */
+#define CXL_DVSEC_REG_LOCATOR					8
+#define   CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET			0xC
+#define     CXL_DVSEC_REG_LOCATOR_BIR_MASK			GENMASK(2, 0)
+#define	    CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK			GENMASK(15, 8)
+#define     CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK		GENMASK(31, 16)
+
+#endif
--

2.34.1

#3 lore raw

PATCH v4 2/10 PCI: switch CXL port DVSEC defines

In reply to [email protected]

From: Srirangan Madhavan <[email protected]>

The PCI core consumes CXL port DVSEC fields for reset handling, so switch it to use the shared CXL PCI header instead of the uapi header. This aligns the core with the header split and keeps internal code from depending on uapi-only definitions.

---
 drivers/pci/pci.c             | 17 +++++++++--------
 include/uapi/linux/pci_regs.h |  5 -----
 2 files changed, 9 insertions(+), 13 deletions(-)
drivers/pci/pci.c +9 -8
index 13dbb405dc31..8bb07e253646 100644
@@ -30,6 +30,7 @@
 #include <asm/dma.h>
 #include <linux/aer.h>
 #include <linux/bitfield.h>
+#include <cxl/pci.h>
 #include "pci.h"
 
 DEFINE_MUTEX(pci_slot_mutex);
@@ -4842,7 +4843,7 @@ static int pci_dev_reset_slot_function(struct pci_dev *dev, bool probe)
 static u16 cxl_port_dvsec(struct pci_dev *dev)
 {
 	return pci_find_dvsec_capability(dev, PCI_VENDOR_ID_CXL,
-					 PCI_DVSEC_CXL_PORT);
+					 CXL_DVSEC_PORT_EXTENSIONS);
 }
 
 static bool cxl_sbr_masked(struct pci_dev *dev)
@@ -4854,7 +4855,7 @@ static bool cxl_sbr_masked(struct pci_dev *dev)
 	if (!dvsec)
 		return false;
 
-	rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_PORT_CTL, &reg);
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_PORT_CTL, &reg);
 	if (rc || PCI_POSSIBLE_ERROR(reg))
 		return false;
 
@@ -4863,7 +4864,7 @@ static bool cxl_sbr_masked(struct pci_dev *dev)
 	 * bit in Bridge Control has no effect.  When 1, the Port generates
 	 * hot reset when the SBR bit is set to 1.
 	 */
-	if (reg & PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR)
+	if (reg & CXL_DVSEC_UNMASK_SBR)
 		return false;
 
 	return true;
@@ -4908,22 +4909,22 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)
 	if (probe)
 		return 0;
 
-	rc = pci_read_config_word(bridge, dvsec + PCI_DVSEC_CXL_PORT_CTL, &reg);
+	rc = pci_read_config_word(bridge, dvsec + CXL_DVSEC_PORT_CTL, &reg);
 	if (rc)
 		return -ENOTTY;
 
-	if (reg & PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR) {
+	if (reg & CXL_DVSEC_UNMASK_SBR) {
 		val = reg;
 	} else {
-		val = reg | PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR;
-		pci_write_config_word(bridge, dvsec + PCI_DVSEC_CXL_PORT_CTL,
+		val = reg | CXL_DVSEC_UNMASK_SBR;
+		pci_write_config_word(bridge, dvsec + CXL_DVSEC_PORT_CTL,
 				      val);
 	}
 
 	rc = pci_reset_bus_function(dev, probe);
 
 	if (reg != val)
-		pci_write_config_word(bridge, dvsec + PCI_DVSEC_CXL_PORT_CTL,
+		pci_write_config_word(bridge, dvsec + CXL_DVSEC_PORT_CTL,
 				      reg);
 
 	return rc;
include/uapi/linux/pci_regs.h +0 -5
index 3add74ae2594..4f9e6dddc282 100644
@@ -1253,11 +1253,6 @@
 #define PCI_DEV3_STA		0x0c	/* Device 3 Status Register */
 #define  PCI_DEV3_STA_SEGMENT	0x8	/* Segment Captured (end-to-end flit-mode detected) */
 
-/* Compute Express Link (CXL r3.1, sec 8.1.5) */
-#define PCI_DVSEC_CXL_PORT				3
-#define PCI_DVSEC_CXL_PORT_CTL				0x0c
-#define PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR		0x00000001
-
 /* Integrity and Data Encryption Extended Capability */
 #define PCI_IDE_CAP			0x04
 #define  PCI_IDE_CAP_LINK		0x1  /* Link IDE Stream Supported */

--
2.34.1

#4 lore raw

Re: [PATCH v4 02/10] PCI: switch CXL port DVSEC defines

Quoted 6 lines · On Tue, 20 Jan 2026 22:26:02 +0000 [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

The PCI core consumes CXL port DVSEC fields for reset handling, so switch it to use the shared CXL PCI header instead of the uapi header. This aligns the core with the header split and keeps internal code from depending on uapi-only definitions.

Why do you think they are uapi only? Masses of kernel code relies on those definitions.

Intent is to just have one source of truth for userspace tools and kernel space ones.

Also, removing anything from those headers will probably break someone's user space so is basically impossible to do safely.

J

Quoted 94 lines
---
 drivers/pci/pci.c             | 17 +++++++++--------
 include/uapi/linux/pci_regs.h |  5 -----
 2 files changed, 9 insertions(+), 13 deletions(-)
drivers/pci/pci.c +9 -8
index 13dbb405dc31..8bb07e253646 100644
@@ -30,6 +30,7 @@
 #include <asm/dma.h>
 #include <linux/aer.h>
 #include <linux/bitfield.h>
+#include <cxl/pci.h>
 #include "pci.h"
 
 DEFINE_MUTEX(pci_slot_mutex);
@@ -4842,7 +4843,7 @@ static int pci_dev_reset_slot_function(struct pci_dev *dev, bool probe)
 static u16 cxl_port_dvsec(struct pci_dev *dev)
 {
 	return pci_find_dvsec_capability(dev, PCI_VENDOR_ID_CXL,
-					 PCI_DVSEC_CXL_PORT);
+					 CXL_DVSEC_PORT_EXTENSIONS);
 }
 
 static bool cxl_sbr_masked(struct pci_dev *dev)
@@ -4854,7 +4855,7 @@ static bool cxl_sbr_masked(struct pci_dev *dev)
 	if (!dvsec)
 		return false;
 
-	rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_PORT_CTL, &reg);
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_PORT_CTL, &reg);
 	if (rc || PCI_POSSIBLE_ERROR(reg))
 		return false;
 
@@ -4863,7 +4864,7 @@ static bool cxl_sbr_masked(struct pci_dev *dev)
 	 * bit in Bridge Control has no effect.  When 1, the Port generates
 	 * hot reset when the SBR bit is set to 1.
 	 */
-	if (reg & PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR)
+	if (reg & CXL_DVSEC_UNMASK_SBR)
 		return false;
 
 	return true;
@@ -4908,22 +4909,22 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)
 	if (probe)
 		return 0;
 
-	rc = pci_read_config_word(bridge, dvsec + PCI_DVSEC_CXL_PORT_CTL, &reg);
+	rc = pci_read_config_word(bridge, dvsec + CXL_DVSEC_PORT_CTL, &reg);
 	if (rc)
 		return -ENOTTY;
 
-	if (reg & PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR) {
+	if (reg & CXL_DVSEC_UNMASK_SBR) {
 		val = reg;
 	} else {
-		val = reg | PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR;
-		pci_write_config_word(bridge, dvsec + PCI_DVSEC_CXL_PORT_CTL,
+		val = reg | CXL_DVSEC_UNMASK_SBR;
+		pci_write_config_word(bridge, dvsec + CXL_DVSEC_PORT_CTL,
 				      val);
 	}
 
 	rc = pci_reset_bus_function(dev, probe);
 
 	if (reg != val)
-		pci_write_config_word(bridge, dvsec + PCI_DVSEC_CXL_PORT_CTL,
+		pci_write_config_word(bridge, dvsec + CXL_DVSEC_PORT_CTL,
 				      reg);
 
 	return rc;
include/uapi/linux/pci_regs.h +0 -6
index 3add74ae2594..4f9e6dddc282 100644
@@ -1253,11 +1253,6 @@
 #define PCI_DEV3_STA		0x0c	/* Device 3 Status Register */
 #define  PCI_DEV3_STA_SEGMENT	0x8	/* Segment Captured (end-to-end flit-mode detected) */
 
-/* Compute Express Link (CXL r3.1, sec 8.1.5) */
-#define PCI_DVSEC_CXL_PORT				3
-#define PCI_DVSEC_CXL_PORT_CTL				0x0c
-#define PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR		0x00000001
-
 /* Integrity and Data Encryption Extended Capability */
 #define PCI_IDE_CAP			0x04
 #define  PCI_IDE_CAP_LINK		0x1  /* Link IDE Stream Supported */
--

2.34.1

#5 lore raw

PATCH v4 3/10 cxl: add type 2 helper and reset DVSEC bits

In reply to [email protected]

From: Srirangan Madhavan <[email protected]>

Introduce a helper to identify CXL Type 2 devices and define the DVSEC reset/cache control bits used by the reset flow.

---
 drivers/cxl/pci.c | 10 ++++++++++
 include/cxl/pci.h | 14 ++++++++++++++
 2 files changed, 24 insertions(+)
drivers/cxl/pci.c +10 -0
index 55c767df4543..b562e607ec46 100644
@@ -1075,6 +1075,16 @@ static pci_ers_result_t cxl_slot_reset(struct pci_dev *pdev)
 	return PCI_ERS_RESULT_RECOVERED;
 }
 
+bool cxl_is_type2_device(struct pci_dev *pdev)
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+
+	if (!cxlds)
+		return false;
+
+	return cxlds->type == CXL_DEVTYPE_DEVMEM;
+}
+
 static void cxl_error_resume(struct pci_dev *pdev)
 {
 	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
include/cxl/pci.h +14 -0
index 728ba0cdd289..71d8de5de948 100644
@@ -14,10 +14,24 @@
 /* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */
 #define CXL_DVSEC_PCIE_DEVICE					0
 #define   CXL_DVSEC_CAP_OFFSET		0xA
+#define     CXL_DVSEC_CACHE_CAPABLE	BIT(0)
 #define     CXL_DVSEC_MEM_CAPABLE	BIT(2)
 #define     CXL_DVSEC_HDM_COUNT_MASK	GENMASK(5, 4)
+#define     CXL_DVSEC_CACHE_WBI_CAPABLE	BIT(6)
+#define     CXL_DVSEC_CXL_RST_CAPABLE	BIT(7)
+#define     CXL_DVSEC_CXL_RST_TIMEOUT_MASK	GENMASK(10, 8)
+#define     CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE	BIT(11)
 #define   CXL_DVSEC_CTRL_OFFSET		0xC
 #define     CXL_DVSEC_MEM_ENABLE	BIT(2)
+#define   CXL_DVSEC_CTRL2_OFFSET	0x10
+#define     CXL_DVSEC_DISABLE_CACHING	BIT(0)
+#define     CXL_DVSEC_INIT_CACHE_WBI	BIT(1)
+#define     CXL_DVSEC_INIT_CXL_RESET	BIT(2)
+#define     CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE	BIT(3)
+#define   CXL_DVSEC_STATUS2_OFFSET	0x12
+#define     CXL_DVSEC_CACHE_INVALID	BIT(0)
+#define     CXL_DVSEC_CXL_RST_COMPLETE	BIT(1)
+#define     CXL_DVSEC_CXL_RESET_ERR	BIT(2)
 #define   CXL_DVSEC_RANGE_SIZE_HIGH(i)	(0x18 + ((i) * 0x10))
 #define   CXL_DVSEC_RANGE_SIZE_LOW(i)	(0x1C + ((i) * 0x10))
 #define     CXL_DVSEC_MEM_INFO_VALID	BIT(0)

--
2.34.1

maintainer
#6 lore raw

Re: [PATCH v4 03/10] cxl: add type 2 helper and reset DVSEC bits

Quoted 4 lines · On 1/20/26 3:26 PM, [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Introduce a helper to identify CXL Type 2 devices and define the DVSEC reset/cache control bits used by the reset flow.

Should probably be 2 separate patches for these 2 things.

Quoted 47 lines
---
 drivers/cxl/pci.c | 10 ++++++++++
 include/cxl/pci.h | 14 ++++++++++++++
 2 files changed, 24 insertions(+)
drivers/cxl/pci.c +10 -0
index 55c767df4543..b562e607ec46 100644
@@ -1075,6 +1075,16 @@ static pci_ers_result_t cxl_slot_reset(struct pci_dev *pdev)
 	return PCI_ERS_RESULT_RECOVERED;
 }
 
+bool cxl_is_type2_device(struct pci_dev *pdev)
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+
+	if (!cxlds)
+		return false;
+
+	return cxlds->type == CXL_DEVTYPE_DEVMEM;
+}
+
 static void cxl_error_resume(struct pci_dev *pdev)
 {
 	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
include/cxl/pci.h +7 -0
index 728ba0cdd289..71d8de5de948 100644
@@ -14,10 +14,24 @@
 /* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */
 #define CXL_DVSEC_PCIE_DEVICE					0
 #define   CXL_DVSEC_CAP_OFFSET		0xA
+#define     CXL_DVSEC_CACHE_CAPABLE	BIT(0)
 #define     CXL_DVSEC_MEM_CAPABLE	BIT(2)
 #define     CXL_DVSEC_HDM_COUNT_MASK	GENMASK(5, 4)
+#define     CXL_DVSEC_CACHE_WBI_CAPABLE	BIT(6)
+#define     CXL_DVSEC_CXL_RST_CAPABLE	BIT(7)
+#define     CXL_DVSEC_CXL_RST_TIMEOUT_MASK	GENMASK(10, 8)
+#define     CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE	BIT(11)
 #define   CXL_DVSEC_CTRL_OFFSET		0xC
 #define     CXL_DVSEC_MEM_ENABLE	BIT(2)
+#define   CXL_DVSEC_CTRL2_OFFSET	0x10
+#define     CXL_DVSEC_DISABLE_CACHING	BIT(0)

include/cxl/pci.h line 37

Quoted 10 lines
include/cxl/pci.h +7 -0
+#define     CXL_DVSEC_INIT_CACHE_WBI	BIT(1)
+#define     CXL_DVSEC_INIT_CXL_RESET	BIT(2)
+#define     CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE	BIT(3)
+#define   CXL_DVSEC_STATUS2_OFFSET	0x12
+#define     CXL_DVSEC_CACHE_INVALID	BIT(0)
+#define     CXL_DVSEC_CXL_RST_COMPLETE	BIT(1)
+#define     CXL_DVSEC_CXL_RESET_ERR	BIT(2)
 #define   CXL_DVSEC_RANGE_SIZE_HIGH(i)	(0x18 + ((i) * 0x10))
 #define   CXL_DVSEC_RANGE_SIZE_LOW(i)	(0x1C + ((i) * 0x10))
 #define     CXL_DVSEC_MEM_INFO_VALID	BIT(0)

Should this chunk go with a different patch where the definitions are being used?

Quoted 3 lines

--
2.34.1

#7 lore raw

Re: [PATCH v4 03/10] cxl: add type 2 helper and reset DVSEC bits

Quoted 7 lines · On Tue, 20 Jan 2026 16:27:33 -0700 Dave Jiang <[email protected]> wrote:
Quoted 4 lines · On 1/20/26 3:26 PM, [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Introduce a helper to identify CXL Type 2 devices and define the DVSEC reset/cache control bits used by the reset flow.

Should probably be 2 separate patches for these 2 things.

Also, follow existing convention and put them new DVSEC defs in the uapi/pci_regs.h file. The rest are moving there shortly.

Given they are now in a uapi file, I also wonder if we should just fill in the rest of the structure definitions as a stand alone patch. A partial set isn't much use to userspace tooling.

Jonathan

Quoted 66 lines
Quoted 48 lines
---
 drivers/cxl/pci.c | 10 ++++++++++
 include/cxl/pci.h | 14 ++++++++++++++
 2 files changed, 24 insertions(+)
drivers/cxl/pci.c +10 -0
index 55c767df4543..b562e607ec46 100644
@@ -1075,6 +1075,16 @@ static pci_ers_result_t cxl_slot_reset(struct pci_dev *pdev)
 	return PCI_ERS_RESULT_RECOVERED;
 }
 
+bool cxl_is_type2_device(struct pci_dev *pdev)
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+
+	if (!cxlds)
+		return false;
+
+	return cxlds->type == CXL_DEVTYPE_DEVMEM;
+}
+
 static void cxl_error_resume(struct pci_dev *pdev)
 {
 	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
include/cxl/pci.h +7 -0
index 728ba0cdd289..71d8de5de948 100644
@@ -14,10 +14,24 @@
 /* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */
 #define CXL_DVSEC_PCIE_DEVICE					0
 #define   CXL_DVSEC_CAP_OFFSET		0xA
+#define     CXL_DVSEC_CACHE_CAPABLE	BIT(0)
 #define     CXL_DVSEC_MEM_CAPABLE	BIT(2)
 #define     CXL_DVSEC_HDM_COUNT_MASK	GENMASK(5, 4)
+#define     CXL_DVSEC_CACHE_WBI_CAPABLE	BIT(6)
+#define     CXL_DVSEC_CXL_RST_CAPABLE	BIT(7)
+#define     CXL_DVSEC_CXL_RST_TIMEOUT_MASK	GENMASK(10, 8)
+#define     CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE	BIT(11)
 #define   CXL_DVSEC_CTRL_OFFSET		0xC
 #define     CXL_DVSEC_MEM_ENABLE	BIT(2)
+#define   CXL_DVSEC_CTRL2_OFFSET	0x10
+#define     CXL_DVSEC_DISABLE_CACHING	BIT(0)

include/cxl/pci.h line 37

Quoted 10 lines
include/cxl/pci.h +7 -0
+#define     CXL_DVSEC_INIT_CACHE_WBI	BIT(1)
+#define     CXL_DVSEC_INIT_CXL_RESET	BIT(2)
+#define     CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE	BIT(3)
+#define   CXL_DVSEC_STATUS2_OFFSET	0x12
+#define     CXL_DVSEC_CACHE_INVALID	BIT(0)
+#define     CXL_DVSEC_CXL_RST_COMPLETE	BIT(1)
+#define     CXL_DVSEC_CXL_RESET_ERR	BIT(2)
 #define   CXL_DVSEC_RANGE_SIZE_HIGH(i)	(0x18 + ((i) * 0x10))
 #define   CXL_DVSEC_RANGE_SIZE_LOW(i)	(0x1C + ((i) * 0x10))
 #define     CXL_DVSEC_MEM_INFO_VALID	BIT(0)  

Should this chunk go with a different patch where the definitions are being used?

Quoted 3 lines

--
2.34.1

#8 lore raw

PATCH v4 4/10 PCI: add CXL reset method

In reply to [email protected]

From: Srirangan Madhavan <[email protected]>

Add a PCI reset method "cxl_reset" that drives the CXL reset sequence using DVSEC controls and timeout encoding. The method is restricted to Type 2 devices, limiting the scope of the changes.

---
 drivers/pci/pci.c   | 104 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h |  10 ++++-
 2 files changed, 113 insertions(+), 1 deletion(-)
drivers/pci/pci.c +104 -0
index 8bb07e253646..e2d5ff25ab67 100644
@@ -4892,6 +4892,109 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)
 	return pci_parent_bus_reset(dev, probe);
 }
 
+static int cxl_reset_init(struct pci_dev *dev, u16 dvsec)
+{
+	/*
+	 * Timeout values ref CXL Spec v3.2 Ch 8 Control and Status Registers,
+	 * under section 8.1.3.1 DVSEC CXL Capability.
+	 */
+	u32 reset_timeouts_ms[] = { 10, 100, 1000, 10000, 100000 };
+	u16 reg;
+	u32 timeout_ms;
+	int rc, ind;
+
+	/* Check if CXL Reset MEM CLR is supported. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CAP_OFFSET, &reg);
+	if (rc)
+		return rc;
+
+	if (reg & CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE) {
+		rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET,
+					  &reg);
+		if (rc)
+			return rc;
+
+		reg |= CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE;
+		pci_write_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, reg);
+	}
+
+	/* Read timeout value. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CAP_OFFSET, &reg);
+	if (rc)
+		return rc;
+	ind = FIELD_GET(CXL_DVSEC_CXL_RST_TIMEOUT_MASK, reg);
+	timeout_ms = reset_timeouts_ms[ind];
+
+	/* Write reset config. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &reg);
+	if (rc)
+		return rc;
+
+	reg |= CXL_DVSEC_INIT_CXL_RESET;
+	pci_write_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, reg);
+
+	/* Wait till timeout and then check reset status is complete. */
+	msleep(timeout_ms);
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_STATUS2_OFFSET, &reg);
+	if (rc)
+		return rc;
+	if (reg & CXL_DVSEC_CXL_RESET_ERR ||
+	    ~reg & CXL_DVSEC_CXL_RST_COMPLETE)
+		return -ETIMEDOUT;
+
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &reg);
+	if (rc)
+		return rc;
+	reg &= (~CXL_DVSEC_DISABLE_CACHING);
+	pci_write_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, reg);
+
+	return 0;
+}
+
+/**
+ * cxl_reset - initiate a cxl reset
+ * @dev: device to reset
+ * @probe: if true, return 0 if device can be reset this way
+ *
+ * Initiate a cxl reset on @dev.
+ */
+static int cxl_reset(struct pci_dev *dev, bool probe)
+{
+	u16 dvsec, reg;
+	int rc;
+
+	dvsec = pci_find_dvsec_capability(dev, PCI_VENDOR_ID_CXL,
+					  CXL_DVSEC_PCIE_DEVICE);
+	if (!dvsec)
+		return -ENOTTY;
+
+	/* Check if CXL Reset is supported. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CAP_OFFSET, &reg);
+	if (rc)
+		return -ENOTTY;
+
+	if ((reg & CXL_DVSEC_CXL_RST_CAPABLE) == 0)
+		return -ENOTTY;
+
+#if !IS_REACHABLE(CONFIG_CXL_PCI)
+	return -ENOTTY;
+#endif
+
+	/*
+	 * Expose CXL reset for Type 2 devices.
+	 */
+	if (!cxl_is_type2_device(dev))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	if (!pci_wait_for_pending_transaction(dev))
+		pci_err(dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
+
+	return cxl_reset_init(dev, dvsec);
+}
+
 static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)
 {
 	struct pci_dev *bridge;
@@ -5016,6 +5119,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = {
 	{ pci_dev_acpi_reset, .name = "acpi" },
 	{ pcie_reset_flr, .name = "flr" },
 	{ pci_af_flr, .name = "af_flr" },
+	{ cxl_reset, .name = "cxl_reset" },
 	{ pci_pm_reset, .name = "pm" },
 	{ pci_reset_bus_function, .name = "bus" },
 	{ cxl_reset_bus_function, .name = "cxl_bus" },
include/linux/pci.h +9 -1
index 864775651c6f..4a8c4767db6e 100644
@@ -51,7 +51,7 @@
 			       PCI_STATUS_PARITY)
 
 /* Number of reset methods used in pci_reset_fn_methods array in pci.c */
-#define PCI_NUM_RESET_METHODS 8
+#define PCI_NUM_RESET_METHODS 9
 
 #define PCI_RESET_PROBE		true
 #define PCI_RESET_DO_RESET	false
@@ -1464,6 +1464,14 @@ int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size,
 
 int pci_select_bars(struct pci_dev *dev, unsigned long flags);
 bool pci_device_is_present(struct pci_dev *pdev);
+#ifdef CONFIG_CXL_PCI
+bool cxl_is_type2_device(struct pci_dev *dev);
+#else
+static inline bool cxl_is_type2_device(struct pci_dev *dev)
+{
+	return false;
+}
+#endif
 void pci_ignore_hotplug(struct pci_dev *dev);
 struct pci_dev *pci_real_dma_dev(struct pci_dev *dev);
 int pci_status_get_and_clear_errors(struct pci_dev *pdev);

--
2.34.1

maintainer
#9 lore raw

Re: [PATCH v4 04/10] PCI: add CXL reset method

Quoted 8 lines · On 1/20/26 3:26 PM, [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Add a PCI reset method "cxl_reset" that drives the CXL reset sequence using DVSEC controls and timeout encoding. The method is restricted to Type 2 devices, limiting the scope of the changes.

Don't think this is needed if it's kbot issue found during the series postings.

Quoted 11 lines
---
 drivers/pci/pci.c   | 104 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h |  10 ++++-
 2 files changed, 113 insertions(+), 1 deletion(-)
drivers/pci/pci.c +0 -0
index 8bb07e253646..e2d5ff25ab67 100644

drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4895

Quoted 4 lines
drivers/pci/pci.c +1 -0
@@ -4892,6 +4892,109 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)
 	return pci_parent_bus_reset(dev, probe);
 }
 
+static int cxl_reset_init(struct pci_dev *dev, u16 dvsec)

cxl_dev_reset() to go with existing reset method naming?

drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4901

Quoted 6 lines
drivers/pci/pci.c +6 -0
+{
+	/*
+	 * Timeout values ref CXL Spec v3.2 Ch 8 Control and Status Registers,
+	 * under section 8.1.3.1 DVSEC CXL Capability.
+	 */
+	u32 reset_timeouts_ms[] = { 10, 100, 1000, 10000, 100000 };

Should this be const?

Quoted 68 lines
drivers/pci/pci.c +68 -0
+	u16 reg;
+	u32 timeout_ms;
+	int rc, ind;
+
+	/* Check if CXL Reset MEM CLR is supported. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CAP_OFFSET, &reg);
+	if (rc)
+		return rc;
+
+	if (reg & CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE) {
+		rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET,
+					  &reg);
+		if (rc)
+			return rc;
+
+		reg |= CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE;
+		pci_write_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, reg);
+	}
+
+	/* Read timeout value. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CAP_OFFSET, &reg);
+	if (rc)
+		return rc;
+	ind = FIELD_GET(CXL_DVSEC_CXL_RST_TIMEOUT_MASK, reg);
+	timeout_ms = reset_timeouts_ms[ind];
+
+	/* Write reset config. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &reg);
+	if (rc)
+		return rc;
+
+	reg |= CXL_DVSEC_INIT_CXL_RESET;
+	pci_write_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, reg);
+
+	/* Wait till timeout and then check reset status is complete. */
+	msleep(timeout_ms);
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_STATUS2_OFFSET, &reg);
+	if (rc)
+		return rc;
+	if (reg & CXL_DVSEC_CXL_RESET_ERR ||
+	    ~reg & CXL_DVSEC_CXL_RST_COMPLETE)
+		return -ETIMEDOUT;
+
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &reg);
+	if (rc)
+		return rc;
+	reg &= (~CXL_DVSEC_DISABLE_CACHING);
+	pci_write_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, reg);
+
+	return 0;
+}
+
+/**
+ * cxl_reset - initiate a cxl reset
+ * @dev: device to reset
+ * @probe: if true, return 0 if device can be reset this way
+ *
+ * Initiate a cxl reset on @dev.
+ */
+static int cxl_reset(struct pci_dev *dev, bool probe)
+{
+	u16 dvsec, reg;
+	int rc;
+
+	dvsec = pci_find_dvsec_capability(dev, PCI_VENDOR_ID_CXL,
+					  CXL_DVSEC_PCIE_DEVICE);
+	if (!dvsec)
+		return -ENOTTY;

drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4979

Quoted 10 lines
drivers/pci/pci.c +10 -0
+
+	/* Check if CXL Reset is supported. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CAP_OFFSET, &reg);
+	if (rc)
+		return -ENOTTY;
+
+	if ((reg & CXL_DVSEC_CXL_RST_CAPABLE) == 0)
+		return -ENOTTY;
+
+#if !IS_REACHABLE(CONFIG_CXL_PCI)

Does this not require CONFIG_CXL_PCI to be built in (not module) to evaluate to 'y'? Also, instead of adding ifdef in the C code, maybe create a helper function for the remaining code within the function below and add an ifdef in the header depending on the config?

Although I think you may want to define cxl_is_type2_device() differently rather than relying on cxl_pci driver data. It probably should check the device type via config space instead to lessen the complications.

DJ

Quoted 60 lines
drivers/pci/pci.c +19 -0
+	return -ENOTTY;
+#endif
+
+	/*
+	 * Expose CXL reset for Type 2 devices.
+	 */
+	if (!cxl_is_type2_device(dev))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	if (!pci_wait_for_pending_transaction(dev))
+		pci_err(dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
+
+	return cxl_reset_init(dev, dvsec);
+}
+
 static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)
 {
 	struct pci_dev *bridge;
@@ -5016,6 +5119,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = {
 	{ pci_dev_acpi_reset, .name = "acpi" },
 	{ pcie_reset_flr, .name = "flr" },
 	{ pci_af_flr, .name = "af_flr" },
+	{ cxl_reset, .name = "cxl_reset" },
 	{ pci_pm_reset, .name = "pm" },
 	{ pci_reset_bus_function, .name = "bus" },
 	{ cxl_reset_bus_function, .name = "cxl_bus" },
include/linux/pci.h +9 -2
index 864775651c6f..4a8c4767db6e 100644
@@ -51,7 +51,7 @@
 			       PCI_STATUS_PARITY)
 
 /* Number of reset methods used in pci_reset_fn_methods array in pci.c */
-#define PCI_NUM_RESET_METHODS 8
+#define PCI_NUM_RESET_METHODS 9
 
 #define PCI_RESET_PROBE		true
 #define PCI_RESET_DO_RESET	false
@@ -1464,6 +1464,14 @@ int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size,
 
 int pci_select_bars(struct pci_dev *dev, unsigned long flags);
 bool pci_device_is_present(struct pci_dev *pdev);
+#ifdef CONFIG_CXL_PCI
+bool cxl_is_type2_device(struct pci_dev *dev);
+#else
+static inline bool cxl_is_type2_device(struct pci_dev *dev)
+{
+	return false;
+}
+#endif
 void pci_ignore_hotplug(struct pci_dev *dev);
 struct pci_dev *pci_real_dma_dev(struct pci_dev *dev);
 int pci_status_get_and_clear_errors(struct pci_dev *pdev);
--

2.34.1

#10 lore raw

Re: [PATCH v4 04/10] PCI: add CXL reset method

In reply to [email protected]
Quoted 36 lines · On Tue, 20 Jan 2026 22:26:04 +0000 [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Add a PCI reset method "cxl_reset" that drives the CXL reset sequence using DVSEC controls and timeout encoding. The method is restricted to Type 2 devices, limiting the scope of the changes.

---
 drivers/pci/pci.c   | 104 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h |  10 ++++-
 2 files changed, 113 insertions(+), 1 deletion(-)
drivers/pci/pci.c +14 -0
index 8bb07e253646..e2d5ff25ab67 100644
@@ -4892,6 +4892,109 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)
 	return pci_parent_bus_reset(dev, probe);
 }
 
+static int cxl_reset_init(struct pci_dev *dev, u16 dvsec)
+{
+	/*
+	 * Timeout values ref CXL Spec v3.2 Ch 8 Control and Status Registers,
+	 * under section 8.1.3.1 DVSEC CXL Capability.
+	 */
+	u32 reset_timeouts_ms[] = { 10, 100, 1000, 10000, 100000 };
+	u16 reg;
+	u32 timeout_ms;
+	int rc, ind;
+
+	/* Check if CXL Reset MEM CLR is supported. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CAP_OFFSET, &reg);
+	if (rc)

drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4918

Quoted 10 lines
drivers/pci/pci.c +10 -0
+		return rc;
+
+	if (reg & CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE) {
+		rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET,
+					  &reg);
+		if (rc)
+			return rc;
+
+		reg |= CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE;
+		pci_write_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, reg);

Be consistent on checking for errors on these pci config accesses.

Pity there isn't a pci_clear_and_set_config_word() like the dword one.

drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4929

Quoted 11 lines
drivers/pci/pci.c +11 -0
+	}
+
+	/* Read timeout value. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CAP_OFFSET, &reg);
+	if (rc)
+		return rc;
+	ind = FIELD_GET(CXL_DVSEC_CXL_RST_TIMEOUT_MASK, reg);
+	timeout_ms = reset_timeouts_ms[ind];
+
+	/* Write reset config. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &reg);

Separate writes needed for this and the MEM_CLR_ENABLE? If not, refactoring to build it up as one value to be written would be good. If separate write is needed then add a comment to make that clear and avoid someone 'tidying' this up later.

Quoted 4 lines
drivers/pci/pci.c +4 -0
+	if (rc)
+		return rc;
+
+	reg |= CXL_DVSEC_INIT_CXL_RESET;

drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4943

Quoted 10 lines
drivers/pci/pci.c +10 -0
+	pci_write_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, reg);
+
+	/* Wait till timeout and then check reset status is complete. */
+	msleep(timeout_ms);
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_STATUS2_OFFSET, &reg);
+	if (rc)
+		return rc;
+	if (reg & CXL_DVSEC_CXL_RESET_ERR ||
+	    ~reg & CXL_DVSEC_CXL_RST_COMPLETE)
+		return -ETIMEDOUT;

I'd split these two conditions. If we saw an reset_err it probably wasn't a timeout so a different return code would be good to indicate that.

drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4948

Quoted 5 lines
drivers/pci/pci.c +5 -0
+
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &reg);
+	if (rc)
+		return rc;
+	reg &= (~CXL_DVSEC_DISABLE_CACHING);

Brackets not needed.

Quoted 26 lines
drivers/pci/pci.c +26 -0
+	pci_write_config_word(dev, dvsec + CXL_DVSEC_CTRL2_OFFSET, reg);
+
+	return 0;
+}
+
+/**
+ * cxl_reset - initiate a cxl reset
+ * @dev: device to reset
+ * @probe: if true, return 0 if device can be reset this way
+ *
+ * Initiate a cxl reset on @dev.
+ */
+static int cxl_reset(struct pci_dev *dev, bool probe)
+{
+	u16 dvsec, reg;
+	int rc;
+
+	dvsec = pci_find_dvsec_capability(dev, PCI_VENDOR_ID_CXL,
+					  CXL_DVSEC_PCIE_DEVICE);
+	if (!dvsec)
+		return -ENOTTY;
+
+	/* Check if CXL Reset is supported. */
+	rc = pci_read_config_word(dev, dvsec + CXL_DVSEC_CAP_OFFSET, &reg);
+	if (rc)
+		return -ENOTTY;

drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4984

Quoted 10 lines
drivers/pci/pci.c +10 -0
+
+	if ((reg & CXL_DVSEC_CXL_RST_CAPABLE) == 0)
+		return -ENOTTY;
+
+#if !IS_REACHABLE(CONFIG_CXL_PCI)
+	return -ENOTTY;
+#endif
+
+	/*
+	 * Expose CXL reset for Type 2 devices.

Agree with Dave. Good to avoid linking to the CXL_PCI driver if possible.

Quoted 45 lines
drivers/pci/pci.c +12 -0
+	 */
+	if (!cxl_is_type2_device(dev))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	if (!pci_wait_for_pending_transaction(dev))
+		pci_err(dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
+
+	return cxl_reset_init(dev, dvsec);
+}
 
include/linux/pci.h +9 -2
index 864775651c6f..4a8c4767db6e 100644
@@ -51,7 +51,7 @@
 			       PCI_STATUS_PARITY)
 
 /* Number of reset methods used in pci_reset_fn_methods array in pci.c */
-#define PCI_NUM_RESET_METHODS 8
+#define PCI_NUM_RESET_METHODS 9
 
 #define PCI_RESET_PROBE		true
 #define PCI_RESET_DO_RESET	false
@@ -1464,6 +1464,14 @@ int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size,
 
 int pci_select_bars(struct pci_dev *dev, unsigned long flags);
 bool pci_device_is_present(struct pci_dev *pdev);
+#ifdef CONFIG_CXL_PCI
+bool cxl_is_type2_device(struct pci_dev *dev);
+#else
+static inline bool cxl_is_type2_device(struct pci_dev *dev)
+{
+	return false;
+}
+#endif
 void pci_ignore_hotplug(struct pci_dev *dev);
 struct pci_dev *pci_real_dma_dev(struct pci_dev *dev);
 int pci_status_get_and_clear_errors(struct pci_dev *pdev);
--

2.34.1

#11 lore raw

Re: [PATCH v4 04/10] PCI: add CXL reset method

In reply to [email protected]
kernel test robot noticed the following build errors:

Hi,

kernel test robot noticed the following build errors:

[auto build test ERROR on pci/next]
[also build test ERROR on pci/for-linus linus/master v6.19-rc6] [cannot apply to next-20260122]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/smadhavan-nvidia-com/cxl-move-DVSEC-defines-to-cxl-pci-header/20260121-071852
base:   https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next
patch link:    https://lore.kernel.org/r/20260120222610.2227109-5-smadhavan%40nvidia.com
patch subject: [PATCH v4 04/10] PCI: add CXL reset method
config: openrisc-allmodconfig (https://download.01.org/0day-ci/archive/20260123/[email protected]/config)
compiler: or1k-linux-gcc (GCC) 15.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260123/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> drivers/cxl/pci.c:1085:6: error: redefinition of 'cxl_is_type2_device'

1085 | bool cxl_is_type2_device(struct pci_dev *pdev)
| ^~~~~~~~~~~~~~~~~~~
In file included from drivers/cxl/pci.c:11:
include/linux/pci.h:1474:20: note: previous definition of 'cxl_is_type2_device' with type 'bool(struct pci_dev *)' {aka '_Bool(struct pci_dev *)'}
1474 | static inline bool cxl_is_type2_device(struct pci_dev *dev)
| ^~~~~~~~~~~~~~~~~~~

vim +/cxl_is_type2_device +1085 drivers/cxl/pci.c

2905cb5236cba6 Dan Williams 2022-11-29 1084

b345d117d51557 Srirangan Madhavan 2026-01-20 @1085  bool cxl_is_type2_device(struct pci_dev *pdev)
b345d117d51557 Srirangan Madhavan 2026-01-20  1086  {
b345d117d51557 Srirangan Madhavan 2026-01-20  1087  	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
b345d117d51557 Srirangan Madhavan 2026-01-20  1088  
b345d117d51557 Srirangan Madhavan 2026-01-20  1089  	if (!cxlds)
b345d117d51557 Srirangan Madhavan 2026-01-20  1090  		return false;
b345d117d51557 Srirangan Madhavan 2026-01-20  1091  
b345d117d51557 Srirangan Madhavan 2026-01-20  1092  	return cxlds->type == CXL_DEVTYPE_DEVMEM;
b345d117d51557 Srirangan Madhavan 2026-01-20  1093  }
b345d117d51557 Srirangan Madhavan 2026-01-20  1094  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#12 lore raw

PATCH v4 5/10 cxl: add reset prepare and region teardown

In reply to [email protected]

From: Srirangan Madhavan <[email protected]>

Prepare a Type 2 device for cxl_reset by validating memory is offline, flushing device caches for region participants, and tearing down decoders under cxl_region_rwsem. The lock stays held across reset to prevent new region creation while reset is in progress.

---
 drivers/cxl/pci.c | 214 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 214 insertions(+)
drivers/cxl/pci.c +214 -0
index b562e607ec46..e4134162e82a 100644
@@ -1085,6 +1085,220 @@ bool cxl_is_type2_device(struct pci_dev *pdev)
 	return cxlds->type == CXL_DEVTYPE_DEVMEM;
 }
 
+static int cxl_check_region_driver_bound(struct device *dev, void *data)
+{
+	struct cxl_decoder *cxld = to_cxl_decoder(dev);
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	guard(rwsem_read)(&cxl_region_rwsem);
+	if (cxld->region && cxld->region->driver)
+		return -EBUSY;
+
+	return 0;
+}
+
+static int cxl_decoder_kill_region_iter(struct device *dev, void *data)
+{
+	struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev);
+	int rc;
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	if (!cxled->cxld.region)
+		return 0;
+
+	cxl_decoder_kill_region_locked(cxled);
+
+	rc = device_for_each_child(&cxled->cxld.dev, NULL,
+				   cxl_check_region_driver_bound);
+	if (rc)
+		return rc;
+
+	return 0;
+}
+
+static int cxl_device_cache_wb_invalidate(struct pci_dev *pdev)
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+	u16 reg, val, cap;
+	int dvsec, rc;
+
+	if (!cxlds)
+		return -ENODEV;
+
+	dvsec = cxlds->cxl_dvsec;
+	if (!dvsec)
+		return -ENODEV;
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAP_OFFSET, &cap);
+	if (rc)
+		return rc;
+
+	if (!(cap & CXL_DVSEC_CACHE_WBI_CAPABLE))
+		return 1;
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &val);
+	if (rc)
+		return rc;
+
+	val |= CXL_DVSEC_INIT_CACHE_WBI;
+	rc = pci_write_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, val);
+	if (rc)
+		return rc;
+
+	do {
+		rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_STATUS2_OFFSET, &reg);
+		if (rc)
+			return rc;
+	} while (!(reg & CXL_DVSEC_CACHE_INVALID));
+
+	return 0;
+}
+
+static int cxl_region_flush_device_caches(struct device *dev, void *data)
+{
+	struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev);
+	struct cxl_region *cxlr = cxled->cxld.region;
+	struct cxl_region_params *p = &cxlr->params;
+	struct pci_dev *target_pdev = data;
+	int i, rc;
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	if (!cxlr || !cxlr->params.res)
+		return 0;
+
+	for (i = 0; i < p->nr_targets; i++) {
+		struct cxl_endpoint_decoder *target_cxled = p->targets[i];
+		struct cxl_memdev *target_cxlmd = cxled_to_memdev(target_cxled);
+		struct cxl_dev_state *target_cxlds = target_cxlmd->cxlds;
+
+		if (!target_cxlds || !target_cxlds->pdev)
+			continue;
+
+		if (target_cxlds->pdev != target_pdev)
+			continue;
+
+		rc = cxl_device_cache_wb_invalidate(target_pdev);
+		if (rc && rc != 1)
+			return rc;
+	}
+
+	return 0;
+}
+
+/**
+ * cxl_reset_prepare_memdev - Prepare CXL device for reset
+ * @pdev: PCI device
+ *
+ * Validates it's safe to reset and tears down regions atomically under lock.
+ * Acquires cxl_region_rwsem and keeps it held throughout reset.
+ *
+ * Return: 0 on success (lock held), -EBUSY if memory online, negative on error
+ */
+static int cxl_reset_prepare_memdev(struct pci_dev *pdev)
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+	struct cxl_memdev *cxlmd;
+	struct cxl_port *endpoint;
+	int rc;
+
+	if (!cxlds)
+		return -ENODEV;
+
+	cxlmd = cxlds->cxlmd;
+	if (!cxlmd)
+		return -ENODEV;
+
+	endpoint = cxlmd->endpoint;
+	if (!endpoint)
+		return 0;
+
+	if (cxl_num_decoders_committed(endpoint) == 0)
+		return 0;
+
+	down_write(&cxl_region_rwsem);
+
+	/* Check and error out if memory is online */
+	rc = device_for_each_child(&endpoint->dev, NULL,
+				   cxl_check_region_driver_bound);
+	if (rc) {
+		up_write(&cxl_region_rwsem);
+		dev_err(&pdev->dev,
+			"Reset blocked: device has active regions with drivers bound\n");
+		return -EBUSY;
+	}
+
+	/* Flush device caches and tear down regions */
+	device_for_each_child(&endpoint->dev, pdev,
+			      cxl_region_flush_device_caches);
+
+	rc = device_for_each_child(&endpoint->dev, NULL,
+				   cxl_decoder_kill_region_iter);
+	if (rc) {
+		up_write(&cxl_region_rwsem);
+		dev_err(&pdev->dev, "Failed to tear down regions: %d\n", rc);
+		return rc;
+	}
+
+	/* Keep cxl_region_rwsem held, released by cleanup function */
+	return 0;
+}
+
+/**
+ * cxl_reset_cleanup_memdev - Release locks after CXL reset
+ * @pdev: PCI device
+ */
+static void cxl_reset_cleanup_memdev(struct pci_dev *pdev)
+{
+	if (lockdep_is_held_type(&cxl_region_rwsem, -1))
+		up_write(&cxl_region_rwsem);
+}
+
+/**
+ * cxl_reset_prepare_device - Prepare CXL device for reset
+ * @pdev: PCI device being reset
+ *
+ * CXL-reset-specific preparation. Validates memory is offline, flushes
+ * device caches, and tears down regions.
+ *
+ * Returns: 0 on success, -EBUSY if memory online, negative on error
+ */
+int cxl_reset_prepare_device(struct pci_dev *pdev)
+{
+	int rc;
+
+	rc = cxl_reset_prepare_memdev(pdev);
+	if (rc) {
+		if (rc == -EBUSY)
+			dev_err(&pdev->dev,
+				"Cannot reset: device has online memory or active regions\n");
+		else
+			dev_err(&pdev->dev,
+				"Failed to prepare device for reset: %d\n", rc);
+		return rc;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");
+
+/**
+ * cxl_reset_cleanup_device - Cleanup after CXL reset
+ * @pdev: PCI device that was reset
+ *
+ * Releases region locks held during reset.
+ */
+void cxl_reset_cleanup_device(struct pci_dev *pdev)
+{
+	cxl_reset_cleanup_memdev(pdev);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_reset_cleanup_device, "CXL");
+
 static void cxl_error_resume(struct pci_dev *pdev)
 {
 	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);

--
2.34.1

#13 lore raw

Re: [PATCH v4 05/10] cxl: add reset prepare and region teardown

Quoted 8 lines · On Tue, 20 Jan 2026 22:26:05 +0000 [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Prepare a Type 2 device for cxl_reset by validating memory is offline, flushing device caches for region participants, and tearing down decoders under cxl_region_rwsem. The lock stays held across reset to prevent new region creation while reset is in progress.

Some minor feedback from a quick look. I'll want to take a closer look when we are closer to merging this.

Quoted 31 lines
---
 drivers/cxl/pci.c | 214 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 214 insertions(+)
drivers/cxl/pci.c +19 -0
index b562e607ec46..e4134162e82a 100644
@@ -1085,6 +1085,220 @@ bool cxl_is_type2_device(struct pci_dev *pdev)
 	return cxlds->type == CXL_DEVTYPE_DEVMEM;
 }
 
+static int cxl_check_region_driver_bound(struct device *dev, void *data)
+{
+	struct cxl_decoder *cxld = to_cxl_decoder(dev);
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	guard(rwsem_read)(&cxl_region_rwsem);
+	if (cxld->region && cxld->region->driver)
+		return -EBUSY;
+
+	return 0;
+}
+
+static int cxl_decoder_kill_region_iter(struct device *dev, void *data)
+{
+	struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev);
+	int rc;
+

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1116

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	if (!cxled->cxld.region)
+		return 0;
+
+	cxl_decoder_kill_region_locked(cxled);
+
+	rc = device_for_each_child(&cxled->cxld.dev, NULL,
+				   cxl_check_region_driver_bound);

return device_for_each_child()

If that doesn't make sense after later patches, fine to leave as it is.

Quoted 15 lines
drivers/cxl/pci.c +15 -0
+	if (rc)
+		return rc;
+
+	return 0;
+}
+
+static int cxl_device_cache_wb_invalidate(struct pci_dev *pdev)
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+	u16 reg, val, cap;
+	int dvsec, rc;
+
+	if (!cxlds)
+		return -ENODEV;
+

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1141

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+	dvsec = cxlds->cxl_dvsec;
+	if (!dvsec)
+		return -ENODEV;
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAP_OFFSET, &cap);
+	if (rc)
+		return rc;
+
+	if (!(cap & CXL_DVSEC_CACHE_WBI_CAPABLE))
+		return 1;

With unusual return value, definitely need docs for this function. Given use below, maybe just return 0?

If there are caches and there is no way to force WB, what does that mean for whether we can reset the device? Feels like maybe this at least deserves a warning print.

My suspicion is that lack of that feature just means there is a device specific way to do it but I'm fine with Linux not supporting that ;)

Quoted 32 lines
drivers/cxl/pci.c +32 -0
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &val);
+	if (rc)
+		return rc;
+
+	val |= CXL_DVSEC_INIT_CACHE_WBI;
+	rc = pci_write_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, val);
+	if (rc)
+		return rc;
+
+	do {
+		rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_STATUS2_OFFSET, &reg);
+		if (rc)
+			return rc;
+	} while (!(reg & CXL_DVSEC_CACHE_INVALID));
+
+	return 0;
+}
+
+static int cxl_region_flush_device_caches(struct device *dev, void *data)
+{
+	struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev);
+	struct cxl_region *cxlr = cxled->cxld.region;
+	struct cxl_region_params *p = &cxlr->params;
+	struct pci_dev *target_pdev = data;
+	int i, rc;
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	if (!cxlr || !cxlr->params.res)
+		return 0;

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1183

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+
+	for (i = 0; i < p->nr_targets; i++) {
+		struct cxl_endpoint_decoder *target_cxled = p->targets[i];
+		struct cxl_memdev *target_cxlmd = cxled_to_memdev(target_cxled);
+		struct cxl_dev_state *target_cxlds = target_cxlmd->cxlds;
+
+		if (!target_cxlds || !target_cxlds->pdev)
+			continue;
+
+		if (target_cxlds->pdev != target_pdev)

Seems like target_pdev == NULL is a bug and if possible should be checked for before doing anything in this function, so you could simplify this as

		if (!target_cxlds || target_clds->pdev != target_pdev)

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1187

Quoted 4 lines
drivers/cxl/pci.c +4 -0
+			continue;
+
+		rc = cxl_device_cache_wb_invalidate(target_pdev);
+		if (rc && rc != 1)

As above, I'm not sure the rc == 1 return is helpful.

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1199

Quoted 12 lines
drivers/cxl/pci.c +12 -0
+			return rc;
+	}
+
+	return 0;
+}
+
+/**
+ * cxl_reset_prepare_memdev - Prepare CXL device for reset
+ * @pdev: PCI device
+ *
+ * Validates it's safe to reset and tears down regions atomically under lock.
+ * Acquires cxl_region_rwsem and keeps it held throughout reset.

That may need some lockdep annotations. Make sure to run a lockdep build.

Quoted 7 lines
drivers/cxl/pci.c +5 -0
+ *
+ * Return: 0 on success (lock held), -EBUSY if memory online, negative on error
+ */
+static int cxl_reset_prepare_memdev(struct pci_dev *pdev)
+{
 
 
maintainer
#14 lore raw

Re: [PATCH v4 05/10] cxl: add reset prepare and region teardown

In reply to [email protected]
Quoted 19 lines · On 1/20/26 3:26 PM, [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Prepare a Type 2 device for cxl_reset by validating memory is offline, flushing device caches for region participants, and tearing down decoders under cxl_region_rwsem. The lock stays held across reset to prevent new region creation while reset is in progress.

---
 drivers/cxl/pci.c | 214 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 214 insertions(+)
drivers/cxl/pci.c +0 -0
index b562e607ec46..e4134162e82a 100644
@@ -1085,6 +1085,220 @@ bool cxl_is_type2_device(struct pci_dev *pdev)
 	return cxlds->type == CXL_DEVTYPE_DEVMEM;
 }

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1096

Quoted 10 lines
drivers/cxl/pci.c +9 -0
 
+static int cxl_check_region_driver_bound(struct device *dev, void *data)
+{
+	struct cxl_decoder *cxld = to_cxl_decoder(dev);
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	guard(rwsem_read)(&cxl_region_rwsem);
+	if (cxld->region && cxld->region->driver)

I think you may need to take the region device lock before checking the region driver. The cxl_region_rwsem won't protect you from region device driver un/binding. While we know cxl_pci is bound since the callback is on a PCI device, there's no gaurantee that the region driver is.

Quoted 7 lines
drivers/cxl/pci.c +7 -0
+		return -EBUSY;
+
+	return 0;
+}
+
+static int cxl_decoder_kill_region_iter(struct device *dev, void *data)
+{

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1113

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+	struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev);
+	int rc;
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	if (!cxled->cxld.region)
+		return 0;
+
+	cxl_decoder_kill_region_locked(cxled);

Where does this function come from? It's not present upstream.

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1116

Quoted 3 lines
drivers/cxl/pci.c +3 -0
+
+	rc = device_for_each_child(&cxled->cxld.dev, NULL,
+				   cxl_check_region_driver_bound);

Why check if driver is bound after region is killed? Also, the check happens in cxl_reset_prepare_memdev() before cxl_decoder_kill_region_iter() is called, and if true fails the cxl_reset_prepare_memdev(). So at this point there shouldn't be any drivers bound right?

Quoted 15 lines
drivers/cxl/pci.c +15 -0
+	if (rc)
+		return rc;
+
+	return 0;
+}
+
+static int cxl_device_cache_wb_invalidate(struct pci_dev *pdev)
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+	u16 reg, val, cap;
+	int dvsec, rc;
+
+	if (!cxlds)
+		return -ENODEV;
+

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1141

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+	dvsec = cxlds->cxl_dvsec;
+	if (!dvsec)
+		return -ENODEV;
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAP_OFFSET, &cap);
+	if (rc)
+		return rc;
+
+	if (!(cap & CXL_DVSEC_CACHE_WBI_CAPABLE))
+		return 1;

Why return 1 when it's ignored anyhow? Can just return 0.

Quoted 52 lines
drivers/cxl/pci.c +52 -0
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &val);
+	if (rc)
+		return rc;
+
+	val |= CXL_DVSEC_INIT_CACHE_WBI;
+	rc = pci_write_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, val);
+	if (rc)
+		return rc;
+
+	do {
+		rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_STATUS2_OFFSET, &reg);
+		if (rc)
+			return rc;
+	} while (!(reg & CXL_DVSEC_CACHE_INVALID));
+
+	return 0;
+}
+
+static int cxl_region_flush_device_caches(struct device *dev, void *data)
+{
+	struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev);
+	struct cxl_region *cxlr = cxled->cxld.region;
+	struct cxl_region_params *p = &cxlr->params;
+	struct pci_dev *target_pdev = data;
+	int i, rc;
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	if (!cxlr || !cxlr->params.res)
+		return 0;
+
+	for (i = 0; i < p->nr_targets; i++) {
+		struct cxl_endpoint_decoder *target_cxled = p->targets[i];
+		struct cxl_memdev *target_cxlmd = cxled_to_memdev(target_cxled);
+		struct cxl_dev_state *target_cxlds = target_cxlmd->cxlds;
+
+		if (!target_cxlds || !target_cxlds->pdev)
+			continue;
+
+		if (target_cxlds->pdev != target_pdev)
+			continue;
+
+		rc = cxl_device_cache_wb_invalidate(target_pdev);
+		if (rc && rc != 1)
+			return rc;
+	}
+
+	return 0;
+}
+

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1203

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+/**
+ * cxl_reset_prepare_memdev - Prepare CXL device for reset
+ * @pdev: PCI device
+ *
+ * Validates it's safe to reset and tears down regions atomically under lock.
+ * Acquires cxl_region_rwsem and keeps it held throughout reset.
+ *
+ * Return: 0 on success (lock held), -EBUSY if memory online, negative on error
+ */
+static int cxl_reset_prepare_memdev(struct pci_dev *pdev)

Given the function name, shouldn't this pass in a cxl_memdev?

Quoted 9 lines
drivers/cxl/pci.c +9 -0
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+	struct cxl_memdev *cxlmd;
+	struct cxl_port *endpoint;
+	int rc;
+
+	if (!cxlds)
+		return -ENODEV;
+

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1222

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+	cxlmd = cxlds->cxlmd;
+	if (!cxlmd)
+		return -ENODEV;
+
+	endpoint = cxlmd->endpoint;
+	if (!endpoint)
+		return 0;
+
+	if (cxl_num_decoders_committed(endpoint) == 0)
+		return 0;

Without any decoders committed, this endpoint shouldn't be part of any region correct? Why bother checking driver bound after this. Seems overkill.

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1226

Quoted 4 lines
drivers/cxl/pci.c +4 -0
+
+	down_write(&cxl_region_rwsem);
+
+	/* Check and error out if memory is online */

Having cxl_region driver bound does not mean memory is online in the sense if it's hot-plugged to the memory subsystem. Are you looking for it to be not system memory here, or that cxl has claimed the region, or if there's just a hardware committed region?

Quoted 12 lines
drivers/cxl/pci.c +12 -0
+	rc = device_for_each_child(&endpoint->dev, NULL,
+				   cxl_check_region_driver_bound);
+	if (rc) {
+		up_write(&cxl_region_rwsem);
+		dev_err(&pdev->dev,
+			"Reset blocked: device has active regions with drivers bound\n");
+		return -EBUSY;
+	}
+
+	/* Flush device caches and tear down regions */
+	device_for_each_child(&endpoint->dev, pdev,
+			      cxl_region_flush_device_caches);

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1248

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+
+	rc = device_for_each_child(&endpoint->dev, NULL,
+				   cxl_decoder_kill_region_iter);
+	if (rc) {
+		up_write(&cxl_region_rwsem);
+		dev_err(&pdev->dev, "Failed to tear down regions: %d\n", rc);
+		return rc;
+	}
+
+	/* Keep cxl_region_rwsem held, released by cleanup function */

May want to annotate the function with __acquires()

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1258

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+	return 0;
+}
+
+/**
+ * cxl_reset_cleanup_memdev - Release locks after CXL reset
+ * @pdev: PCI device
+ */
+static void cxl_reset_cleanup_memdev(struct pci_dev *pdev)
+{
+	if (lockdep_is_held_type(&cxl_region_rwsem, -1))

why check when it's expected that the lock is held? Probably should have lockdep_assert_held() instead. Also may need a __releases() to annotate that this function releases the lock acquires in cxl_reset_prepare_memdev().

DJ

Quoted 49 lines
drivers/cxl/pci.c +43 -1
+		up_write(&cxl_region_rwsem);
+}
+
+/**
+ * cxl_reset_prepare_device - Prepare CXL device for reset
+ * @pdev: PCI device being reset
+ *
+ * CXL-reset-specific preparation. Validates memory is offline, flushes
+ * device caches, and tears down regions.
+ *
+ * Returns: 0 on success, -EBUSY if memory online, negative on error
+ */
+int cxl_reset_prepare_device(struct pci_dev *pdev)
+{
+	int rc;
+
+	rc = cxl_reset_prepare_memdev(pdev);
+	if (rc) {
+		if (rc == -EBUSY)
+			dev_err(&pdev->dev,
+				"Cannot reset: device has online memory or active regions\n");
+		else
+			dev_err(&pdev->dev,
+				"Failed to prepare device for reset: %d\n", rc);
+		return rc;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");
+
+/**
+ * cxl_reset_cleanup_device - Cleanup after CXL reset
+ * @pdev: PCI device that was reset
+ *
+ * Releases region locks held during reset.
+ */
+void cxl_reset_cleanup_device(struct pci_dev *pdev)
+{
+	cxl_reset_cleanup_memdev(pdev);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_reset_cleanup_device, "CXL");
+
 static void cxl_error_resume(struct pci_dev *pdev)
 {
 	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
--

2.34.1

#15 lore raw

PATCH v4 6/10 PCI: wire CXL reset prepare/cleanup

In reply to [email protected]

From: Srirangan Madhavan <[email protected]>

Wire CXL reset preparation and cleanup into the PCI CXL reset path. The flow now validates/offlines regions, performs teardown and cache flushes, then releases the lock on completion or error. This keeps the common reset_prepare flow intact while adding cxl_reset-specific quiesce logic.

---
 drivers/pci/pci.c   | 19 ++++++++++++++++++-
 include/linux/pci.h | 11 +++++++++++
 2 files changed, 29 insertions(+), 1 deletion(-)
drivers/pci/pci.c +18 -1
index e2d5ff25ab67..18047c893b0c 100644
@@ -4989,10 +4989,27 @@ static int cxl_reset(struct pci_dev *dev, bool probe)
 	if (probe)
 		return 0;
 
+	/*
+	 * CXL-reset-specific preparation: validate memory offline,
+	 * tear down regions, flush device caches.
+	 */
+	rc = cxl_reset_prepare_device(dev);
+	if (rc)
+		return rc;
+
 	if (!pci_wait_for_pending_transaction(dev))
 		pci_err(dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
 
-	return cxl_reset_init(dev, dvsec);
+	rc = cxl_reset_init(dev, dvsec);
+	if (rc)
+		goto out_cleanup;
+
+	cxl_reset_cleanup_device(dev);
+	return 0;
+
+out_cleanup:
+	cxl_reset_cleanup_device(dev);
+	return rc;
 }
 
 static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)
include/linux/pci.h +11 -0
index 4a8c4767db6e..c074c2040b28 100644
@@ -1466,11 +1466,22 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags);
 bool pci_device_is_present(struct pci_dev *pdev);
 #ifdef CONFIG_CXL_PCI
 bool cxl_is_type2_device(struct pci_dev *dev);
+int cxl_reset_prepare_device(struct pci_dev *pdev);
+void cxl_reset_cleanup_device(struct pci_dev *pdev);
 #else
 static inline bool cxl_is_type2_device(struct pci_dev *dev)
 {
 	return false;
 }
+
+static inline int cxl_reset_prepare_device(struct pci_dev *pdev)
+{
+	return 0;
+}
+
+static inline void cxl_reset_cleanup_device(struct pci_dev *pdev)
+{
+}
 #endif
 void pci_ignore_hotplug(struct pci_dev *dev);
 struct pci_dev *pci_real_dma_dev(struct pci_dev *dev);

--
2.34.1

maintainer
#16 lore raw

Re: [PATCH v4 06/10] PCI: wire CXL reset prepare/cleanup

Quoted 6 lines · On 1/20/26 3:26 PM, [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Wire CXL reset preparation and cleanup into the PCI CXL reset path. The flow now validates/offlines regions, performs teardown and cache flushes, then releases the lock on completion or error. This keeps the common reset_prepare flow intact while adding cxl_reset-specific quiesce logic.

Can this be moved to the ->reset_prepare() callback of 'pci_error_handlers' rather than directly wire it into PCI core code? I don't think we want build a dependency of cxl core to the PCI core.

Quoted 70 lines
---
 drivers/pci/pci.c   | 19 ++++++++++++++++++-
 include/linux/pci.h | 11 +++++++++++
 2 files changed, 29 insertions(+), 1 deletion(-)
drivers/pci/pci.c +18 -1
index e2d5ff25ab67..18047c893b0c 100644
@@ -4989,10 +4989,27 @@ static int cxl_reset(struct pci_dev *dev, bool probe)
 	if (probe)
 		return 0;
 
+	/*
+	 * CXL-reset-specific preparation: validate memory offline,
+	 * tear down regions, flush device caches.
+	 */
+	rc = cxl_reset_prepare_device(dev);
+	if (rc)
+		return rc;
+
 	if (!pci_wait_for_pending_transaction(dev))
 		pci_err(dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
 
-	return cxl_reset_init(dev, dvsec);
+	rc = cxl_reset_init(dev, dvsec);
+	if (rc)
+		goto out_cleanup;
+
+	cxl_reset_cleanup_device(dev);
+	return 0;
+
+out_cleanup:
+	cxl_reset_cleanup_device(dev);
+	return rc;
 }
 
 static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)
include/linux/pci.h +11 -1
index 4a8c4767db6e..c074c2040b28 100644
@@ -1466,11 +1466,22 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags);
 bool pci_device_is_present(struct pci_dev *pdev);
 #ifdef CONFIG_CXL_PCI
 bool cxl_is_type2_device(struct pci_dev *dev);
+int cxl_reset_prepare_device(struct pci_dev *pdev);
+void cxl_reset_cleanup_device(struct pci_dev *pdev);
 #else
 static inline bool cxl_is_type2_device(struct pci_dev *dev)
 {
 	return false;
 }
+
+static inline int cxl_reset_prepare_device(struct pci_dev *pdev)
+{
+	return 0;
+}
+
+static inline void cxl_reset_cleanup_device(struct pci_dev *pdev)
+{
+}
 #endif
 void pci_ignore_hotplug(struct pci_dev *dev);
 struct pci_dev *pci_real_dma_dev(struct pci_dev *dev);
--

2.34.1

#17 lore raw

Re: [PATCH v4 06/10] PCI: wire CXL reset prepare/cleanup

Quoted 7 lines · On 1/21/26, 2:14 PM, "Dave Jiang" <[email protected]…
Quoted 4 lines

Wire CXL reset preparation and cleanup into the PCI CXL reset path. The flow now validates/offlines regions, performs teardown and cache flushes, then releases the lock on completion or error. This keeps the common reset_prepare flow intact while adding cxl_reset-specific quiesce logic.

Can this be moved to the ->reset_prepare() callback of 'pci_error_handlers' rather than directly wire it into PCI core code? I don't think we want build a dependency of cxl core to the PCI core.

I understand the current implementation is not ideal. But I opted not to insert the call into ->reset_prepare() as pci_error_handler would be invoked for all the other reset methods (cxl_bus, FLR, etc). Would it be okay to include the same reset prepare steps (i.e region teardown, cache flush, etc) in the common reset prepare?

Thank you.

maintainer
#18 lore raw

Re: [PATCH v4 06/10] PCI: wire CXL reset prepare/cleanup

Quoted 10 lines · Srirangan Madhavan
Quoted 5 lines · On 1/21/26, 2:14 PM, "Dave Jiang" <[email protected]…
Quoted 4 lines

Wire CXL reset preparation and cleanup into the PCI CXL reset path. The flow now validates/offlines regions, performs teardown and cache flushes, then releases the lock on completion or error. This keeps the common reset_prepare flow intact while adding cxl_reset-specific quiesce logic.

Quoted 5 lines
Quoted 2 lines

Can this be moved to the ->reset_prepare() callback of 'pci_error_handlers' rather than directly wire it into PCI core code? I don't think we want build a dependency of cxl core to the PCI core.

I understand the current implementation is not ideal. But I opted not to insert the call into ->reset_prepare() as pci_error_handler would be invoked for all the other reset methods (cxl_bus, FLR, etc). Would it be okay to include the same reset prepare steps (i.e region teardown, cache flush, etc) in the common reset prepare?

See my suggestion reply against patch 10/10. Maybe we shouldn't be going to PCI reset path.

Quoted 3 lines

Thank you.

#19 lore raw

Re: [PATCH v4 06/10] PCI: wire CXL reset prepare/cleanup

In reply to [email protected]
kernel test robot noticed the following build errors:

Hi,

kernel test robot noticed the following build errors:

[auto build test ERROR on pci/next]
[also build test ERROR on pci/for-linus linus/master v6.19-rc6] [cannot apply to next-20260123]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/smadhavan-nvidia-com/cxl-move-DVSEC-defines-to-cxl-pci-header/20260121-071852
base:   https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next
patch link:    https://lore.kernel.org/r/20260120222610.2227109-7-smadhavan%40nvidia.com
patch subject: [PATCH v4 06/10] PCI: wire CXL reset prepare/cleanup
config: openrisc-allmodconfig (https://download.01.org/0day-ci/archive/20260124/[email protected]/config)
compiler: or1k-linux-gcc (GCC) 15.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260124/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

drivers/cxl/pci.c:1085:6: error: redefinition of 'cxl_is_type2_device'
1085 | bool cxl_is_type2_device(struct pci_dev *pdev)
| ^~~~~~~~~~~~~~~~~~~
In file included from drivers/cxl/pci.c:11:
include/linux/pci.h:1476:20: note: previous definition of 'cxl_is_type2_device' with type 'bool(struct pci_dev *)' {aka '_Bool(struct pci_dev *)'}
1476 | static inline bool cxl_is_type2_device(struct pci_dev *dev)
| ^~~~~~~~~~~~~~~~~~~
drivers/cxl/pci.c: In function 'cxl_check_region_driver_bound':
drivers/cxl/pci.c:1102:28: error: 'cxl_region_rwsem' undeclared (first use in this function); did you mean 'cxl_region_ref'?

    1102 |         guard(rwsem_read)(&cxl_region_rwsem);
         |                            ^~~~~~~~~~~~~~~~
         |                            cxl_region_ref
   drivers/cxl/pci.c:1102:28: note: each undeclared identifier is reported only once for each function it appears in
   drivers/cxl/pci.c:1103:41: error: 'struct cxl_region' has no member named 'driver'
    1103 |         if (cxld->region && cxld->region->driver)
         |                                         ^~
   drivers/cxl/pci.c: In function 'cxl_decoder_kill_region_iter':
   drivers/cxl/pci.c:1120:9: error: implicit declaration of function 'cxl_decoder_kill_region_locked'; did you mean 'cxl_decoder_kill_region_iter'? [-Wimplicit-function-declaration]
    1120 |         cxl_decoder_kill_region_locked(cxled);
         |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         |         cxl_decoder_kill_region_iter
   drivers/cxl/pci.c: In function 'cxl_region_flush_device_caches':
   drivers/cxl/pci.c:1187:53: error: 'struct cxl_dev_state' has no member named 'pdev'; did you mean 'dev'?
    1187 |                 if (!target_cxlds || !target_cxlds->pdev)
         |                                                     ^~~~
         |                                                     dev
   drivers/cxl/pci.c:1190:35: error: 'struct cxl_dev_state' has no member named 'pdev'; did you mean 'dev'?
    1190 |                 if (target_cxlds->pdev != target_pdev)
         |                                   ^~~~
         |                                   dev
   drivers/cxl/pci.c: In function 'cxl_reset_prepare_memdev':
   drivers/cxl/pci.c:1231:21: error: 'cxl_region_rwsem' undeclared (first use in this function); did you mean 'cxl_region_ref'?
    1231 |         down_write(&cxl_region_rwsem);
         |                     ^~~~~~~~~~~~~~~~
         |                     cxl_region_ref
   In file included from include/linux/spinlock.h:63,
                    from include/linux/mmzone.h:8,
                    from include/linux/gfp.h:7,
                    from include/linux/mm.h:7,
                    from arch/openrisc/include/asm/pgalloc.h:20,
                    from arch/openrisc/include/asm/io.h:18,
                    from include/linux/io.h:12,
                    from include/linux/io-64-nonatomic-lo-hi.h:5,
                    from drivers/cxl/pci.c:4:
   drivers/cxl/pci.c: In function 'cxl_reset_cleanup_memdev':
   drivers/cxl/pci.c:1265:35: error: 'cxl_region_rwsem' undeclared (first use in this function); did you mean 'cxl_region_ref'?
    1265 |         if (lockdep_is_held_type(&cxl_region_rwsem, -1))
         |                                   ^~~~~~~~~~~~~~~~
   include/linux/lockdep.h:253:61: note: in definition of macro 'lockdep_is_held_type'
     253 | #define lockdep_is_held_type(lock, r)   lock_is_held_type(&(lock)->dep_map, (r))
         |                                                             ^~~~
   drivers/cxl/pci.c: At top level:
>> drivers/cxl/pci.c:1278:5: error: redefinition of 'cxl_reset_prepare_device'
    1278 | int cxl_reset_prepare_device(struct pci_dev *pdev)
         |     ^~~~~~~~~~~~~~~~~~~~~~~~
   include/linux/pci.h:1481:19: note: previous definition of 'cxl_reset_prepare_device' with type 'int(struct pci_dev *)'
    1481 | static inline int cxl_reset_prepare_device(struct pci_dev *pdev)
         |                   ^~~~~~~~~~~~~~~~~~~~~~~~
>> drivers/cxl/pci.c:1303:6: error: redefinition of 'cxl_reset_cleanup_device'

1303 | void cxl_reset_cleanup_device(struct pci_dev *pdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~
include/linux/pci.h:1486:20: note: previous definition of 'cxl_reset_cleanup_device' with type 'void(struct pci_dev *)'
1486 | static inline void cxl_reset_cleanup_device(struct pci_dev *pdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~

vim +/cxl_reset_prepare_device +1278 drivers/cxl/pci.c

76e9ed09acad72 Srirangan Madhavan 2026-01-20  1268  
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1269  /**
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1270   * cxl_reset_prepare_device - Prepare CXL device for reset
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1271   * @pdev: PCI device being reset
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1272   *
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1273   * CXL-reset-specific preparation. Validates memory is offline, flushes
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1274   * device caches, and tears down regions.
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1275   *
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1276   * Returns: 0 on success, -EBUSY if memory online, negative on error
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1277   */
76e9ed09acad72 Srirangan Madhavan 2026-01-20 @1278  int cxl_reset_prepare_device(struct pci_dev *pdev)
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1279  {
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1280  	int rc;
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1281  
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1282  	rc = cxl_reset_prepare_memdev(pdev);
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1283  	if (rc) {
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1284  		if (rc == -EBUSY)
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1285  			dev_err(&pdev->dev,
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1286  				"Cannot reset: device has online memory or active regions\n");
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1287  		else
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1288  			dev_err(&pdev->dev,
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1289  				"Failed to prepare device for reset: %d\n", rc);
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1290  		return rc;
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1291  	}
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1292  
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1293  	return 0;
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1294  }
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1295  EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1296  
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1297  /**
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1298   * cxl_reset_cleanup_device - Cleanup after CXL reset
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1299   * @pdev: PCI device that was reset
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1300   *
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1301   * Releases region locks held during reset.
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1302   */
76e9ed09acad72 Srirangan Madhavan 2026-01-20 @1303  void cxl_reset_cleanup_device(struct pci_dev *pdev)
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1304  {
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1305  	cxl_reset_cleanup_memdev(pdev);
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1306  }
76e9ed09acad72 Srirangan Madhavan 2026-01-20  1307  EXPORT_SYMBOL_NS_GPL(cxl_reset_cleanup_device, "CXL");

76e9ed09acad72 Srirangan Madhavan 2026-01-20 1308

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#20 lore raw

PATCH v4 7/10 cxl: add host cache flush and multi-function reset

In reply to [email protected]

From: Srirangan Madhavan <[email protected]>

Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses wbinvd_on_all_cpus() on x86 and VA-based PoC clean+ invalidate on arm64 via memremap() and on_each_cpu(), matching the required ordering before reset.

---
 drivers/cxl/pci.c | 150 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 148 insertions(+), 2 deletions(-)
drivers/cxl/pci.c +148 -2
index e4134162e82a..f9cc452ccb8a 100644
@@ -11,6 +11,10 @@
 #include <linux/pci.h>
 #include <linux/aer.h>
 #include <linux/io.h>
+#include <linux/align.h>
+#include <linux/cache.h>
+#include <linux/cacheflush.h>
+#include <linux/smp.h>
 #include <cxl/mailbox.h>
 #include <cxl/pci.h>
 #include "cxlmem.h"
@@ -1085,6 +1089,71 @@ bool cxl_is_type2_device(struct pci_dev *pdev)
 	return cxlds->type == CXL_DEVTYPE_DEVMEM;
 }
 
+#ifdef CONFIG_ARM64
+struct cxl_cache_flush_ctx {
+	void *va;
+	size_t len;
+};
+
+static void cxl_flush_by_va_local(void *info)
+{
+	struct cxl_cache_flush_ctx *ctx = info;
+
+	dcache_clean_inval_poc((unsigned long)ctx->va,
+			       (unsigned long)ctx->va + ctx->len);
+	asm volatile("dsb ish" ::: "memory");
+}
+#endif
+
+static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data)
+{
+	struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev);
+	struct cxl_region *cxlr = cxled->cxld.region;
+	struct resource *res;
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	if (!cxlr || !cxlr->params.res)
+		return 0;
+
+	res = cxlr->params.res;
+
+#ifdef CONFIG_X86
+	static bool flushed;
+
+	if (!flushed) {
+		wbinvd_on_all_cpus();
+		flushed = true;
+	}
+#elif defined(CONFIG_ARM64)
+	void *va;
+	size_t len, line_size = L1_CACHE_BYTES;
+	phys_addr_t start, end, aligned_start, aligned_end;
+	struct cxl_cache_flush_ctx flush_ctx;
+
+	start = res->start;
+	end = res->end;
+
+	aligned_start = ALIGN_DOWN(start, line_size);
+	aligned_end = ALIGN(end + 1, line_size);
+	len = aligned_end - aligned_start;
+
+	va = memremap(aligned_start, len, MEMREMAP_WB);
+	if (!va) {
+		pr_warn("Failed to map region for cache flush\n");
+		return 0;
+	}
+
+	flush_ctx.va = va;
+	flush_ctx.len = len;
+	on_each_cpu(cxl_flush_by_va_local, &flush_ctx, 1);
+
+	memunmap(va);
+#endif
+	return 0;
+}
+
 static int cxl_check_region_driver_bound(struct device *dev, void *data)
 {
 	struct cxl_decoder *cxld = to_cxl_decoder(dev);
@@ -1245,6 +1314,9 @@ static int cxl_reset_prepare_memdev(struct pci_dev *pdev)
 		return rc;
 	}
 
+	device_for_each_child(&endpoint->dev, NULL,
+			      cxl_region_flush_host_cpu_caches);
+
 	/* Keep cxl_region_rwsem held, released by cleanup function */
 	return 0;
 }
@@ -1259,12 +1331,79 @@ static void cxl_reset_cleanup_memdev(struct pci_dev *pdev)
 		up_write(&cxl_region_rwsem);
 }
 
+static int cxl_reset_prepare_all_functions(struct pci_dev *pdev)
+{
+	struct pci_dev *func_dev;
+	unsigned int devfn;
+	int func, rc;
+	struct pci_dev *prepared_funcs[8] = { NULL };
+	int prepared_count = 0;
+
+	for (func = 0; func < 8; func++) {
+		devfn = PCI_DEVFN(PCI_SLOT(pdev->devfn), func);
+
+		if (devfn == pdev->devfn)
+			continue;
+
+		func_dev = pci_get_slot(pdev->bus, devfn);
+		if (!func_dev)
+			continue;
+
+		if (!cxl_is_type2_device(func_dev)) {
+			pci_dev_put(func_dev);
+			continue;
+		}
+
+		rc = cxl_reset_prepare_memdev(func_dev);
+		if (rc) {
+			pci_dev_put(func_dev);
+			goto cleanup_funcs;
+		}
+
+		prepared_funcs[prepared_count++] = func_dev;
+	}
+
+	return 0;
+
+cleanup_funcs:
+	for (func = 0; func < prepared_count; func++) {
+		if (prepared_funcs[func]) {
+			cxl_reset_cleanup_memdev(prepared_funcs[func]);
+			pci_dev_put(prepared_funcs[func]);
+		}
+	}
+	return rc;
+}
+
+static void cxl_reset_cleanup_all_functions(struct pci_dev *pdev)
+{
+	struct pci_dev *func_dev;
+	unsigned int devfn;
+	int func;
+
+	for (func = 0; func < 8; func++) {
+		devfn = PCI_DEVFN(PCI_SLOT(pdev->devfn), func);
+
+		if (devfn == pdev->devfn)
+			continue;
+
+		func_dev = pci_get_slot(pdev->bus, devfn);
+		if (!func_dev)
+			continue;
+
+		if (cxl_is_type2_device(func_dev))
+			cxl_reset_cleanup_memdev(func_dev);
+
+		pci_dev_put(func_dev);
+	}
+}
+
 /**
  * cxl_reset_prepare_device - Prepare CXL device for reset
  * @pdev: PCI device being reset
  *
  * CXL-reset-specific preparation. Validates memory is offline, flushes
- * device caches, and tears down regions.
+ * device caches, and tears down regions for device and siblings.
  *
  * Returns: 0 on success, -EBUSY if memory online, negative on error
  */
@@ -1283,6 +1422,12 @@ int cxl_reset_prepare_device(struct pci_dev *pdev)
 		return rc;
 	}
 
+	rc = cxl_reset_prepare_all_functions(pdev);
+	if (rc) {
+		cxl_reset_cleanup_memdev(pdev);
+		return rc;
+	}
+
 	return 0;
 }
 EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");
@@ -1291,10 +1436,11 @@ EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");
  * cxl_reset_cleanup_device - Cleanup after CXL reset
  * @pdev: PCI device that was reset
  *
- * Releases region locks held during reset.
+ * Releases region locks for device and all sibling functions.
  */
 void cxl_reset_cleanup_device(struct pci_dev *pdev)
 {
+	cxl_reset_cleanup_all_functions(pdev);
 	cxl_reset_cleanup_memdev(pdev);
 }
 EXPORT_SYMBOL_NS_GPL(cxl_reset_cleanup_device, "CXL");

--
2.34.1

#21 lore raw

Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Quoted 5 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses wbinvd_on_all_cpus() on x86 and VA-based PoC clean+

That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.

Until there are clarifications from Arm we need something like the agents in drivers/cache to do equivalent of wbinvd_on_all_cpus() (be it on a PA range to making a tiny bit less horrible)

Or we need an opt in list for platforms where a flush to PoC is enough.

Needs to some sort of arch_ call as well, not hidden in the cxl driver.

I'd suggest a separate patch to add the necessary infrastructure for arm64 with the relevant maintainers and lists included.

It needs enough work to be something they'd consider that I haven't +CC them on this version.

With that in mind I'll skip reviewing this patch for now.

Jonathan

Quoted 216 lines

invalidate on arm64 via memremap() and on_each_cpu(), matching the required ordering before reset.

---
 drivers/cxl/pci.c | 150 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 148 insertions(+), 2 deletions(-)
drivers/cxl/pci.c +148 -3
index e4134162e82a..f9cc452ccb8a 100644
@@ -11,6 +11,10 @@
 #include <linux/pci.h>
 #include <linux/aer.h>
 #include <linux/io.h>
+#include <linux/align.h>
+#include <linux/cache.h>
+#include <linux/cacheflush.h>
+#include <linux/smp.h>
 #include <cxl/mailbox.h>
 #include <cxl/pci.h>
 #include "cxlmem.h"
@@ -1085,6 +1089,71 @@ bool cxl_is_type2_device(struct pci_dev *pdev)
 	return cxlds->type == CXL_DEVTYPE_DEVMEM;
 }
 
+#ifdef CONFIG_ARM64
+struct cxl_cache_flush_ctx {
+	void *va;
+	size_t len;
+};
+
+static void cxl_flush_by_va_local(void *info)
+{
+	struct cxl_cache_flush_ctx *ctx = info;
+
+	dcache_clean_inval_poc((unsigned long)ctx->va,
+			       (unsigned long)ctx->va + ctx->len);
+	asm volatile("dsb ish" ::: "memory");
+}
+#endif
+
+static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data)
+{
+	struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev);
+	struct cxl_region *cxlr = cxled->cxld.region;
+	struct resource *res;
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	if (!cxlr || !cxlr->params.res)
+		return 0;
+
+	res = cxlr->params.res;
+
+#ifdef CONFIG_X86
+	static bool flushed;
+
+	if (!flushed) {
+		wbinvd_on_all_cpus();
+		flushed = true;
+	}
+#elif defined(CONFIG_ARM64)
+	void *va;
+	size_t len, line_size = L1_CACHE_BYTES;
+	phys_addr_t start, end, aligned_start, aligned_end;
+	struct cxl_cache_flush_ctx flush_ctx;
+
+	start = res->start;
+	end = res->end;
+
+	aligned_start = ALIGN_DOWN(start, line_size);
+	aligned_end = ALIGN(end + 1, line_size);
+	len = aligned_end - aligned_start;
+
+	va = memremap(aligned_start, len, MEMREMAP_WB);
+	if (!va) {
+		pr_warn("Failed to map region for cache flush\n");
+		return 0;
+	}
+
+	flush_ctx.va = va;
+	flush_ctx.len = len;
+	on_each_cpu(cxl_flush_by_va_local, &flush_ctx, 1);
+
+	memunmap(va);
+#endif
+	return 0;
+}
+
 static int cxl_check_region_driver_bound(struct device *dev, void *data)
 {
 	struct cxl_decoder *cxld = to_cxl_decoder(dev);
@@ -1245,6 +1314,9 @@ static int cxl_reset_prepare_memdev(struct pci_dev *pdev)
 		return rc;
 	}
 
+	device_for_each_child(&endpoint->dev, NULL,
+			      cxl_region_flush_host_cpu_caches);
+
 	/* Keep cxl_region_rwsem held, released by cleanup function */
 	return 0;
 }
@@ -1259,12 +1331,79 @@ static void cxl_reset_cleanup_memdev(struct pci_dev *pdev)
 		up_write(&cxl_region_rwsem);
 }
 
+static int cxl_reset_prepare_all_functions(struct pci_dev *pdev)
+{
+	struct pci_dev *func_dev;
+	unsigned int devfn;
+	int func, rc;
+	struct pci_dev *prepared_funcs[8] = { NULL };
+	int prepared_count = 0;
+
+	for (func = 0; func < 8; func++) {
+		devfn = PCI_DEVFN(PCI_SLOT(pdev->devfn), func);
+
+		if (devfn == pdev->devfn)
+			continue;
+
+		func_dev = pci_get_slot(pdev->bus, devfn);
+		if (!func_dev)
+			continue;
+
+		if (!cxl_is_type2_device(func_dev)) {
+			pci_dev_put(func_dev);
+			continue;
+		}
+
+		rc = cxl_reset_prepare_memdev(func_dev);
+		if (rc) {
+			pci_dev_put(func_dev);
+			goto cleanup_funcs;
+		}
+
+		prepared_funcs[prepared_count++] = func_dev;
+	}
+
+	return 0;
+
+cleanup_funcs:
+	for (func = 0; func < prepared_count; func++) {
+		if (prepared_funcs[func]) {
+			cxl_reset_cleanup_memdev(prepared_funcs[func]);
+			pci_dev_put(prepared_funcs[func]);
+		}
+	}
+	return rc;
+}
+
+static void cxl_reset_cleanup_all_functions(struct pci_dev *pdev)
+{
+	struct pci_dev *func_dev;
+	unsigned int devfn;
+	int func;
+
+	for (func = 0; func < 8; func++) {
+		devfn = PCI_DEVFN(PCI_SLOT(pdev->devfn), func);
+
+		if (devfn == pdev->devfn)
+			continue;
+
+		func_dev = pci_get_slot(pdev->bus, devfn);
+		if (!func_dev)
+			continue;
+
+		if (cxl_is_type2_device(func_dev))
+			cxl_reset_cleanup_memdev(func_dev);
+
+		pci_dev_put(func_dev);
+	}
+}
+
 /**
  * cxl_reset_prepare_device - Prepare CXL device for reset
  * @pdev: PCI device being reset
  *
  * CXL-reset-specific preparation. Validates memory is offline, flushes
- * device caches, and tears down regions.
+ * device caches, and tears down regions for device and siblings.
  *
  * Returns: 0 on success, -EBUSY if memory online, negative on error
  */
@@ -1283,6 +1422,12 @@ int cxl_reset_prepare_device(struct pci_dev *pdev)
 		return rc;
 	}
 
+	rc = cxl_reset_prepare_all_functions(pdev);
+	if (rc) {
+		cxl_reset_cleanup_memdev(pdev);
+		return rc;
+	}
+
 	return 0;
 }
 EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");
@@ -1291,10 +1436,11 @@ EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");
  * cxl_reset_cleanup_device - Cleanup after CXL reset
  * @pdev: PCI device that was reset
  *
- * Releases region locks held during reset.
+ * Releases region locks for device and all sibling functions.
  */
 void cxl_reset_cleanup_device(struct pci_dev *pdev)
 {
+	cxl_reset_cleanup_all_functions(pdev);
 	cxl_reset_cleanup_memdev(pdev);
 }
 EXPORT_SYMBOL_NS_GPL(cxl_reset_cleanup_device, "CXL");
--

2.34.1

maintainer
#22 lore raw

Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Quoted 5 lines · Jonathan Cameron
Quoted 2 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Quoted 7 lines
Quoted 3 lines

Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses wbinvd_on_all_cpus() on x86 and VA-based PoC clean+

That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.

... and quite slow for large regions without firmware assistance (CLEAN_INV_MEMREGION from PSCI 1.3).

Quoted 7 lines

Until there are clarifications from Arm we need something like the agents in drivers/cache to do equivalent of wbinvd_on_all_cpus() (be it on a PA range to making a tiny bit less horrible)

Or we need an opt in list for platforms where a flush to PoC is enough.

Needs to some sort of arch_ call as well, not hidden in the cxl driver.

Reset should just be using cpu_cache_invalidate_memregion().

... now, in addition to DCD, we have two users that are beyond the "just once at boot time" use case.

Thanks,
Davidlohr

#23 lore raw

Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Quoted 11 lines · On Wed, 21 Jan 2026 12:27:55 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 9 lines · Jonathan Cameron
Quoted 5 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses wbinvd_on_all_cpus() on x86 and VA-based PoC clean+

Quoted 6 lines
Quoted 3 lines

That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.

... and quite slow for large regions without firmware assistance (CLEAN_INV_MEMREGION from PSCI 1.3).

Just for completeness in case anyone takes this comment out of context.

CLEAN_INV_MEMREGION was dropped after that spec when beyond alpha. See F.b release.
https://developer.arm.com/documentation/den0022/fb/?lang=en
Talk to you to your favorite Arm architecture person for why
(I'll just say it was a good reason).

Also there is no guarantee that firmware interface does it any faster. Implementation defined and all that.

No one jumped on my proposal to write an ACPI AML wrapper spec so for now only option is to write a specific driver if you have a hardware agent that offloads these operations (like the hisi_hha does).

Quoted 5 lines
Quoted 5 lines

Until there are clarifications from Arm we need something like the agents in drivers/cache to do equivalent of wbinvd_on_all_cpus() (be it on a PA range to making a tiny bit less horrible)

Quoted 8 lines
Quoted 3 lines

Or we need an opt in list for platforms where a flush to PoC is enough.

Needs to some sort of arch_ call as well, not hidden in the cxl driver.

Reset should just be using cpu_cache_invalidate_memregion().

... now, in addition to DCD, we have two users that are beyond the "just once at boot time" use case.

or Back-Invalidate to do this from the device side. If anyone has built that yet... Beware physical address based prefetchers though as there be monsters.

Jonathan

Quoted 4 lines

Thanks,
Davidlohr

#24 lore raw

Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

In reply to Jonathan Cameron

Hi Jonathan,

________________________________________
From: Jonathan Cameron <[email protected]>
Sent: Wednesday, January 21, 2026 5:20 AM
To: Srirangan Madhavan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vishal Aslot; Vikram Sethi; Shanker Donthineni; Vidya Sagar; Matt Ochs; Jason Sequeira
Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Quoted 10 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:
Quoted 5 lines

From: Srirangan Madhavan <[email protected]>

Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses wbinvd_on_all_cpus() on x86 and VA-based PoC clean+

That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.

Quoted 7 lines

Until there are clarifications from Arm we need something like the agents in drivers/cache to do equivalent of wbinvd_on_all_cpus() (be it on a PA range to making a tiny bit less horrible)

Or we need an opt in list for platforms where a flush to PoC is enough.

Needs to some sort of arch_ call as well, not hidden in the cxl driver.

Yes, and in fact it cannot be a cache flush by VA while PTE mappings to the memory are valid, as the core can prefetch the lines back right away, and later evictions post device reset can cause device errors as the device snoop filter isn’t aware of host having any lines.

The only way to do this correctly is via a SMC call to Arm Trusted Firmware to handle in a SOC specific way (after the memory has been offlined and removed from the PTEs), since there is no architectural way to flush by PA (set/way isn’t usable in the kernel). There was a SMC call defined for this in the PSCI spec 2 years ago, but was removed for some reason. I had a discussion with some ARM folks maintaining the PSCI and SMCCC specifications and the direction was to bring the SMC call back in SMCCC specification. Like you say, that can be a separate patch series. Adding Souvik for SMCCC specification details once it is available.

Thanks

#25 lore raw

Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Hi,

Quoted 29 lines · Vikram Sethi

Hi Jonathan,

________________________________________
From: Jonathan Cameron <[email protected]>
Sent: Wednesday, January 21, 2026 5:20 AM
To: Srirangan Madhavan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vishal Aslot; Vikram Sethi; Shanker Donthineni; Vidya Sagar; Matt Ochs; Jason Sequeira
Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Quoted 13 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:
Quoted 5 lines

From: Srirangan Madhavan <[email protected]>

Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses wbinvd_on_all_cpus() on x86 and VA-based PoC clean+

That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.

Until there are clarifications from Arm we need something like the agents in drivers/cache to do equivalent of wbinvd_on_all_cpus() (be it on a PA range to making a tiny bit less horrible)

Quoted 7 lines
Quoted 3 lines

Or we need an opt in list for platforms where a flush to PoC is enough.

Needs to some sort of arch_ call as well, not hidden in the cxl driver.

Yes, and in fact it cannot be a cache flush by VA while PTE mappings to the memory are valid, as the core can prefetch the lines back right away, and later evictions post device reset can cause device errors as the device snoop filter isn’t aware of host having any lines.

The only way to do this correctly is via a SMC call to Arm Trusted Firmware to handle in a SOC specific way (after the memory has been offlined and removed from the PTEs), since there is no architectural way to flush by PA (set/way isn’t usable in the kernel). There was a SMC call defined for this in the PSCI spec 2 years ago, but was removed for some reason. I had a discussion with some ARM folks maintaining the PSCI and SMCCC specifications and the direction was to bring the SMC call back in SMCCC specification. Like you say, that can be a separate patch series. Adding Souvik for SMCCC specification details once it is available.

Yes can confirm this is being actively worked on. Something should be available in the public SMCCC spec soon. Will be the same PSCI functions moved to SMCCC but slightly simplified.

Regards,
Souvik

Quoted 3 lines

Thanks

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

A message that is not in the archive (sent off-list or not yet received): PH7PR12MB9175CDFC163843BB497073CEBD96A@PH7PR12MB9175.namprd12.prod.outlook.com

#27 lore raw

Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Quoted 8 lines · On Wed, 21 Jan 2026 20:36:01 +0000 Vikram Sethi <[email protected]> wrote:

Hi Jonathan,

From: Jonathan Cameron <[email protected]>
Date: Wednesday, January 21, 2026 at 5:20 AM
To: Srirangan Madhavan <[email protected]>
Cc: [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, Vishal Aslot <[email protected]>, Vikram Sethi <[email protected]>, Shanker Donthineni <[email protected]>, Vidya Sagar <[email protected]>, Matt Ochs <[email protected]>, Jason Sequeira <[email protected]>
Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Hi Vikram, Happy new year!

Quoted 13 lines
Quoted 8 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:
Quoted 4 lines

From: Srirangan Madhavan <[email protected]>
Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses wbinvd_on_all_cpus() on x86 and VA-based PoC clean+

That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.

snip

Quoted 7 lines
Quoted 1 line

Needs to some sort of arch_ call as well, not hidden in the cxl driver.

Yes, and in fact it cannot be a cache flush by VA while PTE mappings to the memory are valid, as the core can prefetch the lines back right away, and later evictions post device reset can cause device errors as the device snoop filter isn’t aware of host having any lines.

Prefetching should (hopefully) not make any dirty lines. Hopefully no one does clean writebacks (and there is a way to check if they do in the ACPI tables). You need to flush again after to force out that stale stuff though before any demand fetches occur.

+ don't forget PA based prefetchers. Doesn't even need to be a mapping for fetches into distance caches to happen. A given platform might have restrictions on where those PA prefetchers will go but we currently have no way to discover that.

I need to think a bit more on this as there are some scary comments in the spec on CXL.reset such as all CXL.mem reads are dropped (timeout fun). Mind you a device is permitted to do that anyway before cxl.mem is enabled, so hopefully n one times out if a prefetcher hits the device before that's on..

Quoted 11 lines

The only way to do this correctly is via a SMC call to Arm trusted Firmware to handle in SOC specific way (after the memory has been offlined and removed from the PTEs), since there is no architectural way to flush by PA (set/way isn’t usable in the kernel). There was a SMC call defined for this in the PSCI spec 2 years ago, but was removed for some reason. I had a discussion with some ARM folks maintaining the PSCI and SMCCC specifications and the direction was to bring the SMC call back in SMCCC specification. Like you say, that can be a separate patch series. Adding Souvik for SMCCC specification details once it is available.

There are other possible paths which is where drivers/cache stuff came from.+CC James Morse.

From a kernel point of view SMCCC needs to be just one option as a bunch of hardware (I'll only point at ours as not sure what else is public) provide MMIO accessible agents to do this stuff and going via EL3 to talk to an engine the kernel can poke directly is silly.

I'm not against the PSCI thing coming back though if someone needs it. Preferably without the CPU rendezvous stuff though -> or Linux can just reject anyone who does that.

We could also revisit the approach of using an AML op region to issue the SMCCC, thus allowing us to support vendor SMCCC calls (+ a general one if ARM do bring that back) or whatever magic they want to use.
https://lore.kernel.org/all/[email protected]/

Advantage of that is you can wrap SMCCC, MMIO or whatever else you like up and the kernel only needs one driver. Disadvantage is need a spec and only ACPI etc so we need a driver anyway if anyone cares on DT systems...

If you want to mess with that I can dig out the QEMU emulation and rebase that driver. I thought it was kind of cute, just a solution we didn't need at the time!

Jonathan

#28 lore raw

Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Hi Jonathan,
Happy new year!

Quoted 11 lines

From: Jonathan Cameron <[email protected]>
Date: Thursday, January 22, 2026 4:31 AM
To: Srirangan Madhavan <[email protected]>
Cc: [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, Vishal Aslot <[email protected]>, Vikram Sethi <[email protected]>, Shanker Donthineni <[email protected]>, Vidya Sagar <[email protected]>, Matt Ochs <[email protected]>, Jason Sequeira <[email protected]>
Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Prefetching should (hopefully) not make any dirty lines. Hopefully no one does clean writebacks (and there is a way to check if they do in the ACPI tables). You need to flush again after to force out that stale stuff though before any demand fetches occur.

I am aware of some systems which do clean writebacks to device memory. What exact ACPI table has this discovery?

Quoted 5 lines

I need to think a bit more on this as there are some scary comments in the spec on CXL.reset such as all CXL.mem reads are dropped (timeout fun). Mind you a device is permitted to do that anyway before cxl.mem is enabled, so hopefully n one times out if a prefetcher hits the device before that's on..

Yes, any sensible coherent memory device reset will have to include first offlining the memory such that there is no demand or speculative fetch possible, else you get to deal with CXL error isolation fun.

Quoted 4 lines

From a kernel point of view SMCCC needs to be just one option as a bunch of hardware (I'll only point at ours as not sure what else is public) provide MMIO accessible agents to do this stuff and going via EL3 to talk to an engine the kernel can poke directly is silly.

We have a similar custom engine for efficient flushing, but the interface is not available to the kernel, so the SMCCC is preferred for our implementation. Like you say, the AML wrapper for SMCCC is another option, although we also have some device tree based systems where the efficient custom flushing is desirable.

Quoted 3 lines

I'm not against the PSCI thing coming back though if someone needs it. Preferably without the CPU rendezvous stuff though -> or Linux can just reject anyone who does that.

Agreed. I think Souvik also agreed offline that rendezvous is not needed for the SMC based cache flush.

Vikram

#29 lore raw

Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Quoted 10 lines · On Thu, 22 Jan 2026 19:24:49 +0000 Vikram Sethi <[email protected]> wrote:

Hi Jonathan,
Happy new year!

Quoted 5 lines

From: Jonathan Cameron <[email protected]>
Date: Thursday, January 22, 2026 4:31 AM
To: Srirangan Madhavan <[email protected]>
Cc: [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, Vishal Aslot <[email protected]>, Vikram Sethi <[email protected]>, Shanker Donthineni <[email protected]>, Vidya Sagar <[email protected]>, Matt Ochs <[email protected]>, Jason Sequeira <[email protected]>
Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

Quoted 7 lines
Quoted 4 lines

Prefetching should (hopefully) not make any dirty lines. Hopefully no one does clean writebacks (and there is a way to check if they do in the ACPI tables). You need to flush again after to force out that stale stuff though before any demand fetches occur.

I am aware of some systems which do clean writebacks to device memory. What exact ACPI table has this discovery?

CEDT (so in the CXL spec, it's there in 3.2 and 4, I'm too lazy to open earlier specs) Specifically the CXL System Description Structure (CSDS) System Capabilities bit[1]

No Clean Writeback: Specifies the clean writeback beahvior of hte host
- 0 = The host may or may not generate clean writebacks.
- 1 = The host guarantees to never generate clean writeback at the host's cacheline granularity.

Quoted 7 lines
Quoted 6 lines

I need to think a bit more on this as there are some scary comments in the spec on CXL.reset such as all CXL.mem reads are dropped (timeout fun). Mind you a device is permitted to do that anyway before cxl.mem is enabled, so hopefully n one times out if a prefetcher hits the device before that's on..

Quoted 8 lines

Yes, any sensible coherent memory device reset will have to include first offlining the memory such that there is no demand or speculative fetch possible, else you get to deal with CXL error isolation fun.

Quoted 4 lines

From a kernel point of view SMCCC needs to be just one option as a bunch of hardware (I'll only point at ours as not sure what else is public) provide MMIO accessible agents to do this stuff and going via EL3 to talk to an engine the kernel can poke directly is silly.

We have a similar custom engine for efficient flushing, but the interface is not available to the kernel, so the SMCCC is preferred for our implementation. Like you say, the AML wrapper for SMCCC is another option, although we also have some device tree based systems where the efficient custom flushing is desirable.

Great. So all that work they did before dropping the PSCI call was worth doing. I'll keep an eye open for the new spec.

Quoted 6 lines
Quoted 4 lines

I'm not against the PSCI thing coming back though if someone needs it. Preferably without the CPU rendezvous stuff though -> or Linux can just reject anyone who does that.

Agreed. I think Souvik also agreed offline that rendezvous is not needed for the SMC based cache flush.

Great.

Thanks,

Jonathan

Quoted 3 lines

Vikram

maintainer
#30 lore raw

Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

In reply to [email protected]
Quoted 29 lines · On 1/20/26 3:26 PM, [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses wbinvd_on_all_cpus() on x86 and VA-based PoC clean+ invalidate on arm64 via memremap() and on_each_cpu(), matching the required ordering before reset.

---
 drivers/cxl/pci.c | 150 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 148 insertions(+), 2 deletions(-)
drivers/cxl/pci.c +4 -0
index e4134162e82a..f9cc452ccb8a 100644
@@ -11,6 +11,10 @@
 #include <linux/pci.h>
 #include <linux/aer.h>
 #include <linux/io.h>
+#include <linux/align.h>
+#include <linux/cache.h>
+#include <linux/cacheflush.h>
+#include <linux/smp.h>
 #include <cxl/mailbox.h>
 #include <cxl/pci.h>
 #include "cxlmem.h"

drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1092

Quoted 4 lines
drivers/cxl/pci.c +1 -0
@@ -1085,6 +1089,71 @@ bool cxl_is_type2_device(struct pci_dev *pdev)
 	return cxlds->type == CXL_DEVTYPE_DEVMEM;
 }
 
+#ifdef CONFIG_ARM64

Please figure out some arch common way to do this. Really do not want CONFIG_$ARCH ifdefs sprinkled all over CXL code. Also this ARCH specific code probably does not belong under CXL anyhow.

DJ

Quoted 187 lines
drivers/cxl/pci.c +143 -3
+struct cxl_cache_flush_ctx {
+	void *va;
+	size_t len;
+};
+
+static void cxl_flush_by_va_local(void *info)
+{
+	struct cxl_cache_flush_ctx *ctx = info;
+
+	dcache_clean_inval_poc((unsigned long)ctx->va,
+			       (unsigned long)ctx->va + ctx->len);
+	asm volatile("dsb ish" ::: "memory");
+}
+#endif
+
+static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data)
+{
+	struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev);
+	struct cxl_region *cxlr = cxled->cxld.region;
+	struct resource *res;
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	if (!cxlr || !cxlr->params.res)
+		return 0;
+
+	res = cxlr->params.res;
+
+#ifdef CONFIG_X86
+	static bool flushed;
+
+	if (!flushed) {
+		wbinvd_on_all_cpus();
+		flushed = true;
+	}
+#elif defined(CONFIG_ARM64)
+	void *va;
+	size_t len, line_size = L1_CACHE_BYTES;
+	phys_addr_t start, end, aligned_start, aligned_end;
+	struct cxl_cache_flush_ctx flush_ctx;
+
+	start = res->start;
+	end = res->end;
+
+	aligned_start = ALIGN_DOWN(start, line_size);
+	aligned_end = ALIGN(end + 1, line_size);
+	len = aligned_end - aligned_start;
+
+	va = memremap(aligned_start, len, MEMREMAP_WB);
+	if (!va) {
+		pr_warn("Failed to map region for cache flush\n");
+		return 0;
+	}
+
+	flush_ctx.va = va;
+	flush_ctx.len = len;
+	on_each_cpu(cxl_flush_by_va_local, &flush_ctx, 1);
+
+	memunmap(va);
+#endif
+	return 0;
+}
+
 static int cxl_check_region_driver_bound(struct device *dev, void *data)
 {
 	struct cxl_decoder *cxld = to_cxl_decoder(dev);
@@ -1245,6 +1314,9 @@ static int cxl_reset_prepare_memdev(struct pci_dev *pdev)
 		return rc;
 	}
 
+	device_for_each_child(&endpoint->dev, NULL,
+			      cxl_region_flush_host_cpu_caches);
+
 	/* Keep cxl_region_rwsem held, released by cleanup function */
 	return 0;
 }
@@ -1259,12 +1331,79 @@ static void cxl_reset_cleanup_memdev(struct pci_dev *pdev)
 		up_write(&cxl_region_rwsem);
 }
 
+static int cxl_reset_prepare_all_functions(struct pci_dev *pdev)
+{
+	struct pci_dev *func_dev;
+	unsigned int devfn;
+	int func, rc;
+	struct pci_dev *prepared_funcs[8] = { NULL };
+	int prepared_count = 0;
+
+	for (func = 0; func < 8; func++) {
+		devfn = PCI_DEVFN(PCI_SLOT(pdev->devfn), func);
+
+		if (devfn == pdev->devfn)
+			continue;
+
+		func_dev = pci_get_slot(pdev->bus, devfn);
+		if (!func_dev)
+			continue;
+
+		if (!cxl_is_type2_device(func_dev)) {
+			pci_dev_put(func_dev);
+			continue;
+		}
+
+		rc = cxl_reset_prepare_memdev(func_dev);
+		if (rc) {
+			pci_dev_put(func_dev);
+			goto cleanup_funcs;
+		}
+
+		prepared_funcs[prepared_count++] = func_dev;
+	}
+
+	return 0;
+
+cleanup_funcs:
+	for (func = 0; func < prepared_count; func++) {
+		if (prepared_funcs[func]) {
+			cxl_reset_cleanup_memdev(prepared_funcs[func]);
+			pci_dev_put(prepared_funcs[func]);
+		}
+	}
+	return rc;
+}
+
+static void cxl_reset_cleanup_all_functions(struct pci_dev *pdev)
+{
+	struct pci_dev *func_dev;
+	unsigned int devfn;
+	int func;
+
+	for (func = 0; func < 8; func++) {
+		devfn = PCI_DEVFN(PCI_SLOT(pdev->devfn), func);
+
+		if (devfn == pdev->devfn)
+			continue;
+
+		func_dev = pci_get_slot(pdev->bus, devfn);
+		if (!func_dev)
+			continue;
+
+		if (cxl_is_type2_device(func_dev))
+			cxl_reset_cleanup_memdev(func_dev);
+
+		pci_dev_put(func_dev);
+	}
+}
+
 /**
  * cxl_reset_prepare_device - Prepare CXL device for reset
  * @pdev: PCI device being reset
  *
  * CXL-reset-specific preparation. Validates memory is offline, flushes
- * device caches, and tears down regions.
+ * device caches, and tears down regions for device and siblings.
  *
  * Returns: 0 on success, -EBUSY if memory online, negative on error
  */
@@ -1283,6 +1422,12 @@ int cxl_reset_prepare_device(struct pci_dev *pdev)
 		return rc;
 	}
 
+	rc = cxl_reset_prepare_all_functions(pdev);
+	if (rc) {
+		cxl_reset_cleanup_memdev(pdev);
+		return rc;
+	}
+
 	return 0;
 }
 EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");
@@ -1291,10 +1436,11 @@ EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");
  * cxl_reset_cleanup_device - Cleanup after CXL reset
  * @pdev: PCI device that was reset
  *
- * Releases region locks held during reset.
+ * Releases region locks for device and all sibling functions.
  */
 void cxl_reset_cleanup_device(struct pci_dev *pdev)
 {
+	cxl_reset_cleanup_all_functions(pdev);
 	cxl_reset_cleanup_memdev(pdev);
 }
 EXPORT_SYMBOL_NS_GPL(cxl_reset_cleanup_device, "CXL");
--

2.34.1

#31 lore raw

PATCH v4 8/10 cxl: add DVSEC config save/restore

In reply to [email protected]

From: Srirangan Madhavan <[email protected]>

Save and restore CXL DVSEC control registers across reset with CONFIG_LOCK handling so RWL fields are preserved when locked. This maintains device policy and capability state across cxl_reset while avoiding writes to locked fields.

---
 drivers/cxl/pci.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++
 include/cxl/pci.h |  15 +++++++
 2 files changed, 122 insertions(+)
drivers/cxl/pci.c +107 -0
index f9cc452ccb8a..7d6a0ef70b2d 100644
@@ -1154,6 +1154,113 @@ static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data)
 	return 0;
 }
 
+/*
+ * CXL DVSEC register save/restore
+ */
+static int cxl_save_dvsec_state(struct pci_dev *pdev,
+				struct cxl_type2_saved_state *state, int dvsec)
+{
+	int rc;
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL_OFFSET,
+				  &state->dvsec_ctrl);
+	if (rc)
+		return rc;
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET,
+				  &state->dvsec_ctrl2);
+	return rc;
+}
+
+static int cxl_restore_dvsec_state(struct pci_dev *pdev,
+				   const struct cxl_type2_saved_state *state,
+				   int dvsec, bool config_locked)
+{
+	int rc;
+	u16 val_to_restore;
+
+	if (config_locked) {
+		u16 current_val;
+
+		rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL_OFFSET,
+					  &current_val);
+		if (rc)
+			return rc;
+
+		val_to_restore = (current_val & CXL_DVSEC_CTRL_RWL_MASK) |
+				 (state->dvsec_ctrl & ~CXL_DVSEC_CTRL_RWL_MASK);
+	} else {
+		val_to_restore = state->dvsec_ctrl;
+	}
+
+	rc = pci_write_config_word(pdev, dvsec + CXL_DVSEC_CTRL_OFFSET,
+				   val_to_restore);
+	if (rc)
+		return rc;
+
+	rc = pci_write_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET,
+				   state->dvsec_ctrl2);
+	return rc;
+}
+
+/**
+ * cxl_config_save_state - Save CXL configuration state
+ * @pdev: PCI device
+ * @state: Structure to store saved state
+ *
+ * Saves CXL DVSEC state before reset.
+ */
+int cxl_config_save_state(struct pci_dev *pdev,
+			  struct cxl_type2_saved_state *state)
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+	int dvsec;
+
+	if (!cxlds || !state)
+		return -EINVAL;
+
+	memset(state, 0, sizeof(*state));
+
+	dvsec = cxlds->cxl_dvsec;
+	if (!dvsec)
+		return -ENODEV;
+
+	return cxl_save_dvsec_state(pdev, state, dvsec);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_config_save_state, "CXL");
+
+/**
+ * cxl_config_restore_state - Restore CXL configuration state
+ * @pdev: PCI device
+ * @state: Previously saved state
+ *
+ * Restores CXL DVSEC state after reset.
+ */
+int cxl_config_restore_state(struct pci_dev *pdev,
+			     const struct cxl_type2_saved_state *state)
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+	bool config_locked;
+	int rc, dvsec;
+	u16 lock_reg;
+
+	if (!cxlds || !state)
+		return -EINVAL;
+
+	dvsec = cxlds->cxl_dvsec;
+	if (!dvsec)
+		return -ENODEV;
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_LOCK_OFFSET, &lock_reg);
+	if (rc)
+		return rc;
+
+	config_locked = !!(lock_reg & CXL_DVSEC_LOCK_CONFIG_LOCK);
+
+	return cxl_restore_dvsec_state(pdev, state, dvsec, config_locked);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_config_restore_state, "CXL");
+
 static int cxl_check_region_driver_bound(struct device *dev, void *data)
 {
 	struct cxl_decoder *cxld = to_cxl_decoder(dev);
include/cxl/pci.h +15 -0
index 71d8de5de948..2c629ded73cc 100644
@@ -4,6 +4,18 @@
 #ifndef __CXL_ACCEL_PCI_H
 #define __CXL_ACCEL_PCI_H
 
+/* CXL Type 2 device state for save/restore across reset */
+struct cxl_type2_saved_state {
+	/* DVSEC registers */
+	u16 dvsec_ctrl;
+	u16 dvsec_ctrl2;
+};
+
+int cxl_config_save_state(struct pci_dev *pdev,
+			  struct cxl_type2_saved_state *state);
+int cxl_config_restore_state(struct pci_dev *pdev,
+			     const struct cxl_type2_saved_state *state);
+
 /*
  * See section 8.1 Configuration Space Registers in the CXL 2.0
  * Specification. Names are taken straight from the specification with "CXL" and
@@ -23,6 +35,7 @@
 #define     CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE	BIT(11)
 #define   CXL_DVSEC_CTRL_OFFSET		0xC
 #define     CXL_DVSEC_MEM_ENABLE	BIT(2)
+#define     CXL_DVSEC_CTRL_RWL_MASK	0x5FED
 #define   CXL_DVSEC_CTRL2_OFFSET	0x10
 #define     CXL_DVSEC_DISABLE_CACHING	BIT(0)
 #define     CXL_DVSEC_INIT_CACHE_WBI	BIT(1)
@@ -32,6 +45,8 @@
 #define     CXL_DVSEC_CACHE_INVALID	BIT(0)
 #define     CXL_DVSEC_CXL_RST_COMPLETE	BIT(1)
 #define     CXL_DVSEC_CXL_RESET_ERR	BIT(2)
+#define   CXL_DVSEC_LOCK_OFFSET		0x14
+#define     CXL_DVSEC_LOCK_CONFIG_LOCK	BIT(0)
 #define   CXL_DVSEC_RANGE_SIZE_HIGH(i)	(0x18 + ((i) * 0x10))
 #define   CXL_DVSEC_RANGE_SIZE_LOW(i)	(0x1C + ((i) * 0x10))
 #define     CXL_DVSEC_MEM_INFO_VALID	BIT(0)

--
2.34.1

#32 lore raw

Re: [PATCH v4 08/10] cxl: add DVSEC config save/restore

Quoted 6 lines · On Tue, 20 Jan 2026 22:26:08 +0000 [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Save and restore CXL DVSEC control registers across reset with CONFIG_LOCK handling so RWL fields are preserved when locked. This maintains device policy and capability state across cxl_reset while avoiding writes to locked fields.

Add some more info here on what is protected by the lock and preserved across reset. This is odd enough I think that detail is needed. There are other RWL fields outside those registers you are restoring here. e.g. the range registers, various things in extended meta data,

Quoted 21 lines
---
 drivers/cxl/pci.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++
 include/cxl/pci.h |  15 +++++++
 2 files changed, 122 insertions(+)
drivers/cxl/pci.c +6 -0
index f9cc452ccb8a..7d6a0ef70b2d 100644
@@ -1154,6 +1154,113 @@ static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data)
 	return 0;
 }
 
+/*
+ * CXL DVSEC register save/restore
+ */
+static int cxl_save_dvsec_state(struct pci_dev *pdev,
+				struct cxl_type2_saved_state *state, int dvsec)
+{

drivers/cxl/pci.c static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data) · line 1172

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+	int rc;
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL_OFFSET,
+				  &state->dvsec_ctrl);
+	if (rc)
+		return rc;
+
+	rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET,
+				  &state->dvsec_ctrl2);
+	return rc;

Similar to below. Just combine 2 lines into 1.

Quoted 21 lines
drivers/cxl/pci.c +21 -0
+}
+
+static int cxl_restore_dvsec_state(struct pci_dev *pdev,
+				   const struct cxl_type2_saved_state *state,
+				   int dvsec, bool config_locked)
+{
+	int rc;
+	u16 val_to_restore;
+
+	if (config_locked) {
+		u16 current_val;
+
+		rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL_OFFSET,
+					  &current_val);
+		if (rc)
+			return rc;
+
+		val_to_restore = (current_val & CXL_DVSEC_CTRL_RWL_MASK) |
+				 (state->dvsec_ctrl & ~CXL_DVSEC_CTRL_RWL_MASK);
+	} else {
+		val_to_restore = state->dvsec_ctrl;

drivers/cxl/pci.c static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data) · line 1203

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+	}
+
+	rc = pci_write_config_word(pdev, dvsec + CXL_DVSEC_CTRL_OFFSET,
+				   val_to_restore);
+	if (rc)
+		return rc;
+
+	rc = pci_write_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET,
+				   state->dvsec_ctrl2);
+	return rc;

If this isn't expected to get more complex (I haven't checked) then return pci_write_config()

Quoted 9 lines
drivers/cxl/pci.c +9 -0
+}
+
+/**
+ * cxl_config_save_state - Save CXL configuration state
+ * @pdev: PCI device
+ * @state: Structure to store saved state
+ *
+ * Saves CXL DVSEC state before reset.
+ */

drivers/cxl/pci.c static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data) · line 1222

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+int cxl_config_save_state(struct pci_dev *pdev,
+			  struct cxl_type2_saved_state *state)
+{
+	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+	int dvsec;
+
+	if (!cxlds || !state)
+		return -EINVAL;
+
+	memset(state, 0, sizeof(*state));

We need to make sure all registers are read, so why is zeroing helpful?

Quoted 8 lines
drivers/cxl/pci.c +8 -0
+
+	dvsec = cxlds->cxl_dvsec;
+	if (!dvsec)
+		return -ENODEV;
+
+	return cxl_save_dvsec_state(pdev, state, dvsec);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_config_save_state, "CXL");
#33 lore raw

PATCH v4 9/10 PCI: save/restore CXL config around reset

In reply to [email protected]

From: Srirangan Madhavan <[email protected]>

Save PCI and CXL configuration state before cxl_reset and restore it after reset completes. This preserves DVSEC state alongside standard PCI state and avoids losing reset-sensitive CXL configuration.

---
 drivers/pci/pci.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/pci/pci.c +12 -0
index 18047c893b0c..0bc85c4cc5fd 100644
@@ -4960,6 +4960,7 @@ static int cxl_reset_init(struct pci_dev *dev, u16 dvsec)
  */
 static int cxl_reset(struct pci_dev *dev, bool probe)
 {
+	struct cxl_type2_saved_state cxl_state;
 	u16 dvsec, reg;
 	int rc;
 
@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)
 	if (probe)
 		return 0;
 
+	pci_save_state(dev);
+	rc = cxl_config_save_state(dev, &cxl_state);
+	if (rc)
+		pci_warn(dev, "Failed to save CXL config state: %d\n", rc);
+
 	/*
 	 * CXL-reset-specific preparation: validate memory offline,
 	 * tear down regions, flush device caches.
@@ -5004,10 +5010,16 @@ static int cxl_reset(struct pci_dev *dev, bool probe)
 	if (rc)
 		goto out_cleanup;
 
+	pci_restore_state(dev);
+	rc = cxl_config_restore_state(dev, &cxl_state);
+	if (rc)
+		pci_warn(dev, "Failed to restore CXL config state: %d\n", rc);
+
 	cxl_reset_cleanup_device(dev);
 	return 0;
 
 out_cleanup:
+	pci_restore_state(dev);
 	cxl_reset_cleanup_device(dev);
 	return rc;
 }

--
2.34.1

maintainer
#34 lore raw

Re: [PATCH v4 09/10] PCI: save/restore CXL config around reset

Quoted 5 lines · On 1/20/26 3:26 PM, [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Save PCI and CXL configuration state before cxl_reset and restore it after reset completes. This preserves DVSEC state alongside standard PCI state and avoids losing reset-sensitive CXL configuration.

Instead of putting dependency on the cxl core, maybe just move the code in the previous patch here since it's just a few lines of config read/writes. But an explanation of why the DVSEC needs to be preserved on the device regardless of whether a driver is present is needed in the commit log.

DJ

Quoted 50 lines
---
 drivers/pci/pci.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/pci/pci.c +12 -1
index 18047c893b0c..0bc85c4cc5fd 100644
@@ -4960,6 +4960,7 @@ static int cxl_reset_init(struct pci_dev *dev, u16 dvsec)
  */
 static int cxl_reset(struct pci_dev *dev, bool probe)
 {
+	struct cxl_type2_saved_state cxl_state;
 	u16 dvsec, reg;
 	int rc;
 
@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)
 	if (probe)
 		return 0;
 
+	pci_save_state(dev);
+	rc = cxl_config_save_state(dev, &cxl_state);
+	if (rc)
+		pci_warn(dev, "Failed to save CXL config state: %d\n", rc);
+
 	/*
 	 * CXL-reset-specific preparation: validate memory offline,
 	 * tear down regions, flush device caches.
@@ -5004,10 +5010,16 @@ static int cxl_reset(struct pci_dev *dev, bool probe)
 	if (rc)
 		goto out_cleanup;
 
+	pci_restore_state(dev);
+	rc = cxl_config_restore_state(dev, &cxl_state);
+	if (rc)
+		pci_warn(dev, "Failed to restore CXL config state: %d\n", rc);
+
 	cxl_reset_cleanup_device(dev);
 	return 0;
 
 out_cleanup:
+	pci_restore_state(dev);
 	cxl_reset_cleanup_device(dev);
 	return rc;
 }
--

2.34.1

#35 lore raw

Re: [PATCH v4 09/10] PCI: save/restore CXL config around reset

In reply to [email protected]

drivers/pci/pci.c static int cxl_reset(struct pci_dev *dev, bool probe) · line 4997

Quoted 10 lines · On Tue, Jan 20, 2026 at 10:26:09PM +0000, [email protected] wrote:

+++ b/drivers/pci/pci.c

drivers/pci/pci.c +5 -0
@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)
 	if (probe)
 		return 0;
 
+	pci_save_state(dev);
+	rc = cxl_config_save_state(dev, &cxl_state);
+	if (rc)
+		pci_warn(dev, "Failed to save CXL config state: %d\n", rc);
+

Hm, shouldn't the call to cxl_config_save_state() be moved to pci_save_state() (and likewise, cxl_config_restore_state() moved to pci_restore_state())?

E.g. when a DPC event occurs, I assume CXL registers need to be restored as well on recovery, right?

Note that since v6.19-rc1, state is saved on enumeration and thus always available for recovery, see a2f1e22390ac.

As a general remark on this series, it seems to have considerable overlap with Terry's work on AER support for CXL, particularly in patch [01/10], as Jonathan has remarked. Please cc Terry on future submissions and coordinate with him on conflicting parts of your patches:

https://lore.kernel.org/all/[email protected]/

Thanks,

Lukas

#36 lore raw

Re: [PATCH v4 09/10] PCI: save/restore CXL config around reset

Quoted 16 lines · On Thu, 22 Jan 2026 11:01:57 +0100 Lukas Wunner <[email protected]> wrote:

drivers/pci/pci.c static int cxl_reset(struct pci_dev *dev, bool probe) · line 4997

Quoted 10 lines · On Tue, Jan 20, 2026 at 10:26:09PM +0000, [email protected] wrote:

+++ b/drivers/pci/pci.c

drivers/pci/pci.c +5 -0
@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)
 	if (probe)
 		return 0;
 
+	pci_save_state(dev);
+	rc = cxl_config_save_state(dev, &cxl_state);
+	if (rc)
+		pci_warn(dev, "Failed to save CXL config state: %d\n", rc);
+  

Hm, shouldn't the call to cxl_config_save_state() be moved to pci_save_state() (and likewise, cxl_config_restore_state() moved to pci_restore_state())?

Quoted 2 lines

E.g. when a DPC event occurs, I assume CXL registers need to be restored as well on recovery, right?

The CXL spec has some comic language around DPC that basically says "use with care, DPC trigger will bring down physical link, reset devicestate, disrupt CXL.cache and CXL.mem traffic".
or in shorter words
'Good luck'

If a CXL device undergoes DPC high chance you'll either trigger CXL isolation which we aren't handing yet in Linux because we aren't convinced software can really recover form it, or stall a CPU and end up rebooting.

Maybe we'll one day we'll figure this out. Today turn off DPC on CXL ports! :)

J

Quoted 16 lines

Note that since v6.19-rc1, state is saved on enumeration and thus always available for recovery, see a2f1e22390ac.

As a general remark on this series, it seems to have considerable overlap with Terry's work on AER support for CXL, particularly in patch [01/10], as Jonathan has remarked. Please cc Terry on future submissions and coordinate with him on conflicting parts of your patches:

https://lore.kernel.org/all/[email protected]/

Thanks,

Lukas

#37 lore raw

Re: [PATCH v4 09/10] PCI: save/restore CXL config around reset

Quoted 27 lines · On Thu, 22 Jan 2026 10:47:45 +0000 Jonathan Cameron…
Quoted 16 lines · On Thu, 22 Jan 2026 11:01:57 +0100 Lukas Wunner <[email protected]> wrote:

drivers/pci/pci.c static int cxl_reset(struct pci_dev *dev, bool probe) · line 4997

Quoted 10 lines · On Tue, Jan 20, 2026 at 10:26:09PM +0000, [email protected] wrote:

+++ b/drivers/pci/pci.c

drivers/pci/pci.c +5 -0
@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)
 	if (probe)
 		return 0;
 
+	pci_save_state(dev);
+	rc = cxl_config_save_state(dev, &cxl_state);
+	if (rc)
+		pci_warn(dev, "Failed to save CXL config state: %d\n", rc);
+    

Hm, shouldn't the call to cxl_config_save_state() be moved to pci_save_state() (and likewise, cxl_config_restore_state() moved to pci_restore_state())?

Quoted 2 lines

E.g. when a DPC event occurs, I assume CXL registers need to be restored as well on recovery, right?

The CXL spec has some comic language around DPC that basically says "use with care, DPC trigger will bring down physical link, reset devicestate, disrupt CXL.cache and CXL.mem traffic".
or in shorter words
'Good luck'

Quoted 5 lines

If a CXL device undergoes DPC high chance you'll either trigger CXL isolation which we aren't handing yet in Linux because we aren't convinced software can really recover form it, or stall a CPU and end up rebooting.

Maybe we'll one day we'll figure this out. Today turn off DPC on CXL ports! :)

Even if we hand-wave that DPC isn't an issue, save/restore of the PCI state happens at a higher level for every other PCI reset method and we're creating inconsistency here.

PCI-core includes interfaces for saving PCI state, offloading PCI state as an opaque blob, reloading, and restoring that state, and performing resets without saving and restoring state. This has a couple users, including vfio.

If we want similar behavior for CXL type2 devices for a future vfio use case, we shouldn't create unnecessary differentiation here with saving the CXL state separately and making the reset method behave differently. Thanks,

Alex

#38 lore raw

Re: [PATCH v4 09/10] PCI: save/restore CXL config around reset

Quoted 45 lines · On Mon, 26 Jan 2026 15:34:35 -0700 Alex Williamson <[email protected]> wrote:
Quoted 27 lines · On Thu, 22 Jan 2026 10:47:45 +0000 Jonathan Cameron…
Quoted 16 lines · On Thu, 22 Jan 2026 11:01:57 +0100 Lukas Wunner <[email protected]> wrote:

drivers/pci/pci.c static int cxl_reset(struct pci_dev *dev, bool probe) · line 4997

Quoted 10 lines · On Tue, Jan 20, 2026 at 10:26:09PM +0000, [email protected] wrote:

+++ b/drivers/pci/pci.c

drivers/pci/pci.c +5 -0
@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)
 	if (probe)
 		return 0;
 
+	pci_save_state(dev);
+	rc = cxl_config_save_state(dev, &cxl_state);
+	if (rc)
+		pci_warn(dev, "Failed to save CXL config state: %d\n", rc);
+      

Hm, shouldn't the call to cxl_config_save_state() be moved to pci_save_state() (and likewise, cxl_config_restore_state() moved to pci_restore_state())?

Quoted 2 lines

E.g. when a DPC event occurs, I assume CXL registers need to be restored as well on recovery, right?

The CXL spec has some comic language around DPC that basically says "use with care, DPC trigger will bring down physical link, reset devicestate, disrupt CXL.cache and CXL.mem traffic".
or in shorter words
'Good luck'

Quoted 5 lines

If a CXL device undergoes DPC high chance you'll either trigger CXL isolation which we aren't handing yet in Linux because we aren't convinced software can really recover form it, or stall a CPU and end up rebooting.

Maybe we'll one day we'll figure this out. Today turn off DPC on CXL ports! :)

Even if we hand-wave that DPC isn't an issue, save/restore of the PCI state happens at a higher level for every other PCI reset method and we're creating inconsistency here.

PCI-core includes interfaces for saving PCI state, offloading PCI state as an opaque blob, reloading, and restoring that state, and performing resets without saving and restoring state. This has a couple users, including vfio.

Quoted 5 lines

If we want similar behavior for CXL type2 devices for a future vfio use case, we shouldn't create unnecessary differentiation here with saving the CXL state separately and making the reset method behave differently. Thanks,

I'm a bit concerned that, unlike PCI where no traffic flows after reset and restore of basic PCIe stuff, for CXL once you've put the decoders etc back in place, CXL.mem traffic can happen autonomously. It's cacheable and physical address prefetchers on the CPU side may be able wander into it more or less randomly, whether there are page tables yet or not.

This is somewhat similar to PCI devices misbehaving if you enable bus mastering without ensuring they are in a clean state (just in the other direction).

So I'm not sure how safe it is to restore the generic CXL state with out the driver taking control.

I don't think there are tight enough guarantees that devices should be able to survive this if their drivers haven't managed the setup of CXL.mem carefully as they did during driver bind etc. Maybe they had to load a firmware first before there was anything behind a CXL protocol front end.

The drivers can't stop CXL.mem in a prepare reset callback prior to saving state as it may be RWL by an annoying BIOS.

Maybe I'm overly paranoid and all device manufacturers are sensible. Or I missed some spec text that says devices should politely handle traffic turning up before they are ready. If they implement the memory ready checks then we may be fine as hopefully Media Status == Ready doesn't happen until it's safe to enable access (though the spec doesn't actually say that is sufficient that I can find).

I need to do some more digging and maybe a spot of prototyping. Also more than plausible I'm missing a nugget of code in here that makes this all safe.

Jonathan

Quoted 1 line

Alex

#39 lore raw

PATCH v4 10/10 cxl: add HDM decoder and IDE save/restore

In reply to [email protected]

From: Srirangan Madhavan <[email protected]>

Extend state save/restore to HDM decoder and IDE registers for Type 2 devices. The HDM/IDE blocks are located via the component register map, then preserved across reset to retain decoder configuration and IDE policy. This avoids losing HDM/IDE programming when cxl_reset is issued.

---
 drivers/cxl/core/regs.c |   7 ++
 drivers/cxl/cxl.h       |   4 ++
 drivers/cxl/pci.c       | 153 ++++++++++++++++++++++++++++++++++++++--
 include/cxl/pci.h       |  43 +++++++++++
 4 files changed, 201 insertions(+), 6 deletions(-)
drivers/cxl/core/regs.c +7 -0
index ecdb22ae6952..76d6869d82ea 100644
@@ -93,6 +93,12 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
 			length = CXL_RAS_CAPABILITY_LENGTH;
 			rmap = &map->ras;
 			break;
+		case CXL_CM_CAP_CAP_ID_IDE:
+			dev_dbg(dev, "found IDE capability (0x%x)\n",
+				offset);
+			length = CXL_IDE_CAPABILITY_LENGTH;
+			rmap = &map->ide;
+			break;
 		default:
 			dev_dbg(dev, "Unknown CM cap ID: %d (0x%x)\n", cap_id,
 				offset);
@@ -212,6 +218,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
 	} mapinfo[] = {
 		{ &map->component_map.hdm_decoder, &regs->hdm_decoder },
 		{ &map->component_map.ras, &regs->ras },
+		{ &map->component_map.ide, &regs->ide },
 	};
 	int i;
 
drivers/cxl/cxl.h +4 -0
index ba17fa86d249..a7a6b79755b3 100644
@@ -39,8 +39,10 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
 #define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)
 
 #define   CXL_CM_CAP_CAP_ID_RAS 0x2
+#define   CXL_CM_CAP_CAP_ID_IDE 0x4
 #define   CXL_CM_CAP_CAP_ID_HDM 0x5
 #define   CXL_CM_CAP_CAP_HDM_VERSION 1
+#define   CXL_IDE_CAPABILITY_LENGTH 0x20
 
 /* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
 #define CXL_HDM_DECODER_CAP_OFFSET 0x0
@@ -214,6 +216,7 @@ struct cxl_regs {
 	struct_group_tagged(cxl_component_regs, component,
 		void __iomem *hdm_decoder;
 		void __iomem *ras;
+		void __iomem *ide;
 	);
 	/*
 	 * Common set of CXL Device register block base pointers
@@ -256,6 +259,7 @@ struct cxl_reg_map {
 struct cxl_component_reg_map {
 	struct cxl_reg_map hdm_decoder;
 	struct cxl_reg_map ras;
+	struct cxl_reg_map ide;
 };
 
 struct cxl_device_reg_map {
drivers/cxl/pci.c +147 -6

193 lines not on this page · raw message

include/cxl/pci.h +43 -0

60 lines not on this page · raw message

--
2.34.1

#40 lore raw

Re: [PATCH v4 10/10] cxl: add HDM decoder and IDE save/restore

Quoted 8 lines · On Tue, 20 Jan 2026 22:26:10 +0000 [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Extend state save/restore to HDM decoder and IDE registers for Type 2 devices. The HDM/IDE blocks are located via the component register map, then preserved across reset to retain decoder configuration and IDE policy. This avoids losing HDM/IDE programming when cxl_reset is issued.

Split the two up. One patch for HDM, one for IDE.

Been a while since I read the IDE stuff.

Isn't reset going to trip up the encryption to the extent that we need to redo the key exchange etc?

Quoted 12 lines
---
 drivers/cxl/core/regs.c |   7 ++
 drivers/cxl/cxl.h       |   4 ++
 drivers/cxl/pci.c       | 153 ++++++++++++++++++++++++++++++++++++++--
 include/cxl/pci.h       |  43 +++++++++++
 4 files changed, 201 insertions(+), 6 deletions(-)
drivers/cxl/core/regs.c +0 -0
index ecdb22ae6952..76d6869d82ea 100644

drivers/cxl/core/regs.c void cxl_probe_component_regs(struct device *dev, void __iomem *base, · line 98

Quoted 6 lines
drivers/cxl/core/regs.c +3 -0
@@ -93,6 +93,12 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
 			length = CXL_RAS_CAPABILITY_LENGTH;
 			rmap = &map->ras;
 			break;
+		case CXL_CM_CAP_CAP_ID_IDE:
+			dev_dbg(dev, "found IDE capability (0x%x)\n",
+				offset);

Trivial: Fits on one line under 80 chars.

Quoted 14 lines
drivers/cxl/core/regs.c +3 -0
+			length = CXL_IDE_CAPABILITY_LENGTH;
+			rmap = &map->ide;
+			break;
 
drivers/cxl/pci.c +5 -0
index 7d6a0ef70b2d..eb735d0ae175 100644
 
+/*
+ * CXL HDM Decoder register save/restore
+ */
+static int cxl_save_hdm_state(struct cxl_dev_state *cxlds,
+			      struct cxl_type2_saved_state *state)

drivers/cxl/pci.c static int cxl_restore_dvsec_state(struct pci_dev *pdev, · line 1221

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+{
+	void __iomem *hdm = cxlds->regs.hdm_decoder;
+	u32 cap, ctrl;
+	int i, count;
+
+	if (!hdm)
+		return 0;
+
+	cap = readl(hdm + CXL_HDM_DECODER_CAP_OFFSET);
+	count = cap & CXL_HDM_DECODER_COUNT_MASK;

Use FIELD_GET() for this so we don't need to go check the definition to be sure there isn't a shift.

Quoted 13 lines
drivers/cxl/pci.c +13 -0
+	count = min(count, CXL_MAX_DECODERS);
+
+	state->hdm_decoder_count = count;
+	state->hdm_global_ctrl = readl(hdm + CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET);
+
+	for (i = 0; i < count; i++) {
+		struct cxl_hdm_decoder_state *d = &state->decoders[i];
+		u32 base_low, base_high, size_low, size_high;
+		u32 dpa_skip_low, dpa_skip_high;
+
+		base_low = readl(hdm + CXL_HDM_DECODER_BASE_LOW(i));
+		base_high = readl(hdm + CXL_HDM_DECODER_BASE_HIGH(i));
+		size_low = readl(hdm + CXL_HDM_DECODER_SIZE_LOW(i));

drivers/cxl/pci.c static int cxl_restore_dvsec_state(struct pci_dev *pdev, · line 1244

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+		size_high = readl(hdm + CXL_HDM_DECODER_SIZE_HIGH(i));
+		ctrl = readl(hdm + CXL_HDM_DECODER_CTRL(i));
+		dpa_skip_low = readl(hdm + CXL_HDM_DECODER_DPA_SKIP_LOW(i));
+		dpa_skip_high = readl(hdm + CXL_HDM_DECODER_DPA_SKIP_HIGH(i));
+
+		d->base = ((u64)base_high << 32) | base_low;
+		d->size = ((u64)size_high << 32) | size_low;
+		d->ctrl = ctrl;
+		d->dpa_skip = ((u64)dpa_skip_high << 32) | dpa_skip_low;
+		d->enabled = !!(ctrl & CXL_HDM_DECODER_ENABLE);

I'm lazy. Why not just stash the register values as a raw block and put them back? Do we want to put them back if they aren't locked? We ripped down the region so I'd be kind of expecting the type 2 driver for the specific hardware to need to put this stuff back as part of it's own reinit.

Quoted 4 lines
drivers/cxl/pci.c +4 -0
+	}
+
+	return 0;
+}
maintainer
#41 lore raw

Re: [PATCH v4 10/10] cxl: add HDM decoder and IDE save/restore

In reply to [email protected]
Quoted 6 lines · On 1/20/26 3:26 PM, [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Extend state save/restore to HDM decoder and IDE registers for Type 2 devices. The HDM/IDE blocks are located via the component register map, then preserved across reset to retain decoder configuration and IDE policy. This avoids losing HDM/IDE programming when cxl_reset is issued.

So the current implementation tries to save DVSEC, HDM decoders, and IDE registers (MMIO registers). This is not standard device reset and seems more usage specific. Given that is the case and the fact that the implementation keeps trying to pull very CXL specific bits into PCI core, maybe going through the PCI reset path isn't the right place? The whole thing would be significantly simplified if we add a 'reset' sysfs attribute to memdev and just do it from there? Thoughts? That would prevent CXL symbols keep being dragged into PCI and keep everything local for this specific usage. Probably should also add some text on the reasoning of saving of these MMIO registers.

DJ

Quoted 336 lines
---
 drivers/cxl/core/regs.c |   7 ++
 drivers/cxl/cxl.h       |   4 ++
 drivers/cxl/pci.c       | 153 ++++++++++++++++++++++++++++++++++++++--
 include/cxl/pci.h       |  43 +++++++++++
 4 files changed, 201 insertions(+), 6 deletions(-)
drivers/cxl/core/regs.c +7 -0
index ecdb22ae6952..76d6869d82ea 100644
@@ -93,6 +93,12 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
 			length = CXL_RAS_CAPABILITY_LENGTH;
 			rmap = &map->ras;
 			break;
+		case CXL_CM_CAP_CAP_ID_IDE:
+			dev_dbg(dev, "found IDE capability (0x%x)\n",
+				offset);
+			length = CXL_IDE_CAPABILITY_LENGTH;
+			rmap = &map->ide;
+			break;
 		default:
 			dev_dbg(dev, "Unknown CM cap ID: %d (0x%x)\n", cap_id,
 				offset);
@@ -212,6 +218,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
 	} mapinfo[] = {
 		{ &map->component_map.hdm_decoder, &regs->hdm_decoder },
 		{ &map->component_map.ras, &regs->ras },
+		{ &map->component_map.ide, &regs->ide },
 	};
 	int i;
 
drivers/cxl/cxl.h +4 -0
index ba17fa86d249..a7a6b79755b3 100644
@@ -39,8 +39,10 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
 #define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)
 
 #define   CXL_CM_CAP_CAP_ID_RAS 0x2
+#define   CXL_CM_CAP_CAP_ID_IDE 0x4
 #define   CXL_CM_CAP_CAP_ID_HDM 0x5
 #define   CXL_CM_CAP_CAP_HDM_VERSION 1
+#define   CXL_IDE_CAPABILITY_LENGTH 0x20
 
 /* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
 #define CXL_HDM_DECODER_CAP_OFFSET 0x0
@@ -214,6 +216,7 @@ struct cxl_regs {
 	struct_group_tagged(cxl_component_regs, component,
 		void __iomem *hdm_decoder;
 		void __iomem *ras;
+		void __iomem *ide;
 	);
 	/*
 	 * Common set of CXL Device register block base pointers
@@ -256,6 +259,7 @@ struct cxl_reg_map {
 struct cxl_component_reg_map {
 	struct cxl_reg_map hdm_decoder;
 	struct cxl_reg_map ras;
+	struct cxl_reg_map ide;
 };
 
 struct cxl_device_reg_map {
drivers/cxl/pci.c +147 -6
index 7d6a0ef70b2d..eb735d0ae175 100644
@@ -956,7 +956,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		dev_dbg(&pdev->dev, "RAS registers not found\n");
 
 	rc = cxl_map_component_regs(&cxlds->reg_map, &cxlds->regs.component,
-				    BIT(CXL_CM_CAP_CAP_ID_RAS));
+				    BIT(CXL_CM_CAP_CAP_ID_RAS) |
+				    BIT(CXL_CM_CAP_CAP_ID_IDE));
 	if (rc)
 		dev_dbg(&pdev->dev, "Failed to map RAS capability.\n");
 
@@ -1203,18 +1204,126 @@ static int cxl_restore_dvsec_state(struct pci_dev *pdev,
 	return rc;
 }
 
+/*
+ * CXL HDM Decoder register save/restore
+ */
+static int cxl_save_hdm_state(struct cxl_dev_state *cxlds,
+			      struct cxl_type2_saved_state *state)
+{
+	void __iomem *hdm = cxlds->regs.hdm_decoder;
+	u32 cap, ctrl;
+	int i, count;
+
+	if (!hdm)
+		return 0;
+
+	cap = readl(hdm + CXL_HDM_DECODER_CAP_OFFSET);
+	count = cap & CXL_HDM_DECODER_COUNT_MASK;
+	count = min(count, CXL_MAX_DECODERS);
+
+	state->hdm_decoder_count = count;
+	state->hdm_global_ctrl = readl(hdm + CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET);
+
+	for (i = 0; i < count; i++) {
+		struct cxl_hdm_decoder_state *d = &state->decoders[i];
+		u32 base_low, base_high, size_low, size_high;
+		u32 dpa_skip_low, dpa_skip_high;
+
+		base_low = readl(hdm + CXL_HDM_DECODER_BASE_LOW(i));
+		base_high = readl(hdm + CXL_HDM_DECODER_BASE_HIGH(i));
+		size_low = readl(hdm + CXL_HDM_DECODER_SIZE_LOW(i));
+		size_high = readl(hdm + CXL_HDM_DECODER_SIZE_HIGH(i));
+		ctrl = readl(hdm + CXL_HDM_DECODER_CTRL(i));
+		dpa_skip_low = readl(hdm + CXL_HDM_DECODER_DPA_SKIP_LOW(i));
+		dpa_skip_high = readl(hdm + CXL_HDM_DECODER_DPA_SKIP_HIGH(i));
+
+		d->base = ((u64)base_high << 32) | base_low;
+		d->size = ((u64)size_high << 32) | size_low;
+		d->ctrl = ctrl;
+		d->dpa_skip = ((u64)dpa_skip_high << 32) | dpa_skip_low;
+		d->enabled = !!(ctrl & CXL_HDM_DECODER_ENABLE);
+	}
+
+	return 0;
+}
+
+static int cxl_restore_hdm_state(struct cxl_dev_state *cxlds,
+				 const struct cxl_type2_saved_state *state)
+{
+	void __iomem *hdm = cxlds->regs.hdm_decoder;
+	int i;
+
+	if (!hdm || state->hdm_decoder_count == 0)
+		return 0;
+
+	writel(state->hdm_global_ctrl, hdm + CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET);
+
+	for (i = 0; i < state->hdm_decoder_count; i++) {
+		const struct cxl_hdm_decoder_state *d = &state->decoders[i];
+
+		writel((u32)d->base, hdm + CXL_HDM_DECODER_BASE_LOW(i));
+		writel((u32)(d->base >> 32), hdm + CXL_HDM_DECODER_BASE_HIGH(i));
+		writel((u32)d->size, hdm + CXL_HDM_DECODER_SIZE_LOW(i));
+		writel((u32)(d->size >> 32), hdm + CXL_HDM_DECODER_SIZE_HIGH(i));
+		writel(d->ctrl, hdm + CXL_HDM_DECODER_CTRL(i));
+		writel((u32)d->dpa_skip, hdm + CXL_HDM_DECODER_DPA_SKIP_LOW(i));
+		writel((u32)(d->dpa_skip >> 32), hdm + CXL_HDM_DECODER_DPA_SKIP_HIGH(i));
+	}
+
+	return 0;
+}
+
+/*
+ * CXL IDE register save/restore
+ */
+static int cxl_save_ide_state(struct cxl_dev_state *cxlds,
+			      struct cxl_type2_saved_state *state)
+{
+	void __iomem *ide = cxlds->regs.ide;
+	u32 cap;
+
+	if (!ide)
+		return 0;
+
+	cap = readl(ide + CXL_IDE_CAP_OFFSET);
+	if (!(cap & CXL_IDE_CAP_CAPABLE))
+		return 0;
+
+	state->ide_cap = cap;
+	state->ide_ctrl = readl(ide + CXL_IDE_CTRL_OFFSET);
+	state->ide_key_refresh_time = readl(ide + CXL_IDE_KEY_REFRESH_TIME_CTRL_OFFSET);
+	state->ide_truncation_delay = readl(ide + CXL_IDE_TRUNCATION_DELAY_CTRL_OFFSET);
+
+	return 0;
+}
+
+static int cxl_restore_ide_state(struct cxl_dev_state *cxlds,
+				 const struct cxl_type2_saved_state *state)
+{
+	void __iomem *ide = cxlds->regs.ide;
+
+	if (!ide || !(state->ide_cap & CXL_IDE_CAP_CAPABLE))
+		return 0;
+
+	writel(state->ide_ctrl, ide + CXL_IDE_CTRL_OFFSET);
+	writel(state->ide_key_refresh_time, ide + CXL_IDE_KEY_REFRESH_TIME_CTRL_OFFSET);
+	writel(state->ide_truncation_delay, ide + CXL_IDE_TRUNCATION_DELAY_CTRL_OFFSET);
+
+	return 0;
+}
+
 /**
  * cxl_config_save_state - Save CXL configuration state
  * @pdev: PCI device
  * @state: Structure to store saved state
  *
- * Saves CXL DVSEC state before reset.
+ * Saves CXL DVSEC, HDM decoder, and IDE state before reset.
  */
 int cxl_config_save_state(struct pci_dev *pdev,
 			  struct cxl_type2_saved_state *state)
 {
 	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
-	int dvsec;
+	int rc, dvsec;
 
 	if (!cxlds || !state)
 		return -EINVAL;
@@ -1225,7 +1334,23 @@ int cxl_config_save_state(struct pci_dev *pdev,
 	if (!dvsec)
 		return -ENODEV;
 
-	return cxl_save_dvsec_state(pdev, state, dvsec);
+	rc = cxl_save_dvsec_state(pdev, state, dvsec);
+	if (rc)
+		return rc;
+
+	if (cxlds->regs.hdm_decoder) {
+		rc = cxl_save_hdm_state(cxlds, state);
+		if (rc)
+			pci_warn(pdev, "Failed to save HDM state: %d\n", rc);
+	}
+
+	if (cxlds->regs.ide) {
+		rc = cxl_save_ide_state(cxlds, state);
+		if (rc)
+			pci_warn(pdev, "Failed to save IDE state: %d\n", rc);
+	}
+
+	return 0;
 }
 EXPORT_SYMBOL_NS_GPL(cxl_config_save_state, "CXL");
 
@@ -1234,7 +1359,7 @@ EXPORT_SYMBOL_NS_GPL(cxl_config_save_state, "CXL");
  * @pdev: PCI device
  * @state: Previously saved state
  *
- * Restores CXL DVSEC state after reset.
+ * Restores CXL DVSEC, HDM decoder, and IDE state after reset.
  */
 int cxl_config_restore_state(struct pci_dev *pdev,
 			     const struct cxl_type2_saved_state *state)
@@ -1257,7 +1382,23 @@ int cxl_config_restore_state(struct pci_dev *pdev,
 
 	config_locked = !!(lock_reg & CXL_DVSEC_LOCK_CONFIG_LOCK);
 
-	return cxl_restore_dvsec_state(pdev, state, dvsec, config_locked);
+	rc = cxl_restore_dvsec_state(pdev, state, dvsec, config_locked);
+	if (rc)
+		return rc;
+
+	if (cxlds->regs.hdm_decoder && state->hdm_decoder_count > 0) {
+		rc = cxl_restore_hdm_state(cxlds, state);
+		if (rc)
+			pci_warn(pdev, "Failed to restore HDM state: %d\n", rc);
+	}
+
+	if (cxlds->regs.ide && (state->ide_cap & CXL_IDE_CAP_CAPABLE)) {
+		rc = cxl_restore_ide_state(cxlds, state);
+		if (rc)
+			pci_warn(pdev, "Failed to restore IDE state: %d\n", rc);
+	}
+
+	return 0;
 }
 EXPORT_SYMBOL_NS_GPL(cxl_config_restore_state, "CXL");
 
include/cxl/pci.h +43 -1
index 2c629ded73cc..9f2a9ad10d75 100644
@@ -4,11 +4,33 @@
 #ifndef __CXL_ACCEL_PCI_H
 #define __CXL_ACCEL_PCI_H
 
+/* HDM Decoder state for save/restore */
+struct cxl_hdm_decoder_state {
+	u64 base;
+	u64 size;
+	u32 ctrl;
+	u64 dpa_skip;
+	bool enabled;
+};
+
+#define CXL_MAX_DECODERS 10
+
 /* CXL Type 2 device state for save/restore across reset */
 struct cxl_type2_saved_state {
 	/* DVSEC registers */
 	u16 dvsec_ctrl;
 	u16 dvsec_ctrl2;
+
+	/* HDM Decoder registers */
+	u32 hdm_decoder_count;
+	u32 hdm_global_ctrl;
+	struct cxl_hdm_decoder_state decoders[CXL_MAX_DECODERS];
+
+	/* IDE registers */
+	u32 ide_cap;
+	u32 ide_ctrl;
+	u32 ide_key_refresh_time;
+	u32 ide_truncation_delay;
 };
 
 int cxl_config_save_state(struct pci_dev *pdev,
@@ -58,6 +80,27 @@ int cxl_config_restore_state(struct pci_dev *pdev,
 
 #define CXL_DVSEC_RANGE_MAX		2
 
+/* CXL HDM Decoder Capability Structure (Section 8.2.4.20) */
+#define CXL_HDM_DECODER_CAP_OFFSET		0x0
+#define   CXL_HDM_DECODER_COUNT_MASK		GENMASK(3, 0)
+#define CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET	0x4
+#define   CXL_HDM_DECODER_ENABLE		BIT(1)
+/* CXL HDM Decoder n registers (Offset 20h*n + base) */
+#define CXL_HDM_DECODER_BASE_LOW(n)		(0x10 + ((n) * 0x20))
+#define CXL_HDM_DECODER_BASE_HIGH(n)		(0x14 + ((n) * 0x20))
+#define CXL_HDM_DECODER_SIZE_LOW(n)		(0x18 + ((n) * 0x20))
+#define CXL_HDM_DECODER_SIZE_HIGH(n)		(0x1C + ((n) * 0x20))
+#define CXL_HDM_DECODER_CTRL(n)			(0x20 + ((n) * 0x20))
+#define CXL_HDM_DECODER_DPA_SKIP_LOW(n)		(0x24 + ((n) * 0x20))
+#define CXL_HDM_DECODER_DPA_SKIP_HIGH(n)	(0x28 + ((n) * 0x20))
+
+/* CXL IDE Capability Structure (Section 8.2.4.22) */
+#define CXL_IDE_CAP_OFFSET			0x00
+#define   CXL_IDE_CAP_CAPABLE			BIT(0)
+#define CXL_IDE_CTRL_OFFSET			0x04
+#define CXL_IDE_KEY_REFRESH_TIME_CTRL_OFFSET	0x18
+#define CXL_IDE_TRUNCATION_DELAY_CTRL_OFFSET	0x1C
+
 /* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */
 #define CXL_DVSEC_FUNCTION_MAP					2
 
--

2.34.1

maintainer
#42 lore raw
In reply to [email protected]
Quoted 7 lines · On Tue, Jan 20, 2026 at 10:26:00PM +0000, [email protected] wrote:

From: Srirangan Madhavan <[email protected]>

Hi folks!

This patch series introduces support for the CXL Reset method for CXL devices, implementing the reset procedure outlined in CXL Spec [1] v3.2, Sections 9.6 and 9.7.

Hi Srirangan,

Following-up on the base commit for this patch. For the cxl subsystem, a patchset today is expected to be based on 6.19-rc4 (or rc5 or rc6). We get to that because our cxl/next is based on rc4. We ask folks not to base on cxl/next because we want to see conflicts between patchsets.

I like to use the base commit option to automatically append it to the cover letter, but you can just state it in the change log if your method of sending patches doesn't support that.

With this set, there is something else that you are depending upon. Please call that out and point to the commits in some way so we can build on 6.19-rc4 plus those commits plus this patchset.

Thanks!
Alison

Quoted 115 lines

v4 changes:
- Fix CXL reset capability check parentheses warning
- Gate CXL reset path on CONFIG_CXL_PCI reachability

v3 changes:
- Restrict CXL reset to Type 2 devices only
- Add host and device cache flushing for
* all sibling functions on multi-function devices
* all sibling devices in a given region
- Add region teardown and memory online detection before reset
- Add configuration state save/restore (DVSEC, HDM, IDE)
- Split the series by subsystem and functional blocks

v2 changes:
- De-duplicate CXL DVSEC register defines under include/cxl/pci.h
- Fix style-related issues

v1 changes:
- Added cover letter and dropped the RFC

The RFC patches can be found here [2]
v2 patches can be found here [3]

Motivation:
-----------
This change is broadly useful for reasons including but not limited to the following:

- As support for Type 2 devices [4] is being introduced, more devices will require finer-grained reset mechanisms beyond bus-wide reset methods.

- FLR does not affect CXL.cache or CXL.mem protocols, making CXL Reset the preferred method in some cases.

- The CXL spec (Sections 7.2.3 Binding and Unbinding, 9.5 FLR) highlights use cases like function rebinding and error recovery, where CXL Reset is explicitly mentioned.

Change Description:
-------------------

Patch 1: Move CXL DVSEC defines to the CXL PCI header
- Consolidate DVSEC register definitions under include/cxl/pci.h

Patch 2: Switch PCI CXL port DVSEC defines
- Use the shared CXL PCI header in the PCI core

Patch 3: Add Type 2 helper and reset DVSEC bits
- Add helper to identify Type 2 devices
- Define DVSEC reset/cache control bits

Patch 4: Add the CXL reset method in the PCI core
- Implement cxl_reset() method with capability checks and reset sequence
- Restrict to Type 2 devices

Patch 5: Add reset preparation and region teardown
- Implement region validation and teardown before reset
- Add device cache flush for all sibling devices in a given region

Patch 6: Wire CXL reset prepare/cleanup in PCI
- Call CXL reset prepare/cleanup around the core reset flow

Patch 7: Add host CPU cache flush and multi-function support
- Add host CPU cache flush (x86: wbinvd, arm64: dcache_clean_inval_poc)
- Add device cache flush for all sibling functions on multi-function devices

Patch 8: Add DVSEC configuration state save/restore
- Save/restore DVSEC registers (DEVCTL, DEVCTL2) with CONFIG_LOCK handling

Patch 9: Save/restore CXL config around reset
- Save PCI and CXL config before reset and restore afterwards

Patch 10: Add HDM decoder and IDE state save/restore
- Save/restore HDM decoder and IDE register state

The reset sequence: validate device type, check memory offline, tear down regions, flush host CPU caches, flush device caches (all functions), save config state, initiate reset, wait for completion, restore config state.

Command line to test the CXL reset on a capable device:
echo cxl_reset > /sys/bus/pci/devices/<pci_device>/reset_method echo 1 > /sys/bus/pci/devices/<pci_device>/reset

[1] https://computeexpresslink.org/cxl-specification/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/linux-cxl/[email protected]/

Srirangan Madhavan (10):
[PATCH v4 1/10] cxl: move DVSEC defines to cxl pci header [PATCH v4 2/10] PCI: switch CXL port DVSEC defines [PATCH v4 3/10] cxl: add type 2 helper and reset DVSEC bits [PATCH v4 4/10] PCI: add CXL reset method
[PATCH v4 5/10] cxl: add reset prepare and region teardown [PATCH v4 6/10] PCI: wire CXL reset prepare/cleanup [PATCH v4 7/10] cxl: add host cache flush and multi-function reset [PATCH v4 8/10] cxl: add DVSEC config save/restore [PATCH v4 9/10] PCI: save/restore CXL config around reset [PATCH v4 10/10] cxl: add HDM decoder and IDE save/restore

 drivers/cxl/core/pci.c        |   1 +
 drivers/cxl/core/regs.c       |   8 +
 drivers/cxl/cxl.h             |   4 +
 drivers/cxl/cxlpci.h          |  53 ---
 drivers/cxl/pci.c             | 621 +++++++++++++++++++++++++++++++++-
 drivers/pci/pci.c             | 150 +++++++-
 include/cxl/pci.h             | 134 ++++++++
 include/linux/pci.h           |  21 +-
 include/uapi/linux/pci_regs.h |   5 -
 9 files changed, 929 insertions(+), 68 deletions(-)
 create mode 100644 include/cxl/pci.h

--
2.34.1

#43 lore raw
In reply to [email protected]
Quoted 12 lines · On Tue, Jan 20, 2026 at 10:26:00PM +0000, [email protected] wrote:

...

[PATCH v4 1/10] cxl: move DVSEC defines to cxl pci header [PATCH v4 2/10] PCI: switch CXL port DVSEC defines [PATCH v4 3/10] cxl: add type 2 helper and reset DVSEC bits [PATCH v4 4/10] PCI: add CXL reset method
[PATCH v4 5/10] cxl: add reset prepare and region teardown [PATCH v4 6/10] PCI: wire CXL reset prepare/cleanup [PATCH v4 7/10] cxl: add host cache flush and multi-function reset [PATCH v4 8/10] cxl: add DVSEC config save/restore [PATCH v4 9/10] PCI: save/restore CXL config around reset [PATCH v4 10/10] cxl: add HDM decoder and IDE save/restore

Please run "git log --online" on the files you change and match the style of subject lines. In drivers/pci/, the subject lines start with capital letters, e.g.,

PCI: Switch CXL port ...
PCI: Add CXL reset method (but please include the function name)
...

#44 lore raw
In reply to [email protected]
Quoted 10 lines · On Tue, 20 Jan 2026 22:26:00 +0000 <[email protected]> wrote:

Motivation:
-----------
This change is broadly useful for reasons including but not limited to the following:

- As support for Type 2 devices [4] is being introduced, more devices will require finer-grained reset mechanisms beyond bus-wide reset methods.

- FLR does not affect CXL.cache or CXL.mem protocols, making CXL Reset the preferred method in some cases.

This proposal adds cxl_reset to the pci_reset_fn_methods[] array, so while the above suggests there are some use cases that would prefer CXL reset, we're interleaving the capability into the default function for PCI function scoped reset.

I'm concerned about use cases like vfio-pci emulation of FLR, where we currently call through the pci_reset_function() interfaces to take advantage of device specific quirks. Such a use case (guest directed FLR) might reasonably expect the scope of the reset is limited to CXL.io without affecting CXL.cache or CXL.mem, therefore it's not clear that pci_reset_function() necessarily has the correct scope.

OTOH, we also expose a reset ioctl through vfio-pci that could reasonably reset the full state of the device, PCI and CXL. cxl_reset would be appropriate here, however cxl_reset is prioritized below FLR reset in the default reset_methods array, so pci_reset_function() doesn't have the correct scope here either.

Is it really appropriate to consider cxl_reset as just another PCI reset method given the scope difference? Does there need to be a way to specify the scope when calling pci_reset_function() for drivers aware of CXL.mem/cache? Thanks,

Alex

#45 lore raw

Alex Williamson wrote:
[..]

Quoted 4 lines

Is it really appropriate to consider cxl_reset as just another PCI reset method given the scope difference? Does there need to be a way to specify the scope when calling pci_reset_function() for drivers aware of CXL.mem/cache?

Right, FWIW, I am of the opinion that trying to plumb CXL reset through PCI reset is an awkward fit that will be an ongoing source of pain. For the same reason that CXL protocol error handling is designed to have little do with PCIe protocol error handling, CXL is not PCIe. For example, the goal in the error handling case is to arrive at a point where the PCIe error handling implementation can continue to evolve without inflicting the maintenance burden load of "what about CXL?". Notifications are simply shunted over to the CXL core early in the flow. CXL reset does not need any PCI core entanglements as far as I can see.

Once the protocol error handling series has landed there is a potential to extend that with CXL Reset recovery. However, that needs a clear error model defined as to which resets have a chance of recovering *system* operation when CXL.cache/mem fails. In Terry's series, panic / reboot is the recovery, not reset.

#46 lore raw

Hi Dan,

________________________________________
From: [email protected] <[email protected]>
Sent: Tuesday, January 27, 2026 11:02 AM
To: Alex Williamson; Srirangan Madhavan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vishal Aslot; Vikram Sethi; Shanker Donthineni; Vidya Sagar; Matt Ochs; Jason Sequeira
Subject: Re: [PATCH v4 0/10] CXL Reset support for Type 2 devices

Quoted 5 lines

Once the protocol error handling series has landed there is a potential to extend that with CXL Reset recovery. However, that needs a clear error model defined as to which resets have a chance of recovering *system* operation when CXL.cache/mem fails. In Terry's series, panic / reboot is the recovery, not reset.

It's not just about CXL protocol error handling though. Type2 Device passthrough will be a common usecase for CXL reset (entire device is assigned to VM) across different VM assignment.
The cache and mem of the device must be cleared via CXL reset for full device passthrough in addition to the PCIe/CXL.IO reset. Another usecase is when the device is reconfigured either in baremetal or in VM for passthrough usecase. Such usecases often require a reset of the device, and it's cache, mem and not a narrow "memregion" reset, so I'm not in favor of exposing it via CXL.mem sysfs entries. IMO, device sysfs attribute reset method is appropriate for type2 devices. Finally, there is the error usecase, which in the common case is as simple as an uncorrected ECC error in the HDM. While not strictly necessary, it is common practice to reset the device in such cases, to recover the bad page/row via PPR on device reset. You want reset of the memory controller as part of the CXL reset here, FLR is not enough.
Regarding scope, and "recoverability", we have significant experience with recovering "pre-CXL" coherent GB200 devices for device errors: memory ECC or other, by killing the application using device memory, unloading driver, resetting the device, and reloading driver. The CXL protocol error series can use CXL reset series, but I don't see that this series needs to wait for protocol error handling to be merged.

Thanks,
Vikram

#47 lore raw
Quoted 11 lines · Vikram Sethi

Hi Dan,

Quoted 5 lines

Once the protocol error handling series has landed there is a potential to extend that with CXL Reset recovery. However, that needs a clear error model defined as to which resets have a chance of recovering *system* operation when CXL.cache/mem fails. In Terry's series, panic / reboot is the recovery, not reset.

It's not just about CXL protocol error handling though. Type2 Device passthrough will be a common usecase for CXL reset (entire device is assigned to VM) across different VM assignment.

I understand. The point about CXL Protocol Error handling series is that it at least enlightens the PCIe core about the presence of active CXL links.

It is also the case that it much closer to being upstream than this set which still has fundamental questions.

Quoted 7 lines

The cache and mem of the device must be cleared via CXL reset for full device passthrough in addition to the PCIe/CXL.IO reset. Another usecase is when the device is reconfigured either in baremetal or in VM for passthrough usecase. Such usecases often require a reset of the device, and it's cache, mem and not a narrow "memregion" reset, so I'm not in favor of exposing it via CXL.mem sysfs entries. IMO, device sysfs attribute reset method is appropriate for type2 devices.

That case is already handled today with secondary bus reset to completely reset an entire device. That path is problematic for CXL because PCI reset has no idea about how to manage caches or handle memory unplug.

Administrator is responsible for making sure that event is not a surprise memory removal or cache protocol corruption event. That is a level of explosiveness that PCI reset has never needed to consider.

CXL Reset wants to be more surgical than secondary bus reset. The only way to be more surgical is to cooperate with the CXL core that knows whether the explosives have been rendered inert.

Quoted 6 lines

Finally, there is the error usecase, which in the common case is as simple as an uncorrected ECC error in the HDM. While not strictly necessary, it is common practice to reset the device in such cases, to recover the bad page/row via PPR on device reset. You want reset of the memory controller as part of the CXL reset here, FLR is not enough.

??

CXL memory repair is already upstream without CXL Reset, see CONFIG_CXL_EDAC_MEM_REPAIR.

Quoted 6 lines

Regarding scope, and "recoverability", we have significant experience with recovering "pre-CXL" coherent GB200 devices for device errors:
memory ECC or other, by killing the application using device memory, unloading driver, resetting the device, and reloading driver. The CXL protocol error series can use CXL reset series, but I don't see that this series needs to wait for protocol error handling to be merged.

I do want to get to the point where device memory error recovery is a first class citizen. CXL Protocol Error handling just happens to be at the top of the review queue and safe to assume it can be a foundation for further RAS features.

#48 lore raw
Quoted 48 lines · On Tue, 27 Jan 2026 19:42:58 -0800 [email protected] wrote:
Quoted 11 lines · Vikram Sethi

Hi Dan,

Quoted 5 lines

Once the protocol error handling series has landed there is a potential to extend that with CXL Reset recovery. However, that needs a clear error model defined as to which resets have a chance of recovering *system* operation when CXL.cache/mem fails. In Terry's series, panic / reboot is the recovery, not reset.

It's not just about CXL protocol error handling though. Type2 Device passthrough will be a common usecase for CXL reset (entire device is assigned to VM) across different VM assignment.

I understand. The point about CXL Protocol Error handling series is that it at least enlightens the PCIe core about the presence of active CXL links.

It is also the case that it much closer to being upstream than this set which still has fundamental questions.

Quoted 7 lines

The cache and mem of the device must be cleared via CXL reset for full device passthrough in addition to the PCIe/CXL.IO reset. Another usecase is when the device is reconfigured either in baremetal or in VM for passthrough usecase. Such usecases often require a reset of the device, and it's cache, mem and not a narrow "memregion" reset, so I'm not in favor of exposing it via CXL.mem sysfs entries. IMO, device sysfs attribute reset method is appropriate for type2 devices.

That case is already handled today with secondary bus reset to completely reset an entire device. That path is problematic for CXL because PCI reset has no idea about how to manage caches or handle memory unplug.

Administrator is responsible for making sure that event is not a surprise memory removal or cache protocol corruption event. That is a level of explosiveness that PCI reset has never needed to consider.

CXL Reset wants to be more surgical than secondary bus reset. The only way to be more surgical is to cooperate with the CXL core that knows whether the explosives have been rendered inert.

Quoted 6 lines

Finally, there is the error usecase, which in the common case is as simple as an uncorrected ECC error in the HDM. While not strictly necessary, it is common practice to reset the device in such cases, to recover the bad page/row via PPR on device reset. You want reset of the memory controller as part of the CXL reset here, FLR is not enough.

Quoted 4 lines

??

CXL memory repair is already upstream without CXL Reset, see CONFIG_CXL_EDAC_MEM_REPAIR.

To actually do it we rely on tear down of all access to the device (if it it's disruptive).
There is new spec stuff covering asking a device to just do it next reset that would fit more closely with this flow. No support yet I think.

Mind you we are talking type 2, so who knows what people will implement. Nice if they used that part of the spec but I doubt we can rely on it!

Jonathan

Quoted 14 lines
Quoted 7 lines

Regarding scope, and "recoverability", we have significant experience with recovering "pre-CXL" coherent GB200 devices for device errors:
memory ECC or other, by killing the application using device memory, unloading driver, resetting the device, and reloading driver. The CXL protocol error series can use CXL reset series, but I don't see that this series needs to wait for protocol error handling to be merged.

I do want to get to the point where device memory error recovery is a first class citizen. CXL Protocol Error handling just happens to be at the top of the review queue and safe to assume it can be a foundation for further RAS features.