CXL Reset support for Type 2 devices
v4 cover CXL Reset support for Type 2 devices
From: Srirangan Madhavan <[email protected]>
Hi folks!
This patch series introduces support for the CXL Reset method for CXL devices, implementing the reset procedure outlined in CXL Spec [1] v3.2, Sections 9.6 and 9.7.
v4 changes:
- Fix CXL reset capability check parentheses warning
- Gate CXL reset path on CONFIG_CXL_PCI reachability
v3 changes:
- Restrict CXL reset to Type 2 devices only
- Add host and device cache flushing for
* all sibling functions on multi-function devices
* all sibling devices in a given region
- Add region teardown and memory online detection before reset
- Add configuration state save/restore (DVSEC, HDM, IDE)
- Split the series by subsystem and functional blocks
v2 changes:
- De-duplicate CXL DVSEC register defines under include/cxl/pci.h
- Fix style-related issues
v1 changes:
- Added cover letter and dropped the RFC
The RFC patches can be found here [2]
v2 patches can be found here [3]
Motivation:
-----------
This change is broadly useful for reasons including but not limited to the following:
- As support for Type 2 devices [4] is being introduced, more devices will require finer-grained reset mechanisms beyond bus-wide reset methods.
- FLR does not affect CXL.cache or CXL.mem protocols, making CXL Reset the preferred method in some cases.
- The CXL spec (Sections 7.2.3 Binding and Unbinding, 9.5 FLR) highlights use cases like function rebinding and error recovery, where CXL Reset is explicitly mentioned.
Change Description:
-------------------
Patch 1: Move CXL DVSEC defines to the CXL PCI header
- Consolidate DVSEC register definitions under include/cxl/pci.h
Patch 2: Switch PCI CXL port DVSEC defines
- Use the shared CXL PCI header in the PCI core
Patch 3: Add Type 2 helper and reset DVSEC bits
- Add helper to identify Type 2 devices
- Define DVSEC reset/cache control bits
Patch 4: Add the CXL reset method in the PCI core
- Implement cxl_reset() method with capability checks and reset sequence
- Restrict to Type 2 devices
Patch 5: Add reset preparation and region teardown
- Implement region validation and teardown before reset
- Add device cache flush for all sibling devices in a given region
Patch 6: Wire CXL reset prepare/cleanup in PCI
- Call CXL reset prepare/cleanup around the core reset flow
Patch 7: Add host CPU cache flush and multi-function support
- Add host CPU cache flush (x86: wbinvd, arm64: dcache_clean_inval_poc)
- Add device cache flush for all sibling functions on multi-function devices
Patch 8: Add DVSEC configuration state save/restore
- Save/restore DVSEC registers (DEVCTL, DEVCTL2) with CONFIG_LOCK handling
Patch 9: Save/restore CXL config around reset
- Save PCI and CXL config before reset and restore afterwards
Patch 10: Add HDM decoder and IDE state save/restore
- Save/restore HDM decoder and IDE register state
The reset sequence: validate device type, check memory offline, tear down regions, flush host CPU caches, flush device caches (all functions), save config state, initiate reset, wait for completion, restore config state.
Command line to test the CXL reset on a capable device:
echo cxl_reset > /sys/bus/pci/devices/<pci_device>/reset_method echo 1 > /sys/bus/pci/devices/<pci_device>/reset
[1] https://computeexpresslink.org/cxl-specification/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/linux-cxl/[email protected]/
Srirangan Madhavan (10):
[PATCH v4 1/10] cxl: move DVSEC defines to cxl pci header [PATCH v4 2/10] PCI: switch CXL port DVSEC defines [PATCH v4 3/10] cxl: add type 2 helper and reset DVSEC bits [PATCH v4 4/10] PCI: add CXL reset method
[PATCH v4 5/10] cxl: add reset prepare and region teardown [PATCH v4 6/10] PCI: wire CXL reset prepare/cleanup [PATCH v4 7/10] cxl: add host cache flush and multi-function reset [PATCH v4 8/10] cxl: add DVSEC config save/restore [PATCH v4 9/10] PCI: save/restore CXL config around reset [PATCH v4 10/10] cxl: add HDM decoder and IDE save/restore
drivers/cxl/core/pci.c | 1 + drivers/cxl/core/regs.c | 8 + drivers/cxl/cxl.h | 4 + drivers/cxl/cxlpci.h | 53 --- drivers/cxl/pci.c | 621 +++++++++++++++++++++++++++++++++- drivers/pci/pci.c | 150 +++++++- include/cxl/pci.h | 134 ++++++++ include/linux/pci.h | 21 +- include/uapi/linux/pci_regs.h | 5 - 9 files changed, 929 insertions(+), 68 deletions(-) create mode 100644 include/cxl/pci.h
--
2.34.1
PATCH v4 1/10 cxl: move DVSEC defines to cxl pci header
From: Srirangan Madhavan <[email protected]>
CXL DVSEC definitions are shared across PCI core and CXL drivers, so move the register macros into the common CXL PCI header. This keeps the DVSEC surface in one place and avoids duplication as the reset and config helpers build on these offsets and bitfields.
--- drivers/cxl/core/pci.c | 1 + drivers/cxl/core/regs.c | 1 + drivers/cxl/cxlpci.h | 53 ----------------------------------- drivers/cxl/pci.c | 1 + include/cxl/pci.h | 62 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 65 insertions(+), 53 deletions(-) create mode 100644 include/cxl/pci.h
drivers/cxl/core/pci.c +1 -0
@@ -7,6 +7,7 @@#include <linux/pci.h> #include <linux/pci-doe.h> #include <linux/aer.h> +#include <cxl/pci.h> #include <cxlpci.h> #include <cxlmem.h> #include <cxl.h>
drivers/cxl/core/regs.c +1 -0
@@ -4,6 +4,7 @@#include <linux/device.h> #include <linux/slab.h> #include <linux/pci.h> +#include <cxl/pci.h> #include <cxlmem.h> #include <cxlpci.h> #include <pmu.h>
drivers/cxl/cxlpci.h +0 -53
59 lines not on this page · raw message
drivers/cxl/pci.c +1 -0
@@ -12,6 +12,7 @@#include <linux/aer.h> #include <linux/io.h> #include <cxl/mailbox.h> +#include <cxl/pci.h> #include "cxlmem.h" #include "cxlpci.h" #include "cxl.h"
include/cxl/pci.h new file +62 -0
62 lines not on this page · raw message
--
2.34.1
Re: [PATCH v4 01/10] cxl: move DVSEC defines to cxl pci header
Quoted 8 lines · On Tue, 20 Jan 2026 22:26:01 +0000 [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
CXL DVSEC definitions are shared across PCI core and CXL drivers, so move the register macros into the common CXL PCI header. This keeps the DVSEC surface in one place and avoids duplication as the reset and config helpers build on these offsets and bitfields.
Signed-off-by Srirangan Madhavan <[email protected]>
These are moving in:
https://lore.kernel.org/all/[email protected]/
However it is to the main uapi/linux/pci_regs.h file.
*fingers crossed* that should land this cycle and simplify your set a little.
Jonathan
Quoted 181 lines
--- drivers/cxl/core/pci.c | 1 + drivers/cxl/core/regs.c | 1 + drivers/cxl/cxlpci.h | 53 ----------------------------------- drivers/cxl/pci.c | 1 + include/cxl/pci.h | 62 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 65 insertions(+), 53 deletions(-) create mode 100644 include/cxl/pci.hdrivers/cxl/core/pci.c +1
-0@@ -7,6 +7,7 @@#include <linux/pci.h> #include <linux/pci-doe.h> #include <linux/aer.h> +#include <cxl/pci.h> #include <cxlpci.h> #include <cxlmem.h> #include <cxl.h> drivers/cxl/core/regs.c +1
-0@@ -4,6 +4,7 @@#include <linux/device.h> #include <linux/slab.h> #include <linux/pci.h> +#include <cxl/pci.h> #include <cxlmem.h> #include <cxlpci.h> #include <pmu.h> drivers/cxl/cxlpci.h +0
-53@@ -7,59 +7,6 @@#define CXL_MEMORY_PROGIF 0x10 -/* -* See section 8.1 Configuration Space Registers in the CXL 2.0 -* Specification. Names are taken straight from the specification with "CXL" and -* "DVSEC" redundancies removed. When obvious, abbreviations may be used. -*/ -#define PCI_DVSEC_HEADER1_LENGTH_MASK GENMASK(31, 20) - -/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ -#define CXL_DVSEC_PCIE_DEVICE 0 -#define CXL_DVSEC_CAP_OFFSET 0xA -#define CXL_DVSEC_MEM_CAPABLE BIT(2) -#define CXL_DVSEC_HDM_COUNT_MASK GENMASK(5, 4) -#define CXL_DVSEC_CTRL_OFFSET 0xC -#define CXL_DVSEC_MEM_ENABLE BIT(2) -#define CXL_DVSEC_RANGE_SIZE_HIGH (i) (0x18 + (i * 0x10)) -#define CXL_DVSEC_RANGE_SIZE_LOW (i) (0x1C + (i * 0x10)) -#define CXL_DVSEC_MEM_INFO_VALID BIT(0) -#define CXL_DVSEC_MEM_ACTIVE BIT(1) -#define CXL_DVSEC_MEM_SIZE_LOW_MASK GENMASK(31, 28) -#define CXL_DVSEC_RANGE_BASE_HIGH (i) (0x20 + (i * 0x10)) -#define CXL_DVSEC_RANGE_BASE_LOW (i) (0x24 + (i * 0x10)) -#define CXL_DVSEC_MEM_BASE_LOW_MASK GENMASK(31, 28) - -#define CXL_DVSEC_RANGE_MAX 2 - -/* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */ -#define CXL_DVSEC_FUNCTION_MAP 2 - -/* CXL 2.0 8.1.5: CXL 2.0 Extensions DVSEC for Ports */ -#define CXL_DVSEC_PORT_EXTENSIONS 3 - -/* CXL 2.0 8.1.6: GPF DVSEC for CXL Port */ -#define CXL_DVSEC_PORT_GPF 4 -#define CXL_DVSEC_PORT_GPF_PHASE_1_CONTROL_OFFSET 0x0C -#define CXL_DVSEC_PORT_GPF_PHASE_1_TMO_BASE_MASK GENMASK(3, 0) -#define CXL_DVSEC_PORT_GPF_PHASE_1_TMO_SCALE_MASK GENMASK(11, 8) -#define CXL_DVSEC_PORT_GPF_PHASE_2_CONTROL_OFFSET 0xE -#define CXL_DVSEC_PORT_GPF_PHASE_2_TMO_BASE_MASK GENMASK(3, 0) -#define CXL_DVSEC_PORT_GPF_PHASE_2_TMO_SCALE_MASK GENMASK(11, 8) - -/* CXL 2.0 8.1.7: GPF DVSEC for CXL Device */ -#define CXL_DVSEC_DEVICE_GPF 5 - -/* CXL 2.0 8.1.8: PCIe DVSEC for Flex Bus Port */ -#define CXL_DVSEC_PCIE_FLEXBUS_PORT 7 - -/* CXL 2.0 8.1.9: Register Locator DVSEC */ -#define CXL_DVSEC_REG_LOCATOR 8 -#define CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET 0xC -#define CXL_DVSEC_REG_LOCATOR_BIR_MASK GENMASK(2, 0) -#define CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK GENMASK(15, 8) -#define CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK GENMASK(31, 16) -/* * NOTE: Currently all the functions which are enabled for CXL require their * vectors to be in the first 16. Use this as the default max. drivers/cxl/pci.c +1
-0@@ -12,6 +12,7 @@#include <linux/aer.h> #include <linux/io.h> #include <cxl/mailbox.h> +#include <cxl/pci.h> #include "cxlmem.h" #include "cxlpci.h" #include "cxl.h" include/cxl/pci.h new file +62
-1@@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ + +#ifndef __CXL_ACCEL_PCI_H +#define __CXL_ACCEL_PCI_H + +/* +* See section 8.1 Configuration Space Registers in the CXL 2.0 +* Specification. Names are taken straight from the specification with "CXL" and +* "DVSEC" redundancies removed. When obvious, abbreviations may be used. +*/ +#define PCI_DVSEC_HEADER1_LENGTH_MASK GENMASK(31, 20) + +/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ +#define CXL_DVSEC_PCIE_DEVICE 0 +#define CXL_DVSEC_CAP_OFFSET 0xA +#define CXL_DVSEC_MEM_CAPABLE BIT(2) +#define CXL_DVSEC_HDM_COUNT_MASK GENMASK(5, 4) +#define CXL_DVSEC_CTRL_OFFSET 0xC +#define CXL_DVSEC_MEM_ENABLE BIT(2) +#define CXL_DVSEC_RANGE_SIZE_HIGH (i) (0x18 + ((i) * 0x10)) +#define CXL_DVSEC_RANGE_SIZE_LOW (i) (0x1C + ((i) * 0x10)) +#define CXL_DVSEC_MEM_INFO_VALID BIT(0) +#define CXL_DVSEC_MEM_ACTIVE BIT(1) +#define CXL_DVSEC_MEM_SIZE_LOW_MASK GENMASK(31, 28) +#define CXL_DVSEC_RANGE_BASE_HIGH (i) (0x20 + ((i) * 0x10)) +#define CXL_DVSEC_RANGE_BASE_LOW (i) (0x24 + ((i) * 0x10)) +#define CXL_DVSEC_MEM_BASE_LOW_MASK GENMASK(31, 28) + +#define CXL_DVSEC_RANGE_MAX 2 + +/* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */ +#define CXL_DVSEC_FUNCTION_MAP 2 + +/* CXL 2.0 8.1.5: CXL 2.0 Extensions DVSEC for Ports */ +#define CXL_DVSEC_PORT_EXTENSIONS 3 +#define CXL_DVSEC_PORT_CTL 0xC +#define CXL_DVSEC_UNMASK_SBR BIT(0) + +/* CXL 2.0 8.1.6: GPF DVSEC for CXL Port */ +#define CXL_DVSEC_PORT_GPF 4 +#define CXL_DVSEC_PORT_GPF_PHASE_1_CONTROL_OFFSET 0x0C +#define CXL_DVSEC_PORT_GPF_PHASE_1_TMO_BASE_MASK GENMASK(3, 0) +#define CXL_DVSEC_PORT_GPF_PHASE_1_TMO_SCALE_MASK GENMASK(11, 8) +#define CXL_DVSEC_PORT_GPF_PHASE_2_CONTROL_OFFSET 0xE +#define CXL_DVSEC_PORT_GPF_PHASE_2_TMO_BASE_MASK GENMASK(3, 0) +#define CXL_DVSEC_PORT_GPF_PHASE_2_TMO_SCALE_MASK GENMASK(11, 8) + +/* CXL 2.0 8.1.7: GPF DVSEC for CXL Device */ +#define CXL_DVSEC_DEVICE_GPF 5 + +/* CXL 2.0 8.1.8: PCIe DVSEC for Flex Bus Port */ +#define CXL_DVSEC_PCIE_FLEXBUS_PORT 7 + +/* CXL 2.0 8.1.9: Register Locator DVSEC */ +#define CXL_DVSEC_REG_LOCATOR 8 +#define CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET 0xC +#define CXL_DVSEC_REG_LOCATOR_BIR_MASK GENMASK(2, 0) +#define CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK GENMASK(15, 8) +#define CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK GENMASK(31, 16) + +#endif --2.34.1
PATCH v4 2/10 PCI: switch CXL port DVSEC defines
In reply to [email protected]From: Srirangan Madhavan <[email protected]>
The PCI core consumes CXL port DVSEC fields for reset handling, so switch it to use the shared CXL PCI header instead of the uapi header. This aligns the core with the header split and keeps internal code from depending on uapi-only definitions.
--- drivers/pci/pci.c | 17 +++++++++-------- include/uapi/linux/pci_regs.h | 5 ----- 2 files changed, 9 insertions(+), 13 deletions(-)
drivers/pci/pci.c +9 -8
@@ -30,6 +30,7 @@#include <asm/dma.h> #include <linux/aer.h> #include <linux/bitfield.h> +#include <cxl/pci.h> #include "pci.h" DEFINE_MUTEX (pci_slot_mutex);
@@ -4842,7 +4843,7 @@ static int pci_dev_reset_slot_function(struct pci_dev *dev, bool probe)static u16 cxl_port_dvsec (struct pci_dev *dev) {return pci_find_dvsec_capability (dev,PCI_VENDOR_ID_CXL , -PCI_DVSEC_CXL_PORT ); +CXL_DVSEC_PORT_EXTENSIONS ); }static bool cxl_sbr_masked (struct pci_dev *dev)
@@ -4854,7 +4855,7 @@ static bool cxl_sbr_masked(struct pci_dev *dev)if (!dvsec)return false; - rc =pci_read_config_word (dev, dvsec +PCI_DVSEC_CXL_PORT_CTL , ®); + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_PORT_CTL , ®);if (rc ||PCI_POSSIBLE_ERROR (reg))return false;
@@ -4863,7 +4864,7 @@ static bool cxl_sbr_masked(struct pci_dev *dev)* bit in Bridge Control has no effect. When 1, the Port generates * hot reset when the SBR bit is set to 1. */ -if (reg &PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR ) +if (reg &CXL_DVSEC_UNMASK_SBR )return false;return true;
@@ -4908,22 +4909,22 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)if (probe)return 0 ; - rc =pci_read_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , ®); + rc =pci_read_config_word (bridge, dvsec +CXL_DVSEC_PORT_CTL , ®);if (rc)return -ENOTTY ; -if (reg &PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR ) { +if (reg &CXL_DVSEC_UNMASK_SBR ) { val = reg; }else { - val = reg |PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR ; -pci_write_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , + val = reg |CXL_DVSEC_UNMASK_SBR ; +pci_write_config_word (bridge, dvsec +CXL_DVSEC_PORT_CTL , val); } rc =pci_reset_bus_function (dev, probe);if (reg != val) -pci_write_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , +pci_write_config_word (bridge, dvsec +CXL_DVSEC_PORT_CTL , reg);return rc;
include/uapi/linux/pci_regs.h +0 -5
@@ -1253,11 +1253,6 @@#define PCI_DEV3_STA 0x0c/* Device 3 Status Register */ #define PCI_DEV3_STA_SEGMENT 0x8/* Segment Captured (end-to-end flit-mode detected) */ -/* Compute Express Link (CXL r3.1, sec 8.1.5) */ -#define PCI_DVSEC_CXL_PORT 3 -#define PCI_DVSEC_CXL_PORT_CTL 0x0c -#define PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR 0x00000001 -/* Integrity and Data Encryption Extended Capability */ #define PCI_IDE_CAP 0x04#define PCI_IDE_CAP_LINK 0x1/* Link IDE Stream Supported */
--
2.34.1
Re: [PATCH v4 02/10] PCI: switch CXL port DVSEC defines
Quoted 6 lines · On Tue, 20 Jan 2026 22:26:02 +0000 [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
The PCI core consumes CXL port DVSEC fields for reset handling, so switch it to use the shared CXL PCI header instead of the uapi header. This aligns the core with the header split and keeps internal code from depending on uapi-only definitions.
Why do you think they are uapi only? Masses of kernel code relies on those definitions.
Intent is to just have one source of truth for userspace tools and kernel space ones.
Also, removing anything from those headers will probably break someone's user space so is basically impossible to do safely.
J
Quoted 94 lines
Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/pci/pci.c | 17 +++++++++-------- include/uapi/linux/pci_regs.h | 5 ----- 2 files changed, 9 insertions(+), 13 deletions(-)drivers/pci/pci.c +9
-8@@ -30,6 +30,7 @@#include <asm/dma.h> #include <linux/aer.h> #include <linux/bitfield.h> +#include <cxl/pci.h> #include "pci.h" DEFINE_MUTEX (pci_slot_mutex);@@ -4842,7 +4843,7 @@ static int pci_dev_reset_slot_function(struct pci_dev *dev, bool probe)static u16 cxl_port_dvsec (struct pci_dev *dev) {return pci_find_dvsec_capability (dev,PCI_VENDOR_ID_CXL , -PCI_DVSEC_CXL_PORT ); +CXL_DVSEC_PORT_EXTENSIONS ); }static bool cxl_sbr_masked (struct pci_dev *dev)@@ -4854,7 +4855,7 @@ static bool cxl_sbr_masked(struct pci_dev *dev)if (!dvsec)return false; - rc =pci_read_config_word (dev, dvsec +PCI_DVSEC_CXL_PORT_CTL , ®); + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_PORT_CTL , ®);if (rc ||PCI_POSSIBLE_ERROR (reg))return false;@@ -4863,7 +4864,7 @@ static bool cxl_sbr_masked(struct pci_dev *dev)* bit in Bridge Control has no effect. When 1, the Port generates * hot reset when the SBR bit is set to 1. */ -if (reg &PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR ) +if (reg &CXL_DVSEC_UNMASK_SBR )return false;return true;@@ -4908,22 +4909,22 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)if (probe)return 0 ; - rc =pci_read_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , ®); + rc =pci_read_config_word (bridge, dvsec +CXL_DVSEC_PORT_CTL , ®);if (rc)return -ENOTTY ; -if (reg &PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR ) { +if (reg &CXL_DVSEC_UNMASK_SBR ) { val = reg; }else { - val = reg |PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR ; -pci_write_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , + val = reg |CXL_DVSEC_UNMASK_SBR ; +pci_write_config_word (bridge, dvsec +CXL_DVSEC_PORT_CTL , val); } rc =pci_reset_bus_function (dev, probe);if (reg != val) -pci_write_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , +pci_write_config_word (bridge, dvsec +CXL_DVSEC_PORT_CTL , reg);return rc;include/uapi/linux/pci_regs.h +0
-6@@ -1253,11 +1253,6 @@#define PCI_DEV3_STA 0x0c/* Device 3 Status Register */ #define PCI_DEV3_STA_SEGMENT 0x8/* Segment Captured (end-to-end flit-mode detected) */ -/* Compute Express Link (CXL r3.1, sec 8.1.5) */ -#define PCI_DVSEC_CXL_PORT 3 -#define PCI_DVSEC_CXL_PORT_CTL 0x0c -#define PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR 0x00000001 -/* Integrity and Data Encryption Extended Capability */ #define PCI_IDE_CAP 0x04#define PCI_IDE_CAP_LINK 0x1/* Link IDE Stream Supported */ --2.34.1
PATCH v4 3/10 cxl: add type 2 helper and reset DVSEC bits
In reply to [email protected]From: Srirangan Madhavan <[email protected]>
Introduce a helper to identify CXL Type 2 devices and define the DVSEC reset/cache control bits used by the reset flow.
--- drivers/cxl/pci.c | 10 ++++++++++ include/cxl/pci.h | 14 ++++++++++++++ 2 files changed, 24 insertions(+)
drivers/cxl/pci.c +10 -0
@@ -1075,6 +1075,16 @@ static pci_ers_result_t cxl_slot_reset(struct pci_dev *pdev)return PCI_ERS_RESULT_RECOVERED ; } +bool cxl_is_type2_device (struct pci_dev *pdev) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); + +if (!cxlds) +return false; + +return cxlds->type ==CXL_DEVTYPE_DEVMEM ; +} +static void cxl_error_resume (struct pci_dev *pdev) {struct cxl_dev_state *cxlds =pci_get_drvdata (pdev);
include/cxl/pci.h +14 -0
@@ -14,10 +14,24 @@/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ #define CXL_DVSEC_PCIE_DEVICE 0#define CXL_DVSEC_CAP_OFFSET 0xA +#define CXL_DVSEC_CACHE_CAPABLE BIT(0)#define CXL_DVSEC_MEM_CAPABLE BIT(2)#define CXL_DVSEC_HDM_COUNT_MASK GENMASK(5, 4) +#define CXL_DVSEC_CACHE_WBI_CAPABLE BIT(6) +#define CXL_DVSEC_CXL_RST_CAPABLE BIT(7) +#define CXL_DVSEC_CXL_RST_TIMEOUT_MASK GENMASK(10, 8) +#define CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE BIT(11)#define CXL_DVSEC_CTRL_OFFSET 0xC#define CXL_DVSEC_MEM_ENABLE BIT(2) +#define CXL_DVSEC_CTRL2_OFFSET 0x10 +#define CXL_DVSEC_DISABLE_CACHING BIT(0) +#define CXL_DVSEC_INIT_CACHE_WBI BIT(1) +#define CXL_DVSEC_INIT_CXL_RESET BIT(2) +#define CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE BIT(3) +#define CXL_DVSEC_STATUS2_OFFSET 0x12 +#define CXL_DVSEC_CACHE_INVALID BIT(0) +#define CXL_DVSEC_CXL_RST_COMPLETE BIT(1) +#define CXL_DVSEC_CXL_RESET_ERR BIT(2)#define CXL_DVSEC_RANGE_SIZE_HIGH (i) (0x18 + ((i) * 0x10))#define CXL_DVSEC_RANGE_SIZE_LOW (i) (0x1C + ((i) * 0x10))#define CXL_DVSEC_MEM_INFO_VALID BIT(0)
--
2.34.1
Dave Jiang
maintainerRe: [PATCH v4 03/10] cxl: add type 2 helper and reset DVSEC bits
Quoted 4 lines · On 1/20/26 3:26 PM, [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Introduce a helper to identify CXL Type 2 devices and define the DVSEC reset/cache control bits used by the reset flow.
Should probably be 2 separate patches for these 2 things.
Quoted 47 lines
Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/pci.c | 10 ++++++++++ include/cxl/pci.h | 14 ++++++++++++++ 2 files changed, 24 insertions(+)drivers/cxl/pci.c +10
-0@@ -1075,6 +1075,16 @@ static pci_ers_result_t cxl_slot_reset(struct pci_dev *pdev)return PCI_ERS_RESULT_RECOVERED ; } +bool cxl_is_type2_device (struct pci_dev *pdev) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); + +if (!cxlds) +return false; + +return cxlds->type ==CXL_DEVTYPE_DEVMEM ; +} +static void cxl_error_resume (struct pci_dev *pdev) {struct cxl_dev_state *cxlds =pci_get_drvdata (pdev);include/cxl/pci.h +7
-0@@ -14,10 +14,24 @@/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ #define CXL_DVSEC_PCIE_DEVICE 0#define CXL_DVSEC_CAP_OFFSET 0xA +#define CXL_DVSEC_CACHE_CAPABLE BIT(0)#define CXL_DVSEC_MEM_CAPABLE BIT(2)#define CXL_DVSEC_HDM_COUNT_MASK GENMASK(5, 4) +#define CXL_DVSEC_CACHE_WBI_CAPABLE BIT(6) +#define CXL_DVSEC_CXL_RST_CAPABLE BIT(7) +#define CXL_DVSEC_CXL_RST_TIMEOUT_MASK GENMASK(10, 8) +#define CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE BIT(11)#define CXL_DVSEC_CTRL_OFFSET 0xC#define CXL_DVSEC_MEM_ENABLE BIT(2) +#define CXL_DVSEC_CTRL2_OFFSET 0x10 +#define CXL_DVSEC_DISABLE_CACHING BIT(0)
include/cxl/pci.h line 37
Quoted 10 lines
include/cxl/pci.h +7
-0+#define CXL_DVSEC_INIT_CACHE_WBI BIT(1) +#define CXL_DVSEC_INIT_CXL_RESET BIT(2) +#define CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE BIT(3) +#define CXL_DVSEC_STATUS2_OFFSET 0x12 +#define CXL_DVSEC_CACHE_INVALID BIT(0) +#define CXL_DVSEC_CXL_RST_COMPLETE BIT(1) +#define CXL_DVSEC_CXL_RESET_ERR BIT(2)#define CXL_DVSEC_RANGE_SIZE_HIGH (i) (0x18 + ((i) * 0x10))#define CXL_DVSEC_RANGE_SIZE_LOW (i) (0x1C + ((i) * 0x10))#define CXL_DVSEC_MEM_INFO_VALID BIT(0)
Should this chunk go with a different patch where the definitions are being used?
Quoted 3 lines
--
2.34.1
Re: [PATCH v4 03/10] cxl: add type 2 helper and reset DVSEC bits
Quoted 7 lines · On Tue, 20 Jan 2026 16:27:33 -0700 Dave Jiang <[email protected]> wrote:
Quoted 4 lines · On 1/20/26 3:26 PM, [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Introduce a helper to identify CXL Type 2 devices and define the DVSEC reset/cache control bits used by the reset flow.
Should probably be 2 separate patches for these 2 things.
Also, follow existing convention and put them new DVSEC defs in the uapi/pci_regs.h file. The rest are moving there shortly.
Given they are now in a uapi file, I also wonder if we should just fill in the rest of the structure definitions as a stand alone patch. A partial set isn't much use to userspace tooling.
Jonathan
Quoted 66 lines
Quoted 48 lines
Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/pci.c | 10 ++++++++++ include/cxl/pci.h | 14 ++++++++++++++ 2 files changed, 24 insertions(+)drivers/cxl/pci.c +10
-0@@ -1075,6 +1075,16 @@ static pci_ers_result_t cxl_slot_reset(struct pci_dev *pdev)return PCI_ERS_RESULT_RECOVERED ; } +bool cxl_is_type2_device (struct pci_dev *pdev) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); + +if (!cxlds) +return false; + +return cxlds->type ==CXL_DEVTYPE_DEVMEM ; +} +static void cxl_error_resume (struct pci_dev *pdev) {struct cxl_dev_state *cxlds =pci_get_drvdata (pdev);include/cxl/pci.h +7
-0@@ -14,10 +14,24 @@/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ #define CXL_DVSEC_PCIE_DEVICE 0#define CXL_DVSEC_CAP_OFFSET 0xA +#define CXL_DVSEC_CACHE_CAPABLE BIT(0)#define CXL_DVSEC_MEM_CAPABLE BIT(2)#define CXL_DVSEC_HDM_COUNT_MASK GENMASK(5, 4) +#define CXL_DVSEC_CACHE_WBI_CAPABLE BIT(6) +#define CXL_DVSEC_CXL_RST_CAPABLE BIT(7) +#define CXL_DVSEC_CXL_RST_TIMEOUT_MASK GENMASK(10, 8) +#define CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE BIT(11)#define CXL_DVSEC_CTRL_OFFSET 0xC#define CXL_DVSEC_MEM_ENABLE BIT(2) +#define CXL_DVSEC_CTRL2_OFFSET 0x10 +#define CXL_DVSEC_DISABLE_CACHING BIT(0)include/cxl/pci.h line 37
Quoted 10 lines
include/cxl/pci.h +7
-0+#define CXL_DVSEC_INIT_CACHE_WBI BIT(1) +#define CXL_DVSEC_INIT_CXL_RESET BIT(2) +#define CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE BIT(3) +#define CXL_DVSEC_STATUS2_OFFSET 0x12 +#define CXL_DVSEC_CACHE_INVALID BIT(0) +#define CXL_DVSEC_CXL_RST_COMPLETE BIT(1) +#define CXL_DVSEC_CXL_RESET_ERR BIT(2)#define CXL_DVSEC_RANGE_SIZE_HIGH (i) (0x18 + ((i) * 0x10))#define CXL_DVSEC_RANGE_SIZE_LOW (i) (0x1C + ((i) * 0x10))#define CXL_DVSEC_MEM_INFO_VALID BIT(0)Should this chunk go with a different patch where the definitions are being used?
Quoted 3 lines
--
2.34.1
PATCH v4 4/10 PCI: add CXL reset method
In reply to [email protected]From: Srirangan Madhavan <[email protected]>
Add a PCI reset method "cxl_reset" that drives the CXL reset sequence using DVSEC controls and timeout encoding. The method is restricted to Type 2 devices, limiting the scope of the changes.
--- drivers/pci/pci.c | 104 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 10 ++++- 2 files changed, 113 insertions(+), 1 deletion(-)
drivers/pci/pci.c +104 -0
@@ -4892,6 +4892,109 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)return pci_parent_bus_reset (dev, probe); } +static int cxl_reset_init (struct pci_dev *dev,u16 dvsec) +{ +/* +* Timeout values ref CXL Spec v3.2 Ch 8 Control and Status Registers, +* under section 8.1.3.1 DVSEC CXL Capability. +*/ +u32 reset_timeouts_ms[] = {10 ,100 ,1000 ,10000 ,100000 }; +u16 reg; +u32 timeout_ms; +int rc, ind; + +/* Check if CXL Reset MEM CLR is supported. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CAP_OFFSET , ®); +if (rc) +return rc; + +if (reg &CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE ) { + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , + ®); +if (rc) +return rc; + + reg |=CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE ; +pci_write_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , reg); + } + +/* Read timeout value. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CAP_OFFSET , ®); +if (rc) +return rc; + ind =FIELD_GET (CXL_DVSEC_CXL_RST_TIMEOUT_MASK , reg); + timeout_ms = reset_timeouts_ms[ind]; + +/* Write reset config. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , ®); +if (rc) +return rc; + + reg |=CXL_DVSEC_INIT_CXL_RESET ; +pci_write_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , reg); + +/* Wait till timeout and then check reset status is complete. */ +msleep (timeout_ms); + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_STATUS2_OFFSET , ®); +if (rc) +return rc; +if (reg &CXL_DVSEC_CXL_RESET_ERR || + ~reg &CXL_DVSEC_CXL_RST_COMPLETE ) +return -ETIMEDOUT ; + + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , ®); +if (rc) +return rc; + reg &= (~CXL_DVSEC_DISABLE_CACHING ); +pci_write_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , reg); + +return 0 ; +} + +/** +* cxl_reset - initiate a cxl reset +* @dev: device to reset +* @probe: if true, return 0 if device can be reset this way +* +* Initiate a cxl reset on @dev. +*/ +static int cxl_reset (struct pci_dev *dev,bool probe) +{ +u16 dvsec, reg; +int rc; + + dvsec =pci_find_dvsec_capability (dev,PCI_VENDOR_ID_CXL , +CXL_DVSEC_PCIE_DEVICE ); +if (!dvsec) +return -ENOTTY ; + +/* Check if CXL Reset is supported. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CAP_OFFSET , ®); +if (rc) +return -ENOTTY ; + +if ((reg &CXL_DVSEC_CXL_RST_CAPABLE ) ==0 ) +return -ENOTTY ; + +#if !IS_REACHABLE (CONFIG_CXL_PCI ) +return -ENOTTY ; +#endif + +/* +* Expose CXL reset for Type 2 devices. +*/ +if (!cxl_is_type2_device (dev)) +return -ENOTTY ; + +if (probe) +return 0 ; + +if (!pci_wait_for_pending_transaction (dev)) +pci_err (dev,"timed out waiting for pending transaction; performing function level reset anyway\n" ); + +return cxl_reset_init (dev, dvsec); +} +static int cxl_reset_bus_function (struct pci_dev *dev,bool probe) {struct pci_dev *bridge;
@@ -5016,6 +5119,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { pci_dev_acpi_reset, .name = "acpi" }, { pcie_reset_flr, .name = "flr" }, { pci_af_flr, .name = "af_flr" }, + { cxl_reset, .name = "cxl_reset" }, { pci_pm_reset, .name = "pm" }, { pci_reset_bus_function, .name = "bus" }, { cxl_reset_bus_function, .name = "cxl_bus" },
include/linux/pci.h +9 -1
@@ -51,7 +51,7 @@ PCI_STATUS_PARITY)/* Number of reset methods used in pci_reset_fn_methods array in pci.c */ -#define PCI_NUM_RESET_METHODS 8 +#define PCI_NUM_RESET_METHODS 9#define PCI_RESET_PROBE true#define PCI_RESET_DO_RESET false
@@ -1464,6 +1464,14 @@ int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size,int pci_select_bars (struct pci_dev *dev,unsigned long flags);bool pci_device_is_present (struct pci_dev *pdev); +#ifdef CONFIG_CXL_PCI +bool cxl_is_type2_device (struct pci_dev *dev); +#else +static inline bool cxl_is_type2_device (struct pci_dev *dev) +{ +return false; +} +#endif void pci_ignore_hotplug (struct pci_dev *dev);struct pci_dev *pci_real_dma_dev (struct pci_dev *dev);int pci_status_get_and_clear_errors (struct pci_dev *pdev);
--
2.34.1
Dave Jiang
maintainerRe: [PATCH v4 04/10] PCI: add CXL reset method
Quoted 8 lines · On 1/20/26 3:26 PM, [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Add a PCI reset method "
cxl_reset" that drives the CXL reset sequence using DVSEC controls and timeout encoding. The method is restricted to Type 2 devices, limiting the scope of the changes.
Don't think this is needed if it's kbot issue found during the series postings.
Quoted 11 lines
Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/pci/pci.c | 104 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 10 ++++- 2 files changed, 113 insertions(+), 1 deletion(-)drivers/pci/pci.c +0
-0
drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4895
Quoted 4 lines
drivers/pci/pci.c +1
-0@@ -4892,6 +4892,109 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)return pci_parent_bus_reset (dev, probe); } +static int cxl_reset_init (struct pci_dev *dev,u16 dvsec)
cxl_dev_reset() to go with existing reset method naming?
drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4901
Quoted 6 lines
drivers/pci/pci.c +6
-0+{ +/* +* Timeout values ref CXL Spec v3.2 Ch 8 Control and Status Registers, +* under section 8.1.3.1 DVSEC CXL Capability. +*/ +u32 reset_timeouts_ms[] = {10 ,100 ,1000 ,10000 ,100000 };
Should this be const?
Quoted 68 lines
drivers/pci/pci.c +68
-0+u16 reg; +u32 timeout_ms; +int rc, ind; + +/* Check if CXL Reset MEM CLR is supported. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CAP_OFFSET , ®); +if (rc) +return rc; + +if (reg &CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE ) { + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , + ®); +if (rc) +return rc; + + reg |=CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE ; +pci_write_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , reg); + } + +/* Read timeout value. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CAP_OFFSET , ®); +if (rc) +return rc; + ind =FIELD_GET (CXL_DVSEC_CXL_RST_TIMEOUT_MASK , reg); + timeout_ms = reset_timeouts_ms[ind]; + +/* Write reset config. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , ®); +if (rc) +return rc; + + reg |=CXL_DVSEC_INIT_CXL_RESET ; +pci_write_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , reg); + +/* Wait till timeout and then check reset status is complete. */ +msleep (timeout_ms); + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_STATUS2_OFFSET , ®); +if (rc) +return rc; +if (reg &CXL_DVSEC_CXL_RESET_ERR || + ~reg &CXL_DVSEC_CXL_RST_COMPLETE ) +return -ETIMEDOUT ; + + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , ®); +if (rc) +return rc; + reg &= (~CXL_DVSEC_DISABLE_CACHING ); +pci_write_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , reg); + +return 0 ; +} + +/** +* cxl_reset - initiate a cxl reset +* @dev: device to reset +* @probe: if true, return 0 if device can be reset this way +* +* Initiate a cxl reset on @dev. +*/ +static int cxl_reset (struct pci_dev *dev,bool probe) +{ +u16 dvsec, reg; +int rc; + + dvsec =pci_find_dvsec_capability (dev,PCI_VENDOR_ID_CXL , +CXL_DVSEC_PCIE_DEVICE ); +if (!dvsec) +return -ENOTTY ;
drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4979
Quoted 10 lines
drivers/pci/pci.c +10
-0+ +/* Check if CXL Reset is supported. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CAP_OFFSET , ®); +if (rc) +return -ENOTTY ; + +if ((reg &CXL_DVSEC_CXL_RST_CAPABLE ) ==0 ) +return -ENOTTY ; + +#if !IS_REACHABLE (CONFIG_CXL_PCI )
Does this not require CONFIG_CXL_PCI to be built in (not module) to evaluate to 'y'? Also, instead of adding ifdef in the C code, maybe create a helper function for the remaining code within the function below and add an ifdef in the header depending on the config?
Although I think you may want to define cxl_is_type2_device() differently rather than relying on cxl_pci driver data. It probably should check the device type via config space instead to lessen the complications.
DJ
Quoted 60 lines
drivers/pci/pci.c +19
-0+return -ENOTTY ; +#endif + +/* +* Expose CXL reset for Type 2 devices. +*/ +if (!cxl_is_type2_device (dev)) +return -ENOTTY ; + +if (probe) +return 0 ; + +if (!pci_wait_for_pending_transaction (dev)) +pci_err (dev,"timed out waiting for pending transaction; performing function level reset anyway\n" ); + +return cxl_reset_init (dev, dvsec); +} +static int cxl_reset_bus_function (struct pci_dev *dev,bool probe) {struct pci_dev *bridge;@@ -5016,6 +5119,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { pci_dev_acpi_reset, .name = "acpi" }, { pcie_reset_flr, .name = "flr" }, { pci_af_flr, .name = "af_flr" }, + { cxl_reset, .name = "cxl_reset" }, { pci_pm_reset, .name = "pm" }, { pci_reset_bus_function, .name = "bus" }, { cxl_reset_bus_function, .name = "cxl_bus" },include/linux/pci.h +9
-2@@ -51,7 +51,7 @@ PCI_STATUS_PARITY)/* Number of reset methods used in pci_reset_fn_methods array in pci.c */ -#define PCI_NUM_RESET_METHODS 8 +#define PCI_NUM_RESET_METHODS 9#define PCI_RESET_PROBE true#define PCI_RESET_DO_RESET false@@ -1464,6 +1464,14 @@ int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size,int pci_select_bars (struct pci_dev *dev,unsigned long flags);bool pci_device_is_present (struct pci_dev *pdev); +#ifdef CONFIG_CXL_PCI +bool cxl_is_type2_device (struct pci_dev *dev); +#else +static inline bool cxl_is_type2_device (struct pci_dev *dev) +{ +return false; +} +#endif void pci_ignore_hotplug (struct pci_dev *dev);struct pci_dev *pci_real_dma_dev (struct pci_dev *dev);int pci_status_get_and_clear_errors (struct pci_dev *pdev); --2.34.1
Re: [PATCH v4 04/10] PCI: add CXL reset method
In reply to [email protected]Quoted 36 lines · On Tue, 20 Jan 2026 22:26:04 +0000 [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Add a PCI reset method "
cxl_reset" that drives the CXL reset sequence using DVSEC controls and timeout encoding. The method is restricted to Type 2 devices, limiting the scope of the changes.Reported-by kernel test robot <[email protected]>Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/pci/pci.c | 104 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 10 ++++- 2 files changed, 113 insertions(+), 1 deletion(-)drivers/pci/pci.c +14
-0@@ -4892,6 +4892,109 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe)return pci_parent_bus_reset (dev, probe); } +static int cxl_reset_init (struct pci_dev *dev,u16 dvsec) +{ +/* +* Timeout values ref CXL Spec v3.2 Ch 8 Control and Status Registers, +* under section 8.1.3.1 DVSEC CXL Capability. +*/ +u32 reset_timeouts_ms[] = {10 ,100 ,1000 ,10000 ,100000 }; +u16 reg; +u32 timeout_ms; +int rc, ind; + +/* Check if CXL Reset MEM CLR is supported. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CAP_OFFSET , ®); +if (rc)
drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4918
Quoted 10 lines
drivers/pci/pci.c +10
-0+return rc; + +if (reg &CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE ) { + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , + ®); +if (rc) +return rc; + + reg |=CXL_DVSEC_CXL_RST_MEM_CLR_ENABLE ; +pci_write_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , reg);
Be consistent on checking for errors on these pci config accesses.
Pity there isn't a pci_clear_and_set_config_word() like the dword one.
drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4929
Quoted 11 lines
drivers/pci/pci.c +11
-0+ } + +/* Read timeout value. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CAP_OFFSET , ®); +if (rc) +return rc; + ind =FIELD_GET (CXL_DVSEC_CXL_RST_TIMEOUT_MASK , reg); + timeout_ms = reset_timeouts_ms[ind]; + +/* Write reset config. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , ®);
Separate writes needed for this and the MEM_CLR_ENABLE? If not, refactoring to build it up as one value to be written would be good. If separate write is needed then add a comment to make that clear and avoid someone 'tidying' this up later.
Quoted 4 lines
drivers/pci/pci.c +4
-0+if (rc) +return rc; + + reg |=CXL_DVSEC_INIT_CXL_RESET ;
drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4943
Quoted 10 lines
drivers/pci/pci.c +10
-0+pci_write_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , reg); + +/* Wait till timeout and then check reset status is complete. */ +msleep (timeout_ms); + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_STATUS2_OFFSET , ®); +if (rc) +return rc; +if (reg &CXL_DVSEC_CXL_RESET_ERR || + ~reg &CXL_DVSEC_CXL_RST_COMPLETE ) +return -ETIMEDOUT ;
I'd split these two conditions. If we saw an reset_err it probably wasn't a timeout so a different return code would be good to indicate that.
drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4948
Quoted 5 lines
drivers/pci/pci.c +5
-0+ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , ®); +if (rc) +return rc; + reg &= (~CXL_DVSEC_DISABLE_CACHING );
Brackets not needed.
Quoted 26 lines
drivers/pci/pci.c +26
-0+pci_write_config_word (dev, dvsec +CXL_DVSEC_CTRL2_OFFSET , reg); + +return 0 ; +} + +/** +* cxl_reset - initiate a cxl reset +* @dev: device to reset +* @probe: if true, return 0 if device can be reset this way +* +* Initiate a cxl reset on @dev. +*/ +static int cxl_reset (struct pci_dev *dev,bool probe) +{ +u16 dvsec, reg; +int rc; + + dvsec =pci_find_dvsec_capability (dev,PCI_VENDOR_ID_CXL , +CXL_DVSEC_PCIE_DEVICE ); +if (!dvsec) +return -ENOTTY ; + +/* Check if CXL Reset is supported. */ + rc =pci_read_config_word (dev, dvsec +CXL_DVSEC_CAP_OFFSET , ®); +if (rc) +return -ENOTTY ;
drivers/pci/pci.c static int pci_reset_bus_function(struct pci_dev *dev, bool probe) · line 4984
Quoted 10 lines
drivers/pci/pci.c +10
-0+ +if ((reg &CXL_DVSEC_CXL_RST_CAPABLE ) ==0 ) +return -ENOTTY ; + +#if !IS_REACHABLE (CONFIG_CXL_PCI ) +return -ENOTTY ; +#endif + +/* +* Expose CXL reset for Type 2 devices.
Agree with Dave. Good to avoid linking to the CXL_PCI driver if possible.
Quoted 45 lines
drivers/pci/pci.c +12
-0+*/ +if (!cxl_is_type2_device (dev)) +return -ENOTTY ; + +if (probe) +return 0 ; + +if (!pci_wait_for_pending_transaction (dev)) +pci_err (dev,"timed out waiting for pending transaction; performing function level reset anyway\n" ); + +return cxl_reset_init (dev, dvsec); +}include/linux/pci.h +9
-2@@ -51,7 +51,7 @@ PCI_STATUS_PARITY)/* Number of reset methods used in pci_reset_fn_methods array in pci.c */ -#define PCI_NUM_RESET_METHODS 8 +#define PCI_NUM_RESET_METHODS 9#define PCI_RESET_PROBE true#define PCI_RESET_DO_RESET false@@ -1464,6 +1464,14 @@ int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size,int pci_select_bars (struct pci_dev *dev,unsigned long flags);bool pci_device_is_present (struct pci_dev *pdev); +#ifdef CONFIG_CXL_PCI +bool cxl_is_type2_device (struct pci_dev *dev); +#else +static inline bool cxl_is_type2_device (struct pci_dev *dev) +{ +return false; +} +#endif void pci_ignore_hotplug (struct pci_dev *dev);struct pci_dev *pci_real_dma_dev (struct pci_dev *dev);int pci_status_get_and_clear_errors (struct pci_dev *pdev); --2.34.1
kernel test robot
build botRe: [PATCH v4 04/10] PCI: add CXL reset method
In reply to [email protected]kernel test robot noticed the following build errors:
Hi,
kernel test robot noticed the following build errors:
[auto build test ERROR on pci/next]
[also build test ERROR on pci/for-linus linus/master v6.19-rc6] [cannot apply to next-20260122]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/smadhavan-nvidia-com/cxl-move-DVSEC-defines-to-cxl-pci-header/20260121-071852 base: https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next patch link: https://lore.kernel.org/r/20260120222610.2227109-5-smadhavan%40nvidia.com patch subject: [PATCH v4 04/10] PCI: add CXL reset method config: openrisc-allmodconfig (https://download.01.org/0day-ci/archive/20260123/[email protected]/config) compiler: or1k-linux-gcc (GCC) 15.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260123/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> drivers/cxl/pci.c:1085:6: error: redefinition of 'cxl_is_type2_device'
1085 | bool cxl_is_type2_device(struct pci_dev *pdev)
| ^~~~~~~~~~~~~~~~~~~
In file included from drivers/cxl/pci.c:11:
include/linux/pci.h:1474:20: note: previous definition of 'cxl_is_type2_device' with type 'bool(struct pci_dev *)' {aka '_Bool(struct pci_dev *)'}
1474 | static inline bool cxl_is_type2_device(struct pci_dev *dev)
| ^~~~~~~~~~~~~~~~~~~
vim +/cxl_is_type2_device +1085 drivers/cxl/pci.c
2905cb5236cba6 Dan Williams 2022-11-29 1084
b345d117d51557 Srirangan Madhavan 2026-01-20 @1085 bool cxl_is_type2_device(struct pci_dev *pdev)
b345d117d51557 Srirangan Madhavan 2026-01-20 1086 {
b345d117d51557 Srirangan Madhavan 2026-01-20 1087 struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
b345d117d51557 Srirangan Madhavan 2026-01-20 1088
b345d117d51557 Srirangan Madhavan 2026-01-20 1089 if (!cxlds)
b345d117d51557 Srirangan Madhavan 2026-01-20 1090 return false;
b345d117d51557 Srirangan Madhavan 2026-01-20 1091
b345d117d51557 Srirangan Madhavan 2026-01-20 1092 return cxlds->type == CXL_DEVTYPE_DEVMEM;
b345d117d51557 Srirangan Madhavan 2026-01-20 1093 }
b345d117d51557 Srirangan Madhavan 2026-01-20 1094 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v4 5/10 cxl: add reset prepare and region teardown
In reply to [email protected]From: Srirangan Madhavan <[email protected]>
Prepare a Type 2 device for cxl_reset by validating memory is offline, flushing device caches for region participants, and tearing down decoders under cxl_region_rwsem. The lock stays held across reset to prevent new region creation while reset is in progress.
--- drivers/cxl/pci.c | 214 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 214 insertions(+)
drivers/cxl/pci.c +214 -0
@@ -1085,6 +1085,220 @@ bool cxl_is_type2_device(struct pci_dev *pdev)return cxlds->type ==CXL_DEVTYPE_DEVMEM ; } +static int cxl_check_region_driver_bound (struct device *dev,void *data) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev); + +if (!is_endpoint_decoder (dev)) +return 0 ; + +guard (rwsem_read)(&cxl_region_rwsem); +if (cxld->region && cxld->region->driver) +return -EBUSY ; + +return 0 ; +} + +static int cxl_decoder_kill_region_iter (struct device *dev,void *data) +{ +struct cxl_endpoint_decoder *cxled =to_cxl_endpoint_decoder (dev); +int rc; + +if (!is_endpoint_decoder (dev)) +return 0 ; + +if (!cxled->cxld.region) +return 0 ; + +cxl_decoder_kill_region_locked (cxled); + + rc =device_for_each_child (&cxled->cxld.dev,NULL , + cxl_check_region_driver_bound); +if (rc) +return rc; + +return 0 ; +} + +static int cxl_device_cache_wb_invalidate (struct pci_dev *pdev) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +u16 reg, val, cap; +int dvsec, rc; + +if (!cxlds) +return -ENODEV ; + + dvsec = cxlds->cxl_dvsec; +if (!dvsec) +return -ENODEV ; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CAP_OFFSET , &cap); +if (rc) +return rc; + +if (!(cap &CXL_DVSEC_CACHE_WBI_CAPABLE )) +return 1 ; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CTRL2_OFFSET , &val); +if (rc) +return rc; + + val |=CXL_DVSEC_INIT_CACHE_WBI ; + rc =pci_write_config_word (pdev, dvsec +CXL_DVSEC_CTRL2_OFFSET , val); +if (rc) +return rc; + +do { + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_STATUS2_OFFSET , ®); +if (rc) +return rc; + }while (!(reg &CXL_DVSEC_CACHE_INVALID )); + +return 0 ; +} + +static int cxl_region_flush_device_caches (struct device *dev,void *data) +{ +struct cxl_endpoint_decoder *cxled =to_cxl_endpoint_decoder (dev); +struct cxl_region *cxlr = cxled->cxld.region; +struct cxl_region_params *p = &cxlr->params; +struct pci_dev *target_pdev = data; +int i, rc; + +if (!is_endpoint_decoder (dev)) +return 0 ; + +if (!cxlr || !cxlr->params.res) +return 0 ; + +for (i =0 ; i < p->nr_targets; i++) { +struct cxl_endpoint_decoder *target_cxled = p->targets[i]; +struct cxl_memdev *target_cxlmd =cxled_to_memdev (target_cxled); +struct cxl_dev_state *target_cxlds = target_cxlmd->cxlds; + +if (!target_cxlds || !target_cxlds->pdev) +continue ; + +if (target_cxlds->pdev != target_pdev) +continue ; + + rc =cxl_device_cache_wb_invalidate (target_pdev); +if (rc && rc !=1 ) +return rc; + } + +return 0 ; +} + +/** +* cxl_reset_prepare_memdev - Prepare CXL device for reset +* @pdev: PCI device +* +* Validates it's safe to reset and tears down regions atomically under lock. +* Acquires cxl_region_rwsem and keeps it held throughout reset. +* +* Return: 0 on success (lock held), -EBUSY if memory online, negative on error +*/ +static int cxl_reset_prepare_memdev (struct pci_dev *pdev) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +struct cxl_memdev *cxlmd; +struct cxl_port *endpoint; +int rc; + +if (!cxlds) +return -ENODEV ; + + cxlmd = cxlds->cxlmd; +if (!cxlmd) +return -ENODEV ; + + endpoint = cxlmd->endpoint; +if (!endpoint) +return 0 ; + +if (cxl_num_decoders_committed (endpoint) ==0 ) +return 0 ; + +down_write (&cxl_region_rwsem); + +/* Check and error out if memory is online */ + rc =device_for_each_child (&endpoint->dev,NULL , + cxl_check_region_driver_bound); +if (rc) { +up_write (&cxl_region_rwsem); +dev_err (&pdev->dev, +"Reset blocked: device has active regions with drivers bound\n" ); +return -EBUSY ; + } + +/* Flush device caches and tear down regions */ +device_for_each_child (&endpoint->dev, pdev, + cxl_region_flush_device_caches); + + rc =device_for_each_child (&endpoint->dev,NULL , + cxl_decoder_kill_region_iter); +if (rc) { +up_write (&cxl_region_rwsem); +dev_err (&pdev->dev,"Failed to tear down regions: %d\n" , rc); +return rc; + } + +/* Keep cxl_region_rwsem held, released by cleanup function */ +return 0 ; +} + +/** +* cxl_reset_cleanup_memdev - Release locks after CXL reset +* @pdev: PCI device +*/ +static void cxl_reset_cleanup_memdev (struct pci_dev *pdev) +{ +if (lockdep_is_held_type (&cxl_region_rwsem,-1 )) +up_write (&cxl_region_rwsem); +} + +/** +* cxl_reset_prepare_device - Prepare CXL device for reset +* @pdev: PCI device being reset +* +* CXL-reset-specific preparation. Validates memory is offline, flushes +* device caches, and tears down regions. +* +* Returns: 0 on success, -EBUSY if memory online, negative on error +*/ +int cxl_reset_prepare_device (struct pci_dev *pdev) +{ +int rc; + + rc =cxl_reset_prepare_memdev (pdev); +if (rc) { +if (rc == -EBUSY ) +dev_err (&pdev->dev, +"Cannot reset: device has online memory or active regions\n" ); +else +dev_err (&pdev->dev, +"Failed to prepare device for reset: %d\n" , rc); +return rc; + } + +return 0 ; +} +EXPORT_SYMBOL_NS_GPL (cxl_reset_prepare_device,"CXL" ); + +/** +* cxl_reset_cleanup_device - Cleanup after CXL reset +* @pdev: PCI device that was reset +* +* Releases region locks held during reset. +*/ +void cxl_reset_cleanup_device (struct pci_dev *pdev) +{ +cxl_reset_cleanup_memdev (pdev); +} +EXPORT_SYMBOL_NS_GPL (cxl_reset_cleanup_device,"CXL" ); +static void cxl_error_resume (struct pci_dev *pdev) {struct cxl_dev_state *cxlds =pci_get_drvdata (pdev);
--
2.34.1
Re: [PATCH v4 05/10] cxl: add reset prepare and region teardown
Quoted 8 lines · On Tue, 20 Jan 2026 22:26:05 +0000 [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Prepare a Type 2 device for
cxl_resetby validating memory is offline, flushing device caches for region participants, and tearing down decoders undercxl_region_rwsem. The lock stays held across reset to prevent new region creation while reset is in progress.Signed-off-by Srirangan Madhavan <[email protected]>
Some minor feedback from a quick look. I'll want to take a closer look when we are closer to merging this.
Quoted 31 lines
--- drivers/cxl/pci.c | 214 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 214 insertions(+)drivers/cxl/pci.c +19
-0@@ -1085,6 +1085,220 @@ bool cxl_is_type2_device(struct pci_dev *pdev)return cxlds->type ==CXL_DEVTYPE_DEVMEM ; } +static int cxl_check_region_driver_bound (struct device *dev,void *data) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev); + +if (!is_endpoint_decoder (dev)) +return 0 ; + +guard (rwsem_read)(&cxl_region_rwsem); +if (cxld->region && cxld->region->driver) +return -EBUSY ; + +return 0 ; +} + +static int cxl_decoder_kill_region_iter (struct device *dev,void *data) +{ +struct cxl_endpoint_decoder *cxled =to_cxl_endpoint_decoder (dev); +int rc; +
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1116
Quoted 10 lines
drivers/cxl/pci.c +10
-0+if (!is_endpoint_decoder (dev)) +return 0 ; + +if (!cxled->cxld.region) +return 0 ; + +cxl_decoder_kill_region_locked (cxled); + + rc =device_for_each_child (&cxled->cxld.dev,NULL , + cxl_check_region_driver_bound);
return device_for_each_child()
If that doesn't make sense after later patches, fine to leave as it is.
Quoted 15 lines
drivers/cxl/pci.c +15
-0+if (rc) +return rc; + +return 0 ; +} + +static int cxl_device_cache_wb_invalidate (struct pci_dev *pdev) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +u16 reg, val, cap; +int dvsec, rc; + +if (!cxlds) +return -ENODEV ; +
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1141
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ dvsec = cxlds->cxl_dvsec; +if (!dvsec) +return -ENODEV ; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CAP_OFFSET , &cap); +if (rc) +return rc; + +if (!(cap &CXL_DVSEC_CACHE_WBI_CAPABLE )) +return 1 ;
With unusual return value, definitely need docs for this function. Given use below, maybe just return 0?
If there are caches and there is no way to force WB, what does that mean for whether we can reset the device? Feels like maybe this at least deserves a warning print.
My suspicion is that lack of that feature just means there is a device specific way to do it but I'm fine with Linux not supporting that ;)
Quoted 32 lines
drivers/cxl/pci.c +32
-0+ + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CTRL2_OFFSET , &val); +if (rc) +return rc; + + val |=CXL_DVSEC_INIT_CACHE_WBI ; + rc =pci_write_config_word (pdev, dvsec +CXL_DVSEC_CTRL2_OFFSET , val); +if (rc) +return rc; + +do { + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_STATUS2_OFFSET , ®); +if (rc) +return rc; + }while (!(reg &CXL_DVSEC_CACHE_INVALID )); + +return 0 ; +} + +static int cxl_region_flush_device_caches (struct device *dev,void *data) +{ +struct cxl_endpoint_decoder *cxled =to_cxl_endpoint_decoder (dev); +struct cxl_region *cxlr = cxled->cxld.region; +struct cxl_region_params *p = &cxlr->params; +struct pci_dev *target_pdev = data; +int i, rc; + +if (!is_endpoint_decoder (dev)) +return 0 ; + +if (!cxlr || !cxlr->params.res) +return 0 ;
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1183
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ +for (i =0 ; i < p->nr_targets; i++) { +struct cxl_endpoint_decoder *target_cxled = p->targets[i]; +struct cxl_memdev *target_cxlmd =cxled_to_memdev (target_cxled); +struct cxl_dev_state *target_cxlds = target_cxlmd->cxlds; + +if (!target_cxlds || !target_cxlds->pdev) +continue ; + +if (target_cxlds->pdev != target_pdev)
Seems like target_pdev == NULL is a bug and if possible should be checked for before doing anything in this function, so you could simplify this as
if (!target_cxlds || target_clds->pdev != target_pdev)
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1187
Quoted 4 lines
drivers/cxl/pci.c +4
-0+continue ; + + rc =cxl_device_cache_wb_invalidate (target_pdev); +if (rc && rc !=1 )
As above, I'm not sure the rc == 1 return is helpful.
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1199
Quoted 12 lines
drivers/cxl/pci.c +12
-0+return rc; + } + +return 0 ; +} + +/** +* cxl_reset_prepare_memdev - Prepare CXL device for reset +* @pdev: PCI device +* +* Validates it's safe to reset and tears down regions atomically under lock. +* Acquires cxl_region_rwsem and keeps it held throughout reset.
That may need some lockdep annotations. Make sure to run a lockdep build.
Quoted 7 lines
drivers/cxl/pci.c +5
-0+* +* Return: 0 on success (lock held), -EBUSY if memory online, negative on error +*/ +static int cxl_reset_prepare_memdev (struct pci_dev *pdev) +{
Dave Jiang
maintainerRe: [PATCH v4 05/10] cxl: add reset prepare and region teardown
In reply to [email protected]Quoted 19 lines · On 1/20/26 3:26 PM, [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Prepare a Type 2 device for
cxl_resetby validating memory is offline, flushing device caches for region participants, and tearing down decoders undercxl_region_rwsem. The lock stays held across reset to prevent new region creation while reset is in progress.Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/pci.c | 214 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 214 insertions(+)drivers/cxl/pci.c +0
-0@@ -1085,6 +1085,220 @@ bool cxl_is_type2_device(struct pci_dev *pdev)return cxlds->type ==CXL_DEVTYPE_DEVMEM ; }
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1096
Quoted 10 lines
drivers/cxl/pci.c +9
-0+static int cxl_check_region_driver_bound (struct device *dev,void *data) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev); + +if (!is_endpoint_decoder (dev)) +return 0 ; + +guard (rwsem_read)(&cxl_region_rwsem); +if (cxld->region && cxld->region->driver)
I think you may need to take the region device lock before checking the region driver. The cxl_region_rwsem won't protect you from region device driver un/binding. While we know cxl_pci is bound since the callback is on a PCI device, there's no gaurantee that the region driver is.
Quoted 7 lines
drivers/cxl/pci.c +7
-0+return -EBUSY ; + +return 0 ; +} + +static int cxl_decoder_kill_region_iter (struct device *dev,void *data) +{
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1113
Quoted 10 lines
drivers/cxl/pci.c +10
-0+struct cxl_endpoint_decoder *cxled =to_cxl_endpoint_decoder (dev); +int rc; + +if (!is_endpoint_decoder (dev)) +return 0 ; + +if (!cxled->cxld.region) +return 0 ; + +cxl_decoder_kill_region_locked (cxled);
Where does this function come from? It's not present upstream.
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1116
Quoted 3 lines
drivers/cxl/pci.c +3
-0+ + rc =device_for_each_child (&cxled->cxld.dev,NULL , + cxl_check_region_driver_bound);
Why check if driver is bound after region is killed? Also, the check happens in cxl_reset_prepare_memdev() before cxl_decoder_kill_region_iter() is called, and if true fails the cxl_reset_prepare_memdev(). So at this point there shouldn't be any drivers bound right?
Quoted 15 lines
drivers/cxl/pci.c +15
-0+if (rc) +return rc; + +return 0 ; +} + +static int cxl_device_cache_wb_invalidate (struct pci_dev *pdev) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +u16 reg, val, cap; +int dvsec, rc; + +if (!cxlds) +return -ENODEV ; +
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1141
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ dvsec = cxlds->cxl_dvsec; +if (!dvsec) +return -ENODEV ; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CAP_OFFSET , &cap); +if (rc) +return rc; + +if (!(cap &CXL_DVSEC_CACHE_WBI_CAPABLE )) +return 1 ;
Why return 1 when it's ignored anyhow? Can just return 0.
Quoted 52 lines
drivers/cxl/pci.c +52
-0+ + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CTRL2_OFFSET , &val); +if (rc) +return rc; + + val |=CXL_DVSEC_INIT_CACHE_WBI ; + rc =pci_write_config_word (pdev, dvsec +CXL_DVSEC_CTRL2_OFFSET , val); +if (rc) +return rc; + +do { + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_STATUS2_OFFSET , ®); +if (rc) +return rc; + }while (!(reg &CXL_DVSEC_CACHE_INVALID )); + +return 0 ; +} + +static int cxl_region_flush_device_caches (struct device *dev,void *data) +{ +struct cxl_endpoint_decoder *cxled =to_cxl_endpoint_decoder (dev); +struct cxl_region *cxlr = cxled->cxld.region; +struct cxl_region_params *p = &cxlr->params; +struct pci_dev *target_pdev = data; +int i, rc; + +if (!is_endpoint_decoder (dev)) +return 0 ; + +if (!cxlr || !cxlr->params.res) +return 0 ; + +for (i =0 ; i < p->nr_targets; i++) { +struct cxl_endpoint_decoder *target_cxled = p->targets[i]; +struct cxl_memdev *target_cxlmd =cxled_to_memdev (target_cxled); +struct cxl_dev_state *target_cxlds = target_cxlmd->cxlds; + +if (!target_cxlds || !target_cxlds->pdev) +continue ; + +if (target_cxlds->pdev != target_pdev) +continue ; + + rc =cxl_device_cache_wb_invalidate (target_pdev); +if (rc && rc !=1 ) +return rc; + } + +return 0 ; +} +
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1203
Quoted 10 lines
drivers/cxl/pci.c +10
-0+/** +* cxl_reset_prepare_memdev - Prepare CXL device for reset +* @pdev: PCI device +* +* Validates it's safe to reset and tears down regions atomically under lock. +* Acquires cxl_region_rwsem and keeps it held throughout reset. +* +* Return: 0 on success (lock held), -EBUSY if memory online, negative on error +*/ +static int cxl_reset_prepare_memdev (struct pci_dev *pdev)
Given the function name, shouldn't this pass in a cxl_memdev?
Quoted 9 lines
drivers/cxl/pci.c +9
-0+{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +struct cxl_memdev *cxlmd; +struct cxl_port *endpoint; +int rc; + +if (!cxlds) +return -ENODEV ; +
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1222
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ cxlmd = cxlds->cxlmd; +if (!cxlmd) +return -ENODEV ; + + endpoint = cxlmd->endpoint; +if (!endpoint) +return 0 ; + +if (cxl_num_decoders_committed (endpoint) ==0 ) +return 0 ;
Without any decoders committed, this endpoint shouldn't be part of any region correct? Why bother checking driver bound after this. Seems overkill.
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1226
Quoted 4 lines
drivers/cxl/pci.c +4
-0+ +down_write (&cxl_region_rwsem); + +/* Check and error out if memory is online */
Having cxl_region driver bound does not mean memory is online in the sense if it's hot-plugged to the memory subsystem. Are you looking for it to be not system memory here, or that cxl has claimed the region, or if there's just a hardware committed region?
Quoted 12 lines
drivers/cxl/pci.c +12
-0+ rc =device_for_each_child (&endpoint->dev,NULL , + cxl_check_region_driver_bound); +if (rc) { +up_write (&cxl_region_rwsem); +dev_err (&pdev->dev, +"Reset blocked: device has active regions with drivers bound\n" ); +return -EBUSY ; + } + +/* Flush device caches and tear down regions */ +device_for_each_child (&endpoint->dev, pdev, + cxl_region_flush_device_caches);
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1248
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ + rc =device_for_each_child (&endpoint->dev,NULL , + cxl_decoder_kill_region_iter); +if (rc) { +up_write (&cxl_region_rwsem); +dev_err (&pdev->dev,"Failed to tear down regions: %d\n" , rc); +return rc; + } + +/* Keep cxl_region_rwsem held, released by cleanup function */
May want to annotate the function with __acquires()
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1258
Quoted 10 lines
drivers/cxl/pci.c +10
-0+return 0 ; +} + +/** +* cxl_reset_cleanup_memdev - Release locks after CXL reset +* @pdev: PCI device +*/ +static void cxl_reset_cleanup_memdev (struct pci_dev *pdev) +{ +if (lockdep_is_held_type (&cxl_region_rwsem,-1 ))
why check when it's expected that the lock is held? Probably should have lockdep_assert_held() instead. Also may need a __releases() to annotate that this function releases the lock acquires in cxl_reset_prepare_memdev().
DJ
Quoted 49 lines
drivers/cxl/pci.c +43
-1+up_write (&cxl_region_rwsem); +} + +/** +* cxl_reset_prepare_device - Prepare CXL device for reset +* @pdev: PCI device being reset +* +* CXL-reset-specific preparation. Validates memory is offline, flushes +* device caches, and tears down regions. +* +* Returns: 0 on success, -EBUSY if memory online, negative on error +*/ +int cxl_reset_prepare_device (struct pci_dev *pdev) +{ +int rc; + + rc =cxl_reset_prepare_memdev (pdev); +if (rc) { +if (rc == -EBUSY ) +dev_err (&pdev->dev, +"Cannot reset: device has online memory or active regions\n" ); +else +dev_err (&pdev->dev, +"Failed to prepare device for reset: %d\n" , rc); +return rc; + } + +return 0 ; +} +EXPORT_SYMBOL_NS_GPL (cxl_reset_prepare_device,"CXL" ); + +/** +* cxl_reset_cleanup_device - Cleanup after CXL reset +* @pdev: PCI device that was reset +* +* Releases region locks held during reset. +*/ +void cxl_reset_cleanup_device (struct pci_dev *pdev) +{ +cxl_reset_cleanup_memdev (pdev); +} +EXPORT_SYMBOL_NS_GPL (cxl_reset_cleanup_device,"CXL" ); +static void cxl_error_resume (struct pci_dev *pdev) {struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); --2.34.1
PATCH v4 6/10 PCI: wire CXL reset prepare/cleanup
In reply to [email protected]From: Srirangan Madhavan <[email protected]>
Wire CXL reset preparation and cleanup into the PCI CXL reset path. The flow now validates/offlines regions, performs teardown and cache flushes, then releases the lock on completion or error. This keeps the common reset_prepare flow intact while adding cxl_reset-specific quiesce logic.
--- drivers/pci/pci.c | 19 ++++++++++++++++++- include/linux/pci.h | 11 +++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-)
drivers/pci/pci.c +18 -1
@@ -4989,10 +4989,27 @@ static int cxl_reset(struct pci_dev *dev, bool probe)if (probe)return 0 ; +/* +* CXL-reset-specific preparation: validate memory offline, +* tear down regions, flush device caches. +*/ + rc =cxl_reset_prepare_device (dev); +if (rc) +return rc; +if (!pci_wait_for_pending_transaction (dev))pci_err (dev,"timed out waiting for pending transaction; performing function level reset anyway\n" ); -return cxl_reset_init (dev, dvsec); + rc =cxl_reset_init (dev, dvsec); +if (rc) + goto out_cleanup; + +cxl_reset_cleanup_device (dev); +return 0 ; + +out_cleanup: +cxl_reset_cleanup_device (dev); +return rc; }static int cxl_reset_bus_function (struct pci_dev *dev,bool probe)
include/linux/pci.h +11 -0
@@ -1466,11 +1466,22 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags);bool pci_device_is_present (struct pci_dev *pdev);#ifdef CONFIG_CXL_PCI bool cxl_is_type2_device (struct pci_dev *dev); +int cxl_reset_prepare_device (struct pci_dev *pdev); +void cxl_reset_cleanup_device (struct pci_dev *pdev);#else static inline bool cxl_is_type2_device (struct pci_dev *dev) {return false; } + +static inline int cxl_reset_prepare_device (struct pci_dev *pdev) +{ +return 0 ; +} + +static inline void cxl_reset_cleanup_device (struct pci_dev *pdev) +{ +}#endif void pci_ignore_hotplug (struct pci_dev *dev);struct pci_dev *pci_real_dma_dev (struct pci_dev *dev);
--
2.34.1
Dave Jiang
maintainerRe: [PATCH v4 06/10] PCI: wire CXL reset prepare/cleanup
Quoted 6 lines · On 1/20/26 3:26 PM, [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Wire CXL reset preparation and cleanup into the PCI CXL reset path. The flow now validates/offlines regions, performs teardown and cache flushes, then releases the lock on completion or error. This keeps the common
reset_prepareflow intact while adding cxl_reset-specific quiesce logic.
Can this be moved to the ->reset_prepare() callback of 'pci_error_handlers' rather than directly wire it into PCI core code? I don't think we want build a dependency of cxl core to the PCI core.
Quoted 70 lines
Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/pci/pci.c | 19 ++++++++++++++++++- include/linux/pci.h | 11 +++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-)drivers/pci/pci.c +18
-1@@ -4989,10 +4989,27 @@ static int cxl_reset(struct pci_dev *dev, bool probe)if (probe)return 0 ; +/* +* CXL-reset-specific preparation: validate memory offline, +* tear down regions, flush device caches. +*/ + rc =cxl_reset_prepare_device (dev); +if (rc) +return rc; +if (!pci_wait_for_pending_transaction (dev))pci_err (dev,"timed out waiting for pending transaction; performing function level reset anyway\n" ); -return cxl_reset_init (dev, dvsec); + rc =cxl_reset_init (dev, dvsec); +if (rc) + goto out_cleanup; + +cxl_reset_cleanup_device (dev); +return 0 ; + +out_cleanup: +cxl_reset_cleanup_device (dev); +return rc; }static int cxl_reset_bus_function (struct pci_dev *dev,bool probe)include/linux/pci.h +11
-1@@ -1466,11 +1466,22 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags);bool pci_device_is_present (struct pci_dev *pdev);#ifdef CONFIG_CXL_PCI bool cxl_is_type2_device (struct pci_dev *dev); +int cxl_reset_prepare_device (struct pci_dev *pdev); +void cxl_reset_cleanup_device (struct pci_dev *pdev);#else static inline bool cxl_is_type2_device (struct pci_dev *dev) {return false; } + +static inline int cxl_reset_prepare_device (struct pci_dev *pdev) +{ +return 0 ; +} + +static inline void cxl_reset_cleanup_device (struct pci_dev *pdev) +{ +}#endif void pci_ignore_hotplug (struct pci_dev *dev);struct pci_dev *pci_real_dma_dev (struct pci_dev *dev); --2.34.1
Re: [PATCH v4 06/10] PCI: wire CXL reset prepare/cleanup
Quoted 7 lines · On 1/21/26, 2:14 PM, "Dave Jiang" <[email protected]…
Quoted 4 lines
Wire CXL reset preparation and cleanup into the PCI CXL reset path. The flow now validates/offlines regions, performs teardown and cache flushes, then releases the lock on completion or error. This keeps the common
reset_prepareflow intact while adding cxl_reset-specific quiesce logic.Can this be moved to the ->reset_prepare() callback of '
pci_error_handlers' rather than directly wire it into PCI core code? I don't think we want build a dependency of cxl core to the PCI core.
I understand the current implementation is not ideal. But I opted not to insert the call into ->reset_prepare() as pci_error_handler would be invoked for all the other reset methods (cxl_bus, FLR, etc). Would it be okay to include the same reset prepare steps (i.e region teardown, cache flush, etc) in the common reset prepare?
Thank you.
Dave Jiang
maintainerRe: [PATCH v4 06/10] PCI: wire CXL reset prepare/cleanup
Quoted 10 lines · Srirangan Madhavan
Quoted 5 lines · On 1/21/26, 2:14 PM, "Dave Jiang" <[email protected]…
Quoted 4 lines
Wire CXL reset preparation and cleanup into the PCI CXL reset path. The flow now validates/offlines regions, performs teardown and cache flushes, then releases the lock on completion or error. This keeps the common
reset_prepareflow intact while adding cxl_reset-specific quiesce logic.
Quoted 5 lines
Quoted 2 lines
Can this be moved to the ->reset_prepare() callback of '
pci_error_handlers' rather than directly wire it into PCI core code? I don't think we want build a dependency of cxl core to the PCI core.I understand the current implementation is not ideal. But I opted not to insert the call into ->reset_prepare() as
pci_error_handlerwould be invoked for all the other reset methods (cxl_bus, FLR, etc). Would it be okay to include the same reset prepare steps (i.e region teardown, cache flush, etc) in the common reset prepare?
See my suggestion reply against patch 10/10. Maybe we shouldn't be going to PCI reset path.
Quoted 3 lines
Thank you.
kernel test robot
build botRe: [PATCH v4 06/10] PCI: wire CXL reset prepare/cleanup
In reply to [email protected]kernel test robot noticed the following build errors:
Hi,
kernel test robot noticed the following build errors:
[auto build test ERROR on pci/next]
[also build test ERROR on pci/for-linus linus/master v6.19-rc6] [cannot apply to next-20260123]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/smadhavan-nvidia-com/cxl-move-DVSEC-defines-to-cxl-pci-header/20260121-071852 base: https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next patch link: https://lore.kernel.org/r/20260120222610.2227109-7-smadhavan%40nvidia.com patch subject: [PATCH v4 06/10] PCI: wire CXL reset prepare/cleanup config: openrisc-allmodconfig (https://download.01.org/0day-ci/archive/20260124/[email protected]/config) compiler: or1k-linux-gcc (GCC) 15.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260124/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
drivers/cxl/pci.c:1085:6: error: redefinition of 'cxl_is_type2_device'
1085 | bool cxl_is_type2_device(struct pci_dev *pdev)
| ^~~~~~~~~~~~~~~~~~~
In file included from drivers/cxl/pci.c:11:
include/linux/pci.h:1476:20: note: previous definition of 'cxl_is_type2_device' with type 'bool(struct pci_dev *)' {aka '_Bool(struct pci_dev *)'}
1476 | static inline bool cxl_is_type2_device(struct pci_dev *dev)
| ^~~~~~~~~~~~~~~~~~~
drivers/cxl/pci.c: In function 'cxl_check_region_driver_bound':
drivers/cxl/pci.c:1102:28: error: 'cxl_region_rwsem' undeclared (first use in this function); did you mean 'cxl_region_ref'?
1102 | guard(rwsem_read)(&cxl_region_rwsem);
| ^~~~~~~~~~~~~~~~
| cxl_region_ref
drivers/cxl/pci.c:1102:28: note: each undeclared identifier is reported only once for each function it appears in
drivers/cxl/pci.c:1103:41: error: 'struct cxl_region' has no member named 'driver'
1103 | if (cxld->region && cxld->region->driver)
| ^~
drivers/cxl/pci.c: In function 'cxl_decoder_kill_region_iter':
drivers/cxl/pci.c:1120:9: error: implicit declaration of function 'cxl_decoder_kill_region_locked'; did you mean 'cxl_decoder_kill_region_iter'? [-Wimplicit-function-declaration]
1120 | cxl_decoder_kill_region_locked(cxled);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| cxl_decoder_kill_region_iter
drivers/cxl/pci.c: In function 'cxl_region_flush_device_caches':
drivers/cxl/pci.c:1187:53: error: 'struct cxl_dev_state' has no member named 'pdev'; did you mean 'dev'?
1187 | if (!target_cxlds || !target_cxlds->pdev)
| ^~~~
| dev
drivers/cxl/pci.c:1190:35: error: 'struct cxl_dev_state' has no member named 'pdev'; did you mean 'dev'?
1190 | if (target_cxlds->pdev != target_pdev)
| ^~~~
| dev
drivers/cxl/pci.c: In function 'cxl_reset_prepare_memdev':
drivers/cxl/pci.c:1231:21: error: 'cxl_region_rwsem' undeclared (first use in this function); did you mean 'cxl_region_ref'?
1231 | down_write(&cxl_region_rwsem);
| ^~~~~~~~~~~~~~~~
| cxl_region_ref
In file included from include/linux/spinlock.h:63,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:7,
from include/linux/mm.h:7,
from arch/openrisc/include/asm/pgalloc.h:20,
from arch/openrisc/include/asm/io.h:18,
from include/linux/io.h:12,
from include/linux/io-64-nonatomic-lo-hi.h:5,
from drivers/cxl/pci.c:4:
drivers/cxl/pci.c: In function 'cxl_reset_cleanup_memdev':
drivers/cxl/pci.c:1265:35: error: 'cxl_region_rwsem' undeclared (first use in this function); did you mean 'cxl_region_ref'?
1265 | if (lockdep_is_held_type(&cxl_region_rwsem, -1))
| ^~~~~~~~~~~~~~~~
include/linux/lockdep.h:253:61: note: in definition of macro 'lockdep_is_held_type'
253 | #define lockdep_is_held_type(lock, r) lock_is_held_type(&(lock)->dep_map, (r))
| ^~~~
drivers/cxl/pci.c: At top level:
>> drivers/cxl/pci.c:1278:5: error: redefinition of 'cxl_reset_prepare_device'
1278 | int cxl_reset_prepare_device(struct pci_dev *pdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~
include/linux/pci.h:1481:19: note: previous definition of 'cxl_reset_prepare_device' with type 'int(struct pci_dev *)'
1481 | static inline int cxl_reset_prepare_device(struct pci_dev *pdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~
>> drivers/cxl/pci.c:1303:6: error: redefinition of 'cxl_reset_cleanup_device' 1303 | void cxl_reset_cleanup_device(struct pci_dev *pdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~
include/linux/pci.h:1486:20: note: previous definition of 'cxl_reset_cleanup_device' with type 'void(struct pci_dev *)'
1486 | static inline void cxl_reset_cleanup_device(struct pci_dev *pdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~
vim +/cxl_reset_prepare_device +1278 drivers/cxl/pci.c
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1268
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1269 /**
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1270 * cxl_reset_prepare_device - Prepare CXL device for reset
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1271 * @pdev: PCI device being reset
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1272 *
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1273 * CXL-reset-specific preparation. Validates memory is offline, flushes
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1274 * device caches, and tears down regions.
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1275 *
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1276 * Returns: 0 on success, -EBUSY if memory online, negative on error
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1277 */
76e9ed09acad72 Srirangan Madhavan 2026-01-20 @1278 int cxl_reset_prepare_device(struct pci_dev *pdev)
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1279 {
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1280 int rc;
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1281
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1282 rc = cxl_reset_prepare_memdev(pdev);
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1283 if (rc) {
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1284 if (rc == -EBUSY)
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1285 dev_err(&pdev->dev,
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1286 "Cannot reset: device has online memory or active regions\n");
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1287 else
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1288 dev_err(&pdev->dev,
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1289 "Failed to prepare device for reset: %d\n", rc);
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1290 return rc;
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1291 }
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1292
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1293 return 0;
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1294 }
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1295 EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1296
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1297 /**
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1298 * cxl_reset_cleanup_device - Cleanup after CXL reset
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1299 * @pdev: PCI device that was reset
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1300 *
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1301 * Releases region locks held during reset.
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1302 */
76e9ed09acad72 Srirangan Madhavan 2026-01-20 @1303 void cxl_reset_cleanup_device(struct pci_dev *pdev)
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1304 {
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1305 cxl_reset_cleanup_memdev(pdev);
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1306 }
76e9ed09acad72 Srirangan Madhavan 2026-01-20 1307 EXPORT_SYMBOL_NS_GPL(cxl_reset_cleanup_device, "CXL");76e9ed09acad72 Srirangan Madhavan 2026-01-20 1308
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v4 7/10 cxl: add host cache flush and multi-function reset
In reply to [email protected]From: Srirangan Madhavan <[email protected]>
Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses wbinvd_on_all_cpus() on x86 and VA-based PoC clean+ invalidate on arm64 via memremap() and on_each_cpu(), matching the required ordering before reset.
--- drivers/cxl/pci.c | 150 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 148 insertions(+), 2 deletions(-)
drivers/cxl/pci.c +148 -2
@@ -11,6 +11,10 @@#include <linux/pci.h> #include <linux/aer.h> #include <linux/io.h> +#include <linux/align.h> +#include <linux/cache.h> +#include <linux/cacheflush.h> +#include <linux/smp.h> #include <cxl/mailbox.h> #include <cxl/pci.h> #include "cxlmem.h"
@@ -1085,6 +1089,71 @@ bool cxl_is_type2_device(struct pci_dev *pdev)return cxlds->type ==CXL_DEVTYPE_DEVMEM ; } +#ifdef CONFIG_ARM64 +struct cxl_cache_flush_ctx { +void *va; +size_t len; +}; + +static void cxl_flush_by_va_local (void *info) +{ +struct cxl_cache_flush_ctx *ctx = info; + +dcache_clean_inval_poc ((unsigned long )ctx->va, + (unsigned long )ctx->va + ctx->len); + asmvolatile ("dsb ish" :::"memory" ); +} +#endif + +static int cxl_region_flush_host_cpu_caches (struct device *dev,void *data) +{ +struct cxl_endpoint_decoder *cxled =to_cxl_endpoint_decoder (dev); +struct cxl_region *cxlr = cxled->cxld.region; +struct resource *res; + +if (!is_endpoint_decoder (dev)) +return 0 ; + +if (!cxlr || !cxlr->params.res) +return 0 ; + + res = cxlr->params.res; + +#ifdef CONFIG_X86 +static bool flushed; + +if (!flushed) { +wbinvd_on_all_cpus (); + flushed = true; + } +#elif defined(CONFIG_ARM64 ) +void *va; +size_t len, line_size =L1_CACHE_BYTES ; +phys_addr_t start, end, aligned_start, aligned_end; +struct cxl_cache_flush_ctx flush_ctx; + + start = res->start; + end = res->end; + + aligned_start =ALIGN_DOWN (start, line_size); + aligned_end =ALIGN (end +1 , line_size); + len = aligned_end - aligned_start; + + va =memremap (aligned_start, len,MEMREMAP_WB ); +if (!va) { +pr_warn ("Failed to map region for cache flush\n" ); +return 0 ; + } + + flush_ctx.va = va; + flush_ctx.len = len; +on_each_cpu (cxl_flush_by_va_local, &flush_ctx,1 ); + +memunmap (va); +#endif +return 0 ; +} +static int cxl_check_region_driver_bound (struct device *dev,void *data) {struct cxl_decoder *cxld =to_cxl_decoder (dev);
@@ -1245,6 +1314,9 @@ static int cxl_reset_prepare_memdev(struct pci_dev *pdev)return rc; } +device_for_each_child (&endpoint->dev,NULL , + cxl_region_flush_host_cpu_caches); +/* Keep cxl_region_rwsem held, released by cleanup function */ return 0 ; }
@@ -1259,12 +1331,79 @@ static void cxl_reset_cleanup_memdev(struct pci_dev *pdev)up_write (&cxl_region_rwsem); } +static int cxl_reset_prepare_all_functions (struct pci_dev *pdev) +{ +struct pci_dev *func_dev; +unsigned int devfn; +int func, rc; +struct pci_dev *prepared_funcs[8 ] = {NULL }; +int prepared_count =0 ; + +for (func =0 ; func <8 ; func++) { + devfn =PCI_DEVFN (PCI_SLOT (pdev->devfn), func); + +if (devfn == pdev->devfn) +continue ; + + func_dev =pci_get_slot (pdev->bus, devfn); +if (!func_dev) +continue ; + +if (!cxl_is_type2_device (func_dev)) { +pci_dev_put (func_dev); +continue ; + } + + rc =cxl_reset_prepare_memdev (func_dev); +if (rc) { +pci_dev_put (func_dev); + goto cleanup_funcs; + } + + prepared_funcs[prepared_count++] = func_dev; + } + +return 0 ; + +cleanup_funcs: +for (func =0 ; func < prepared_count; func++) { +if (prepared_funcs[func]) { +cxl_reset_cleanup_memdev (prepared_funcs[func]); +pci_dev_put (prepared_funcs[func]); + } + } +return rc; +} + +static void cxl_reset_cleanup_all_functions (struct pci_dev *pdev) +{ +struct pci_dev *func_dev; +unsigned int devfn; +int func; + +for (func =0 ; func <8 ; func++) { + devfn =PCI_DEVFN (PCI_SLOT (pdev->devfn), func); + +if (devfn == pdev->devfn) +continue ; + + func_dev =pci_get_slot (pdev->bus, devfn); +if (!func_dev) +continue ; + +if (cxl_is_type2_device (func_dev)) +cxl_reset_cleanup_memdev (func_dev); + +pci_dev_put (func_dev); + } +} +/** * cxl_reset_prepare_device - Prepare CXL device for reset * @pdev: PCI device being reset * * CXL-reset-specific preparation. Validates memory is offline, flushes -* device caches, and tears down regions. +* device caches, and tears down regions for device and siblings. * * Returns: 0 on success, -EBUSY if memory online, negative on error */
@@ -1283,6 +1422,12 @@ int cxl_reset_prepare_device(struct pci_dev *pdev)return rc; } + rc =cxl_reset_prepare_all_functions (pdev); +if (rc) { +cxl_reset_cleanup_memdev (pdev); +return rc; + } +return 0 ; }EXPORT_SYMBOL_NS_GPL (cxl_reset_prepare_device,"CXL" );
@@ -1291,10 +1436,11 @@ EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");* cxl_reset_cleanup_device - Cleanup after CXL reset * @pdev: PCI device that was reset * -* Releases region locks held during reset. +* Releases region locks for device and all sibling functions. */ void cxl_reset_cleanup_device (struct pci_dev *pdev) { +cxl_reset_cleanup_all_functions (pdev);cxl_reset_cleanup_memdev (pdev); }EXPORT_SYMBOL_NS_GPL (cxl_reset_cleanup_device,"CXL" );
--
2.34.1
Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
Quoted 5 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses
wbinvd_on_all_cpus()on x86 and VA-based PoC clean+
That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.
Until there are clarifications from Arm we need something like the agents in drivers/cache to do equivalent of wbinvd_on_all_cpus() (be it on a PA range to making a tiny bit less horrible)
Or we need an opt in list for platforms where a flush to PoC is enough.
Needs to some sort of arch_ call as well, not hidden in the cxl driver.
I'd suggest a separate patch to add the necessary infrastructure for arm64 with the relevant maintainers and lists included.
It needs enough work to be something they'd consider that I haven't +CC them on this version.
With that in mind I'll skip reviewing this patch for now.
Jonathan
Quoted 216 lines
invalidate on arm64 via
memremap()andon_each_cpu(), matching the required ordering before reset.Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/pci.c | 150 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 148 insertions(+), 2 deletions(-)drivers/cxl/pci.c +148
-3@@ -11,6 +11,10 @@#include <linux/pci.h> #include <linux/aer.h> #include <linux/io.h> +#include <linux/align.h> +#include <linux/cache.h> +#include <linux/cacheflush.h> +#include <linux/smp.h> #include <cxl/mailbox.h> #include <cxl/pci.h> #include "cxlmem.h" @@ -1085,6 +1089,71 @@ bool cxl_is_type2_device(struct pci_dev *pdev)return cxlds->type ==CXL_DEVTYPE_DEVMEM ; } +#ifdef CONFIG_ARM64 +struct cxl_cache_flush_ctx { +void *va; +size_t len; +}; + +static void cxl_flush_by_va_local (void *info) +{ +struct cxl_cache_flush_ctx *ctx = info; + +dcache_clean_inval_poc ((unsigned long )ctx->va, + (unsigned long )ctx->va + ctx->len); + asmvolatile ("dsb ish" :::"memory" ); +} +#endif + +static int cxl_region_flush_host_cpu_caches (struct device *dev,void *data) +{ +struct cxl_endpoint_decoder *cxled =to_cxl_endpoint_decoder (dev); +struct cxl_region *cxlr = cxled->cxld.region; +struct resource *res; + +if (!is_endpoint_decoder (dev)) +return 0 ; + +if (!cxlr || !cxlr->params.res) +return 0 ; + + res = cxlr->params.res; + +#ifdef CONFIG_X86 +static bool flushed; + +if (!flushed) { +wbinvd_on_all_cpus (); + flushed = true; + } +#elif defined(CONFIG_ARM64 ) +void *va; +size_t len, line_size =L1_CACHE_BYTES ; +phys_addr_t start, end, aligned_start, aligned_end; +struct cxl_cache_flush_ctx flush_ctx; + + start = res->start; + end = res->end; + + aligned_start =ALIGN_DOWN (start, line_size); + aligned_end =ALIGN (end +1 , line_size); + len = aligned_end - aligned_start; + + va =memremap (aligned_start, len,MEMREMAP_WB ); +if (!va) { +pr_warn ("Failed to map region for cache flush\n" ); +return 0 ; + } + + flush_ctx.va = va; + flush_ctx.len = len; +on_each_cpu (cxl_flush_by_va_local, &flush_ctx,1 ); + +memunmap (va); +#endif +return 0 ; +} +static int cxl_check_region_driver_bound (struct device *dev,void *data) {struct cxl_decoder *cxld =to_cxl_decoder (dev);@@ -1245,6 +1314,9 @@ static int cxl_reset_prepare_memdev(struct pci_dev *pdev)return rc; } +device_for_each_child (&endpoint->dev,NULL , + cxl_region_flush_host_cpu_caches); +/* Keep cxl_region_rwsem held, released by cleanup function */ return 0 ; }@@ -1259,12 +1331,79 @@ static void cxl_reset_cleanup_memdev(struct pci_dev *pdev)up_write (&cxl_region_rwsem); } +static int cxl_reset_prepare_all_functions (struct pci_dev *pdev) +{ +struct pci_dev *func_dev; +unsigned int devfn; +int func, rc; +struct pci_dev *prepared_funcs[8 ] = {NULL }; +int prepared_count =0 ; + +for (func =0 ; func <8 ; func++) { + devfn =PCI_DEVFN (PCI_SLOT (pdev->devfn), func); + +if (devfn == pdev->devfn) +continue ; + + func_dev =pci_get_slot (pdev->bus, devfn); +if (!func_dev) +continue ; + +if (!cxl_is_type2_device (func_dev)) { +pci_dev_put (func_dev); +continue ; + } + + rc =cxl_reset_prepare_memdev (func_dev); +if (rc) { +pci_dev_put (func_dev); + goto cleanup_funcs; + } + + prepared_funcs[prepared_count++] = func_dev; + } + +return 0 ; + +cleanup_funcs: +for (func =0 ; func < prepared_count; func++) { +if (prepared_funcs[func]) { +cxl_reset_cleanup_memdev (prepared_funcs[func]); +pci_dev_put (prepared_funcs[func]); + } + } +return rc; +} + +static void cxl_reset_cleanup_all_functions (struct pci_dev *pdev) +{ +struct pci_dev *func_dev; +unsigned int devfn; +int func; + +for (func =0 ; func <8 ; func++) { + devfn =PCI_DEVFN (PCI_SLOT (pdev->devfn), func); + +if (devfn == pdev->devfn) +continue ; + + func_dev =pci_get_slot (pdev->bus, devfn); +if (!func_dev) +continue ; + +if (cxl_is_type2_device (func_dev)) +cxl_reset_cleanup_memdev (func_dev); + +pci_dev_put (func_dev); + } +} +/** * cxl_reset_prepare_device - Prepare CXL device for reset * @pdev: PCI device being reset * * CXL-reset-specific preparation. Validates memory is offline, flushes -* device caches, and tears down regions. +* device caches, and tears down regions for device and siblings. * * Returns: 0 on success, -EBUSY if memory online, negative on error */ @@ -1283,6 +1422,12 @@ int cxl_reset_prepare_device(struct pci_dev *pdev)return rc; } + rc =cxl_reset_prepare_all_functions (pdev); +if (rc) { +cxl_reset_cleanup_memdev (pdev); +return rc; + } +return 0 ; }EXPORT_SYMBOL_NS_GPL (cxl_reset_prepare_device,"CXL" );@@ -1291,10 +1436,11 @@ EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");* cxl_reset_cleanup_device - Cleanup after CXL reset * @pdev: PCI device that was reset * -* Releases region locks held during reset. +* Releases region locks for device and all sibling functions. */ void cxl_reset_cleanup_device (struct pci_dev *pdev) { +cxl_reset_cleanup_all_functions (pdev);cxl_reset_cleanup_memdev (pdev); }EXPORT_SYMBOL_NS_GPL (cxl_reset_cleanup_device,"CXL" ); --2.34.1
Davidlohr Bueso
maintainerRe: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
Quoted 5 lines · Jonathan Cameron
Quoted 2 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Quoted 7 lines
Quoted 3 lines
Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses
wbinvd_on_all_cpus()on x86 and VA-based PoC clean+That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.
... and quite slow for large regions without firmware assistance (CLEAN_INV_MEMREGION from PSCI 1.3).
Quoted 7 lines
Until there are clarifications from Arm we need something like the agents in
drivers/cacheto do equivalent ofwbinvd_on_all_cpus()(be it on a PA range to making a tiny bit less horrible)Or we need an opt in list for platforms where a flush to PoC is enough.
Needs to some sort of
arch_call as well, not hidden in the cxl driver.
Reset should just be using cpu_cache_invalidate_memregion().
... now, in addition to DCD, we have two users that are beyond the "just once at boot time" use case.
Thanks,
Davidlohr
Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
Quoted 11 lines · On Wed, 21 Jan 2026 12:27:55 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 9 lines · Jonathan Cameron
Quoted 5 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses
wbinvd_on_all_cpus()on x86 and VA-based PoC clean+
Quoted 6 lines
Quoted 3 lines
That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.
... and quite slow for large regions without firmware assistance (
CLEAN_INV_MEMREGIONfrom PSCI 1.3).
Just for completeness in case anyone takes this comment out of context.
CLEAN_INV_MEMREGION was dropped after that spec when beyond alpha. See F.b release.
https://developer.arm.com/documentation/den0022/fb/?lang=en
Talk to you to your favorite Arm architecture person for why
(I'll just say it was a good reason).
Also there is no guarantee that firmware interface does it any faster. Implementation defined and all that.
No one jumped on my proposal to write an ACPI AML wrapper spec so for now only option is to write a specific driver if you have a hardware agent that offloads these operations (like the hisi_hha does).
Quoted 5 lines
Quoted 5 lines
Until there are clarifications from Arm we need something like the agents in
drivers/cacheto do equivalent ofwbinvd_on_all_cpus()(be it on a PA range to making a tiny bit less horrible)
Quoted 8 lines
Quoted 3 lines
Or we need an opt in list for platforms where a flush to PoC is enough.
Needs to some sort of
arch_call as well, not hidden in the cxl driver.Reset should just be using
cpu_cache_invalidate_memregion().... now, in addition to DCD, we have two users that are beyond the "just once at boot time" use case.
or Back-Invalidate to do this from the device side. If anyone has built that yet... Beware physical address based prefetchers though as there be monsters.
Jonathan
Quoted 4 lines
Thanks,
Davidlohr
Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
In reply to Jonathan CameronHi Jonathan,
________________________________________
From: Jonathan Cameron <[email protected]>
Sent: Wednesday, January 21, 2026 5:20 AM
To: Srirangan Madhavan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vishal Aslot; Vikram Sethi; Shanker Donthineni; Vidya Sagar; Matt Ochs; Jason Sequeira
Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
Quoted 10 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:
Quoted 5 lines
From: Srirangan Madhavan <[email protected]>
Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses
wbinvd_on_all_cpus()on x86 and VA-based PoC clean+That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.
Quoted 7 lines
Until there are clarifications from Arm we need something like the agents in
drivers/cacheto do equivalent ofwbinvd_on_all_cpus()(be it on a PA range to making a tiny bit less horrible)Or we need an opt in list for platforms where a flush to PoC is enough.
Needs to some sort of
arch_call as well, not hidden in the cxl driver.
Yes, and in fact it cannot be a cache flush by VA while PTE mappings to the memory are valid, as the core can prefetch the lines back right away, and later evictions post device reset can cause device errors as the device snoop filter isn’t aware of host having any lines.
The only way to do this correctly is via a SMC call to Arm Trusted Firmware to handle in a SOC specific way (after the memory has been offlined and removed from the PTEs), since there is no architectural way to flush by PA (set/way isn’t usable in the kernel). There was a SMC call defined for this in the PSCI spec 2 years ago, but was removed for some reason. I had a discussion with some ARM folks maintaining the PSCI and SMCCC specifications and the direction was to bring the SMC call back in SMCCC specification. Like you say, that can be a separate patch series. Adding Souvik for SMCCC specification details once it is available.
Thanks
Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
Hi,
Quoted 29 lines · Vikram Sethi
Hi Jonathan,
________________________________________
From: Jonathan Cameron <[email protected]>
Sent: Wednesday, January 21, 2026 5:20 AM
To: Srirangan Madhavan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vishal Aslot; Vikram Sethi; Shanker Donthineni; Vidya Sagar; Matt Ochs; Jason Sequeira
Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function resetQuoted 13 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:
Quoted 5 lines
From: Srirangan Madhavan <[email protected]>
Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses
wbinvd_on_all_cpus()on x86 and VA-based PoC clean+That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.
Until there are clarifications from Arm we need something like the agents in
drivers/cacheto do equivalent ofwbinvd_on_all_cpus()(be it on a PA range to making a tiny bit less horrible)
Quoted 7 lines
Quoted 3 lines
Or we need an opt in list for platforms where a flush to PoC is enough.
Needs to some sort of
arch_call as well, not hidden in the cxl driver.Yes, and in fact it cannot be a cache flush by VA while PTE mappings to the memory are valid, as the core can prefetch the lines back right away, and later evictions post device reset can cause device errors as the device snoop filter isn’t aware of host having any lines.
The only way to do this correctly is via a SMC call to Arm Trusted Firmware to handle in a SOC specific way (after the memory has been offlined and removed from the PTEs), since there is no architectural way to flush by PA (set/way isn’t usable in the kernel). There was a SMC call defined for this in the PSCI spec 2 years ago, but was removed for some reason. I had a discussion with some ARM folks maintaining the PSCI and SMCCC specifications and the direction was to bring the SMC call back in SMCCC specification. Like you say, that can be a separate patch series. Adding Souvik for SMCCC specification details once it is available.
Yes can confirm this is being actively worked on. Something should be available in the public SMCCC spec soon. Will be the same PSCI functions moved to SMCCC but slightly simplified.
Regards,
Souvik
Quoted 3 lines
Thanks
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
A message that is not in the archive (sent off-list or not yet received): PH7PR12MB9175CDFC163843BB497073CEBD96A@PH7PR12MB9175.namprd12.prod.outlook.com
Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
Quoted 8 lines · On Wed, 21 Jan 2026 20:36:01 +0000 Vikram Sethi <[email protected]> wrote:
Hi Jonathan,
From: Jonathan Cameron <[email protected]>
Date: Wednesday, January 21, 2026 at 5:20 AM
To: Srirangan Madhavan <[email protected]>
Cc: [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, Vishal Aslot <[email protected]>, Vikram Sethi <[email protected]>, Shanker Donthineni <[email protected]>, Vidya Sagar <[email protected]>, Matt Ochs <[email protected]>, Jason Sequeira <[email protected]>
Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
Hi Vikram, Happy new year!
Quoted 13 lines
Quoted 8 lines · On Tue, 20 Jan 2026 22:26:07 +0000 [email protected] wrote:
Quoted 4 lines
From: Srirangan Madhavan <[email protected]>
Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance useswbinvd_on_all_cpus()on x86 and VA-based PoC clean+That's not sufficient in general on arm64. It might flush far enough or there might be buffers beyond the point of coherence that are not flushed.
snip
Quoted 7 lines
Quoted 1 line
Needs to some sort of
arch_call as well, not hidden in the cxl driver.Yes, and in fact it cannot be a cache flush by VA while PTE mappings to the memory are valid, as the core can prefetch the lines back right away, and later evictions post device reset can cause device errors as the device snoop filter isn’t aware of host having any lines.
Prefetching should (hopefully) not make any dirty lines. Hopefully no one does clean writebacks (and there is a way to check if they do in the ACPI tables). You need to flush again after to force out that stale stuff though before any demand fetches occur.
+ don't forget PA based prefetchers. Doesn't even need to be a mapping for fetches into distance caches to happen. A given platform might have restrictions on where those PA prefetchers will go but we currently have no way to discover that.
I need to think a bit more on this as there are some scary comments in the spec on CXL.reset such as all CXL.mem reads are dropped (timeout fun). Mind you a device is permitted to do that anyway before cxl.mem is enabled, so hopefully n one times out if a prefetcher hits the device before that's on..
Quoted 11 lines
The only way to do this correctly is via a SMC call to Arm trusted Firmware to handle in SOC specific way (after the memory has been offlined and removed from the PTEs), since there is no architectural way to flush by PA (set/way isn’t usable in the kernel). There was a SMC call defined for this in the PSCI spec 2 years ago, but was removed for some reason. I had a discussion with some ARM folks maintaining the PSCI and SMCCC specifications and the direction was to bring the SMC call back in SMCCC specification. Like you say, that can be a separate patch series. Adding Souvik for SMCCC specification details once it is available.
There are other possible paths which is where drivers/cache stuff came from.+CC James Morse.
From a kernel point of view SMCCC needs to be just one option as a bunch of hardware (I'll only point at ours as not sure what else is public) provide MMIO accessible agents to do this stuff and going via EL3 to talk to an engine the kernel can poke directly is silly.
I'm not against the PSCI thing coming back though if someone needs it. Preferably without the CPU rendezvous stuff though -> or Linux can just reject anyone who does that.
We could also revisit the approach of using an AML op region to issue the SMCCC, thus allowing us to support vendor SMCCC calls (+ a general one if ARM do bring that back) or whatever magic they want to use.
https://lore.kernel.org/all/[email protected]/
Advantage of that is you can wrap SMCCC, MMIO or whatever else you like up and the kernel only needs one driver. Disadvantage is need a spec and only ACPI etc so we need a driver anyway if anyone cares on DT systems...
If you want to mess with that I can dig out the QEMU emulation and rebase that driver. I thought it was kind of cute, just a solution we didn't need at the time!
Jonathan
Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
Hi Jonathan,
Happy new year!
Quoted 11 lines
From: Jonathan Cameron <[email protected]>
Date: Thursday, January 22, 2026 4:31 AM
To: Srirangan Madhavan <[email protected]>
Cc: [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, Vishal Aslot <[email protected]>, Vikram Sethi <[email protected]>, Shanker Donthineni <[email protected]>, Vidya Sagar <[email protected]>, Matt Ochs <[email protected]>, Jason Sequeira <[email protected]>
Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function resetPrefetching should (hopefully) not make any dirty lines. Hopefully no one does clean writebacks (and there is a way to check if they do in the ACPI tables). You need to flush again after to force out that stale stuff though before any demand fetches occur.
I am aware of some systems which do clean writebacks to device memory. What exact ACPI table has this discovery?
Quoted 5 lines
I need to think a bit more on this as there are some scary comments in the spec on CXL.reset such as all CXL.mem reads are dropped (timeout fun). Mind you a device is permitted to do that anyway before cxl.mem is enabled, so hopefully n one times out if a prefetcher hits the device before that's on..
Yes, any sensible coherent memory device reset will have to include first offlining the memory such that there is no demand or speculative fetch possible, else you get to deal with CXL error isolation fun.
Quoted 4 lines
From a kernel point of view SMCCC needs to be just one option as a bunch of hardware (I'll only point at ours as not sure what else is public) provide MMIO accessible agents to do this stuff and going via EL3 to talk to an engine the kernel can poke directly is silly.
We have a similar custom engine for efficient flushing, but the interface is not available to the kernel, so the SMCCC is preferred for our implementation. Like you say, the AML wrapper for SMCCC is another option, although we also have some device tree based systems where the efficient custom flushing is desirable.
Quoted 3 lines
I'm not against the PSCI thing coming back though if someone needs it. Preferably without the CPU rendezvous stuff though -> or Linux can just reject anyone who does that.
Agreed. I think Souvik also agreed offline that rendezvous is not needed for the SMC based cache flush.
Vikram
Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
Quoted 10 lines · On Thu, 22 Jan 2026 19:24:49 +0000 Vikram Sethi <[email protected]> wrote:
Hi Jonathan,
Happy new year!Quoted 5 lines
From: Jonathan Cameron <[email protected]>
Date: Thursday, January 22, 2026 4:31 AM
To: Srirangan Madhavan <[email protected]>
Cc: [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, [email protected] <[email protected]>, Vishal Aslot <[email protected]>, Vikram Sethi <[email protected]>, Shanker Donthineni <[email protected]>, Vidya Sagar <[email protected]>, Matt Ochs <[email protected]>, Jason Sequeira <[email protected]>
Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
Quoted 7 lines
Quoted 4 lines
Prefetching should (hopefully) not make any dirty lines. Hopefully no one does clean writebacks (and there is a way to check if they do in the ACPI tables). You need to flush again after to force out that stale stuff though before any demand fetches occur.
I am aware of some systems which do clean writebacks to device memory. What exact ACPI table has this discovery?
CEDT (so in the CXL spec, it's there in 3.2 and 4, I'm too lazy to open earlier specs) Specifically the CXL System Description Structure (CSDS) System Capabilities bit[1]
No Clean Writeback: Specifies the clean writeback beahvior of hte host
- 0 = The host may or may not generate clean writebacks.
- 1 = The host guarantees to never generate clean writeback at the host's cacheline granularity.
Quoted 7 lines
Quoted 6 lines
I need to think a bit more on this as there are some scary comments in the spec on CXL.reset such as all CXL.mem reads are dropped (timeout fun). Mind you a device is permitted to do that anyway before cxl.mem is enabled, so hopefully n one times out if a prefetcher hits the device before that's on..
Quoted 8 lines
Yes, any sensible coherent memory device reset will have to include first offlining the memory such that there is no demand or speculative fetch possible, else you get to deal with CXL error isolation fun.
Quoted 4 lines
From a kernel point of view SMCCC needs to be just one option as a bunch of hardware (I'll only point at ours as not sure what else is public) provide MMIO accessible agents to do this stuff and going via EL3 to talk to an engine the kernel can poke directly is silly.
We have a similar custom engine for efficient flushing, but the interface is not available to the kernel, so the SMCCC is preferred for our implementation. Like you say, the AML wrapper for SMCCC is another option, although we also have some device tree based systems where the efficient custom flushing is desirable.
Great. So all that work they did before dropping the PSCI call was worth doing. I'll keep an eye open for the new spec.
Quoted 6 lines
Quoted 4 lines
I'm not against the PSCI thing coming back though if someone needs it. Preferably without the CPU rendezvous stuff though -> or Linux can just reject anyone who does that.
Agreed. I think Souvik also agreed offline that rendezvous is not needed for the SMC based cache flush.
Great.
Thanks,
Jonathan
Quoted 3 lines
Vikram
Dave Jiang
maintainerRe: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
In reply to [email protected]Quoted 29 lines · On 1/20/26 3:26 PM, [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Flush host CPU caches for mapped HDM ranges after teardown and prepare sibling Type 2 functions on multi-function devices. The host cache maintenance uses
wbinvd_on_all_cpus()on x86 and VA-based PoC clean+ invalidate on arm64 viamemremap()andon_each_cpu(), matching the required ordering before reset.Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/pci.c | 150 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 148 insertions(+), 2 deletions(-)drivers/cxl/pci.c +4
-0@@ -11,6 +11,10 @@#include <linux/pci.h> #include <linux/aer.h> #include <linux/io.h> +#include <linux/align.h> +#include <linux/cache.h> +#include <linux/cacheflush.h> +#include <linux/smp.h> #include <cxl/mailbox.h> #include <cxl/pci.h> #include "cxlmem.h"
drivers/cxl/pci.c bool cxl_is_type2_device(struct pci_dev *pdev) · line 1092
Quoted 4 lines
drivers/cxl/pci.c +1
-0@@ -1085,6 +1089,71 @@ bool cxl_is_type2_device(struct pci_dev *pdev)return cxlds->type ==CXL_DEVTYPE_DEVMEM ; } +#ifdef CONFIG_ARM64
Please figure out some arch common way to do this. Really do not want CONFIG_$ARCH ifdefs sprinkled all over CXL code. Also this ARCH specific code probably does not belong under CXL anyhow.
DJ
Quoted 187 lines
drivers/cxl/pci.c +143
-3+struct cxl_cache_flush_ctx { +void *va; +size_t len; +}; + +static void cxl_flush_by_va_local (void *info) +{ +struct cxl_cache_flush_ctx *ctx = info; + +dcache_clean_inval_poc ((unsigned long )ctx->va, + (unsigned long )ctx->va + ctx->len); + asmvolatile ("dsb ish" :::"memory" ); +} +#endif + +static int cxl_region_flush_host_cpu_caches (struct device *dev,void *data) +{ +struct cxl_endpoint_decoder *cxled =to_cxl_endpoint_decoder (dev); +struct cxl_region *cxlr = cxled->cxld.region; +struct resource *res; + +if (!is_endpoint_decoder (dev)) +return 0 ; + +if (!cxlr || !cxlr->params.res) +return 0 ; + + res = cxlr->params.res; + +#ifdef CONFIG_X86 +static bool flushed; + +if (!flushed) { +wbinvd_on_all_cpus (); + flushed = true; + } +#elif defined(CONFIG_ARM64 ) +void *va; +size_t len, line_size =L1_CACHE_BYTES ; +phys_addr_t start, end, aligned_start, aligned_end; +struct cxl_cache_flush_ctx flush_ctx; + + start = res->start; + end = res->end; + + aligned_start =ALIGN_DOWN (start, line_size); + aligned_end =ALIGN (end +1 , line_size); + len = aligned_end - aligned_start; + + va =memremap (aligned_start, len,MEMREMAP_WB ); +if (!va) { +pr_warn ("Failed to map region for cache flush\n" ); +return 0 ; + } + + flush_ctx.va = va; + flush_ctx.len = len; +on_each_cpu (cxl_flush_by_va_local, &flush_ctx,1 ); + +memunmap (va); +#endif +return 0 ; +} +static int cxl_check_region_driver_bound (struct device *dev,void *data) {struct cxl_decoder *cxld =to_cxl_decoder (dev);@@ -1245,6 +1314,9 @@ static int cxl_reset_prepare_memdev(struct pci_dev *pdev)return rc; } +device_for_each_child (&endpoint->dev,NULL , + cxl_region_flush_host_cpu_caches); +/* Keep cxl_region_rwsem held, released by cleanup function */ return 0 ; }@@ -1259,12 +1331,79 @@ static void cxl_reset_cleanup_memdev(struct pci_dev *pdev)up_write (&cxl_region_rwsem); } +static int cxl_reset_prepare_all_functions (struct pci_dev *pdev) +{ +struct pci_dev *func_dev; +unsigned int devfn; +int func, rc; +struct pci_dev *prepared_funcs[8 ] = {NULL }; +int prepared_count =0 ; + +for (func =0 ; func <8 ; func++) { + devfn =PCI_DEVFN (PCI_SLOT (pdev->devfn), func); + +if (devfn == pdev->devfn) +continue ; + + func_dev =pci_get_slot (pdev->bus, devfn); +if (!func_dev) +continue ; + +if (!cxl_is_type2_device (func_dev)) { +pci_dev_put (func_dev); +continue ; + } + + rc =cxl_reset_prepare_memdev (func_dev); +if (rc) { +pci_dev_put (func_dev); + goto cleanup_funcs; + } + + prepared_funcs[prepared_count++] = func_dev; + } + +return 0 ; + +cleanup_funcs: +for (func =0 ; func < prepared_count; func++) { +if (prepared_funcs[func]) { +cxl_reset_cleanup_memdev (prepared_funcs[func]); +pci_dev_put (prepared_funcs[func]); + } + } +return rc; +} + +static void cxl_reset_cleanup_all_functions (struct pci_dev *pdev) +{ +struct pci_dev *func_dev; +unsigned int devfn; +int func; + +for (func =0 ; func <8 ; func++) { + devfn =PCI_DEVFN (PCI_SLOT (pdev->devfn), func); + +if (devfn == pdev->devfn) +continue ; + + func_dev =pci_get_slot (pdev->bus, devfn); +if (!func_dev) +continue ; + +if (cxl_is_type2_device (func_dev)) +cxl_reset_cleanup_memdev (func_dev); + +pci_dev_put (func_dev); + } +} +/** * cxl_reset_prepare_device - Prepare CXL device for reset * @pdev: PCI device being reset * * CXL-reset-specific preparation. Validates memory is offline, flushes -* device caches, and tears down regions. +* device caches, and tears down regions for device and siblings. * * Returns: 0 on success, -EBUSY if memory online, negative on error */ @@ -1283,6 +1422,12 @@ int cxl_reset_prepare_device(struct pci_dev *pdev)return rc; } + rc =cxl_reset_prepare_all_functions (pdev); +if (rc) { +cxl_reset_cleanup_memdev (pdev); +return rc; + } +return 0 ; }EXPORT_SYMBOL_NS_GPL (cxl_reset_prepare_device,"CXL" );@@ -1291,10 +1436,11 @@ EXPORT_SYMBOL_NS_GPL(cxl_reset_prepare_device, "CXL");* cxl_reset_cleanup_device - Cleanup after CXL reset * @pdev: PCI device that was reset * -* Releases region locks held during reset. +* Releases region locks for device and all sibling functions. */ void cxl_reset_cleanup_device (struct pci_dev *pdev) { +cxl_reset_cleanup_all_functions (pdev);cxl_reset_cleanup_memdev (pdev); }EXPORT_SYMBOL_NS_GPL (cxl_reset_cleanup_device,"CXL" ); --2.34.1
PATCH v4 8/10 cxl: add DVSEC config save/restore
In reply to [email protected]From: Srirangan Madhavan <[email protected]>
Save and restore CXL DVSEC control registers across reset with CONFIG_LOCK handling so RWL fields are preserved when locked. This maintains device policy and capability state across cxl_reset while avoiding writes to locked fields.
--- drivers/cxl/pci.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++ include/cxl/pci.h | 15 +++++++ 2 files changed, 122 insertions(+)
drivers/cxl/pci.c +107 -0
@@ -1154,6 +1154,113 @@ static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data)return 0 ; } +/* +* CXL DVSEC register save/restore +*/ +static int cxl_save_dvsec_state (struct pci_dev *pdev, +struct cxl_type2_saved_state *state,int dvsec) +{ +int rc; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CTRL_OFFSET , + &state->dvsec_ctrl); +if (rc) +return rc; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CTRL2_OFFSET , + &state->dvsec_ctrl2); +return rc; +} + +static int cxl_restore_dvsec_state (struct pci_dev *pdev, +const struct cxl_type2_saved_state *state, +int dvsec,bool config_locked) +{ +int rc; +u16 val_to_restore; + +if (config_locked) { +u16 current_val; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CTRL_OFFSET , + ¤t_val); +if (rc) +return rc; + + val_to_restore = (current_val &CXL_DVSEC_CTRL_RWL_MASK ) | + (state->dvsec_ctrl & ~CXL_DVSEC_CTRL_RWL_MASK ); + }else { + val_to_restore = state->dvsec_ctrl; + } + + rc =pci_write_config_word (pdev, dvsec +CXL_DVSEC_CTRL_OFFSET , + val_to_restore); +if (rc) +return rc; + + rc =pci_write_config_word (pdev, dvsec +CXL_DVSEC_CTRL2_OFFSET , + state->dvsec_ctrl2); +return rc; +} + +/** +* cxl_config_save_state - Save CXL configuration state +* @pdev: PCI device +* @state: Structure to store saved state +* +* Saves CXL DVSEC state before reset. +*/ +int cxl_config_save_state (struct pci_dev *pdev, +struct cxl_type2_saved_state *state) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +int dvsec; + +if (!cxlds || !state) +return -EINVAL ; + +memset (state,0 ,sizeof (*state)); + + dvsec = cxlds->cxl_dvsec; +if (!dvsec) +return -ENODEV ; + +return cxl_save_dvsec_state (pdev, state, dvsec); +} +EXPORT_SYMBOL_NS_GPL (cxl_config_save_state,"CXL" ); + +/** +* cxl_config_restore_state - Restore CXL configuration state +* @pdev: PCI device +* @state: Previously saved state +* +* Restores CXL DVSEC state after reset. +*/ +int cxl_config_restore_state (struct pci_dev *pdev, +const struct cxl_type2_saved_state *state) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +bool config_locked; +int rc, dvsec; +u16 lock_reg; + +if (!cxlds || !state) +return -EINVAL ; + + dvsec = cxlds->cxl_dvsec; +if (!dvsec) +return -ENODEV ; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_LOCK_OFFSET , &lock_reg); +if (rc) +return rc; + + config_locked = !!(lock_reg &CXL_DVSEC_LOCK_CONFIG_LOCK ); + +return cxl_restore_dvsec_state (pdev, state, dvsec, config_locked); +} +EXPORT_SYMBOL_NS_GPL (cxl_config_restore_state,"CXL" ); +static int cxl_check_region_driver_bound (struct device *dev,void *data) {struct cxl_decoder *cxld =to_cxl_decoder (dev);
include/cxl/pci.h +15 -0
@@ -4,6 +4,18 @@#ifndef __CXL_ACCEL_PCI_H#define __CXL_ACCEL_PCI_H +/* CXL Type 2 device state for save/restore across reset */ +struct cxl_type2_saved_state { +/* DVSEC registers */ +u16 dvsec_ctrl; +u16 dvsec_ctrl2; +}; + +int cxl_config_save_state (struct pci_dev *pdev, +struct cxl_type2_saved_state *state); +int cxl_config_restore_state (struct pci_dev *pdev, +const struct cxl_type2_saved_state *state); +/* * See section 8.1 Configuration Space Registers in the CXL 2.0 * Specification. Names are taken straight from the specification with "CXL" and
@@ -23,6 +35,7 @@#define CXL_DVSEC_CXL_RST_MEM_CLR_CAPABLE BIT(11)#define CXL_DVSEC_CTRL_OFFSET 0xC#define CXL_DVSEC_MEM_ENABLE BIT(2) +#define CXL_DVSEC_CTRL_RWL_MASK 0x5FED#define CXL_DVSEC_CTRL2_OFFSET 0x10#define CXL_DVSEC_DISABLE_CACHING BIT(0)#define CXL_DVSEC_INIT_CACHE_WBI BIT(1)
@@ -32,6 +45,8 @@#define CXL_DVSEC_CACHE_INVALID BIT(0)#define CXL_DVSEC_CXL_RST_COMPLETE BIT(1)#define CXL_DVSEC_CXL_RESET_ERR BIT(2) +#define CXL_DVSEC_LOCK_OFFSET 0x14 +#define CXL_DVSEC_LOCK_CONFIG_LOCK BIT(0)#define CXL_DVSEC_RANGE_SIZE_HIGH (i) (0x18 + ((i) * 0x10))#define CXL_DVSEC_RANGE_SIZE_LOW (i) (0x1C + ((i) * 0x10))#define CXL_DVSEC_MEM_INFO_VALID BIT(0)
--
2.34.1
Re: [PATCH v4 08/10] cxl: add DVSEC config save/restore
Quoted 6 lines · On Tue, 20 Jan 2026 22:26:08 +0000 [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Save and restore CXL DVSEC control registers across reset with
CONFIG_LOCKhandling so RWL fields are preserved when locked. This maintains device policy and capability state acrosscxl_resetwhile avoiding writes to locked fields.
Add some more info here on what is protected by the lock and preserved across reset. This is odd enough I think that detail is needed. There are other RWL fields outside those registers you are restoring here. e.g. the range registers, various things in extended meta data,
Quoted 21 lines
Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/pci.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++ include/cxl/pci.h | 15 +++++++ 2 files changed, 122 insertions(+)drivers/cxl/pci.c +6
-0@@ -1154,6 +1154,113 @@ static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data)return 0 ; } +/* +* CXL DVSEC register save/restore +*/ +static int cxl_save_dvsec_state (struct pci_dev *pdev, +struct cxl_type2_saved_state *state,int dvsec) +{
drivers/cxl/pci.c static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data) · line 1172
Quoted 10 lines
drivers/cxl/pci.c +10
-0+int rc; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CTRL_OFFSET , + &state->dvsec_ctrl); +if (rc) +return rc; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CTRL2_OFFSET , + &state->dvsec_ctrl2); +return rc;
Similar to below. Just combine 2 lines into 1.
Quoted 21 lines
drivers/cxl/pci.c +21
-0+} + +static int cxl_restore_dvsec_state (struct pci_dev *pdev, +const struct cxl_type2_saved_state *state, +int dvsec,bool config_locked) +{ +int rc; +u16 val_to_restore; + +if (config_locked) { +u16 current_val; + + rc =pci_read_config_word (pdev, dvsec +CXL_DVSEC_CTRL_OFFSET , + ¤t_val); +if (rc) +return rc; + + val_to_restore = (current_val &CXL_DVSEC_CTRL_RWL_MASK ) | + (state->dvsec_ctrl & ~CXL_DVSEC_CTRL_RWL_MASK ); + }else { + val_to_restore = state->dvsec_ctrl;
drivers/cxl/pci.c static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data) · line 1203
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ } + + rc =pci_write_config_word (pdev, dvsec +CXL_DVSEC_CTRL_OFFSET , + val_to_restore); +if (rc) +return rc; + + rc =pci_write_config_word (pdev, dvsec +CXL_DVSEC_CTRL2_OFFSET , + state->dvsec_ctrl2); +return rc;
If this isn't expected to get more complex (I haven't checked) then return pci_write_config()
Quoted 9 lines
drivers/cxl/pci.c +9
-0+} + +/** +* cxl_config_save_state - Save CXL configuration state +* @pdev: PCI device +* @state: Structure to store saved state +* +* Saves CXL DVSEC state before reset. +*/
drivers/cxl/pci.c static int cxl_region_flush_host_cpu_caches(struct device *dev, void *data) · line 1222
Quoted 10 lines
drivers/cxl/pci.c +10
-0+int cxl_config_save_state (struct pci_dev *pdev, +struct cxl_type2_saved_state *state) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +int dvsec; + +if (!cxlds || !state) +return -EINVAL ; + +memset (state,0 ,sizeof (*state));
We need to make sure all registers are read, so why is zeroing helpful?
Quoted 8 lines
drivers/cxl/pci.c +8
-0+ + dvsec = cxlds->cxl_dvsec; +if (!dvsec) +return -ENODEV ; + +return cxl_save_dvsec_state (pdev, state, dvsec); +} +EXPORT_SYMBOL_NS_GPL (cxl_config_save_state,"CXL" );
PATCH v4 9/10 PCI: save/restore CXL config around reset
In reply to [email protected]From: Srirangan Madhavan <[email protected]>
Save PCI and CXL configuration state before cxl_reset and restore it after reset completes. This preserves DVSEC state alongside standard PCI state and avoids losing reset-sensitive CXL configuration.
--- drivers/pci/pci.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
drivers/pci/pci.c +12 -0
@@ -4960,6 +4960,7 @@ static int cxl_reset_init(struct pci_dev *dev, u16 dvsec)*/ static int cxl_reset (struct pci_dev *dev,bool probe) { +struct cxl_type2_saved_state cxl_state;u16 dvsec, reg;int rc;
@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)if (probe)return 0 ; +pci_save_state (dev); + rc =cxl_config_save_state (dev, &cxl_state); +if (rc) +pci_warn (dev,"Failed to save CXL config state: %d\n" , rc); +/* * CXL-reset-specific preparation: validate memory offline, * tear down regions, flush device caches.
@@ -5004,10 +5010,16 @@ static int cxl_reset(struct pci_dev *dev, bool probe)if (rc) goto out_cleanup; +pci_restore_state (dev); + rc =cxl_config_restore_state (dev, &cxl_state); +if (rc) +pci_warn (dev,"Failed to restore CXL config state: %d\n" , rc); +cxl_reset_cleanup_device (dev);return 0 ; out_cleanup: +pci_restore_state (dev);cxl_reset_cleanup_device (dev);return rc; }
--
2.34.1
Dave Jiang
maintainerRe: [PATCH v4 09/10] PCI: save/restore CXL config around reset
Quoted 5 lines · On 1/20/26 3:26 PM, [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Save PCI and CXL configuration state before
cxl_resetand restore it after reset completes. This preserves DVSEC state alongside standard PCI state and avoids losing reset-sensitive CXL configuration.
Instead of putting dependency on the cxl core, maybe just move the code in the previous patch here since it's just a few lines of config read/writes. But an explanation of why the DVSEC needs to be preserved on the device regardless of whether a driver is present is needed in the commit log.
DJ
Quoted 50 lines
Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/pci/pci.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)drivers/pci/pci.c +12
-1@@ -4960,6 +4960,7 @@ static int cxl_reset_init(struct pci_dev *dev, u16 dvsec)*/ static int cxl_reset (struct pci_dev *dev,bool probe) { +struct cxl_type2_saved_state cxl_state;u16 dvsec, reg;int rc;@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)if (probe)return 0 ; +pci_save_state (dev); + rc =cxl_config_save_state (dev, &cxl_state); +if (rc) +pci_warn (dev,"Failed to save CXL config state: %d\n" , rc); +/* * CXL-reset-specific preparation: validate memory offline, * tear down regions, flush device caches. @@ -5004,10 +5010,16 @@ static int cxl_reset(struct pci_dev *dev, bool probe)if (rc) goto out_cleanup; +pci_restore_state (dev); + rc =cxl_config_restore_state (dev, &cxl_state); +if (rc) +pci_warn (dev,"Failed to restore CXL config state: %d\n" , rc); +cxl_reset_cleanup_device (dev);return 0 ; out_cleanup: +pci_restore_state (dev);cxl_reset_cleanup_device (dev);return rc; } --2.34.1
Re: [PATCH v4 09/10] PCI: save/restore CXL config around reset
In reply to [email protected]drivers/pci/pci.c static int cxl_reset(struct pci_dev *dev, bool probe) · line 4997
Quoted 10 lines · On Tue, Jan 20, 2026 at 10:26:09PM +0000, [email protected] wrote:
+++
b/drivers/pci/pci.cdrivers/pci/pci.c +5
-0@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)if (probe)return 0 ; +pci_save_state (dev); + rc =cxl_config_save_state (dev, &cxl_state); +if (rc) +pci_warn (dev,"Failed to save CXL config state: %d\n" , rc); +
Hm, shouldn't the call to cxl_config_save_state() be moved to pci_save_state() (and likewise, cxl_config_restore_state() moved to pci_restore_state())?
E.g. when a DPC event occurs, I assume CXL registers need to be restored as well on recovery, right?
Note that since v6.19-rc1, state is saved on enumeration and thus always available for recovery, see a2f1e22390ac.
As a general remark on this series, it seems to have considerable overlap with Terry's work on AER support for CXL, particularly in patch [01/10], as Jonathan has remarked. Please cc Terry on future submissions and coordinate with him on conflicting parts of your patches:
https://lore.kernel.org/all/[email protected]/
Thanks,
Lukas
Re: [PATCH v4 09/10] PCI: save/restore CXL config around reset
Quoted 16 lines · On Thu, 22 Jan 2026 11:01:57 +0100 Lukas Wunner <[email protected]> wrote:
drivers/pci/pci.c static int cxl_reset(struct pci_dev *dev, bool probe) · line 4997
Quoted 10 lines · On Tue, Jan 20, 2026 at 10:26:09PM +0000, [email protected] wrote:
+++
b/drivers/pci/pci.cdrivers/pci/pci.c +5
-0@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)if (probe)return 0 ; +pci_save_state (dev); + rc =cxl_config_save_state (dev, &cxl_state); +if (rc) +pci_warn (dev,"Failed to save CXL config state: %d\n" , rc); +Hm, shouldn't the call to
cxl_config_save_state()be moved topci_save_state()(and likewise,cxl_config_restore_state()moved topci_restore_state())?
Quoted 2 lines
E.g. when a DPC event occurs, I assume CXL registers need to be restored as well on recovery, right?
The CXL spec has some comic language around DPC that basically says "use with care, DPC trigger will bring down physical link, reset devicestate, disrupt CXL.cache and CXL.mem traffic".
or in shorter words
'Good luck'
If a CXL device undergoes DPC high chance you'll either trigger CXL isolation which we aren't handing yet in Linux because we aren't convinced software can really recover form it, or stall a CPU and end up rebooting.
Maybe we'll one day we'll figure this out. Today turn off DPC on CXL ports! :)
J
Quoted 16 lines
Note that since v6.19-rc1, state is saved on enumeration and thus always available for recovery, see
a2f1e22390ac.As a general remark on this series, it seems to have considerable overlap with Terry's work on AER support for CXL, particularly in patch [01/10], as Jonathan has remarked. Please cc Terry on future submissions and coordinate with him on conflicting parts of your patches:
https://lore.kernel.org/all/[email protected]/
Thanks,
Lukas
Re: [PATCH v4 09/10] PCI: save/restore CXL config around reset
Quoted 27 lines · On Thu, 22 Jan 2026 10:47:45 +0000 Jonathan Cameron…
Quoted 16 lines · On Thu, 22 Jan 2026 11:01:57 +0100 Lukas Wunner <[email protected]> wrote:
drivers/pci/pci.c static int cxl_reset(struct pci_dev *dev, bool probe) · line 4997
Quoted 10 lines · On Tue, Jan 20, 2026 at 10:26:09PM +0000, [email protected] wrote:
+++
b/drivers/pci/pci.cdrivers/pci/pci.c +5
-0@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)if (probe)return 0 ; +pci_save_state (dev); + rc =cxl_config_save_state (dev, &cxl_state); +if (rc) +pci_warn (dev,"Failed to save CXL config state: %d\n" , rc); +Hm, shouldn't the call to
cxl_config_save_state()be moved topci_save_state()(and likewise,cxl_config_restore_state()moved topci_restore_state())?Quoted 2 lines
E.g. when a DPC event occurs, I assume CXL registers need to be restored as well on recovery, right?
The CXL spec has some comic language around DPC that basically says "use with care, DPC trigger will bring down physical link, reset devicestate, disrupt CXL.cache and CXL.mem traffic".
or in shorter words
'Good luck'
Quoted 5 lines
If a CXL device undergoes DPC high chance you'll either trigger CXL isolation which we aren't handing yet in Linux because we aren't convinced software can really recover form it, or stall a CPU and end up rebooting.
Maybe we'll one day we'll figure this out. Today turn off DPC on CXL ports! :)
Even if we hand-wave that DPC isn't an issue, save/restore of the PCI state happens at a higher level for every other PCI reset method and we're creating inconsistency here.
PCI-core includes interfaces for saving PCI state, offloading PCI state as an opaque blob, reloading, and restoring that state, and performing resets without saving and restoring state. This has a couple users, including vfio.
If we want similar behavior for CXL type2 devices for a future vfio use case, we shouldn't create unnecessary differentiation here with saving the CXL state separately and making the reset method behave differently. Thanks,
Alex
Re: [PATCH v4 09/10] PCI: save/restore CXL config around reset
Quoted 45 lines · On Mon, 26 Jan 2026 15:34:35 -0700 Alex Williamson <[email protected]> wrote:
Quoted 27 lines · On Thu, 22 Jan 2026 10:47:45 +0000 Jonathan Cameron…
Quoted 16 lines · On Thu, 22 Jan 2026 11:01:57 +0100 Lukas Wunner <[email protected]> wrote:
drivers/pci/pci.c static int cxl_reset(struct pci_dev *dev, bool probe) · line 4997
Quoted 10 lines · On Tue, Jan 20, 2026 at 10:26:09PM +0000, [email protected] wrote:
+++
b/drivers/pci/pci.cdrivers/pci/pci.c +5
-0@@ -4989,6 +4990,11 @@ static int cxl_reset(struct pci_dev *dev, bool probe)if (probe)return 0 ; +pci_save_state (dev); + rc =cxl_config_save_state (dev, &cxl_state); +if (rc) +pci_warn (dev,"Failed to save CXL config state: %d\n" , rc); +Hm, shouldn't the call to
cxl_config_save_state()be moved topci_save_state()(and likewise,cxl_config_restore_state()moved topci_restore_state())?Quoted 2 lines
E.g. when a DPC event occurs, I assume CXL registers need to be restored as well on recovery, right?
The CXL spec has some comic language around DPC that basically says "use with care, DPC trigger will bring down physical link, reset devicestate, disrupt CXL.cache and CXL.mem traffic".
or in shorter words
'Good luck'Quoted 5 lines
If a CXL device undergoes DPC high chance you'll either trigger CXL isolation which we aren't handing yet in Linux because we aren't convinced software can really recover form it, or stall a CPU and end up rebooting.
Maybe we'll one day we'll figure this out. Today turn off DPC on CXL ports! :)
Even if we hand-wave that DPC isn't an issue, save/restore of the PCI state happens at a higher level for every other PCI reset method and we're creating inconsistency here.
PCI-core includes interfaces for saving PCI state, offloading PCI state as an opaque blob, reloading, and restoring that state, and performing resets without saving and restoring state. This has a couple users, including vfio.
Quoted 5 lines
If we want similar behavior for CXL type2 devices for a future vfio use case, we shouldn't create unnecessary differentiation here with saving the CXL state separately and making the reset method behave differently. Thanks,
I'm a bit concerned that, unlike PCI where no traffic flows after reset and restore of basic PCIe stuff, for CXL once you've put the decoders etc back in place, CXL.mem traffic can happen autonomously. It's cacheable and physical address prefetchers on the CPU side may be able wander into it more or less randomly, whether there are page tables yet or not.
This is somewhat similar to PCI devices misbehaving if you enable bus mastering without ensuring they are in a clean state (just in the other direction).
So I'm not sure how safe it is to restore the generic CXL state with out the driver taking control.
I don't think there are tight enough guarantees that devices should be able to survive this if their drivers haven't managed the setup of CXL.mem carefully as they did during driver bind etc. Maybe they had to load a firmware first before there was anything behind a CXL protocol front end.
The drivers can't stop CXL.mem in a prepare reset callback prior to saving state as it may be RWL by an annoying BIOS.
Maybe I'm overly paranoid and all device manufacturers are sensible. Or I missed some spec text that says devices should politely handle traffic turning up before they are ready. If they implement the memory ready checks then we may be fine as hopefully Media Status == Ready doesn't happen until it's safe to enable access (though the spec doesn't actually say that is sufficient that I can find).
I need to do some more digging and maybe a spot of prototyping. Also more than plausible I'm missing a nugget of code in here that makes this all safe.
Jonathan
Quoted 1 line
Alex
PATCH v4 10/10 cxl: add HDM decoder and IDE save/restore
In reply to [email protected]From: Srirangan Madhavan <[email protected]>
Extend state save/restore to HDM decoder and IDE registers for Type 2 devices. The HDM/IDE blocks are located via the component register map, then preserved across reset to retain decoder configuration and IDE policy. This avoids losing HDM/IDE programming when cxl_reset is issued.
--- drivers/cxl/core/regs.c | 7 ++ drivers/cxl/cxl.h | 4 ++ drivers/cxl/pci.c | 153 ++++++++++++++++++++++++++++++++++++++-- include/cxl/pci.h | 43 +++++++++++ 4 files changed, 201 insertions(+), 6 deletions(-)
drivers/cxl/core/regs.c +7 -0
@@ -93,6 +93,12 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, length =CXL_RAS_CAPABILITY_LENGTH ; rmap = &map->ras;break ; +case CXL_CM_CAP_CAP_ID_IDE : +dev_dbg (dev,"found IDE capability (0x%x)\n" , + offset); + length =CXL_IDE_CAPABILITY_LENGTH ; + rmap = &map->ide; +break ;default :dev_dbg (dev,"Unknown CM cap ID: %d (0x%x)\n" , cap_id, offset);
@@ -212,6 +218,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map, } mapinfo[] = { { &map->component_map.hdm_decoder, ®s->hdm_decoder }, { &map->component_map.ras, ®s->ras }, + { &map->component_map.ide, ®s->ide }, }; int i;
drivers/cxl/cxl.h +4 -0
@@ -39,8 +39,10 @@ extern const struct nvdimm_security_ops *cxl_security_ops;#define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)#define CXL_CM_CAP_CAP_ID_RAS 0x2 +#define CXL_CM_CAP_CAP_ID_IDE 0x4#define CXL_CM_CAP_CAP_ID_HDM 0x5#define CXL_CM_CAP_CAP_HDM_VERSION 1 +#define CXL_IDE_CAPABILITY_LENGTH 0x20/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */ #define CXL_HDM_DECODER_CAP_OFFSET 0x0
@@ -214,6 +216,7 @@ struct cxl_regs {struct_group_tagged (cxl_component_regs, component, void __iomem *hdm_decoder;void __iomem *ras; +void __iomem *ide; );/* * Common set of CXL Device register block base pointers
@@ -256,6 +259,7 @@ struct cxl_reg_map {struct cxl_component_reg_map {struct cxl_reg_map hdm_decoder;struct cxl_reg_map ras; +struct cxl_reg_map ide; };struct cxl_device_reg_map {
drivers/cxl/pci.c +147 -6
193 lines not on this page · raw message
include/cxl/pci.h +43 -0
60 lines not on this page · raw message
--
2.34.1
Re: [PATCH v4 10/10] cxl: add HDM decoder and IDE save/restore
Quoted 8 lines · On Tue, 20 Jan 2026 22:26:10 +0000 [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Extend state save/restore to HDM decoder and IDE registers for Type 2 devices. The HDM/IDE blocks are located via the component register map, then preserved across reset to retain decoder configuration and IDE policy. This avoids losing HDM/IDE programming when
cxl_resetis issued.Signed-off-by Srirangan Madhavan <[email protected]>
Split the two up. One patch for HDM, one for IDE.
Been a while since I read the IDE stuff.
Isn't reset going to trip up the encryption to the extent that we need to redo the key exchange etc?
Quoted 12 lines
--- drivers/cxl/core/regs.c | 7 ++ drivers/cxl/cxl.h | 4 ++ drivers/cxl/pci.c | 153 ++++++++++++++++++++++++++++++++++++++-- include/cxl/pci.h | 43 +++++++++++ 4 files changed, 201 insertions(+), 6 deletions(-)drivers/cxl/core/regs.c +0
-0
drivers/cxl/core/regs.c void cxl_probe_component_regs(struct device *dev, void __iomem *base, · line 98
Quoted 6 lines
drivers/cxl/core/regs.c +3
-0@@ -93,6 +93,12 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, length =CXL_RAS_CAPABILITY_LENGTH ; rmap = &map->ras;break ; +case CXL_CM_CAP_CAP_ID_IDE : +dev_dbg (dev,"found IDE capability (0x%x)\n" , + offset);
Trivial: Fits on one line under 80 chars.
Quoted 14 lines
drivers/cxl/core/regs.c +3
-0+ length =CXL_IDE_CAPABILITY_LENGTH ; + rmap = &map->ide; +break ;drivers/cxl/pci.c +5
-0+/* +* CXL HDM Decoder register save/restore +*/ +static int cxl_save_hdm_state (struct cxl_dev_state *cxlds, +struct cxl_type2_saved_state *state)
drivers/cxl/pci.c static int cxl_restore_dvsec_state(struct pci_dev *pdev, · line 1221
Quoted 10 lines
drivers/cxl/pci.c +10
-0+{ +void __iomem *hdm = cxlds->regs.hdm_decoder; +u32 cap, ctrl; +int i, count; + +if (!hdm) +return 0 ; + + cap =readl (hdm +CXL_HDM_DECODER_CAP_OFFSET ); + count = cap &CXL_HDM_DECODER_COUNT_MASK ;
Use FIELD_GET() for this so we don't need to go check the definition to be sure there isn't a shift.
Quoted 13 lines
drivers/cxl/pci.c +13
-0+ count =min (count,CXL_MAX_DECODERS ); + + state->hdm_decoder_count = count; + state->hdm_global_ctrl =readl (hdm +CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET ); + +for (i =0 ; i < count; i++) { +struct cxl_hdm_decoder_state *d = &state->decoders[i]; +u32 base_low, base_high, size_low, size_high; +u32 dpa_skip_low, dpa_skip_high; + + base_low =readl (hdm +CXL_HDM_DECODER_BASE_LOW (i)); + base_high =readl (hdm +CXL_HDM_DECODER_BASE_HIGH (i)); + size_low =readl (hdm +CXL_HDM_DECODER_SIZE_LOW (i));
drivers/cxl/pci.c static int cxl_restore_dvsec_state(struct pci_dev *pdev, · line 1244
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ size_high =readl (hdm +CXL_HDM_DECODER_SIZE_HIGH (i)); + ctrl =readl (hdm +CXL_HDM_DECODER_CTRL (i)); + dpa_skip_low =readl (hdm +CXL_HDM_DECODER_DPA_SKIP_LOW (i)); + dpa_skip_high =readl (hdm +CXL_HDM_DECODER_DPA_SKIP_HIGH (i)); + + d->base = ((u64 )base_high <<32 ) | base_low; + d->size = ((u64 )size_high <<32 ) | size_low; + d->ctrl = ctrl; + d->dpa_skip = ((u64 )dpa_skip_high <<32 ) | dpa_skip_low; + d->enabled = !!(ctrl &CXL_HDM_DECODER_ENABLE );
I'm lazy. Why not just stash the register values as a raw block and put them back? Do we want to put them back if they aren't locked? We ripped down the region so I'd be kind of expecting the type 2 driver for the specific hardware to need to put this stuff back as part of it's own reinit.
Quoted 4 lines
drivers/cxl/pci.c +4
-0+ } + +return 0 ; +}
Dave Jiang
maintainerRe: [PATCH v4 10/10] cxl: add HDM decoder and IDE save/restore
In reply to [email protected]Quoted 6 lines · On 1/20/26 3:26 PM, [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Extend state save/restore to HDM decoder and IDE registers for Type 2 devices. The HDM/IDE blocks are located via the component register map, then preserved across reset to retain decoder configuration and IDE policy. This avoids losing HDM/IDE programming when
cxl_resetis issued.
So the current implementation tries to save DVSEC, HDM decoders, and IDE registers (MMIO registers). This is not standard device reset and seems more usage specific. Given that is the case and the fact that the implementation keeps trying to pull very CXL specific bits into PCI core, maybe going through the PCI reset path isn't the right place? The whole thing would be significantly simplified if we add a 'reset' sysfs attribute to memdev and just do it from there? Thoughts? That would prevent CXL symbols keep being dragged into PCI and keep everything local for this specific usage. Probably should also add some text on the reasoning of saving of these MMIO registers.
DJ
Quoted 336 lines
Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/core/regs.c | 7 ++ drivers/cxl/cxl.h | 4 ++ drivers/cxl/pci.c | 153 ++++++++++++++++++++++++++++++++++++++-- include/cxl/pci.h | 43 +++++++++++ 4 files changed, 201 insertions(+), 6 deletions(-)drivers/cxl/core/regs.c +7
-0@@ -93,6 +93,12 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, length =CXL_RAS_CAPABILITY_LENGTH ; rmap = &map->ras;break ; +case CXL_CM_CAP_CAP_ID_IDE : +dev_dbg (dev,"found IDE capability (0x%x)\n" , + offset); + length =CXL_IDE_CAPABILITY_LENGTH ; + rmap = &map->ide; +break ;default :dev_dbg (dev,"Unknown CM cap ID: %d (0x%x)\n" , cap_id, offset);@@ -212,6 +218,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map, } mapinfo[] = { { &map->component_map.hdm_decoder, ®s->hdm_decoder }, { &map->component_map.ras, ®s->ras }, + { &map->component_map.ide, ®s->ide }, }; int i;drivers/cxl/cxl.h +4
-0@@ -39,8 +39,10 @@ extern const struct nvdimm_security_ops *cxl_security_ops;#define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)#define CXL_CM_CAP_CAP_ID_RAS 0x2 +#define CXL_CM_CAP_CAP_ID_IDE 0x4#define CXL_CM_CAP_CAP_ID_HDM 0x5#define CXL_CM_CAP_CAP_HDM_VERSION 1 +#define CXL_IDE_CAPABILITY_LENGTH 0x20/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */ #define CXL_HDM_DECODER_CAP_OFFSET 0x0@@ -214,6 +216,7 @@ struct cxl_regs {struct_group_tagged (cxl_component_regs, component, void __iomem *hdm_decoder;void __iomem *ras; +void __iomem *ide; );/* * Common set of CXL Device register block base pointers @@ -256,6 +259,7 @@ struct cxl_reg_map {struct cxl_component_reg_map {struct cxl_reg_map hdm_decoder;struct cxl_reg_map ras; +struct cxl_reg_map ide; };struct cxl_device_reg_map {drivers/cxl/pci.c +147
-6@@ -956,7 +956,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)dev_dbg (&pdev->dev,"RAS registers not found\n" ); rc =cxl_map_component_regs (&cxlds->reg_map, &cxlds->regs.component, -BIT (CXL_CM_CAP_CAP_ID_RAS )); +BIT (CXL_CM_CAP_CAP_ID_RAS ) | +BIT (CXL_CM_CAP_CAP_ID_IDE ));if (rc)dev_dbg (&pdev->dev,"Failed to map RAS capability.\n" );@@ -1203,18 +1204,126 @@ static int cxl_restore_dvsec_state(struct pci_dev *pdev,return rc; } +/* +* CXL HDM Decoder register save/restore +*/ +static int cxl_save_hdm_state (struct cxl_dev_state *cxlds, +struct cxl_type2_saved_state *state) +{ +void __iomem *hdm = cxlds->regs.hdm_decoder; +u32 cap, ctrl; +int i, count; + +if (!hdm) +return 0 ; + + cap =readl (hdm +CXL_HDM_DECODER_CAP_OFFSET ); + count = cap &CXL_HDM_DECODER_COUNT_MASK ; + count =min (count,CXL_MAX_DECODERS ); + + state->hdm_decoder_count = count; + state->hdm_global_ctrl =readl (hdm +CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET ); + +for (i =0 ; i < count; i++) { +struct cxl_hdm_decoder_state *d = &state->decoders[i]; +u32 base_low, base_high, size_low, size_high; +u32 dpa_skip_low, dpa_skip_high; + + base_low =readl (hdm +CXL_HDM_DECODER_BASE_LOW (i)); + base_high =readl (hdm +CXL_HDM_DECODER_BASE_HIGH (i)); + size_low =readl (hdm +CXL_HDM_DECODER_SIZE_LOW (i)); + size_high =readl (hdm +CXL_HDM_DECODER_SIZE_HIGH (i)); + ctrl =readl (hdm +CXL_HDM_DECODER_CTRL (i)); + dpa_skip_low =readl (hdm +CXL_HDM_DECODER_DPA_SKIP_LOW (i)); + dpa_skip_high =readl (hdm +CXL_HDM_DECODER_DPA_SKIP_HIGH (i)); + + d->base = ((u64 )base_high <<32 ) | base_low; + d->size = ((u64 )size_high <<32 ) | size_low; + d->ctrl = ctrl; + d->dpa_skip = ((u64 )dpa_skip_high <<32 ) | dpa_skip_low; + d->enabled = !!(ctrl &CXL_HDM_DECODER_ENABLE ); + } + +return 0 ; +} + +static int cxl_restore_hdm_state (struct cxl_dev_state *cxlds, +const struct cxl_type2_saved_state *state) +{ +void __iomem *hdm = cxlds->regs.hdm_decoder; +int i; + +if (!hdm || state->hdm_decoder_count ==0 ) +return 0 ; + +writel (state->hdm_global_ctrl, hdm +CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET ); + +for (i =0 ; i < state->hdm_decoder_count; i++) { +const struct cxl_hdm_decoder_state *d = &state->decoders[i]; + +writel ((u32 )d->base, hdm +CXL_HDM_DECODER_BASE_LOW (i)); +writel ((u32)(d->base >>32 ), hdm +CXL_HDM_DECODER_BASE_HIGH (i)); +writel ((u32 )d->size, hdm +CXL_HDM_DECODER_SIZE_LOW (i)); +writel ((u32)(d->size >>32 ), hdm +CXL_HDM_DECODER_SIZE_HIGH (i)); +writel (d->ctrl, hdm +CXL_HDM_DECODER_CTRL (i)); +writel ((u32 )d->dpa_skip, hdm +CXL_HDM_DECODER_DPA_SKIP_LOW (i)); +writel ((u32)(d->dpa_skip >>32 ), hdm +CXL_HDM_DECODER_DPA_SKIP_HIGH (i)); + } + +return 0 ; +} + +/* +* CXL IDE register save/restore +*/ +static int cxl_save_ide_state (struct cxl_dev_state *cxlds, +struct cxl_type2_saved_state *state) +{ +void __iomem *ide = cxlds->regs.ide; +u32 cap; + +if (!ide) +return 0 ; + + cap =readl (ide +CXL_IDE_CAP_OFFSET ); +if (!(cap &CXL_IDE_CAP_CAPABLE )) +return 0 ; + + state->ide_cap = cap; + state->ide_ctrl =readl (ide +CXL_IDE_CTRL_OFFSET ); + state->ide_key_refresh_time =readl (ide +CXL_IDE_KEY_REFRESH_TIME_CTRL_OFFSET ); + state->ide_truncation_delay =readl (ide +CXL_IDE_TRUNCATION_DELAY_CTRL_OFFSET ); + +return 0 ; +} + +static int cxl_restore_ide_state (struct cxl_dev_state *cxlds, +const struct cxl_type2_saved_state *state) +{ +void __iomem *ide = cxlds->regs.ide; + +if (!ide || !(state->ide_cap &CXL_IDE_CAP_CAPABLE )) +return 0 ; + +writel (state->ide_ctrl, ide +CXL_IDE_CTRL_OFFSET ); +writel (state->ide_key_refresh_time, ide +CXL_IDE_KEY_REFRESH_TIME_CTRL_OFFSET ); +writel (state->ide_truncation_delay, ide +CXL_IDE_TRUNCATION_DELAY_CTRL_OFFSET ); + +return 0 ; +} +/** * cxl_config_save_state - Save CXL configuration state * @pdev: PCI device * @state: Structure to store saved state * -* Saves CXL DVSEC state before reset. +* Saves CXL DVSEC, HDM decoder, and IDE state before reset. */ int cxl_config_save_state (struct pci_dev *pdev,struct cxl_type2_saved_state *state) {struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); -int dvsec; +int rc, dvsec;if (!cxlds || !state)return -EINVAL ;@@ -1225,7 +1334,23 @@ int cxl_config_save_state(struct pci_dev *pdev,if (!dvsec)return -ENODEV ; -return cxl_save_dvsec_state (pdev, state, dvsec); + rc =cxl_save_dvsec_state (pdev, state, dvsec); +if (rc) +return rc; + +if (cxlds->regs.hdm_decoder) { + rc =cxl_save_hdm_state (cxlds, state); +if (rc) +pci_warn (pdev,"Failed to save HDM state: %d\n" , rc); + } + +if (cxlds->regs.ide) { + rc =cxl_save_ide_state (cxlds, state); +if (rc) +pci_warn (pdev,"Failed to save IDE state: %d\n" , rc); + } + +return 0 ; }EXPORT_SYMBOL_NS_GPL (cxl_config_save_state,"CXL" );@@ -1234,7 +1359,7 @@ EXPORT_SYMBOL_NS_GPL(cxl_config_save_state, "CXL");* @pdev: PCI device * @state: Previously saved state * -* Restores CXL DVSEC state after reset. +* Restores CXL DVSEC, HDM decoder, and IDE state after reset. */ int cxl_config_restore_state (struct pci_dev *pdev,const struct cxl_type2_saved_state *state)@@ -1257,7 +1382,23 @@ int cxl_config_restore_state(struct pci_dev *pdev, config_locked = !!(lock_reg &CXL_DVSEC_LOCK_CONFIG_LOCK ); -return cxl_restore_dvsec_state (pdev, state, dvsec, config_locked); + rc =cxl_restore_dvsec_state (pdev, state, dvsec, config_locked); +if (rc) +return rc; + +if (cxlds->regs.hdm_decoder && state->hdm_decoder_count >0 ) { + rc =cxl_restore_hdm_state (cxlds, state); +if (rc) +pci_warn (pdev,"Failed to restore HDM state: %d\n" , rc); + } + +if (cxlds->regs.ide && (state->ide_cap &CXL_IDE_CAP_CAPABLE )) { + rc =cxl_restore_ide_state (cxlds, state); +if (rc) +pci_warn (pdev,"Failed to restore IDE state: %d\n" , rc); + } + +return 0 ; }EXPORT_SYMBOL_NS_GPL (cxl_config_restore_state,"CXL" );include/cxl/pci.h +43
-1@@ -4,11 +4,33 @@#ifndef __CXL_ACCEL_PCI_H#define __CXL_ACCEL_PCI_H +/* HDM Decoder state for save/restore */ +struct cxl_hdm_decoder_state { +u64 base; +u64 size; +u32 ctrl; +u64 dpa_skip; +bool enabled; +}; + +#define CXL_MAX_DECODERS 10 +/* CXL Type 2 device state for save/restore across reset */ struct cxl_type2_saved_state {/* DVSEC registers */ u16 dvsec_ctrl;u16 dvsec_ctrl2; + +/* HDM Decoder registers */ +u32 hdm_decoder_count; +u32 hdm_global_ctrl; +struct cxl_hdm_decoder_state decoders[CXL_MAX_DECODERS ]; + +/* IDE registers */ +u32 ide_cap; +u32 ide_ctrl; +u32 ide_key_refresh_time; +u32 ide_truncation_delay; }; int cxl_config_save_state(struct pci_dev *pdev,@@ -58,6 +80,27 @@ int cxl_config_restore_state(struct pci_dev *pdev,#define CXL_DVSEC_RANGE_MAX 2 +/* CXL HDM Decoder Capability Structure (Section 8.2.4.20) */ +#define CXL_HDM_DECODER_CAP_OFFSET 0x0 +#define CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0) +#define CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET 0x4 +#define CXL_HDM_DECODER_ENABLE BIT(1) +/* CXL HDM Decoder n registers (Offset 20h*n + base) */ +#define CXL_HDM_DECODER_BASE_LOW (n) (0x10 + ((n) * 0x20)) +#define CXL_HDM_DECODER_BASE_HIGH (n) (0x14 + ((n) * 0x20)) +#define CXL_HDM_DECODER_SIZE_LOW (n) (0x18 + ((n) * 0x20)) +#define CXL_HDM_DECODER_SIZE_HIGH (n) (0x1C + ((n) * 0x20)) +#define CXL_HDM_DECODER_CTRL (n) (0x20 + ((n) * 0x20)) +#define CXL_HDM_DECODER_DPA_SKIP_LOW (n) (0x24 + ((n) * 0x20)) +#define CXL_HDM_DECODER_DPA_SKIP_HIGH (n) (0x28 + ((n) * 0x20)) + +/* CXL IDE Capability Structure (Section 8.2.4.22) */ +#define CXL_IDE_CAP_OFFSET 0x00 +#define CXL_IDE_CAP_CAPABLE BIT(0) +#define CXL_IDE_CTRL_OFFSET 0x04 +#define CXL_IDE_KEY_REFRESH_TIME_CTRL_OFFSET 0x18 +#define CXL_IDE_TRUNCATION_DELAY_CTRL_OFFSET 0x1C +/* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */ #define CXL_DVSEC_FUNCTION_MAP 2 --2.34.1
Alison Schofield
maintainerQuoted 7 lines · On Tue, Jan 20, 2026 at 10:26:00PM +0000, [email protected] wrote:
From: Srirangan Madhavan <[email protected]>
Hi folks!
This patch series introduces support for the CXL Reset method for CXL devices, implementing the reset procedure outlined in CXL Spec [1] v3.2, Sections 9.6 and 9.7.
Hi Srirangan,
Following-up on the base commit for this patch. For the cxl subsystem, a patchset today is expected to be based on 6.19-rc4 (or rc5 or rc6). We get to that because our cxl/next is based on rc4. We ask folks not to base on cxl/next because we want to see conflicts between patchsets.
I like to use the base commit option to automatically append it to the cover letter, but you can just state it in the change log if your method of sending patches doesn't support that.
With this set, there is something else that you are depending upon. Please call that out and point to the commits in some way so we can build on 6.19-rc4 plus those commits plus this patchset.
Thanks!
Alison
Quoted 115 lines
v4 changes:
- Fix CXL reset capability check parentheses warning
- Gate CXL reset path onCONFIG_CXL_PCIreachabilityv3 changes:
- Restrict CXL reset to Type 2 devices only
- Add host and device cache flushing for
* all sibling functions on multi-function devices
* all sibling devices in a given region
- Add region teardown and memory online detection before reset
- Add configuration state save/restore (DVSEC, HDM, IDE)
- Split the series by subsystem and functional blocksv2 changes:
- De-duplicate CXL DVSEC register defines underinclude/cxl/pci.h
- Fix style-related issuesv1 changes:
- Added cover letter and dropped the RFCThe RFC patches can be found here [2]
v2 patches can be found here [3]Motivation:
-----------
This change is broadly useful for reasons including but not limited to the following:- As support for Type 2 devices [4] is being introduced, more devices will require finer-grained reset mechanisms beyond bus-wide reset methods.
- FLR does not affect CXL.cache or CXL.mem protocols, making CXL Reset the preferred method in some cases.
- The CXL spec (Sections 7.2.3 Binding and Unbinding, 9.5 FLR) highlights use cases like function rebinding and error recovery, where CXL Reset is explicitly mentioned.
Change Description:
-------------------Patch 1: Move CXL DVSEC defines to the CXL PCI header
- Consolidate DVSEC register definitions underinclude/cxl/pci.hPatch 2: Switch PCI CXL port DVSEC defines
- Use the shared CXL PCI header in the PCI corePatch 3: Add Type 2 helper and reset DVSEC bits
- Add helper to identify Type 2 devices
- Define DVSEC reset/cache control bitsPatch 4: Add the CXL reset method in the PCI core
- Implementcxl_reset()method with capability checks and reset sequence
- Restrict to Type 2 devicesPatch 5: Add reset preparation and region teardown
- Implement region validation and teardown before reset
- Add device cache flush for all sibling devices in a given regionPatch 6: Wire CXL reset prepare/cleanup in PCI
- Call CXL reset prepare/cleanup around the core reset flowPatch 7: Add host CPU cache flush and multi-function support
- Add host CPU cache flush (x86: wbinvd, arm64:dcache_clean_inval_poc)
- Add device cache flush for all sibling functions on multi-function devicesPatch 8: Add DVSEC configuration state save/restore
- Save/restore DVSEC registers (DEVCTL, DEVCTL2) withCONFIG_LOCKhandlingPatch 9: Save/restore CXL config around reset
- Save PCI and CXL config before reset and restore afterwardsPatch 10: Add HDM decoder and IDE state save/restore
- Save/restore HDM decoder and IDE register stateThe reset sequence: validate device type, check memory offline, tear down regions, flush host CPU caches, flush device caches (all functions), save config state, initiate reset, wait for completion, restore config state.
Command line to test the CXL reset on a capable device:
echocxl_reset> /sys/bus/pci/devices/<pci_device>/reset_method echo 1 > /sys/bus/pci/devices/<pci_device>/reset[1] https://computeexpresslink.org/cxl-specification/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/linux-cxl/[email protected]/Srirangan Madhavan (10):
[PATCH v4 1/10] cxl: move DVSEC defines to cxl pci header [PATCH v4 2/10] PCI: switch CXL port DVSEC defines [PATCH v4 3/10] cxl: add type 2 helper and reset DVSEC bits [PATCH v4 4/10] PCI: add CXL reset method
[PATCH v4 5/10] cxl: add reset prepare and region teardown [PATCH v4 6/10] PCI: wire CXL reset prepare/cleanup [PATCH v4 7/10] cxl: add host cache flush and multi-function reset [PATCH v4 8/10] cxl: add DVSEC config save/restore [PATCH v4 9/10] PCI: save/restore CXL config around reset [PATCH v4 10/10] cxl: add HDM decoder and IDE save/restoredrivers/cxl/core/pci.c | 1 + drivers/cxl/core/regs.c | 8 + drivers/cxl/cxl.h | 4 + drivers/cxl/cxlpci.h | 53 --- drivers/cxl/pci.c | 621 +++++++++++++++++++++++++++++++++- drivers/pci/pci.c | 150 +++++++- include/cxl/pci.h | 134 ++++++++ include/linux/pci.h | 21 +- include/uapi/linux/pci_regs.h | 5 - 9 files changed, 929 insertions(+), 68 deletions(-) create mode 100644 include/cxl/pci.h--
2.34.1
Quoted 12 lines · On Tue, Jan 20, 2026 at 10:26:00PM +0000, [email protected] wrote:
...
[PATCH v4 1/10] cxl: move DVSEC defines to cxl pci header [PATCH v4 2/10] PCI: switch CXL port DVSEC defines [PATCH v4 3/10] cxl: add type 2 helper and reset DVSEC bits [PATCH v4 4/10] PCI: add CXL reset method
[PATCH v4 5/10] cxl: add reset prepare and region teardown [PATCH v4 6/10] PCI: wire CXL reset prepare/cleanup [PATCH v4 7/10] cxl: add host cache flush and multi-function reset [PATCH v4 8/10] cxl: add DVSEC config save/restore [PATCH v4 9/10] PCI: save/restore CXL config around reset [PATCH v4 10/10] cxl: add HDM decoder and IDE save/restore
Please run "git log --online" on the files you change and match the style of subject lines. In drivers/pci/, the subject lines start with capital letters, e.g.,
PCI: Switch CXL port ...
PCI: Add CXL reset method (but please include the function name)
...
Quoted 10 lines · On Tue, 20 Jan 2026 22:26:00 +0000 <[email protected]> wrote:
Motivation:
-----------
This change is broadly useful for reasons including but not limited to the following:- As support for Type 2 devices [4] is being introduced, more devices will require finer-grained reset mechanisms beyond bus-wide reset methods.
- FLR does not affect CXL.cache or CXL.mem protocols, making CXL Reset the preferred method in some cases.
This proposal adds cxl_reset to the pci_reset_fn_methods[] array, so while the above suggests there are some use cases that would prefer CXL reset, we're interleaving the capability into the default function for PCI function scoped reset.
I'm concerned about use cases like vfio-pci emulation of FLR, where we currently call through the pci_reset_function() interfaces to take advantage of device specific quirks. Such a use case (guest directed FLR) might reasonably expect the scope of the reset is limited to CXL.io without affecting CXL.cache or CXL.mem, therefore it's not clear that pci_reset_function() necessarily has the correct scope.
OTOH, we also expose a reset ioctl through vfio-pci that could reasonably reset the full state of the device, PCI and CXL. cxl_reset would be appropriate here, however cxl_reset is prioritized below FLR reset in the default reset_methods array, so pci_reset_function() doesn't have the correct scope here either.
Is it really appropriate to consider cxl_reset as just another PCI reset method given the scope difference? Does there need to be a way to specify the scope when calling pci_reset_function() for drivers aware of CXL.mem/cache? Thanks,
Alex
Alex Williamson wrote:
[..]
Quoted 4 lines
Is it really appropriate to consider
cxl_resetas just another PCI reset method given the scope difference? Does there need to be a way to specify the scope when callingpci_reset_function()for drivers aware of CXL.mem/cache?
Right, FWIW, I am of the opinion that trying to plumb CXL reset through PCI reset is an awkward fit that will be an ongoing source of pain. For the same reason that CXL protocol error handling is designed to have little do with PCIe protocol error handling, CXL is not PCIe. For example, the goal in the error handling case is to arrive at a point where the PCIe error handling implementation can continue to evolve without inflicting the maintenance burden load of "what about CXL?". Notifications are simply shunted over to the CXL core early in the flow. CXL reset does not need any PCI core entanglements as far as I can see.
Once the protocol error handling series has landed there is a potential to extend that with CXL Reset recovery. However, that needs a clear error model defined as to which resets have a chance of recovering *system* operation when CXL.cache/mem fails. In Terry's series, panic / reboot is the recovery, not reset.
Hi Dan,
________________________________________
From: [email protected] <[email protected]>
Sent: Tuesday, January 27, 2026 11:02 AM
To: Alex Williamson; Srirangan Madhavan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vishal Aslot; Vikram Sethi; Shanker Donthineni; Vidya Sagar; Matt Ochs; Jason Sequeira
Subject: Re: [PATCH v4 0/10] CXL Reset support for Type 2 devices
Quoted 5 lines
Once the protocol error handling series has landed there is a potential to extend that with CXL Reset recovery. However, that needs a clear error model defined as to which resets have a chance of recovering *system* operation when CXL.cache/mem fails. In Terry's series, panic / reboot is the recovery, not reset.
It's not just about CXL protocol error handling though. Type2 Device passthrough will be a common usecase for CXL reset (entire device is assigned to VM) across different VM assignment.
The cache and mem of the device must be cleared via CXL reset for full device passthrough in addition to the PCIe/CXL.IO reset. Another usecase is when the device is reconfigured either in baremetal or in VM for passthrough usecase. Such usecases often require a reset of the device, and it's cache, mem and not a narrow "memregion" reset, so I'm not in favor of exposing it via CXL.mem sysfs entries. IMO, device sysfs attribute reset method is appropriate for type2 devices. Finally, there is the error usecase, which in the common case is as simple as an uncorrected ECC error in the HDM. While not strictly necessary, it is common practice to reset the device in such cases, to recover the bad page/row via PPR on device reset. You want reset of the memory controller as part of the CXL reset here, FLR is not enough.
Regarding scope, and "recoverability", we have significant experience with recovering "pre-CXL" coherent GB200 devices for device errors: memory ECC or other, by killing the application using device memory, unloading driver, resetting the device, and reloading driver. The CXL protocol error series can use CXL reset series, but I don't see that this series needs to wait for protocol error handling to be merged.
Thanks,
Vikram
Quoted 11 lines · Vikram Sethi
Hi Dan,
Quoted 5 lines
Once the protocol error handling series has landed there is a potential to extend that with CXL Reset recovery. However, that needs a clear error model defined as to which resets have a chance of recovering *system* operation when CXL.cache/mem fails. In Terry's series, panic / reboot is the recovery, not reset.
It's not just about CXL protocol error handling though. Type2 Device passthrough will be a common usecase for CXL reset (entire device is assigned to VM) across different VM assignment.
I understand. The point about CXL Protocol Error handling series is that it at least enlightens the PCIe core about the presence of active CXL links.
It is also the case that it much closer to being upstream than this set which still has fundamental questions.
Quoted 7 lines
The cache and mem of the device must be cleared via CXL reset for full device passthrough in addition to the PCIe/CXL.IO reset. Another usecase is when the device is reconfigured either in baremetal or in VM for passthrough usecase. Such usecases often require a reset of the device, and it's cache, mem and not a narrow "memregion" reset, so I'm not in favor of exposing it via CXL.mem sysfs entries. IMO, device sysfs attribute reset method is appropriate for type2 devices.
That case is already handled today with secondary bus reset to completely reset an entire device. That path is problematic for CXL because PCI reset has no idea about how to manage caches or handle memory unplug.
Administrator is responsible for making sure that event is not a surprise memory removal or cache protocol corruption event. That is a level of explosiveness that PCI reset has never needed to consider.
CXL Reset wants to be more surgical than secondary bus reset. The only way to be more surgical is to cooperate with the CXL core that knows whether the explosives have been rendered inert.
Quoted 6 lines
Finally, there is the error usecase, which in the common case is as simple as an uncorrected ECC error in the HDM. While not strictly necessary, it is common practice to reset the device in such cases, to recover the bad page/row via PPR on device reset. You want reset of the memory controller as part of the CXL reset here, FLR is not enough.
??
CXL memory repair is already upstream without CXL Reset, see CONFIG_CXL_EDAC_MEM_REPAIR.
Quoted 6 lines
Regarding scope, and "recoverability", we have significant experience with recovering "pre-CXL" coherent GB200 devices for device errors:
memory ECC or other, by killing the application using device memory, unloading driver, resetting the device, and reloading driver. The CXL protocol error series can use CXL reset series, but I don't see that this series needs to wait for protocol error handling to be merged.
I do want to get to the point where device memory error recovery is a first class citizen. CXL Protocol Error handling just happens to be at the top of the review queue and safe to assume it can be a foundation for further RAS features.
Quoted 48 lines · On Tue, 27 Jan 2026 19:42:58 -0800 [email protected] wrote:
Quoted 11 lines · Vikram Sethi
Hi Dan,
Quoted 5 lines
Once the protocol error handling series has landed there is a potential to extend that with CXL Reset recovery. However, that needs a clear error model defined as to which resets have a chance of recovering *system* operation when CXL.cache/mem fails. In Terry's series, panic / reboot is the recovery, not reset.
It's not just about CXL protocol error handling though. Type2 Device passthrough will be a common usecase for CXL reset (entire device is assigned to VM) across different VM assignment.
I understand. The point about CXL Protocol Error handling series is that it at least enlightens the PCIe core about the presence of active CXL links.
It is also the case that it much closer to being upstream than this set which still has fundamental questions.
Quoted 7 lines
The cache and mem of the device must be cleared via CXL reset for full device passthrough in addition to the PCIe/CXL.IO reset. Another usecase is when the device is reconfigured either in baremetal or in VM for passthrough usecase. Such usecases often require a reset of the device, and it's cache, mem and not a narrow "memregion" reset, so I'm not in favor of exposing it via CXL.mem sysfs entries. IMO, device sysfs attribute reset method is appropriate for type2 devices.
That case is already handled today with secondary bus reset to completely reset an entire device. That path is problematic for CXL because PCI reset has no idea about how to manage caches or handle memory unplug.
Administrator is responsible for making sure that event is not a surprise memory removal or cache protocol corruption event. That is a level of explosiveness that PCI reset has never needed to consider.
CXL Reset wants to be more surgical than secondary bus reset. The only way to be more surgical is to cooperate with the CXL core that knows whether the explosives have been rendered inert.
Quoted 6 lines
Finally, there is the error usecase, which in the common case is as simple as an uncorrected ECC error in the HDM. While not strictly necessary, it is common practice to reset the device in such cases, to recover the bad page/row via PPR on device reset. You want reset of the memory controller as part of the CXL reset here, FLR is not enough.
Quoted 4 lines
??
CXL memory repair is already upstream without CXL Reset, see
CONFIG_CXL_EDAC_MEM_REPAIR.
To actually do it we rely on tear down of all access to the device (if it it's disruptive).
There is new spec stuff covering asking a device to just do it next reset that would fit more closely with this flow. No support yet I think.
Mind you we are talking type 2, so who knows what people will implement. Nice if they used that part of the spec but I doubt we can rely on it!
Jonathan
Quoted 14 lines
Quoted 7 lines
Regarding scope, and "recoverability", we have significant experience with recovering "pre-CXL" coherent GB200 devices for device errors:
memory ECC or other, by killing the application using device memory, unloading driver, resetting the device, and reloading driver. The CXL protocol error series can use CXL reset series, but I don't see that this series needs to wait for protocol error handling to be merged.I do want to get to the point where device memory error recovery is a first class citizen. CXL Protocol Error handling just happens to be at the top of the review queue and safe to assume it can be a foundation for further RAS features.