PCI/CXL: Add CXL reset support for Type 2 devices
Subsystems (3)
v12 cover PCI/CXL: Add CXL reset support for Type 2 devices
Hi folks!
This series adds CXL Reset support for CXL Type 2 devices through the existing PCI reset_method ABI. The reset sequence follows the CXL 4.0 specification [1], including CXL.cache disable, optional cache writeback, CXL Reset Memory Clear, ResetComplete polling, and ResetError reporting.
The userspace ABI is the existing PCI reset interface:
/sys/bus/pci/devices/.../reset_method
/sys/bus/pci/devices/.../resetUserspace can select "cxl_reset" in reset_method and then trigger reset through the existing reset attribute.
CXL Reset is exposed conservatively. The target must be a single-function CXL Type 2 device with no enabled VFs, CXL.cache and CXL.mem must be enabled, and CXL Reset Memory Clear must be supported. The kernel must also have cached HDM state sufficient to validate affected ranges and restore decoder programming.
For reset through the sysfs reset attribute, the generic PCI path calls the driver's reset_prepare() callback when provided and disables PCI I/O, memory decoding, and bus mastering. The CXL reset path then temporarily reserves each affected HDM range exclusively for the duration of the reset, synchronizes CPU and device caches, and blocks IOMMU translations where applicable. It restores HDM decoder state and CXL Device DVSEC Cache and Mem Enable controls before PCI operation resumes. The path does not offline memory or revoke mappings; it relies on CXL.mem users retaining conflicting iomem resources while access remains active.
Following Dan's v6 feedback, this uses the existing PCI reset_method interface instead of adding a memdev-specific sysfs ABI.
v12 changes from v11 [2]:
- Reject cxl_reset and cxl_bus reset when the PF has enabled VFs, including VFs enabled after reset-method probing.
- Require CXL Reset Memory Clear support and request memory clearing so the generic PCI reset does not preserve tenant memory.
- Cache the CXL Device DVSEC control register during early PCI discovery and restore its CXL.cache and CXL.mem enable bits after CXL Reset and CXL bus reset.
- Initialize the early PCI HDM cache only from pci_bus_add_device(), removing the later CXL core retry and its lifecycle locking.
- Reapply and verify a cached HDM decoder lock after decoder restoration and commit, while leaving surviving locked committed state unchanged.
- Convert temporary early HDM-cache ownership to scoped cleanup.
- Document the reset-safety contract, including the iomem ownership requirement and the limits around memory offlining and mapping revocation.
Motivation:
-----------
- Type 2 devices need a CXL-specific reset mechanism beyond existing PCI reset methods.
- FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is the architectural reset mechanism for those protocols.
- The PCI reset_method ABI lets userspace select CXL Reset before falling back to broader bus reset methods.
Change Description:
-------------------
Patch 1: cxl: Move HDM decoder programming helpers
- Introduce a decoder settings snapshot collected under the DPA lock.
- Move low-level settings-based HDM programming into resource.c.
Patch 2: cxl: Make HDM commit helpers available to reset code
- Build generic HDM resource helpers under CONFIG_CXL_RESET.
- Export the lock and commit helpers needed when cxl_core is modular.
Patch 3: cxl: Share HDM decoder decode logic
- Share HDM decoder decode and validation between normal CXL core enumeration and early PCI HDM cache setup.
- Decode zero-size committed decoder state without imposing caller policy.
Patch 4: cxl: Cache decoder settings on PCI devices
- Cache CXL core HDM decoder settings in pci_dev->hdm.
- Refresh the cache as decoders are enumerated, committed, or reset.
Patch 5: cxl: Cache endpoint decoder settings during PCI enumeration
- Snapshot endpoint HDM and CXL Device DVSEC control state during PCI enumeration.
- Cache the HDM register locator as BAR-relative metadata and preserve raw decoder state for reset restore.
- Keep early cache initialization in the PCI bus-add path only.
Patch 6: cxl: Add CXL Device Reset helper
- Add the internal DVSEC reset sequence.
- Disable CXL.cache, perform cache writeback where supported, initiate CXL Reset with Memory Clear, and wait for completion.
Patch 7: cxl: Validate HDM ranges before CXL reset
- Collect enabled cached HDM ranges, reject busy ranges, and perform CPU cache invalidation when a runtime backend is available.
Patch 8: PCI/CXL: Reject CXL Reset on multifunction devices
- Conservatively reject CXL Reset for multifunction devices and PFs with enabled VFs.
Patch 9: cxl: Restore CXL state after PCI reset
- Restore cached global and per-decoder HDM state, decoder locks, and CXL Device DVSEC protocol enables while the target IOMMU block is active.
Patch 10: PCI/CXL: Expose CXL Reset as a PCI reset method
- Add "cxl_reset" to the PCI reset-method table ahead of FLR.
Patch 11: Documentation/ABI: Document CXL Reset PCI reset method
- Document the reset method and its safety contract.
Patch 12: PCI/CXL: Restore CXL state after CXL bus reset
- Restore cached CXL HDM and Device DVSEC state after cxl_bus reset.
- Reject multifunction devices and PFs with enabled VFs.
The CPU cache invalidation step depends on cpu_cache_invalidate_memregion() support for the affected address ranges. If no runtime provider is available, the kernel emits a warning and continues after the affected HDM ranges have been reserved.
Example:
echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method
echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/resetTesting:
- Applied all 12 patches sequentially to a fresh v7.3-rc1 source tree with CONFIG_CXL_BUS=m and CONFIG_CXL_RESET=y. Every intermediate patch passed make vmlinux modules and make M=tools/testing/cxl/.
- Built and booted the series on an arm64 NVIDIA NVL72 system and successfully exercised CXL Reset through the PCI reset sysfs ABI.
- Exercised cxl_bus reset with CXL state restoration on all four GPUs in an NVL72 system; all four resets completed successfully.
- Earlier revision testing ran 100 iterations of cxl_reset.
References:
[1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
[2] https://lore.kernel.org/linux-cxl/[email protected]/
[3] https://lore.kernel.org/linux-cxl/[email protected]/
[4] https://lore.kernel.org/linux-cxl/[email protected]/
[5] https://lore.kernel.org/linux-cxl/[email protected]/
Srirangan Madhavan (12):
cxl: Move HDM decoder programming helpers
cxl: Make HDM commit helpers available to reset code
cxl: Share HDM decoder decode logic
cxl: Cache decoder settings on PCI devices
cxl: Cache endpoint decoder settings during PCI enumeration
cxl: Add CXL Device Reset helper
cxl: Validate HDM ranges before CXL reset PCI/CXL: Reject CXL Reset on multifunction devices
cxl: Restore CXL state after PCI reset
PCI/CXL: Expose CXL Reset as a PCI reset method Documentation/ABI: Document CXL Reset PCI reset method PCI/CXL: Restore CXL state after CXL bus reset
Documentation/ABI/testing/sysfs-bus-pci | 25 + drivers/cxl/Kconfig | 4 + drivers/cxl/core/Makefile | 2 +- drivers/cxl/core/core.h | 8 + drivers/cxl/core/hdm.c | 301 +++---- drivers/cxl/core/pci.c | 22 +- drivers/cxl/core/port.c | 19 + drivers/cxl/core/regs.c | 4 + drivers/cxl/core/resource.c | 1400 +++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 41 - drivers/pci/bus.c | 2 + drivers/pci/pci.c | 8 + drivers/pci/probe.c | 2 + include/cxl/cxl.h | 120 +- include/linux/pci.h | 8 +- include/uapi/linux/pci_regs.h | 14 + tools/testing/cxl/Kbuild | 1 - 17 files changed, 1769 insertions(+), 212 deletions(-) create mode 100644 drivers/cxl/core/resource.c
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
Signature · 1 line
2.43.0
PATCH v12 1/12 cxl: Move HDM decoder programming helpers
Introduce struct cxl_decoder_settings as a snapshot of HDM decoder programming state. Collect endpoint skip and switch target-list state in hdm.c while holding the DPA lock, then pass the snapshot to generic commit helpers.
Move the low-level settings-based register programming into resource.c so later reset restore code can reuse it without depending on private decoder types.
--- drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/hdm.c | 159 +++++++++++------------------------- drivers/cxl/core/resource.c | 112 +++++++++++++++++++++++++ drivers/cxl/cxl.h | 41 ---------- include/cxl/cxl.h | 65 ++++++++++++++- tools/testing/cxl/Kbuild | 1 + 7 files changed, 227 insertions(+), 155 deletions(-) create mode 100644 drivers/cxl/core/resource.c
drivers/cxl/core/Makefile +1 -0
@@ -6,6 +6,7 @@ ccflags-y += -I$(srctree)/drivers/cxl CFLAGS_trace.o = -DTRACE_INCLUDE_PATH=. -I$(src) cxl_core-y := port.o +cxl_core-y += resource.o cxl_core-y += pmem.o cxl_core-y += regs.o cxl_core-y += memdev.o
drivers/cxl/core/core.h +3 -0
@@ -210,8 +210,11 @@ static inline void devm_cxl_dport_ras_setup(struct cxl_dport *dport) { }int cxl_gpf_port_setup (struct cxl_dport *dport);struct cxl_hdm ; +struct cxl_decoder_settings ;int cxl_hdm_decode_init (struct cxl_dev_state *cxlds,struct cxl_hdm *cxlhdm,struct cxl_endpoint_dvsec_info *info); +int cxl_commit_start (void __iomem *hdm,struct cxl_decoder_settings *settings); +int cxl_commit_wait (void __iomem *hdm,struct cxl_decoder_settings *settings);int cxl_port_get_possible_dports (struct cxl_port *port);#ifdef CONFIG_CXL_FEATURES
drivers/cxl/core/hdm.c +46 -113
212 lines not on this page · raw message
drivers/cxl/core/resource.c new file +112 -0
112 lines not on this page · raw message
drivers/cxl/cxl.h +0 -41
49 lines not on this page · raw message
include/cxl/cxl.h +64 -1
79 lines not on this page · raw message
tools/testing/cxl/Kbuild +1 -0
@@ -60,6 +60,7 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o +cxl_core-y += $(CXL_CORE_SRC)/resource.o cxl_core-y += $(CXL_CORE_SRC)/pmu.o cxl_core-y += $(CXL_CORE_SRC)/cdat.o cxl_core-$(CONFIG_TRACING) += $(CXL_CORE_SRC)/trace.o
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 01/12] cxl: Move HDM decoder programming helpers
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines
Introduce struct
cxl_decoder_settingsas a snapshot of HDM decoder programming state. Collect endpoint skip and switch target-list state inhdm.cwhile holding the DPA lock, then pass the snapshot to generic commit helpers.Move the low-level settings-based register programming into
resource.cso later reset restore code can reuse it without depending on private decoder types.Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Jonathan Cameron
maintainerRe: [PATCH v12 01/12] cxl: Move HDM decoder programming helpers
In reply to Srirangan MadhavanQuoted 8 lines · On Thu, 10 Sep 2026 07:07:57 +0000 Srirangan Madhavan <[email protected]>…
Introduce struct
cxl_decoder_settingsas a snapshot of HDM decoder programming state. Collect endpoint skip and switch target-list state inhdm.cwhile holding the DPA lock, then pass the snapshot to generic commit helpers.Move the low-level settings-based register programming into
resource.cso later reset restore code can reuse it without depending on private decoder types.
This feels like two things in one patch. How much of this could be done by code movement in a precursor patch then real changes in a follow up?
I'd like it to be a lot more obvious what has changed.
Or failing that, introduce changes and then move the code in a follow up patch. Would still be easier to review than this combined patch.
Hopefully doing move and changes separately will also stop git diff randomly interleaving old and new functions that are unrelated.
I've commented on some things but the diff is nasty enough I'll wait for a split version to take a closer look at the bulk of this.
Jonathan
Quoted 149 lines
Signed-off-by Srirangan Madhavan <[email protected]>--- drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/hdm.c | 159 +++++++++++------------------------- drivers/cxl/core/resource.c | 112 +++++++++++++++++++++++++ drivers/cxl/cxl.h | 41 ---------- include/cxl/cxl.h | 65 ++++++++++++++- tools/testing/cxl/Kbuild | 1 + 7 files changed, 227 insertions(+), 155 deletions(-) create mode 100644 drivers/cxl/core/resource.cdrivers/cxl/core/Makefile +1
-0@@ -6,6 +6,7 @@ ccflags-y += -I$(srctree)/drivers/cxl CFLAGS_trace.o = -DTRACE_INCLUDE_PATH=. -I$(src) cxl_core-y := port.o +cxl_core-y += resource.o cxl_core-y += pmem.o cxl_core-y += regs.o cxl_core-y += memdev.odrivers/cxl/core/core.h +3
-0@@ -210,8 +210,11 @@ static inline void devm_cxl_dport_ras_setup(struct cxl_dport *dport) { }int cxl_gpf_port_setup (struct cxl_dport *dport);struct cxl_hdm ; +struct cxl_decoder_settings ;int cxl_hdm_decode_init (struct cxl_dev_state *cxlds,struct cxl_hdm *cxlhdm,struct cxl_endpoint_dvsec_info *info); +int cxl_commit_start (void __iomem *hdm,struct cxl_decoder_settings *settings); +int cxl_commit_wait (void __iomem *hdm,struct cxl_decoder_settings *settings);int cxl_port_get_possible_dports (struct cxl_port *port);#ifdef CONFIG_CXL_FEATURES drivers/cxl/core/hdm.c +13
-74@@ -16,11 +16,6 @@* for enumerating these registers and capabilities. */ -struct cxl_rwsem cxl_rwsem = { - .region =__RWSEM_INITIALIZER (cxl_rwsem.region), - .dpa =__RWSEM_INITIALIZER (cxl_rwsem.dpa), -}; -static int add_hdm_decoder (struct cxl_port *port,struct cxl_decoder *cxld) {int rc;@@ -679,122 +674,52 @@ int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, u64 size)return devm_add_action_or_reset (&port->dev, cxl_dpa_release, cxled); } -static void cxld_set_interleave (struct cxl_decoder *cxld,u32 *ctrl) +static u64 cxl_switch_get_target_list (struct cxl_switch_decoder *cxlsd) { -u16 eig; -u8 eiw; - -/* -* Input validation ensures these warns never fire, but otherwise -* suppress unititalized variable usage warnings. -*/ -if (WARN_ONCE (ways_to_eiw (cxld->interleave_ways, &eiw), -"invalid interleave_ways: %d\n" , cxld->interleave_ways)) -return ; -if (WARN_ONCE (granularity_to_eig (cxld->interleave_granularity, &eig), -"invalid interleave_granularity: %d\n" , - cxld->interleave_granularity)) -return ; - -u32p_replace_bits (ctrl, eig,CXL_HDM_DECODER0_CTRL_IG_MASK ); -u32p_replace_bits (ctrl, eiw,CXL_HDM_DECODER0_CTRL_IW_MASK ); - *ctrl |=CXL_HDM_DECODER0_CTRL_COMMIT ; -} - -static void cxld_set_type (struct cxl_decoder *cxld,u32 *ctrl) -{ -u32p_replace_bits (ctrl, - !!(cxld->target_type ==CXL_DECODER_HOSTONLYMEM ), -CXL_HDM_DECODER0_CTRL_HOSTONLY ); -} +struct cxl_decoder *cxld = &cxlsd->cxld; +u64 targets =0 ; +int ways =min (cxld->interleave_ways, cxlsd->nr_targets); -static void cxlsd_set_targets (struct cxl_switch_decoder *cxlsd,u64 *tgt) -{ -struct cxl_dport **t = &cxlsd->target[0 ]; -int ways = cxlsd->cxld.interleave_ways; - - *tgt =FIELD_PREP (GENMASK (7 ,0 ), t[0 ]->port_id); -if (ways >1 ) - *tgt |=FIELD_PREP (GENMASK (15 ,8 ), t[1 ]->port_id); -if (ways >2 ) - *tgt |=FIELD_PREP (GENMASK (23 ,16 ), t[2 ]->port_id); -if (ways >3 ) - *tgt |=FIELD_PREP (GENMASK (31 ,24 ), t[3 ]->port_id); -if (ways >4 ) - *tgt |=FIELD_PREP (GENMASK_ULL (39 ,32 ), t[4 ]->port_id); -if (ways >5 ) - *tgt |=FIELD_PREP (GENMASK_ULL (47 ,40 ), t[5 ]->port_id); -if (ways >6 ) - *tgt |=FIELD_PREP (GENMASK_ULL (55 ,48 ), t[6 ]->port_id); -if (ways >7 ) - *tgt |=FIELD_PREP (GENMASK_ULL (63 ,56 ), t[7 ]->port_id); -} +/* target_map[] holds the raw list before target[] is resolved. */ +for (int i =0 ; i < ways && i <8 ; i++) { +u8 port_id; -/* -* Per CXL 2.0 8.2.5.12.20 Committing Decoder Programming, hardware must set -* committed or error within 10ms, but just be generous with 20ms to account for -* clock skew and other marginal behavior -*/ -#define COMMIT_TIMEOUT_MS 20 -static int cxld_await_commit (void __iomem *hdm,int id) -{ -u32 ctrl; -int i; +if (cxlsd->target[i]) + port_id = cxlsd->target[i]->port_id; +else + port_id = cxld->target_map[i]; -for (i =0 ; i <COMMIT_TIMEOUT_MS ; i++) { - ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); -if (FIELD_GET (CXL_HDM_DECODER0_CTRL_COMMIT_ERROR , ctrl)) { - ctrl &= ~CXL_HDM_DECODER0_CTRL_COMMIT ; -writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); -return -EIO ; - } -if (FIELD_GET (CXL_HDM_DECODER0_CTRL_COMMITTED , ctrl)) -return 0 ; -fsleep (1000 ); + targets |= (u64 )port_id << (i *8 ); } -return -ETIMEDOUT ; +return targets;
drivers/cxl/core/hdm.c int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, u64 size) · line 701
Quoted 10 lines
drivers/cxl/core/hdm.c +3
-4} -static void setup_hw_decoder (struct cxl_decoder *cxld,void __iomem *hdm) +static void cxl_decoder_snapshot (struct cxl_decoder *cxld, +struct cxl_decoder_settings *settings) { -int id = cxld->id; -u64 base, size; -u32 ctrl; +lockdep_assert_held (&cxl_rwsem.dpa);
This messed up diff is an example of why moves and code changes should not be in the same patch. Hopefully if it's just moves we won't get this mess.
Quoted 52 lines
drivers/cxl/core/hdm.c +14
-27-/* common decoder settings */ - ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (cxld->id)); -cxld_set_interleave (cxld, &ctrl); -cxld_set_type (cxld, &ctrl); - base = cxld->hpa_range.start; - size =range_len (&cxld->hpa_range); + *settings = (struct cxl_decoder_settings ) { + .id = cxld->id, + .hpa_range = cxld->hpa_range, + .interleave_ways = cxld->interleave_ways, + .interleave_granularity = cxld->interleave_granularity, + .target_type = cxld->target_type, + .flags = cxld->flags, + }; -writel (upper_32_bits (base), hdm +CXL_HDM_DECODER0_BASE_HIGH_OFFSET (id)); -writel (lower_32_bits (base), hdm +CXL_HDM_DECODER0_BASE_LOW_OFFSET (id)); -writel (upper_32_bits (size), hdm +CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (id)); -writel (lower_32_bits (size), hdm +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (id)); +if (is_endpoint_decoder (&cxld->dev)) { +struct cxl_endpoint_decoder *cxled = +to_cxl_endpoint_decoder (&cxld->dev); -if (is_switch_decoder (&cxld->dev)) { + settings->target_or_skip = cxled->skip; + }else if (is_switch_decoder (&cxld->dev)) {struct cxl_switch_decoder *cxlsd =to_cxl_switch_decoder (&cxld->dev); -void __iomem *tl_hi = hdm +CXL_HDM_DECODER0_TL_HIGH (id); -void __iomem *tl_lo = hdm +CXL_HDM_DECODER0_TL_LOW (id); -u64 targets; -cxlsd_set_targets (cxlsd, &targets); -writel (upper_32_bits (targets), tl_hi); -writel (lower_32_bits (targets), tl_lo); - }else { -struct cxl_endpoint_decoder *cxled = -to_cxl_endpoint_decoder (&cxld->dev); -void __iomem *sk_hi = hdm +CXL_HDM_DECODER0_SKIP_HIGH (id); -void __iomem *sk_lo = hdm +CXL_HDM_DECODER0_SKIP_LOW (id); - -writel (upper_32_bits (cxled->skip), sk_hi); -writel (lower_32_bits (cxled->skip), sk_lo); + settings->target_or_skip =cxl_switch_get_target_list (cxlsd); } - -writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); }static int cxl_decoder_commit (struct cxl_decoder *cxld)
drivers/cxl/core/hdm.c static int cxl_decoder_commit(struct cxl_decoder *cxld) · line 731
Quoted 6 lines
drivers/cxl/core/hdm.c +2
-1@@ -802,7 +727,8 @@ static int cxl_decoder_commit(struct cxl_decoder *cxld)struct cxl_port *port =to_cxl_port (cxld->dev.parent);struct cxl_hdm *cxlhdm =dev_get_drvdata (&port->dev);void __iomem *hdm = cxlhdm->regs.hdm_decoder; -int id = cxld->id, rc; +struct cxl_decoder_settings settings; +int id = cxld->id, rc =0 ;
I guess this rc = 0 is for compilers who can't tell always enter scoped_guard().
Wasn't that problem fixed by:
https://lore.kernel.org/all/[email protected]/
?
Quoted 15 lines
drivers/cxl/core/hdm.c +0
-4if (cxld->flags &CXL_DECODER_F_ENABLE )return 0 ;@@ -815,10 +741,6 @@ static int cxl_decoder_commit(struct cxl_decoder *cxld)return -EBUSY ; } -/* -* For endpoint decoders hosted on CXL memory devices that -* support the sanitize operation, make sure sanitize is not in-flight. -*/ if (is_endpoint_decoder (&cxld->dev)) {struct cxl_endpoint_decoder *cxled =to_cxl_endpoint_decoder (&cxld->dev);
drivers/cxl/core/hdm.c static int cxl_decoder_commit(struct cxl_decoder *cxld) · line 753
Quoted 6 lines
drivers/cxl/core/hdm.c +3
-0@@ -826,6 +748,10 @@ static int cxl_decoder_commit(struct cxl_decoder *cxld)struct cxl_memdev_state *mds =to_cxl_memdev_state (cxlmd->cxlds); +/* +* For endpoint decoders hosted on CXL memory devices that +* support the sanitize operation, make sure sanitize is not in-flight.
Odd line wrapping and I'm not sure why the move is here as you aren't touching the relevant code. If it makes sense for other reasons pull it out as a precursor patch.
drivers/cxl/core/hdm.c static int cxl_decoder_commit(struct cxl_decoder *cxld) · line 765
Quoted 13 lines
drivers/cxl/core/hdm.c +4
-2+*/ if (mds && mds->security.sanitize_active) { dev_dbg(&cxlmd->dev, "attempted to commit %s during sanitize\n",@@ -834,10 +760,17 @@ static int cxl_decoder_commit(struct cxl_decoder *cxld) } } - scoped_guard(rwsem_read, &cxl_rwsem.dpa) -setup_hw_decoder (cxld, hdm); + scoped_guard(rwsem_read, &cxl_rwsem.dpa) { +cxl_decoder_snapshot (cxld, &settings); + rc =cxl_commit_start (hdm, &settings);
Given this never returns anything not 0, make that void and skip the error checks that follows. We can bring it back if that changes after this series.
Quoted 36 lines
drivers/cxl/core/hdm.c +7
-1+ } +if (rc) { +dev_dbg (&port->dev,"%s: error %d committing decoder\n" , +dev_name (&cxld->dev), rc); +return rc; + } - rc =cxld_await_commit (hdm, cxld->id); + rc =cxl_commit_wait (hdm, &settings);if (rc) {dev_dbg (&port->dev,"%s: error %d committing decoder\n" ,dev_name (&cxld->dev), rc);drivers/cxl/core/resource.c new file +18
-0@@ -0,0 +1,112 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2026 NVIDIA Corporation & Affiliates */ +#include <linux/delay.h> +#include <linux/bug.h> +#include <linux/errno.h> +#include <linux/export.h> +#include <linux/kernel.h> + +#include "cxl.h" +#include "core.h" + +struct cxl_rwsem cxl_rwsem = { + .region =__RWSEM_INITIALIZER (cxl_rwsem.region), + .dpa =__RWSEM_INITIALIZER (cxl_rwsem.dpa), +}; + +static void cxld_set_interleave (struct cxl_decoder_settings *settings,u32 *ctrl) +{
drivers/cxl/core/resource.c line 28
Quoted 10 lines
drivers/cxl/core/resource.c new file +10
-0+u16 eig; +u8 eiw; + +/* +* Input validation ensures these warns never fire, but otherwise +* suppress uninitialized variable usage warnings. +*/ +if (WARN_ONCE (ways_to_eiw (settings->interleave_ways, &eiw), +"invalid interleave_ways: %d\n" , + settings->interleave_ways))
Keep old wrapping. It's fine to go over 80 chars and in this case the reason is to make it easier to see no actual change in the code movement.
Quoted 20 lines
drivers/cxl/core/resource.c new file +20
-0+return ; +if (WARN_ONCE (granularity_to_eig (settings->interleave_granularity, &eig), +"invalid interleave_granularity: %d\n" , + settings->interleave_granularity)) +return ; + +u32p_replace_bits (ctrl, eig,CXL_HDM_DECODER0_CTRL_IG_MASK ); +u32p_replace_bits (ctrl, eiw,CXL_HDM_DECODER0_CTRL_IW_MASK ); + *ctrl |=CXL_HDM_DECODER0_CTRL_COMMIT ; +} + +static void cxld_set_type (struct cxl_decoder_settings *settings,u32 *ctrl) +{ +u32p_replace_bits (ctrl, + !!(settings->target_type ==CXL_DECODER_HOSTONLYMEM ), +CXL_HDM_DECODER0_CTRL_HOSTONLY ); +} + +/* +* Per CXL 2.0 8.2.5.12.20 Committing Decoder Programming, hardware must set
drivers/cxl/core/resource.c line 58
Quoted 10 lines
drivers/cxl/core/resource.c new file +10
-0+* committed or error within 10ms, but just be generous with 20ms to account for +* clock skew and other marginal behavior. +*/ +#define COMMIT_TIMEOUT_MS 20 +static int cxld_await_commit (void __iomem *hdm,int id) +{ +u32 ctrl; +int i; + +for (i =0 ; i <COMMIT_TIMEOUT_MS ; i++) {
for (unsigned int i; i <...
Mind you if this were just code movement leaving it alone better and maybe chasing with a tidy up patch at somepoint.
Quoted 28 lines
drivers/cxl/core/resource.c new file +28
-0+ ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); +if (FIELD_GET (CXL_HDM_DECODER0_CTRL_COMMIT_ERROR , ctrl)) { + ctrl &= ~CXL_HDM_DECODER0_CTRL_COMMIT ; +writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); +return -EIO ; + } +if (FIELD_GET (CXL_HDM_DECODER0_CTRL_COMMITTED , ctrl)) +return 0 ; +fsleep (1000 ); + } + +return -ETIMEDOUT ; +} + +static int setup_hw_decoder (void __iomem *hdm, +struct cxl_decoder_settings *settings) +{ +int id = settings->id; +u64 target_or_skip; +u64 base, size; +u32 ctrl; + + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); +cxld_set_interleave (settings, &ctrl); +cxld_set_type (settings, &ctrl); + base = settings->hpa_range.start; + size =range_len (&settings->hpa_range); + target_or_skip = settings->target_or_skip;
drivers/cxl/core/resource.c line 96
Quoted 10 lines
drivers/cxl/core/resource.c new file +10
-0+ +writel (upper_32_bits (base), hdm +CXL_HDM_DECODER0_BASE_HIGH_OFFSET (id)); +writel (lower_32_bits (base), hdm +CXL_HDM_DECODER0_BASE_LOW_OFFSET (id)); +writel (upper_32_bits (size), hdm +CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (id)); +writel (lower_32_bits (size), hdm +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (id)); +/* Target-list and endpoint-skip registers alias the same slot. */ +writel (upper_32_bits (target_or_skip), + hdm +CXL_HDM_DECODER0_TL_HIGH (id)); +writel (lower_32_bits (target_or_skip), + hdm +CXL_HDM_DECODER0_TL_LOW (id));
I'd not worry about going one or two chars over 80. Just put these on one line each.
drivers/cxl/core/resource.c line 100
Quoted 4 lines
drivers/cxl/core/resource.c new file +4
-0+ +writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); + +return 0 ;
Does this function ever return non zero? I couldn't spot it in later patches in this series. If not, void and that might propagate to the callers - I haven't checked.
Quoted 43 lines
drivers/cxl/core/resource.c new file +1
-0+}include/cxl/cxl.h +23
-1@@ -5,8 +5,10 @@#ifndef __CXL_CXL_H__#define __CXL_CXL_H__ +#include <linux/device.h> #include <linux/node.h> #include <linux/ioport.h> +#include <linux/range.h> #include <cxl/mailbox.h> /** @@ -23,7 +25,68 @@ enum cxl_devtype { CXL_DEVTYPE_CLASSMEM, }; -struct device ; +struct cxl_region ; + +enum cxl_decoder_type { +CXL_DECODER_DEVMEM =2 , +CXL_DECODER_HOSTONLYMEM =3 , +}; + +/** +* struct cxl_decoder_settings - CXL HDM decoder programming snapshot +* @id: decoder id +* @hpa_range: Host physical address range mapped by this decoder +* @target_or_skip: switch target list or endpoint skip register value +* @interleave_ways: number of cxl_dports in this decode +* @interleave_granularity: data stride per dport +* @target_type: accelerator vs expander (type2 vs type3) selector +* @flags: memory type capabilities and locking +*/ +struct cxl_decoder_settings { +int id; +struct range hpa_range; +u64 target_or_skip;
include/cxl/cxl.h enum cxl_devtype { · line 58
Quoted 10 lines
include/cxl/cxl.h +10
-0+int interleave_ways; +int interleave_granularity; +enum cxl_decoder_type target_type; +unsigned long flags; +}; + +/* +* Current specification goes up to 8, double that seems a reasonable +* software max for the foreseeable future +*/
That comment has been wrong for a while (for EPs at least).
Would you mind adding a precursor patch to just delete it, ideally with a spec reference to say when it changed. Anyhow definitely don't want to move it! *cough* It's almost like whoever wrote that comment knew that the spec was changing to increase this 16 :)
Quoted 17 lines
include/cxl/cxl.h +17
-0+#define CXL_DECODER_MAX_INTERLEAVE 16 + +/** +* struct cxl_decoder - Common CXL HDM Decoder Attributes +* @dev: this decoder's device +* @id: kernel device name id +* @hpa_range: Host physical address range mapped by this decoder +* @interleave_ways: number of cxl_dports in this decode +* @interleave_granularity: data stride per dport +* @target_type: accelerator vs expander (type2 vs type3) selector +* @flags: memory type capabilities and locking +* @region: currently assigned region for this decoder +* @target_map: cached copy of hardware port-id list, available at init +* before all @dport objects have been instantiated. While +* dport id is 8bit, CFMWS interleave targets are 32bits. +* @commit: device/decoder-type specific callback to commit settings to hw +* @reset: device/decoder-type specific callback to reset hw settings
include/cxl/cxl.h enum cxl_devtype { · line 85
Quoted 10 lines
include/cxl/cxl.h +10
-0+*/ +struct cxl_decoder { +struct device dev; +int id; +struct range hpa_range; +int interleave_ways; +int interleave_granularity; +enum cxl_decoder_type target_type; +unsigned long flags; +struct cxl_region *region;
This reorders fields from where it came from. If that makes sense, separate patch with an explanation of why. For a code move patch
we want absolutely no unnecessary changes.
Quoted 5 lines
include/cxl/cxl.h +4
-0+u32 target_map[CXL_DECODER_MAX_INTERLEAVE ]; +int (*commit)(struct cxl_decoder *cxld); +void (*reset)(struct cxl_decoder *cxld); +};
Re: [PATCH v12 01/12] cxl: Move HDM decoder programming helpers
Quoted 15 lines · Jonathan Cameron
Quoted 8 lines
Introduce struct
cxl_decoder_settingsas a snapshot of HDM decoder programming state. Collect endpoint skip and switch target-list state inhdm.cwhile holding the DPA lock, then pass the snapshot to generic commit helpers.Move the low-level settings-based register programming into
resource.cso later reset restore code can reuse it without depending on private decoder types.This feels like two things in one patch. How much of this could be done by code movement in a precursor patch then real changes in a follow up?
I'd like it to be a lot more obvious what has changed.
Or failing that, introduce changes and then move the code in a follow up patch. Would still be easier to review than this combined patch.
Quoted 7 lines
Hopefully doing move and changes separately will also stop git diff randomly interleaving old and new functions that are unrelated.
I've commented on some things but the diff is nasty enough I'll wait for a split version to take a closer look at the bulk of this.
Jonathan
I split this up in v13:
- patch 1 removes the stale 8-way interleave comment, noting that CXL
2.0 introduced 16-way interleaving;
- patch 3 is the mechanical movement of the HDM programming helpers;
- patch 4 moves the decoder declarations without reordering their
fields; and
- patch 5 contains the settings-related semantic changes.
I preserved the existing formatting and loop declarations in the
movement patches, made the setup helper return void, and removed unrelated cleanup from the mechanical moves.
Signature · 2 lines
Regards, Srirangan
PATCH v12 2/12 cxl: Make HDM commit helpers available to reset code
In reply to Srirangan MadhavanBuild the generic HDM resource helpers under CONFIG_CXL_RESET so PCI reset paths can use them when cxl_core is modular. Export the lock and commit entry points consumed by cxl_core, and leave resource.o out of the mock cxl_core module once it is built into the kernel.
--- drivers/cxl/Kconfig | 4 ++++ drivers/cxl/core/Makefile | 2 +- drivers/cxl/core/resource.c | 3 +++ tools/testing/cxl/Kbuild | 1 - 4 files changed, 8 insertions(+), 2 deletions(-)
drivers/cxl/Kconfig +4 -0
@@ -6,6 +6,7 @@ menuconfig CXL_BUS select FW_UPLOAD select PCI_DOE select FIRMWARE_TABLE + select CXL_RESET select NUMA_KEEP_MEMINFO if NUMA_MEMBLKS select FWCTL if CXL_FEATURES help
@@ -243,4 +244,7 @@ config CXL_ATL depends on CXL_REGION depends on ACPI_PRMT && AMD_NB +config CXL_RESET + bool + endif
drivers/cxl/core/Makefile +1 -1
@@ -1,12 +1,12 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CXL_BUS) += cxl_core.o +obj-$(CONFIG_CXL_RESET) += resource.o obj-$(CONFIG_CXL_SUSPEND) += suspend.o ccflags-y += -I$(srctree)/drivers/cxl CFLAGS_trace.o = -DTRACE_INCLUDE_PATH=. -I$(src) cxl_core-y := port.o -cxl_core-y += resource.o cxl_core-y += pmem.o cxl_core-y += regs.o cxl_core-y += memdev.o
drivers/cxl/core/resource.c +3 -0
@@ -13,6 +13,7 @@ struct cxl_rwsem cxl_rwsem = { .region = __RWSEM_INITIALIZER(cxl_rwsem.region), .dpa = __RWSEM_INITIALIZER(cxl_rwsem.dpa), }; +EXPORT_SYMBOL_FOR_MODULES (cxl_rwsem,"cxl_core" );static void cxld_set_interleave (struct cxl_decoder_settings *settings,u32 *ctrl) {
@@ -105,8 +106,10 @@ int cxl_commit_start(void __iomem *hdm, struct cxl_decoder_settings *settings)lockdep_assert_held (&cxl_rwsem.dpa);return setup_hw_decoder (hdm, settings); } +EXPORT_SYMBOL_FOR_MODULES (cxl_commit_start,"cxl_core" );int cxl_commit_wait (void __iomem *hdm,struct cxl_decoder_settings *settings) {return cxld_await_commit (hdm, settings->id); } +EXPORT_SYMBOL_FOR_MODULES (cxl_commit_wait,"cxl_core" );
tools/testing/cxl/Kbuild +0 -1
@@ -60,7 +60,6 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o -cxl_core-y += $(CXL_CORE_SRC)/resource.o cxl_core-y += $(CXL_CORE_SRC)/pmu.o cxl_core-y += $(CXL_CORE_SRC)/cdat.o cxl_core-$(CONFIG_TRACING) += $(CXL_CORE_SRC)/trace.o
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 02/12] cxl: Make HDM commit helpers available to reset code
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Adding `config CXL_RESET` inside the `if CXL_BUS` block causes a Kconfig unmet direct dependency…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Adding config CXL_RESET inside the if CXL_BUS block causes a Kconfig unmet direct dependency warning when CXL_BUS=m. --
commit 8c6d2d2149249d98c40c726d32df18f750f68999
Author: Srirangan Madhavan <[email protected]>
cxl: Make HDM commit helpers available to reset code
This commit builds generic HDM resource helpers under CONFIG_CXL_RESET so PCI reset paths can use them when cxl_core is modular. It exports the necessary lock and commit entry points consumed by cxl_core.
Quoted 12 lines
drivers/cxl/Kconfig +1
-0@@ -6,6 +6,7 @@ menuconfig CXL_BUS select FW_UPLOAD select PCI_DOE select FIRMWARE_TABLE + select CXL_RESET select NUMA_KEEP_MEMINFO if NUMA_MEMBLKS select FWCTL if CXL_FEATURES help
[ ... ]
drivers/cxl/Kconfig config CXL_ATL · line 250
Quoted 8 lines
drivers/cxl/Kconfig +3
-0@@ -243,4 +244,7 @@ config CXL_ATL depends on CXL_REGION depends on ACPI_PRMT && AMD_NB +config CXL_RESET + bool + endif
[Severity: Low]
Will defining config CXL_RESET inside the if CXL_BUS block cause an unmet direct dependency warning during build when CONFIG_CXL_BUS=m?
Because CXL_BUS uses select CXL_RESET, but CXL_RESET inherently inherits the if CXL_BUS direct dependency, configuring CONFIG_CXL_BUS=m forces the boolean target CXL_RESET to y while its direct dependency evaluates to m.
Could this be resolved by moving config CXL_RESET outside the if CXL_BUS block?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v12 3/12 cxl: Share HDM decoder decode logic
In reply to Srirangan MadhavanMove HDM decoder register decoding into a helper shared by normal CXL core enumeration and early PCI HDM cache setup. This keeps validation of base, range overflow, interleave, target type, and enable state in one place before adding another HDM parser.
Keep caller-owned policy out of the decode helper. A committed zero-size decoder now decodes successfully, while init_hdm_decoder() retains its existing zero-size rejection.
Preserve endpoint DPA state ownership by using the decoded skip value as a local input to devm_cxl_dpa_reserve(). The reservation helper updates cxled->skip under cxl_rwsem.dpa.
--- drivers/cxl/core/core.h | 4 ++ drivers/cxl/core/hdm.c | 75 +++++++++++++------------------------ drivers/cxl/core/port.c | 19 ++++++++++ drivers/cxl/core/resource.c | 43 +++++++++++++++++++++ 4 files changed, 91 insertions(+), 50 deletions(-)
drivers/cxl/core/core.h +4 -0
@@ -157,6 +157,7 @@ long cxl_pci_get_latency(struct pci_dev *pdev);int cxl_pci_get_bandwidth (struct pci_dev *pdev,struct access_coordinate *c);int cxl_port_get_switch_dport_bandwidth (struct cxl_port *port,struct access_coordinate *c); +struct pci_dev *cxl_port_get_uport_pci_dev (struct cxl_port *port);static inline struct device *port_to_host (struct cxl_port *port) {
@@ -215,6 +216,9 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, struct cxl_endpoint_dvsec_info *info);int cxl_commit_start (void __iomem *hdm,struct cxl_decoder_settings *settings);int cxl_commit_wait (void __iomem *hdm,struct cxl_decoder_settings *settings); +int cxl_hdm_decode_decoder (struct cxl_decoder_settings *settings,int id, +u32 ctrl,u64 base,u64 size,u64 target_or_skip, +bool *committed);int cxl_port_get_possible_dports (struct cxl_port *port);#ifdef CONFIG_CXL_FEATURES
drivers/cxl/core/hdm.c +25 -50
116 lines not on this page · raw message
drivers/cxl/core/port.c +19 -0
@@ -34,6 +34,25 @@static DEFINE_IDA (cxl_port_ida);static DEFINE_XARRAY (cxl_root_buses); +struct pci_dev *cxl_port_get_uport_pci_dev (struct cxl_port *port) +{ +struct device *uport = port->uport_dev; +struct device *host; + +if (is_cxl_memdev (uport)) { +struct cxl_memdev *cxlmd =to_cxl_memdev (uport); + + host = cxlmd->dev.parent; + }else { + host = uport; + } + +if (!host || !dev_is_pci (host)) +return NULL ; + +return pci_dev_get (to_pci_dev (host)); +} +/* * The terminal device in PCI is NULL and @platform_bus * for platform devices (for cxl_test)
drivers/cxl/core/resource.c +43 -0
46 lines not on this page · raw message
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 03/12] cxl: Share HDM decoder decode logic
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
Move HDM decoder register decoding into a helper shared by normal CXL core enumeration and early PCI HDM cache setup. This keeps validation of base, range overflow, interleave, target type, and enable state in one place before adding another HDM parser.
Keep caller-owned policy out of the decode helper. A committed zero-size decoder now decodes successfully, while
init_hdm_decoder()retains its existing zero-size rejection.
Quoted 6 lines
Preserve endpoint DPA state ownership by using the decoded skip value as a local input to
devm_cxl_dpa_reserve(). The reservation helper updatescxled->skipunder cxl_rwsem.dpa.Reported-by Fenghua Yu <[email protected]>Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Jonathan Cameron
maintainerRe: [PATCH v12 03/12] cxl: Share HDM decoder decode logic
In reply to Srirangan MadhavanQuoted 5 lines · On Thu, 10 Sep 2026 07:07:59 +0000 Srirangan Madhavan <[email protected]>…
Move HDM decoder register decoding into a helper shared by normal CXL core enumeration and early PCI HDM cache setup. This keeps validation of base, range overflow, interleave, target type, and enable state in one place before adding another HDM parser.
Hi Srirangan,
Quoted 2 lines
Keep caller-owned policy out of the decode helper. A committed zero-size decoder now decodes successfully, while
init_hdm_decoder()retains its
I'd not use decodes for that second bit given it's a decoder. Choose another word - it definitely isn't doing any decoding.
Quoted 7 lines
existing zero-size rejection.
Preserve endpoint DPA state ownership by using the decoded skip value as a local input to
devm_cxl_dpa_reserve(). The reservation helper updatescxled->skipunder cxl_rwsem.dpa.Reported-by Fenghua Yu <[email protected]>
Add a of Closes tag for the report so we can see exactly what it is referring to. I'm guessing the zero length decoders?
Quoted 1 line
Signed-off-by Srirangan Madhavan <[email protected]>
Quite a bit of feedback on how this is done. Maybe I'll get convinced in later patches but as it stands this is making the code less readable. If it is useable in the cxl_decoder and we can lose the local structure than it becomes more convincing.
Quoted 22 lines
drivers/cxl/core/hdm.c +2
-5@@ -907,14 +907,11 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, {struct cxl_endpoint_decoder *cxled =NULL ;u64 size, base, skip, dpa_size, lo, hi; +struct cxl_decoder_settings settings;bool committed;u32 remainder;int i, rc; -u32 ctrl; -union { -u64 value; -unsigned char target_id[8 ]; - } target_list; +u32 ctrl, tl_low, tl_high;if (should_emulate_decoders (info)) return cxl_setup_hdm_decoder_from_dvsec(port, cxld, dpa_base,
drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 928
Quoted 6 lines
drivers/cxl/core/hdm.c +2
-1@@ -927,35 +924,33 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, lo =readl (hdm +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (which)); hi =readl (hdm +CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (which)); size = (hi <<32 ) + lo; - committed = !!(ctrl &CXL_HDM_DECODER0_CTRL_COMMITTED ); + tl_low =readl (hdm +CXL_HDM_DECODER0_TL_LOW (which)); + tl_high =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (which));
Be consistent on either combining these into local variables or not. Right now this is the only one handled in the parameters for the next call.
Quoted 15 lines
drivers/cxl/core/hdm.c +9
-4+ rc =cxl_hdm_decode_decoder (&settings, which, ctrl, base, size, + ((u64 )tl_high <<32 ) | tl_low, &committed); +if (rc) { +dev_warn (&port->dev, +"decoder%d.%d: Invalid decoder configuration (ctrl: %#x): %d\n" , + port->id, cxld->id, ctrl, rc); +return rc; + } + cxld->commit = cxl_decoder_commit; cxld->reset = cxl_decoder_reset; - -if (!committed) - size =0 ; -if (base ==U64_MAX || size ==U64_MAX ) {
drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 945
Quoted 10 lines
drivers/cxl/core/hdm.c +6
-4-dev_warn (&port->dev,"decoder%d.%d: Invalid resource range\n" , - port->id, cxld->id); -return -ENXIO ; - } + cxld->hpa_range = settings.hpa_range; + cxld->interleave_ways = settings.interleave_ways; + cxld->interleave_granularity = settings.interleave_granularity; + cxld->target_type = settings.target_type; + cxld->flags = settings.flags; + size =range_len (&cxld->hpa_range);
If this settings field matches cxld fields so well, why not embed one in there and write to that directly? Without that I'm seeing little benefit in using the settings structure in here. It is complicating the code and the only deduplication is a tiny number of checks.
Quoted 47 lines
drivers/cxl/core/hdm.c +2
-29if (info) cxled =to_cxl_endpoint_decoder (&cxld->dev); - cxld->hpa_range = (struct range ) { - .start = base, - .end = base + size -1 , - }; +if (!cxled && cxld->interleave_ways >8 ) +return -ENXIO ;/* decoders are enabled if committed */ if (committed) { - cxld->flags |=CXL_DECODER_F_ENABLE ; -if (ctrl &CXL_HDM_DECODER0_CTRL_LOCK ) - cxld->flags |=CXL_DECODER_F_LOCK ; -if (FIELD_GET (CXL_HDM_DECODER0_CTRL_HOSTONLY , ctrl)) - cxld->target_type =CXL_DECODER_HOSTONLYMEM ; -else - cxld->target_type =CXL_DECODER_DEVMEM ; -guard (rwsem_write)(&cxl_rwsem.region);if (cxld->id !=cxl_num_decoders_committed (port)) { dev_warn(&port->dev,@@ -995,33 +990,15 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (which)); } } - rc =eiw_to_ways (FIELD_GET (CXL_HDM_DECODER0_CTRL_IW_MASK , ctrl), - &cxld->interleave_ways); -if (rc) { -dev_warn (&port->dev, -"decoder%d.%d: Invalid interleave ways (ctrl: %#x)\n" , - port->id, cxld->id, ctrl); -return rc; - } - rc =eig_to_granularity (FIELD_GET (CXL_HDM_DECODER0_CTRL_IG_MASK , ctrl), - &cxld->interleave_granularity); -if (rc) { -dev_warn (&port->dev, -"decoder%d.%d: Invalid interleave granularity (ctrl: %#x)\n" , - port->id, cxld->id, ctrl); -return rc; - } -dev_dbg (&port->dev,"decoder%d.%d: range: %#llx-%#llx iw: %d ig: %d\n" , port->id, cxld->id, cxld->hpa_range.start, cxld->hpa_range.end, cxld->interleave_ways, cxld->interleave_granularity);
drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1001
Quoted 10 lines
drivers/cxl/core/hdm.c +3
-4if (!cxled) { - lo =readl (hdm +CXL_HDM_DECODER0_TL_LOW (which)); - hi =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (which)); - target_list.value = (hi <<32 ) + lo;for (i =0 ; i < cxld->interleave_ways; i++) - cxld->target_map[i] = target_list.target_id[i]; + cxld->target_map[i] = i <4 ? + (tl_low >> (i *8 )) &0xff : + (tl_high >> ((i -4 ) *8 )) &0xff ;
Can't we keep the type punning and readability it brings? Also why is the one thing that is still using the non settings path to get to values? I'm not that convinced it makes sense to do any of this with your new settings structure but it needs to be consistent at least (like skip is below).
Quoted 29 lines
drivers/cxl/core/hdm.c +1
-3return 0 ; }@@ -1036,9 +1013,7 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, port->id, cxld->id, size, cxld->interleave_ways);return -ENXIO ; } - lo =readl (hdm +CXL_HDM_DECODER0_SKIP_LOW (which)); - hi =readl (hdm +CXL_HDM_DECODER0_SKIP_HIGH (which)); - skip = (hi <<32 ) + lo; + skip = settings.target_or_skip; rc =devm_cxl_dpa_reserve (cxled, *dpa_base + skip, dpa_size, skip);if (rc) { dev_err(&port->dev,drivers/cxl/core/port.c +7
-0@@ -34,6 +34,25 @@static DEFINE_IDA (cxl_port_ida);static DEFINE_XARRAY (cxl_root_buses); +struct pci_dev *cxl_port_get_uport_pci_dev (struct cxl_port *port) +{ +struct device *uport = port->uport_dev; +struct device *host; + +if (is_cxl_memdev (uport)) { +struct cxl_memdev *cxlmd =to_cxl_memdev (uport);
drivers/cxl/core/port.c line 53
Quoted 10 lines
drivers/cxl/core/port.c +10
-0+ + host = cxlmd->dev.parent; + }else { + host = uport; + } + +if (!host || !dev_is_pci (host)) +return NULL ; + +return pci_dev_get (to_pci_dev (host));
Very nearly same code in read_cdata_data()
If you want this helper here, then introduce if first refactoring that code to show the helper is useful then use it here as well. So basically put that as a precursor with a note that it will get reuse in this patch.
Quoted 10 lines
drivers/cxl/core/port.c +2
-0+} +/* * The terminal device in PCI is NULL and @platform_bus * for platform devices (for cxl_test) drivers/cxl/core/resource.c +0
-0
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 117
Quoted 5 lines
drivers/cxl/core/resource.c +2
-0@@ -113,3 +113,46 @@ int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings)return cxld_await_commit (hdm, settings->id); }EXPORT_SYMBOL_FOR_MODULES (cxl_commit_wait,"cxl_core" ); + +int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id,
As in the description, this is too many decode given unrelated things they are talking about. cxl_hdm_parse_decoder() maybe or cxl_hdm_unpack_decoder() or cxl_hdm_decoder_fill_settings() though then you'd need to put committed in there.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 128
Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+ u32 ctrl, u64 base, u64 size, u64 target_or_skip, + bool *committed) +{ + bool enabled = FIELD_GET(CXL_HDM_DECODER0_CTRL_COMMITTED, ctrl); +int rc; + + *settings = (struct cxl_decoder_settings) { + .id = id, + .target_or_skip = target_or_skip, + .target_type = FIELD_GET(CXL_HDM_DECODER0_CTRL_HOSTONLY, ctrl) ? + CXL_DECODER_HOSTONLYMEM : CXL_DECODER_DEVMEM,
Do we have paths where an early exit needs the partly filled in structure? I'm assuming not. In which case I'd shunt this down a bit to where you can fill in more in one go.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 132
Quoted 4 lines
drivers/cxl/core/resource.c +4
-0+ }; + +if (committed) + *committed = enabled;
I'm not sure why committed is special and doesn't go in the settings.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 142
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+if (!enabled) + size =0 ; +if (base ==U64_MAX || size ==U64_MAX || + (size && base >U64_MAX - (size -1 ))) +return -ENXIO ; + + settings->hpa_range = (struct range ) { + .start = base, + .end = base + size -1 , + };
With a bit of reorg, this can be filled in along with the stuff above reducing the zeroing then overwriting that is going on currently.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 147
Quoted 5 lines
drivers/cxl/core/resource.c +5
-0+if (enabled) { + settings->flags =CXL_DECODER_F_ENABLE ; +if (ctrl &CXL_HDM_DECODER0_CTRL_LOCK ) + settings->flags |=CXL_DECODER_F_LOCK ; + }
If you used a local for building flags, this could also be rolled in.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 155
Quoted 8 lines
drivers/cxl/core/resource.c +8
-0+ + rc =eiw_to_ways (FIELD_GET (CXL_HDM_DECODER0_CTRL_IW_MASK , ctrl), + &settings->interleave_ways); +if (rc) +return rc; + + return eig_to_granularity(FIELD_GET(CXL_HDM_DECODER0_CTRL_IG_MASK, + ctrl),
Go long for readability.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 156
Quoted 1 line
drivers/cxl/core/resource.c +1
-0+ &settings->interleave_granularity);
Locals for these as well and it becomes
...
rc = eig_to_granularity(FIELD_GET(CXL_HDM_DECODER0_CTRL_IG_MASK, ctrl), &ig);
if (rc)
return rc;
*settings = (struct cxl_decoder_settings) {
.id = id,
.hpa_range = {
.start = base,
.end = base + size - 1,
},
.target_or_skip = target_or_skip,
.interleave_ways = iw,
.interleave_granularity = ig,
.target_type = FIELD_GET(CXL_HDM_DECODER0_CTRL_HOSTONLY, ctrl) ?
CXL_DECODER_HOSTONLYMEM : CXL_DECODER_DEVMEM,
.flags = flags,
};
return 0;
}Quoted 1 line
drivers/cxl/core/resource.c +1
-0+EXPORT_SYMBOL_FOR_MODULES (cxl_hdm_decode_decoder,"cxl_core" );
Re: [PATCH v12 03/12] cxl: Share HDM decoder decode logic
Quoted 308 lines · Jonathan Cameron
External email: Use caution opening links or attachments
Quoted 5 lines · On Thu, 10 Sep 2026 07:07:59 +0000 Srirangan Madhavan <[email protected]>…
Move HDM decoder register decoding into a helper shared by normal CXL core enumeration and early PCI HDM cache setup. This keeps validation of base, range overflow, interleave, target type, and enable state in one place before adding another HDM parser.
Hi Srirangan,
Quoted 2 lines
Keep caller-owned policy out of the decode helper. A committed zero-size decoder now decodes successfully, while
init_hdm_decoder()retains itsI'd not use decodes for that second bit given it's a decoder. Choose another word - it definitely isn't doing any decoding.
Quoted 7 lines
existing zero-size rejection.
Preserve endpoint DPA state ownership by using the decoded skip value as a local input to
devm_cxl_dpa_reserve(). The reservation helper updatescxled->skipunder cxl_rwsem.dpa.Reported-by Fenghua Yu <[email protected]>Add a of Closes tag for the report so we can see exactly what it is referring to. I'm guessing the zero length decoders?
Quoted 1 line
Signed-off-by Srirangan Madhavan <[email protected]>Quite a bit of feedback on how this is done. Maybe I'll get convinced in later patches but as it stands this is making the code less readable. If it is useable in the
cxl_decoderand we can lose the local structure than it becomes more convincing.Quoted 22 lines
drivers/cxl/core/hdm.c +2
-5@@ -907,14 +907,11 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, {struct cxl_endpoint_decoder *cxled =NULL ;u64 size, base, skip, dpa_size, lo, hi; +struct cxl_decoder_settings settings;bool committed;u32 remainder;int i, rc; -u32 ctrl; -union { -u64 value; -unsigned char target_id[8 ]; - } target_list; +u32 ctrl, tl_low, tl_high;if (should_emulate_decoders (info)) return cxl_setup_hdm_decoder_from_dvsec(port, cxld, dpa_base,drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 928
Quoted 6 lines
drivers/cxl/core/hdm.c +2
-1@@ -927,35 +924,33 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, lo =readl (hdm +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (which)); hi =readl (hdm +CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (which)); size = (hi <<32 ) + lo; - committed = !!(ctrl &CXL_HDM_DECODER0_CTRL_COMMITTED ); + tl_low =readl (hdm +CXL_HDM_DECODER0_TL_LOW (which)); + tl_high =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (which));Be consistent on either combining these into local variables or not. Right now this is the only one handled in the parameters for the next call.
Quoted 15 lines
drivers/cxl/core/hdm.c +9
-4+ rc =cxl_hdm_decode_decoder (&settings, which, ctrl, base, size, + ((u64 )tl_high <<32 ) | tl_low, &committed); +if (rc) { +dev_warn (&port->dev, +"decoder%d.%d: Invalid decoder configuration (ctrl: %#x): %d\n" , + port->id, cxld->id, ctrl, rc); +return rc; + } + cxld->commit = cxl_decoder_commit; cxld->reset = cxl_decoder_reset; - -if (!committed) - size =0 ; -if (base ==U64_MAX || size ==U64_MAX ) {drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 945
Quoted 10 lines
drivers/cxl/core/hdm.c +6
-4-dev_warn (&port->dev,"decoder%d.%d: Invalid resource range\n" , - port->id, cxld->id); -return -ENXIO ; - } + cxld->hpa_range = settings.hpa_range; + cxld->interleave_ways = settings.interleave_ways; + cxld->interleave_granularity = settings.interleave_granularity; + cxld->target_type = settings.target_type; + cxld->flags = settings.flags; + size =range_len (&cxld->hpa_range);If this settings field matches cxld fields so well, why not embed one in there and write to that directly? Without that I'm seeing little benefit in using the settings structure in here. It is complicating the code and the only deduplication is a tiny number of checks.
Quoted 47 lines
drivers/cxl/core/hdm.c +2
-29if (info) cxled =to_cxl_endpoint_decoder (&cxld->dev); - cxld->hpa_range = (struct range ) { - .start = base, - .end = base + size -1 , - }; +if (!cxled && cxld->interleave_ways >8 ) +return -ENXIO ;/* decoders are enabled if committed */ if (committed) { - cxld->flags |=CXL_DECODER_F_ENABLE ; -if (ctrl &CXL_HDM_DECODER0_CTRL_LOCK ) - cxld->flags |=CXL_DECODER_F_LOCK ; -if (FIELD_GET (CXL_HDM_DECODER0_CTRL_HOSTONLY , ctrl)) - cxld->target_type =CXL_DECODER_HOSTONLYMEM ; -else - cxld->target_type =CXL_DECODER_DEVMEM ; -guard (rwsem_write)(&cxl_rwsem.region);if (cxld->id !=cxl_num_decoders_committed (port)) { dev_warn(&port->dev,@@ -995,33 +990,15 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (which)); } } - rc =eiw_to_ways (FIELD_GET (CXL_HDM_DECODER0_CTRL_IW_MASK , ctrl), - &cxld->interleave_ways); -if (rc) { -dev_warn (&port->dev, -"decoder%d.%d: Invalid interleave ways (ctrl: %#x)\n" , - port->id, cxld->id, ctrl); -return rc; - } - rc =eig_to_granularity (FIELD_GET (CXL_HDM_DECODER0_CTRL_IG_MASK , ctrl), - &cxld->interleave_granularity); -if (rc) { -dev_warn (&port->dev, -"decoder%d.%d: Invalid interleave granularity (ctrl: %#x)\n" , - port->id, cxld->id, ctrl); -return rc; - } -dev_dbg (&port->dev,"decoder%d.%d: range: %#llx-%#llx iw: %d ig: %d\n" , port->id, cxld->id, cxld->hpa_range.start, cxld->hpa_range.end, cxld->interleave_ways, cxld->interleave_granularity);drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1001
Quoted 10 lines
drivers/cxl/core/hdm.c +3
-4if (!cxled) { - lo =readl (hdm +CXL_HDM_DECODER0_TL_LOW (which)); - hi =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (which)); - target_list.value = (hi <<32 ) + lo;for (i =0 ; i < cxld->interleave_ways; i++) - cxld->target_map[i] = target_list.target_id[i]; + cxld->target_map[i] = i <4 ? + (tl_low >> (i *8 )) &0xff : + (tl_high >> ((i -4 ) *8 )) &0xff ;Can't we keep the type punning and readability it brings? Also why is the one thing that is still using the non settings path to get to values? I'm not that convinced it makes sense to do any of this with your new settings structure but it needs to be consistent at least (like skip is below).
Quoted 29 lines
drivers/cxl/core/hdm.c +1
-3return 0 ; }@@ -1036,9 +1013,7 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, port->id, cxld->id, size, cxld->interleave_ways);return -ENXIO ; } - lo =readl (hdm +CXL_HDM_DECODER0_SKIP_LOW (which)); - hi =readl (hdm +CXL_HDM_DECODER0_SKIP_HIGH (which)); - skip = (hi <<32 ) + lo; + skip = settings.target_or_skip; rc =devm_cxl_dpa_reserve (cxled, *dpa_base + skip, dpa_size, skip);if (rc) { dev_err(&port->dev,drivers/cxl/core/port.c +7
-0@@ -34,6 +34,25 @@static DEFINE_IDA (cxl_port_ida);static DEFINE_XARRAY (cxl_root_buses); +struct pci_dev *cxl_port_get_uport_pci_dev (struct cxl_port *port) +{ +struct device *uport = port->uport_dev; +struct device *host; + +if (is_cxl_memdev (uport)) { +struct cxl_memdev *cxlmd =to_cxl_memdev (uport);drivers/cxl/core/port.c line 53
Quoted 10 lines
drivers/cxl/core/port.c +10
-0+ + host = cxlmd->dev.parent; + }else { + host = uport; + } + +if (!host || !dev_is_pci (host)) +return NULL ; + +return pci_dev_get (to_pci_dev (host));Very nearly same code in
read_cdata_data()If you want this helper here, then introduce if first refactoring that code to show the helper is useful then use it here as well. So basically put that as a precursor with a note that it will get reuse in this patch.
Quoted 10 lines
drivers/cxl/core/port.c +2
-0+} +/* * The terminal device in PCI is NULL and @platform_bus * for platform devices (for cxl_test) drivers/cxl/core/resource.c +0
-0drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 117
Quoted 5 lines
drivers/cxl/core/resource.c +2
-0@@ -113,3 +113,46 @@ int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings)return cxld_await_commit (hdm, settings->id); }EXPORT_SYMBOL_FOR_MODULES (cxl_commit_wait,"cxl_core" ); + +int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id,As in the description, this is too many decode given unrelated things they are talking about.
cxl_hdm_parse_decoder()maybe orcxl_hdm_unpack_decoder()orcxl_hdm_decoder_fill_settings()though then you'd need to put committed in there.drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 128
Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+ u32 ctrl, u64 base, u64 size, u64 target_or_skip, + bool *committed) +{ + bool enabled = FIELD_GET(CXL_HDM_DECODER0_CTRL_COMMITTED, ctrl); +int rc; + + *settings = (struct cxl_decoder_settings) { + .id = id, + .target_or_skip = target_or_skip, + .target_type = FIELD_GET(CXL_HDM_DECODER0_CTRL_HOSTONLY, ctrl) ? + CXL_DECODER_HOSTONLYMEM : CXL_DECODER_DEVMEM,Do we have paths where an early exit needs the partly filled in structure? I'm assuming not. In which case I'd shunt this down a bit to where you can fill in more in one go.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 132
Quoted 4 lines
drivers/cxl/core/resource.c +4
-0+ }; + +if (committed) + *committed = enabled;I'm not sure why committed is special and doesn't go in the settings.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 142
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+if (!enabled) + size =0 ; +if (base ==U64_MAX || size ==U64_MAX || + (size && base >U64_MAX - (size -1 ))) +return -ENXIO ; + + settings->hpa_range = (struct range ) { + .start = base, + .end = base + size -1 , + };With a bit of reorg, this can be filled in along with the stuff above reducing the zeroing then overwriting that is going on currently.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 147
Quoted 5 lines
drivers/cxl/core/resource.c +5
-0+if (enabled) { + settings->flags =CXL_DECODER_F_ENABLE ; +if (ctrl &CXL_HDM_DECODER0_CTRL_LOCK ) + settings->flags |=CXL_DECODER_F_LOCK ; + }If you used a local for building flags, this could also be rolled in.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 155
Quoted 8 lines
drivers/cxl/core/resource.c +8
-0+ + rc =eiw_to_ways (FIELD_GET (CXL_HDM_DECODER0_CTRL_IW_MASK , ctrl), + &settings->interleave_ways); +if (rc) +return rc; + + return eig_to_granularity(FIELD_GET(CXL_HDM_DECODER0_CTRL_IG_MASK, + ctrl),Go long for readability.
drivers/cxl/core/resource.c int cxl_commit_wait(void __iomem *hdm, struct cxl_decoder_settings *settings) · line 156
Quoted 1 line
drivers/cxl/core/resource.c +1
-0+ &settings->interleave_granularity);Locals for these as well and it becomes
...rc = eig_to_granularity(FIELD_GET(CXL_HDM_DECODER0_CTRL_IG_MASK, ctrl), &ig); if (rc) return rc; *settings = (struct cxl_decoder_settings) { .id = id, .hpa_range = { .start = base, .end = base + size - 1, }, .target_or_skip = target_or_skip, .interleave_ways = iw, .interleave_granularity = ig,
Quoted 10 lines
.target_type = FIELD_GET(CXL_HDM_DECODER0_CTRL_HOSTONLY, ctrl) ? CXL_DECODER_HOSTONLYMEM : CXL_DECODER_DEVMEM, .flags = flags, }; return 0; }Quoted 1 line
drivers/cxl/core/resource.c +1
-0+EXPORT_SYMBOL_FOR_MODULES (cxl_hdm_decode_decoder,"cxl_core" );
I reworked this across v13 patches 2 and 7 based on the comments in
this review. The shared PCI-device lookup is now a separate precursor, and the decoder helper is now cxl_hdm_unpack_decoder().
The unpack helper validates locals before publishing the settings in
one assignment, represents committed state in the settings flags, reads the target registers consistently, and retains the target-list union.
https://lore.kernel.org/linux-cxl/[email protected]/
https://lore.kernel.org/linux-cxl/[email protected]/
Signature · 2 lines
Regards, Srirangan
PATCH v12 4/12 cxl: Cache decoder settings on PCI devices
In reply to Srirangan MadhavanAdd CXL core plumbing to refresh a PCI device HDM decoder cache when decoders are enumerated, committed, or reset. PCI reset paths can use this snapshot to restore HDM programming without walking CXL topology during reset recovery.
The cache is populated by PCI-side discovery in a follow-on patch. Until then, the CXL core update path is a no-op when no PCI HDM cache is present.
--- drivers/cxl/core/hdm.c | 68 +++++++++++++++++++++++++++++++++++++++++- include/cxl/cxl.h | 12 ++++++++ include/linux/pci.h | 6 ++++ 3 files changed, 85 insertions(+), 1 deletion(-)
drivers/cxl/core/hdm.c +67 -1
@@ -16,6 +16,9 @@* for enumerating these registers and capabilities. */ +static void cxl_decoder_snapshot (struct cxl_decoder *cxld, +struct cxl_decoder_settings *settings); +static int add_hdm_decoder (struct cxl_port *port,struct cxl_decoder *cxld) {int rc;
@@ -84,6 +87,62 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) cxlhdm->iw_cap_mask |=BIT (16 ); } +static bool __cxl_pci_hdm_decoder_count_match (struct pci_dev *pdev, +int decoder_count) +{ +struct cxl_hdm_info *info; +bool match = true; + +down_read (&cxl_rwsem.dpa); + info = pdev->hdm; +if (info) { +if (info->decoder_count != decoder_count) { +pci_warn (pdev, +"CXL HDM cache decoder count mismatch: cached=%d hdm=%d\n" , + info->decoder_count, decoder_count); + match = false; + } + } +up_read (&cxl_rwsem.dpa); + +return match; +} + +static bool cxl_pci_hdm_decoder_count_match (struct cxl_hdm *cxlhdm) +{ + struct pci_dev *pdev __free(pci_dev_put) = +cxl_port_get_uport_pci_dev (cxlhdm->port); + +if (!pdev) +return true; + +return __cxl_pci_hdm_decoder_count_match (pdev, cxlhdm->decoder_count); +} + +static void cxl_hdm_save_decoder_info (struct cxl_hdm *cxlhdm, +struct cxl_decoder *cxld) +{ + struct pci_dev *pdev __free(pci_dev_put) = +cxl_port_get_uport_pci_dev (cxlhdm->port); +struct cxl_decoder_settings *settings; +struct cxl_hdm_info *info; + +if (!pdev) +return ; + +guard (rwsem_write)(&cxl_rwsem.dpa); + info = pdev->hdm; +if (!info || cxld->id >= info->decoder_count) +return ; + + settings = &info->settings[cxld->id]; + *settings = (struct cxl_decoder_settings ) { + .id = cxld->id, + }; +if (cxld->flags &CXL_DECODER_F_ENABLE ) +cxl_decoder_snapshot (cxld, settings); +} +static bool should_emulate_decoders (struct cxl_endpoint_dvsec_info *info) {struct cxl_hdm *cxlhdm;
@@ -778,6 +837,7 @@ static int cxl_decoder_commit(struct cxl_decoder *cxld) } port->commit_end++; cxld->flags |=CXL_DECODER_F_ENABLE ; +cxl_hdm_save_decoder_info (cxlhdm, cxld);return 0 ; }
@@ -850,6 +910,7 @@ static void cxl_decoder_reset(struct cxl_decoder *cxld)writel (0 , hdm +CXL_HDM_DECODER0_BASE_LOW_OFFSET (id)); cxld->flags &= ~CXL_DECODER_F_ENABLE ; +cxl_hdm_save_decoder_info (cxlhdm, cxld);/* Userspace is now responsible for reconfiguring this decoder */ if (is_endpoint_decoder (&cxld->dev)) {
@@ -1068,11 +1129,15 @@ static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,struct cxl_port *port = cxlhdm->port;int i;u64 dpa_base =0 ; +int rc;cxl_settle_decoders (cxlhdm); +if (!cxl_pci_hdm_decoder_count_match (cxlhdm)) +return -ENXIO ; +for (i =0 ; i < cxlhdm->decoder_count; i++) { -int rc, target_count = cxlhdm->target_count; +int target_count = cxlhdm->target_count;struct cxl_decoder *cxld;if (is_cxl_endpoint (port)) {
@@ -1107,6 +1172,7 @@ static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,put_device (&cxld->dev);return rc; } +cxl_hdm_save_decoder_info (cxlhdm, cxld); rc =add_hdm_decoder (port, cxld);if (rc) { dev_warn(&port->dev,
include/cxl/cxl.h +12 -0
@@ -133,6 +133,18 @@ struct cxl_regs { ); }; +#define CXL_HDM_DECODER_MAX_COUNT 32 + +/** +* struct cxl_hdm_info - PCI device HDM decoder programming cache +* @decoder_count: number of decoder settings entries +* @settings: cached per-decoder programming state +*/ +struct cxl_hdm_info { +int decoder_count; +struct cxl_decoder_settings settings[CXL_HDM_DECODER_MAX_COUNT ]; +}; +struct cxl_reg_map {bool valid;int id;
include/linux/pci.h +6 -0
@@ -339,6 +339,9 @@ struct pcie_link_state;struct pci_sriov ;struct pci_p2pdma ;struct rcec_ea ; +#ifdef CONFIG_CXL_RESET +struct cxl_hdm_info ; +#endif /* struct pci_dev - describes a PCI device *
@@ -566,6 +569,9 @@ struct pci_dev {#ifdef CONFIG_PCI_DOE struct xarray doe_mbs;/* Data Object Exchange mailboxes */ #endif +#ifdef CONFIG_CXL_RESET +struct cxl_hdm_info *hdm;/* CXL HDM decoder reset state */ +#endif #ifdef CONFIG_PCI_NPEM struct npem *npem;/* Native PCIe Enclosure Management */ #endif
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 04/12] cxl: Cache decoder settings on PCI devices
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines
Add CXL core plumbing to refresh a PCI device HDM decoder cache when decoders are enumerated, committed, or reset. PCI reset paths can use this snapshot to restore HDM programming without walking CXL topology during reset recovery.
The cache is populated by PCI-side discovery in a follow-on patch. Until then, the CXL core update path is a no-op when no PCI HDM cache is present.
Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Jonathan Cameron
maintainerRe: [PATCH v12 04/12] cxl: Cache decoder settings on PCI devices
In reply to Srirangan MadhavanQuoted 10 lines · On Thu, 10 Sep 2026 07:08:00 +0000 Srirangan Madhavan <[email protected]>…
Add CXL core plumbing to refresh a PCI device HDM decoder cache when decoders are enumerated, committed, or reset. PCI reset paths can use this snapshot to restore HDM programming without walking CXL topology during reset recovery.
The cache is populated by PCI-side discovery in a follow-on patch. Until then, the CXL core update path is a no-op when no PCI HDM cache is present.
Signed-off-by Srirangan Madhavan <[email protected]>
Various comments inline
Thanks,
Jonathan
Quoted 11 lines
--- drivers/cxl/core/hdm.c | 68 +++++++++++++++++++++++++++++++++++++++++- include/cxl/cxl.h | 12 ++++++++ include/linux/pci.h | 6 ++++ 3 files changed, 85 insertions(+), 1 deletion(-)drivers/cxl/core/hdm.c +0
-0
drivers/cxl/core/hdm.c static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) · line 96
Quoted 10 lines
drivers/cxl/core/hdm.c +7
-0@@ -16,6 +16,9 @@ * for enumerating these registers and capabilities. +static bool __cxl_pci_hdm_decoder_count_match (struct pci_dev *pdev, +int decoder_count) +{ +struct cxl_hdm_info *info; +bool match = true; + +down_read (&cxl_rwsem.dpa);
guard()
drivers/cxl/core/hdm.c static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) · line 98
Quoted 2 lines
drivers/cxl/core/hdm.c +2
-0+ info = pdev->hdm; +if (info) {
If !info fails the number of decoders definitely didn't match - so maybe print something in that path too.
so far, the local info variable is less readable than pdev->hdm. Maybe it becomes more useful later in series.
Quoted 12 lines
drivers/cxl/core/hdm.c +11
-0+if (info->decoder_count != decoder_count) { +pci_warn (pdev, +"CXL HDM cache decoder count mismatch: cached=%d hdm=%d\n" , + info->decoder_count, decoder_count); + match = false; + } + } +up_read (&cxl_rwsem.dpa); + +return match; +}
drivers/cxl/core/hdm.c static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) · line 119
Quoted 10 lines
drivers/cxl/core/hdm.c +10
-0+ +static bool cxl_pci_hdm_decoder_count_match (struct cxl_hdm *cxlhdm) +{ + struct pci_dev *pdev __free(pci_dev_put) = +cxl_port_get_uport_pci_dev (cxlhdm->port); + +if (!pdev) +return true; + +return __cxl_pci_hdm_decoder_count_match (pdev, cxlhdm->decoder_count);
I went looking and seems like this is the only call. Just bring the implementation inline here.
Quoted 13 lines
drivers/cxl/core/hdm.c +13
-0+} + +static void cxl_hdm_save_decoder_info (struct cxl_hdm *cxlhdm, +struct cxl_decoder *cxld) +{ + struct pci_dev *pdev __free(pci_dev_put) = +cxl_port_get_uport_pci_dev (cxlhdm->port); +struct cxl_decoder_settings *settings; +struct cxl_hdm_info *info; + +if (!pdev) +return ; +
drivers/cxl/core/hdm.c static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) · line 142
Quoted 10 lines
drivers/cxl/core/hdm.c +10
-0+guard (rwsem_write)(&cxl_rwsem.dpa); + info = pdev->hdm; +if (!info || cxld->id >= info->decoder_count) +return ; + + settings = &info->settings[cxld->id]; + *settings = (struct cxl_decoder_settings ) { + .id = cxld->id, + }; +if (cxld->flags &CXL_DECODER_F_ENABLE )
Why is it bad to snapshot a non enabled decoder? Is it pointless or harmful. Add a comment.
Quoted 12 lines
drivers/cxl/core/hdm.c +2
-0+cxl_decoder_snapshot (cxld, settings); +}include/cxl/cxl.h +1
-0@@ -133,6 +133,18 @@ struct cxl_regs { ); }; +#define CXL_HDM_DECODER_MAX_COUNT 32
include/cxl/cxl.h struct cxl_regs { · line 146
Quoted 10 lines
include/cxl/cxl.h +10
-0+ +/** +* struct cxl_hdm_info - PCI device HDM decoder programming cache +* @decoder_count: number of decoder settings entries +* @settings: cached per-decoder programming state +*/ +struct cxl_hdm_info { +int decoder_count; +struct cxl_decoder_settings settings[CXL_HDM_DECODER_MAX_COUNT ]; +};
This is quite a big structure. Can't we do
int decoder_count; struct cxl_decodet setting settings[] __counted_by(decoder_count);
and allocate the necessary size using struct_size()?
I doubt that many devices will do all 32 decoders - particularly as the spec used to allow a smaller number IIRC.
Quoted 28 lines
include/cxl/cxl.h +1
-0+struct cxl_reg_map {bool valid;int id;include/linux/pci.h +6
-0@@ -339,6 +339,9 @@ struct pcie_link_state;struct pci_sriov ;struct pci_p2pdma ;struct rcec_ea ; +#ifdef CONFIG_CXL_RESET +struct cxl_hdm_info ; +#endif /* struct pci_dev - describes a PCI device * @@ -566,6 +569,9 @@ struct pci_dev {#ifdef CONFIG_PCI_DOE struct xarray doe_mbs;/* Data Object Exchange mailboxes */ #endif +#ifdef CONFIG_CXL_RESET +struct cxl_hdm_info *hdm;/* CXL HDM decoder reset state */ +#endif #ifdef CONFIG_PCI_NPEM struct npem *npem;/* Native PCIe Enclosure Management */ #endif
Re: [PATCH v12 04/12] cxl: Cache decoder settings on PCI devices
Quoted 153 lines · Jonathan Cameron
External email: Use caution opening links or attachments
Quoted 10 lines · On Thu, 10 Sep 2026 07:08:00 +0000 Srirangan Madhavan <[email protected]>…
Add CXL core plumbing to refresh a PCI device HDM decoder cache when decoders are enumerated, committed, or reset. PCI reset paths can use this snapshot to restore HDM programming without walking CXL topology during reset recovery.
The cache is populated by PCI-side discovery in a follow-on patch. Until then, the CXL core update path is a no-op when no PCI HDM cache is present.
Signed-off-by Srirangan Madhavan <[email protected]>Various comments inline
Thanks,
Jonathan
Quoted 11 lines
--- drivers/cxl/core/hdm.c | 68 +++++++++++++++++++++++++++++++++++++++++- include/cxl/cxl.h | 12 ++++++++ include/linux/pci.h | 6 ++++ 3 files changed, 85 insertions(+), 1 deletion(-)drivers/cxl/core/hdm.c +0
-0drivers/cxl/core/hdm.c static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) · line 96
Quoted 10 lines
drivers/cxl/core/hdm.c +7
-0@@ -16,6 +16,9 @@ * for enumerating these registers and capabilities. +static bool __cxl_pci_hdm_decoder_count_match (struct pci_dev *pdev, +int decoder_count) +{ +struct cxl_hdm_info *info; +bool match = true; + +down_read (&cxl_rwsem.dpa);
guard()drivers/cxl/core/hdm.c static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) · line 98
Quoted 2 lines
drivers/cxl/core/hdm.c +2
-0+ info = pdev->hdm; +if (info) {If !info fails the number of decoders definitely didn't match - so maybe print something in that path too.
so far, the local info variable is less readable than
pdev->hdm. Maybe it becomes more useful later in series.Quoted 12 lines
drivers/cxl/core/hdm.c +11
-0+if (info->decoder_count != decoder_count) { +pci_warn (pdev, +"CXL HDM cache decoder count mismatch: cached=%d hdm=%d\n" , + info->decoder_count, decoder_count); + match = false; + } + } +up_read (&cxl_rwsem.dpa); + +return match; +}drivers/cxl/core/hdm.c static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) · line 119
Quoted 10 lines
drivers/cxl/core/hdm.c +10
-0+ +static bool cxl_pci_hdm_decoder_count_match (struct cxl_hdm *cxlhdm) +{ + struct pci_dev *pdev __free(pci_dev_put) = +cxl_port_get_uport_pci_dev (cxlhdm->port); + +if (!pdev) +return true; + +return __cxl_pci_hdm_decoder_count_match (pdev, cxlhdm->decoder_count);I went looking and seems like this is the only call. Just bring the implementation inline here.
Quoted 13 lines
drivers/cxl/core/hdm.c +13
-0+} + +static void cxl_hdm_save_decoder_info (struct cxl_hdm *cxlhdm, +struct cxl_decoder *cxld) +{ + struct pci_dev *pdev __free(pci_dev_put) = +cxl_port_get_uport_pci_dev (cxlhdm->port); +struct cxl_decoder_settings *settings; +struct cxl_hdm_info *info; + +if (!pdev) +return ; +drivers/cxl/core/hdm.c static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) · line 142
Quoted 10 lines
drivers/cxl/core/hdm.c +10
-0+guard (rwsem_write)(&cxl_rwsem.dpa); + info = pdev->hdm; +if (!info || cxld->id >= info->decoder_count) +return ; + + settings = &info->settings[cxld->id]; + *settings = (struct cxl_decoder_settings ) { + .id = cxld->id, + }; +if (cxld->flags &CXL_DECODER_F_ENABLE )Why is it bad to snapshot a non enabled decoder? Is it pointless or harmful. Add a comment.
Quoted 12 lines
drivers/cxl/core/hdm.c +2
-0+cxl_decoder_snapshot (cxld, settings); +}include/cxl/cxl.h +1
-0@@ -133,6 +133,18 @@ struct cxl_regs { ); }; +#define CXL_HDM_DECODER_MAX_COUNT 32include/cxl/cxl.h struct cxl_regs { · line 146
Quoted 10 lines
include/cxl/cxl.h +10
-0+ +/** +* struct cxl_hdm_info - PCI device HDM decoder programming cache +* @decoder_count: number of decoder settings entries +* @settings: cached per-decoder programming state +*/ +struct cxl_hdm_info { +int decoder_count; +struct cxl_decoder_settings settings[CXL_HDM_DECODER_MAX_COUNT ]; +};This is quite a big structure. Can't we do
int decoder_count; struct cxl_decodet setting settings[] __counted_by(decoder_count);and allocate the necessary size using
struct_size()?I doubt that many devices will do all 32 decoders - particularly as the spec used to allow a smaller number IIRC.
Quoted 16 lines
include/cxl/cxl.h +1
-0+struct cxl_reg_map {bool valid;int id;include/linux/pci.h +3
-0@@ -339,6 +339,9 @@ struct pcie_link_state;struct pci_sriov ;struct pci_p2pdma ;struct rcec_ea ; +#ifdef CONFIG_CXL_RESET +struct cxl_hdm_info ; +#endif
Quoted 13 lines
include/linux/pci.h struct pci_dev { · line 577
Quoted 12 lines
include/linux/pci.h +3
-0/* struct pci_dev - describes a PCI device * @@ -566,6 +569,9 @@ struct pci_dev {#ifdef CONFIG_PCI_DOE struct xarray doe_mbs;/* Data Object Exchange mailboxes */ #endif +#ifdef CONFIG_CXL_RESET +struct cxl_hdm_info *hdm;/* CXL HDM decoder reset state */ +#endif #ifdef CONFIG_PCI_NPEM struct npem *npem;/* Native PCIe Enclosure Management */ #endif
I addressed these comments in v13 patch 8:
- replaced the manual locking with guard();
- removed the single-use count-match wrapper;
- added the explanation for not snapshotting stale state from a
disabled decoder; and
- changed cxl_hdm_info to use a flexible array allocated withstruct_size().
For the missing pdev->hdm case, I did not report a decoder-count mismatch. With the revised flow, no cache means there is no state to refresh; reset will subsequently be rejected because no snapshot is available. A mismatch is reported only when a cache exists and its decoder count differs.
https://lore.kernel.org/linux-cxl/[email protected]/
Signature · 2 lines
Regards, Srirangan
PATCH v12 5/12 cxl: Cache endpoint decoder settings during PCI enumeration
In reply to Srirangan MadhavanPopulate pci_dev->hdm for CXL.mem functions from pci_bus_add_device(), after final PCI fixups and state save but before driver binding. This gives driver-free reset paths an early HDM snapshot while avoiding the pre-resource-assignment window in PCI capability initialization.
Use the CXL Register Locator BAR Indicator to find the component register BAR, reject unassigned, disabled, or zero memory BAR resources before temporarily enabling Memory Space, and restore the original PCI_COMMAND value before returning. Cache the CXL Device DVSEC control register with the HDM state for reset recovery before a driver can alter it.
CXL core refreshes the cache as decoders are committed or reset, and keeps the cached DVSEC control synchronized when CXL.mem is enabled or disabled. Move the register helpers into the built-in CONFIG_CXL_RESET set so the early cache path is available without cxl_core, and keep the cxl-test mock core from building a duplicate regs.o.
--- drivers/cxl/core/Makefile | 3 +- drivers/cxl/core/core.h | 1 + drivers/cxl/core/hdm.c | 45 ++--- drivers/cxl/core/pci.c | 22 ++- drivers/cxl/core/regs.c | 4 + drivers/cxl/core/resource.c | 336 ++++++++++++++++++++++++++++++++++++ drivers/pci/bus.c | 2 + drivers/pci/probe.c | 2 + include/cxl/cxl.h | 30 ++++ tools/testing/cxl/Kbuild | 1 - 10 files changed, 417 insertions(+), 29 deletions(-)
drivers/cxl/core/Makefile +1 -2
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CXL_BUS) += cxl_core.o -obj-$(CONFIG_CXL_RESET) += resource.o +obj-$(CONFIG_CXL_RESET) += regs.o resource.o obj-$(CONFIG_CXL_SUSPEND) += suspend.o ccflags-y += -I$(srctree)/drivers/cxl
@@ -8,7 +8,6 @@ CFLAGS_trace.o = -DTRACE_INCLUDE_PATH=. -I$(src) cxl_core-y := port.o cxl_core-y += pmem.o -cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.o cxl_core-y += pci.o
drivers/cxl/core/core.h +1 -0
@@ -158,6 +158,7 @@ int cxl_pci_get_bandwidth(struct pci_dev *pdev, struct access_coordinate *c);int cxl_port_get_switch_dport_bandwidth (struct cxl_port *port,struct access_coordinate *c);struct pci_dev *cxl_port_get_uport_pci_dev (struct cxl_port *port); +int cxl_pci_get_device_dvsec_cap (struct pci_dev *pdev,int dvsec,u16 *cap);static inline struct device *port_to_host (struct cxl_port *port) {
drivers/cxl/core/hdm.c +23 -22
75 lines not on this page · raw message
drivers/cxl/core/pci.c +18 -4
@@ -180,6 +180,16 @@ int cxl_await_media_ready(struct cxl_dev_state *cxlds) }EXPORT_SYMBOL_NS_GPL (cxl_await_media_ready,"CXL" ); +static void cxl_hdm_cache_dvsec_ctrl (struct pci_dev *pdev,u16 ctrl) +{ +guard (rwsem_write)(&cxl_rwsem.dpa); +if (!pdev->hdm) +return ; + + pdev->hdm->dvsec_ctrl = ctrl; + pdev->hdm->dvsec_ctrl_valid = true; +} +static int cxl_set_mem_enable (struct cxl_dev_state *cxlds,u16 val) {struct pci_dev *pdev =to_pci_dev (cxlds->dev);
@@ -191,8 +201,10 @@ static int cxl_set_mem_enable(struct cxl_dev_state *cxlds, u16 val)if (rc)return pcibios_err_to_errno (rc); -if ((ctrl &PCI_DVSEC_CXL_MEM_ENABLE ) == val) +if ((ctrl &PCI_DVSEC_CXL_MEM_ENABLE ) == val) { +cxl_hdm_cache_dvsec_ctrl (pdev, ctrl);return 1 ; + } ctrl &= ~PCI_DVSEC_CXL_MEM_ENABLE ; ctrl |= val;
@@ -200,6 +212,7 @@ static int cxl_set_mem_enable(struct cxl_dev_state *cxlds, u16 val)if (rc)return pcibios_err_to_errno (rc); +cxl_hdm_cache_dvsec_ctrl (pdev, ctrl);return 0 ; }
@@ -274,9 +287,10 @@ int cxl_dvsec_rr_decode(struct cxl_dev_state *cxlds,return -ENXIO ; } - rc =pci_read_config_word (pdev, d +PCI_DVSEC_CXL_CAP , &cap); -if (rc) -return pcibios_err_to_errno (rc); + rc =cxl_pci_get_device_dvsec_cap (pdev, d, &cap); +if (rc <0 ) +return rc; + d = rc;if (!(cap &PCI_DVSEC_CXL_MEM_CAPABLE )) {dev_dbg (dev,"Not MEM Capable\n" );
drivers/cxl/core/regs.c +4 -0
@@ -200,6 +200,7 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,return ret_val; } +EXPORT_SYMBOL_NS_GPL (devm_cxl_iomap_block,"CXL" ); int cxl_map_component_regs(const struct cxl_register_map *map, struct cxl_component_regs *regs,
@@ -518,6 +519,7 @@ u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb)return offset; } +EXPORT_SYMBOL_NS_GPL (cxl_rcrb_to_aer,"CXL" );static resource_size_t cxl_rcrb_to_linkcap (struct device *dev,struct cxl_dport *dport) {
@@ -634,6 +636,7 @@ resource_size_t __rcrb_to_component(struct device *dev, struct cxl_rcrb_info *rireturn component_reg_phys; } +EXPORT_SYMBOL_NS_GPL (__rcrb_to_component,"CXL" );resource_size_t cxl_rcd_component_reg_phys (struct device *dev,struct cxl_dport *dport)
@@ -642,3 +645,4 @@ resource_size_t cxl_rcd_component_reg_phys(struct device *dev,return CXL_RESOURCE_NONE ;return __rcrb_to_component (dev, &dport->rcrb,CXL_RCRB_UPSTREAM ); } +EXPORT_SYMBOL_NS_GPL (cxl_rcd_component_reg_phys,"CXL" );
drivers/cxl/core/resource.c +336 -0
348 lines not on this page · raw message
drivers/pci/bus.c +2 -0
@@ -18,6 +18,7 @@#include <linux/pm_runtime.h> #include <linux/proc_fs.h> #include <linux/slab.h> +#include <cxl/cxl.h> #include "pci.h"
@@ -359,6 +360,7 @@ void pci_bus_add_device(struct pci_dev *dev)/* Save config space for error recoverability */ pci_save_state (dev); +pci_cxl_hdm_init (dev);/* * Enable runtime PM, which potentially allows the device to
drivers/pci/probe.c +2 -0
@@ -24,6 +24,7 @@#include <linux/pm_runtime.h> #include <linux/bitfield.h> #include <trace/events/pci.h> +#include <cxl/cxl.h> #include "pci.h" static struct resource busn_resource = {
@@ -2484,6 +2485,7 @@ static void pci_release_dev(struct device *dev)struct pci_dev *pci_dev; pci_dev =to_pci_dev (dev); +pci_cxl_hdm_release (pci_dev);pci_release_capabilities (pci_dev);pci_release_of_node (pci_dev);pcibios_release_device (pci_dev);
include/cxl/cxl.h +30 -0
@@ -26,6 +26,7 @@ enum cxl_devtype { };struct cxl_region ; +struct pci_dev ;enum cxl_decoder_type {CXL_DECODER_DEVMEM =2 ,
@@ -135,16 +136,45 @@ struct cxl_regs {#define CXL_HDM_DECODER_MAX_COUNT 32 +struct cxl_hdm_decoder_state ; +/** * struct cxl_hdm_info - PCI device HDM decoder programming cache * @decoder_count: number of decoder settings entries +* @hdm_bar: BAR containing the HDM decoder registers +* @hdm_offset: HDM decoder register offset relative to @hdm_bar +* @hdm_size: HDM decoder register resource size +* @global_ctrl: cached HDM decoder global control register +* @dvsec_ctrl: cached CXL Device DVSEC control register +* @dvsec_ctrl_valid: @dvsec_ctrl contains valid state +* @decoder_state: cached raw per-decoder register state * @settings: cached per-decoder programming state */ struct cxl_hdm_info {int decoder_count; +int hdm_bar; +resource_size_t hdm_offset; +resource_size_t hdm_size; +u32 global_ctrl; +u16 dvsec_ctrl; +bool dvsec_ctrl_valid; +struct cxl_hdm_decoder_state *decoder_state;struct cxl_decoder_settings settings[CXL_HDM_DECODER_MAX_COUNT ]; }; +#ifdef CONFIG_CXL_RESET +void pci_cxl_hdm_init (struct pci_dev *pdev); +void pci_cxl_hdm_release (struct pci_dev *pdev); +#else +static inline void pci_cxl_hdm_init (struct pci_dev *pdev) +{ +} + +static inline void pci_cxl_hdm_release (struct pci_dev *pdev) +{ +} +#endif +struct cxl_reg_map {bool valid;int id;
tools/testing/cxl/Kbuild +0 -1
@@ -55,7 +55,6 @@ obj-m += cxl_core.o cxl_core-y := $(CXL_CORE_SRC)/port.o cxl_core-y += $(CXL_CORE_SRC)/pmem.o -cxl_core-y += $(CXL_CORE_SRC)/regs.o cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 05/12] cxl: Cache endpoint decoder settings during PCI enumeration
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Populate
pci_dev->hdmfor CXL.mem functions frompci_bus_add_device(), after final PCI fixups and state save but before driver binding. This gives driver-free reset paths an early HDM snapshot while avoiding the pre-resource-assignment window in PCI capability initialization.Use the CXL Register Locator BAR Indicator to find the component register BAR, reject unassigned, disabled, or zero memory BAR resources before temporarily enabling Memory Space, and restore the original
PCI_COMMANDvalue before returning. Cache the CXL Device DVSEC control register with the HDM state for reset recovery before a driver can alter it.
Quoted 7 lines
CXL core refreshes the cache as decoders are committed or reset, and keeps the cached DVSEC control synchronized when CXL.mem is enabled or disabled. Move the register helpers into the built-in
CONFIG_CXL_RESETset so the early cache path is available withoutcxl_core, and keep the cxl-test mock core from building a duplicate regs.o.Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Jonathan Cameron
maintainerRe: [PATCH v12 05/12] cxl: Cache endpoint decoder settings during PCI enumeration
In reply to Srirangan MadhavanQuoted 11 lines · On Thu, 10 Sep 2026 07:08:01 +0000 Srirangan Madhavan <[email protected]>…
Populate
pci_dev->hdmfor CXL.mem functions frompci_bus_add_device(), after final PCI fixups and state save but before driver binding. This gives driver-free reset paths an early HDM snapshot while avoiding the pre-resource-assignment window in PCI capability initialization.Use the CXL Register Locator BAR Indicator to find the component register BAR, reject unassigned, disabled, or zero memory BAR resources before temporarily enabling Memory Space, and restore the original
PCI_COMMANDvalue before returning. Cache the CXL Device DVSEC control register with the HDM state for reset recovery before a driver can alter it.
Quoted 7 lines
CXL core refreshes the cache as decoders are committed or reset, and keeps the cached DVSEC control synchronized when CXL.mem is enabled or disabled. Move the register helpers into the built-in
CONFIG_CXL_RESETset so the early cache path is available withoutcxl_core, and keep the cxl-test mock core from building a duplicate regs.o.Signed-off-by Srirangan Madhavan <[email protected]>
Hi Srirangan,
Unless I'm reading this wrong, this has evolved to the point that it needs a step back and a rethink. There is complexity in here I don't think you need at all. I may well be missing something though given it's Friday evening!
Jonathan
Quoted 44 lines
drivers/cxl/core/resource.c +26
-0@@ -2,9 +2,17 @@/* Copyright (c) 2026 NVIDIA Corporation & Affiliates */ #include <linux/delay.h> #include <linux/bug.h> +#include <linux/bitfield.h> +#include <linux/cleanup.h> #include <linux/errno.h> #include <linux/export.h> +#include <linux/io.h> +#include <linux/ioport.h> #include <linux/kernel.h> +#include <linux/pci.h> +#include <linux/slab.h> + +#include <cxl/pci.h> #include "cxl.h" #include "core.h" @@ -156,3 +164,331 @@ int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, &settings->interleave_granularity); }EXPORT_SYMBOL_FOR_MODULES (cxl_hdm_decode_decoder,"cxl_core" ); + +struct cxl_hdm_decoder_state { +u32 ctrl; +u32 base_low; +u32 base_high; +u32 size_low; +u32 size_high; +u32 target_low; +u32 target_high; +}; + +static void cxl_pci_hdm_info_free (struct cxl_hdm_info *info) +{ +if (!info) +return ; + +kfree (info->decoder_state); +kfree (info);
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 194
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+} + +DEFINE_FREE(cxl_pci_hdm_info, struct cxl_hdm_info *, + cxl_pci_hdm_info_free(_T)) + +void pci_cxl_hdm_release (struct pci_dev *pdev) +{ +struct cxl_hdm_info *info = pdev->hdm; + + pdev->hdm =NULL ;
If the order here matters, you need a barrier or WRITE_ONCE() might do it.
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 198
Quoted 4 lines
drivers/cxl/core/resource.c +4
-0+cxl_pci_hdm_info_free (info); +} + +static bool cxl_pci_bar_usable (struct pci_dev *pdev,int bar)
This doesn't seem to have anything CXL specific about it. Maybe give it a more generic name and move it to pci.c? However, see later - I'm not sure you need this.
Quoted 19 lines
drivers/cxl/core/resource.c +19
-0+{ +struct resource *res = &pdev->resource[bar]; + +if (!pci_resource_len (pdev, bar)) +return false; +if (res->flags & (IORESOURCE_UNSET |IORESOURCE_DISABLED )) +return false; +if (resource_type (res) !=IORESOURCE_MEM ) +return false; +if (!res->start || !res->end) +return false; + +return true; +} + +static int cxl_pci_hdm_find_bar (struct pci_dev *pdev,resource_size_t hdm_start, +resource_size_t hdm_size,int *bar, +resource_size_t *offset) +{
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 227
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+resource_size_t hdm_end; + +if (!hdm_size) +return -EINVAL ; + + hdm_end = hdm_start + hdm_size -1 ; +if (hdm_end < hdm_start) +return -EINVAL ; + +for (int i =0 ; i <PCI_STD_NUM_BARS ; i++) {
This feels unduly painful and overly specific to this case. It is just looking for a resource to bar and offset.
It is not CXL specific so maybe ask Bjorn if such a helper might go in pci.c (assuming not discussed and dismissed in earlier rounds of review!) I'd still like it to mention hdm or anything even if
local to here.
static int cxl_pci_find_resource_bar(struct pci_dev *pdev,
resource_size_t start, resource_size_t size, int *bar, resource_size_t *offset)
Noted later, you can skip this entirely as this is going in a circle.
Quoted 33 lines
drivers/cxl/core/resource.c +32
-0+struct resource *res = &pdev->resource[i]; + +if (!cxl_pci_bar_usable (pdev, i)) +continue ; +if (hdm_start < res->start || hdm_end > res->end) +continue ; + +if (bar) + *bar = i; +if (offset) + *offset = hdm_start - res->start; +return 0 ; + } + +return -ENODEV ; +} + +static void cxl_pci_hdm_read_decoder_state (struct cxl_hdm_decoder_state *state, +void __iomem *hdm,int id) +{ + state->ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); + state->base_low =readl (hdm +CXL_HDM_DECODER0_BASE_LOW_OFFSET (id)); + state->base_high =readl (hdm +CXL_HDM_DECODER0_BASE_HIGH_OFFSET (id)); + state->size_low =readl (hdm +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (id)); + state->size_high =readl (hdm +CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (id)); + state->target_low =readl (hdm +CXL_HDM_DECODER0_TL_LOW (id)); + state->target_high =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (id)); +} + +static int cxl_pci_hdm_read_decoder(struct pci_dev *pdev, + struct cxl_hdm_decoder_state *state, + struct cxl_decoder_settings *settings,
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 295
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ void __iomem *hdm, int id) +{ +u64 target_or_skip, base, size; +int rc; + +cxl_pci_hdm_read_decoder_state (state, hdm, id); + + base = ((u64 )state->base_high <<32 ) | state->base_low; + size = ((u64 )state->size_high <<32 ) | state->size_low; + target_or_skip = ((u64 )state->target_high <<32 ) | state->target_low;
I'm not sure I get why we cache the registers and the stuff derived from them. Why isn't one source of info enough? Or do the have different lifetimes? If they do then add a comment to structure definition on that.
Quoted 24 lines
drivers/cxl/core/resource.c +23
-0+ + rc =cxl_hdm_decode_decoder (settings, id, state->ctrl, base, size, + target_or_skip,NULL ); +if (rc) { +pci_err (pdev,"CXL HDM decoder %d has invalid configuration: %d\n" , + id, rc); +return rc; + } +return 0 ; +} + +static int __cxl_pci_hdm_read_info (struct pci_dev *pdev, +struct cxl_register_map *map, +struct cxl_hdm_info *info) +{ +struct cxl_decoder_settings *settings; +int decoder_count; +int rc; + + rc =cxl_setup_regs (map); +if (rc) +return rc; +
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 372
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+if (!map->component_map.hdm_decoder.valid) +return -ENODEV ; + + void __iomem *hdm __free(cxl_hdm_iounmap) = +cxl_pci_hdm_map (pdev, map, info); +if (IS_ERR (hdm)) +return PTR_ERR (no_free_ptr (hdm)); + + decoder_count =cxl_hdm_decoder_count (readl (hdm + +CXL_HDM_DECODER_CAP_OFFSET ));
Go long on lines like this. As long as you stay only a bit over 80 no one will mind.
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 379
Quoted 7 lines
drivers/cxl/core/resource.c +7
-0+if (decoder_count <0 ) +return decoder_count; + +if (decoder_count >ARRAY_SIZE (info->settings)) +return -ENXIO ; + + if (CXL_HDM_DECODER0_CTRL_OFFSET(decoder_count - 1) + 0x10 >
That 0x10 needs to be a define or other useful code. I have no idea what it is...
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 390
Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+ info->hdm_size) { +pci_err (pdev, +"CXL HDM decoder count exceeds mapped register block\n" ); +return -ENXIO ; + } + + info->decoder_count = decoder_count; + info->global_ctrl =readl (hdm +CXL_HDM_DECODER_CTRL_OFFSET ); + info->decoder_state =kcalloc (decoder_count, +sizeof (*info->decoder_state), +GFP_KERNEL );
That's same size as settings, so put the two of them a struct at the end of cxl_hdm_info with a short name (e.g. decoder[].state, decoder[].settings) and use a struct_size() allocation for them both.
Quoted 41 lines
drivers/cxl/core/resource.c +41
-0+if (!info->decoder_state) +return -ENOMEM ; + + settings = info->settings; +for (int i =0 ; i < info->decoder_count; i++) { + rc =cxl_pci_hdm_read_decoder (pdev, &info->decoder_state[i], + &settings[i], hdm, i); +if (rc) +return rc; + } + +return 0 ; +} + +static int cxl_pci_hdm_read_info (struct pci_dev *pdev, +struct cxl_register_map *map, +struct cxl_hdm_info *info) +{ +bool restore_command; +u16 command; +int rc, rc2; + +guard (pci_dev)(pdev); + + rc =pci_read_config_word (pdev,PCI_COMMAND , &command); +if (rc) +return pcibios_err_to_errno (rc); + + restore_command = !(command &PCI_COMMAND_MEMORY ); +if (restore_command) { + rc =pci_write_config_word (pdev,PCI_COMMAND , + command |PCI_COMMAND_MEMORY ); +if (rc) +return pcibios_err_to_errno (rc); + } + + rc =__cxl_pci_hdm_read_info (pdev, map, info); + +if (!restore_command) +return rc; +
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 440
Quoted 10 lines
drivers/cxl/core/resource.c +9
-0+ rc2 =pci_write_config_word (pdev,PCI_COMMAND , command); +if (rc2) { + rc2 =pcibios_err_to_errno (rc2); +pci_err (pdev, +"failed to restore PCI_COMMAND after CXL HDM cache init: %d\n" , + rc2); +if (!rc) + rc = rc2; + }
Dance is more complex to read than just duplicating a little.
if (rc) goto reset_command_reg return pci_write_config_word(pdev, PCI_COMMAND, command);
reset_command_reg:
if (pci_write_config_word(pdev, PCI_COMMAND, command) != PCIBIOS_SUCCESSFUL) pci_err(pdev, ...);
I doubt we care about what return of that is given we are on fire.
return rc;
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 447
Quoted 7 lines
drivers/cxl/core/resource.c +7
-0+ +return rc; +} + +static int __pci_cxl_hdm_init (struct pci_dev *pdev) +{ +struct cxl_register_map map = {0 };
The 0 doesn't add anything = { };
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 456
Quoted 9 lines
drivers/cxl/core/resource.c +9
-0+int dvsec; +int rc; + +if (!cxl_pci_hdm_capable (pdev)) +return -ENOTTY ; + + rc =cxl_find_regblock (pdev,CXL_REGLOC_RBI_COMPONENT , &map); +if (rc) +return rc;
So this reads the regblock locator to get where the component regs are by decodeing the bar and offset then finding the resource form that...
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 460
Quoted 4 lines
drivers/cxl/core/resource.c +4
-0+ + rc =cxl_pci_hdm_find_bar (pdev, map.resource, map.max_size,NULL ,NULL ); +if (rc) +return rc;
This takes the resource and finds the bar and offset?
Going in circles. Can't you pull a helper out of the start of cxl_decode_regblock() and get the bar and offset directly. Even if you need the sanity checks along the way, I think you can do them more directly and only go in one direction.
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 465
Quoted 5 lines
drivers/cxl/core/resource.c +5
-0+ + struct cxl_hdm_info *info __free(cxl_pci_hdm_info) = +kzalloc_obj (*info,GFP_KERNEL ); +if (!info) +return -ENOMEM ;
Why allocate here when so much can still fail? We don't need it for a few more calls.
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 477
Quoted 12 lines
drivers/cxl/core/resource.c +12
-0+ + dvsec =pci_find_dvsec_capability (pdev,PCI_VENDOR_ID_CXL , +PCI_DVSEC_CXL_DEVICE ); +if (!dvsec) +return -ENOTTY ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL , + &info->dvsec_ctrl); +if (rc) +return pcibios_err_to_errno (rc); + info->dvsec_ctrl_valid = true; +
I'd make this read and allocate - having found the size so we don't allocate more than necessary. If any crazy device changes number of decoders on reset then we can just clamp it and print a rude message.
Quoted 17 lines
drivers/cxl/core/resource.c +17
-0+ rc =cxl_pci_hdm_read_info (pdev, &map, info); +if (rc) +return rc; + + pdev->hdm =no_free_ptr (info); + +return 0 ; +} + +void pci_cxl_hdm_init (struct pci_dev *pdev) +{ +int rc; + + rc =__pci_cxl_hdm_init (pdev); +if (rc && rc != -ENOTTY && rc != -ENODEV ) +pci_dbg (pdev,"CXL HDM cache init failed: %d\n" , rc); +}
Re: [PATCH v12 05/12] cxl: Cache endpoint decoder settings during PCI enumeration
Quoted 391 lines · Jonathan Cameron
External email: Use caution opening links or attachments
Quoted 11 lines · On Thu, 10 Sep 2026 07:08:01 +0000 Srirangan Madhavan <[email protected]>…
Populate
pci_dev->hdmfor CXL.mem functions frompci_bus_add_device(), after final PCI fixups and state save but before driver binding. This gives driver-free reset paths an early HDM snapshot while avoiding the pre-resource-assignment window in PCI capability initialization.Use the CXL Register Locator BAR Indicator to find the component register BAR, reject unassigned, disabled, or zero memory BAR resources before temporarily enabling Memory Space, and restore the original
PCI_COMMANDvalue before returning. Cache the CXL Device DVSEC control register with the HDM state for reset recovery before a driver can alter it.Quoted 7 lines
CXL core refreshes the cache as decoders are committed or reset, and keeps the cached DVSEC control synchronized when CXL.mem is enabled or disabled. Move the register helpers into the built-in
CONFIG_CXL_RESETset so the early cache path is available withoutcxl_core, and keep the cxl-test mock core from building a duplicate regs.o.Signed-off-by Srirangan Madhavan <[email protected]>Hi Srirangan,
Unless I'm reading this wrong, this has evolved to the point that it needs a step back and a rethink. There is complexity in here I don't think you need at all. I may well be missing something though given it's Friday evening!
Jonathan
Quoted 44 lines
drivers/cxl/core/resource.c +26
-0@@ -2,9 +2,17 @@/* Copyright (c) 2026 NVIDIA Corporation & Affiliates */ #include <linux/delay.h> #include <linux/bug.h> +#include <linux/bitfield.h> +#include <linux/cleanup.h> #include <linux/errno.h> #include <linux/export.h> +#include <linux/io.h> +#include <linux/ioport.h> #include <linux/kernel.h> +#include <linux/pci.h> +#include <linux/slab.h> + +#include <cxl/pci.h> #include "cxl.h" #include "core.h" @@ -156,3 +164,331 @@ int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, &settings->interleave_granularity); }EXPORT_SYMBOL_FOR_MODULES (cxl_hdm_decode_decoder,"cxl_core" ); + +struct cxl_hdm_decoder_state { +u32 ctrl; +u32 base_low; +u32 base_high; +u32 size_low; +u32 size_high; +u32 target_low; +u32 target_high; +}; + +static void cxl_pci_hdm_info_free (struct cxl_hdm_info *info) +{ +if (!info) +return ; + +kfree (info->decoder_state); +kfree (info);drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 194
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+} + +DEFINE_FREE(cxl_pci_hdm_info, struct cxl_hdm_info *, + cxl_pci_hdm_info_free(_T)) + +void pci_cxl_hdm_release (struct pci_dev *pdev) +{ +struct cxl_hdm_info *info = pdev->hdm; + + pdev->hdm =NULL ;If the order here matters, you need a barrier or
WRITE_ONCE()might do it.drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 198
Quoted 4 lines
drivers/cxl/core/resource.c +4
-0+cxl_pci_hdm_info_free (info); +} + +static bool cxl_pci_bar_usable (struct pci_dev *pdev,int bar)This doesn't seem to have anything CXL specific about it. Maybe give it a more generic name and move it to
pci.c? However, see later - I'm not sure you need this.Quoted 19 lines
drivers/cxl/core/resource.c +19
-0+{ +struct resource *res = &pdev->resource[bar]; + +if (!pci_resource_len (pdev, bar)) +return false; +if (res->flags & (IORESOURCE_UNSET |IORESOURCE_DISABLED )) +return false; +if (resource_type (res) !=IORESOURCE_MEM ) +return false; +if (!res->start || !res->end) +return false; + +return true; +} + +static int cxl_pci_hdm_find_bar (struct pci_dev *pdev,resource_size_t hdm_start, +resource_size_t hdm_size,int *bar, +resource_size_t *offset) +{drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 227
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+resource_size_t hdm_end; + +if (!hdm_size) +return -EINVAL ; + + hdm_end = hdm_start + hdm_size -1 ; +if (hdm_end < hdm_start) +return -EINVAL ; + +for (int i =0 ; i <PCI_STD_NUM_BARS ; i++) {This feels unduly painful and overly specific to this case. It is just looking for a resource to bar and offset.
It is not CXL specific so maybe ask Bjorn if such a helper might go in
pci.c(assuming not discussed and dismissed in earlier rounds of review!) I'd still like it to mention hdm or anything even if
local to here.
static int cxl_pci_find_resource_bar(structpci_dev*pdev,
resource_size_tstart,resource_size_tsize, int *bar,resource_size_t*offset)Noted later, you can skip this entirely as this is going in a circle.
Quoted 33 lines
drivers/cxl/core/resource.c +32
-0+struct resource *res = &pdev->resource[i]; + +if (!cxl_pci_bar_usable (pdev, i)) +continue ; +if (hdm_start < res->start || hdm_end > res->end) +continue ; + +if (bar) + *bar = i; +if (offset) + *offset = hdm_start - res->start; +return 0 ; + } + +return -ENODEV ; +} + +static void cxl_pci_hdm_read_decoder_state (struct cxl_hdm_decoder_state *state, +void __iomem *hdm,int id) +{ + state->ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); + state->base_low =readl (hdm +CXL_HDM_DECODER0_BASE_LOW_OFFSET (id)); + state->base_high =readl (hdm +CXL_HDM_DECODER0_BASE_HIGH_OFFSET (id)); + state->size_low =readl (hdm +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (id)); + state->size_high =readl (hdm +CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (id)); + state->target_low =readl (hdm +CXL_HDM_DECODER0_TL_LOW (id)); + state->target_high =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (id)); +} + +static int cxl_pci_hdm_read_decoder(struct pci_dev *pdev, + struct cxl_hdm_decoder_state *state, + struct cxl_decoder_settings *settings,drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 295
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ void __iomem *hdm, int id) +{ +u64 target_or_skip, base, size; +int rc; + +cxl_pci_hdm_read_decoder_state (state, hdm, id); + + base = ((u64 )state->base_high <<32 ) | state->base_low; + size = ((u64 )state->size_high <<32 ) | state->size_low; + target_or_skip = ((u64 )state->target_high <<32 ) | state->target_low;I'm not sure I get why we cache the registers and the stuff derived from them. Why isn't one source of info enough? Or do the have different lifetimes? If they do then add a comment to structure definition on that.
Quoted 24 lines
drivers/cxl/core/resource.c +23
-0+ + rc =cxl_hdm_decode_decoder (settings, id, state->ctrl, base, size, + target_or_skip,NULL ); +if (rc) { +pci_err (pdev,"CXL HDM decoder %d has invalid configuration: %d\n" , + id, rc); +return rc; + } +return 0 ; +} + +static int __cxl_pci_hdm_read_info (struct pci_dev *pdev, +struct cxl_register_map *map, +struct cxl_hdm_info *info) +{ +struct cxl_decoder_settings *settings; +int decoder_count; +int rc; + + rc =cxl_setup_regs (map); +if (rc) +return rc; +drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 372
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+if (!map->component_map.hdm_decoder.valid) +return -ENODEV ; + + void __iomem *hdm __free(cxl_hdm_iounmap) = +cxl_pci_hdm_map (pdev, map, info); +if (IS_ERR (hdm)) +return PTR_ERR (no_free_ptr (hdm)); + + decoder_count =cxl_hdm_decoder_count (readl (hdm + +CXL_HDM_DECODER_CAP_OFFSET ));Go long on lines like this. As long as you stay only a bit over 80 no one will mind.
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 379
Quoted 7 lines
drivers/cxl/core/resource.c +7
-0+if (decoder_count <0 ) +return decoder_count; + +if (decoder_count >ARRAY_SIZE (info->settings)) +return -ENXIO ; + + if (CXL_HDM_DECODER0_CTRL_OFFSET(decoder_count - 1) + 0x10 >That
0x10needs to be a define or other useful code. I have no idea what it is...drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 390
Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+ info->hdm_size) { +pci_err (pdev, +"CXL HDM decoder count exceeds mapped register block\n" ); +return -ENXIO ; + } + + info->decoder_count = decoder_count; + info->global_ctrl =readl (hdm +CXL_HDM_DECODER_CTRL_OFFSET ); + info->decoder_state =kcalloc (decoder_count, +sizeof (*info->decoder_state), +GFP_KERNEL );That's same size as settings, so put the two of them a struct at the end of
cxl_hdm_infowith a short name (e.g. decoder[].state, decoder[].settings) and use astruct_size()allocation for them both.Quoted 41 lines
drivers/cxl/core/resource.c +41
-0+if (!info->decoder_state) +return -ENOMEM ; + + settings = info->settings; +for (int i =0 ; i < info->decoder_count; i++) { + rc =cxl_pci_hdm_read_decoder (pdev, &info->decoder_state[i], + &settings[i], hdm, i); +if (rc) +return rc; + } + +return 0 ; +} + +static int cxl_pci_hdm_read_info (struct pci_dev *pdev, +struct cxl_register_map *map, +struct cxl_hdm_info *info) +{ +bool restore_command; +u16 command; +int rc, rc2; + +guard (pci_dev)(pdev); + + rc =pci_read_config_word (pdev,PCI_COMMAND , &command); +if (rc) +return pcibios_err_to_errno (rc); + + restore_command = !(command &PCI_COMMAND_MEMORY ); +if (restore_command) { + rc =pci_write_config_word (pdev,PCI_COMMAND , + command |PCI_COMMAND_MEMORY ); +if (rc) +return pcibios_err_to_errno (rc); + } + + rc =__cxl_pci_hdm_read_info (pdev, map, info); + +if (!restore_command) +return rc; +drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 440
Quoted 10 lines
drivers/cxl/core/resource.c +9
-0+ rc2 =pci_write_config_word (pdev,PCI_COMMAND , command); +if (rc2) { + rc2 =pcibios_err_to_errno (rc2); +pci_err (pdev, +"failed to restore PCI_COMMAND after CXL HDM cache init: %d\n" , + rc2); +if (!rc) + rc = rc2; + }Dance is more complex to read than just duplicating a little.
if (rc) goto reset_command_reg return pci_write_config_word(pdev, PCI_COMMAND, command);
reset_command_reg:if (pci_write_config_word(pdev, PCI_COMMAND, command) != PCIBIOS_SUCCESSFUL) pci_err(pdev, ...);I doubt we care about what return of that is given we are on fire.
return rc;drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 447
Quoted 7 lines
drivers/cxl/core/resource.c +7
-0+ +return rc; +} + +static int __pci_cxl_hdm_init (struct pci_dev *pdev) +{ +struct cxl_register_map map = {0 };The 0 doesn't add anything = { };
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 456
Quoted 9 lines
drivers/cxl/core/resource.c +9
-0+int dvsec; +int rc; + +if (!cxl_pci_hdm_capable (pdev)) +return -ENOTTY ; + + rc =cxl_find_regblock (pdev,CXL_REGLOC_RBI_COMPONENT , &map); +if (rc) +return rc;So this reads the regblock locator to get where the component regs are by decodeing the bar and offset then finding the resource form that...
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 460
Quoted 4 lines
drivers/cxl/core/resource.c +4
-0+ + rc =cxl_pci_hdm_find_bar (pdev, map.resource, map.max_size,NULL ,NULL ); +if (rc) +return rc;This takes the resource and finds the bar and offset?
Going in circles. Can't you pull a helper out of the start of
cxl_decode_regblock()and get the bar and offset directly. Even if you need the sanity checks along the way, I think you can do them more directly and only go in one direction.drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 465
Quoted 5 lines
drivers/cxl/core/resource.c +5
-0+ + struct cxl_hdm_info *info __free(cxl_pci_hdm_info) = +kzalloc_obj (*info,GFP_KERNEL ); +if (!info) +return -ENOMEM ;Why allocate here when so much can still fail? We don't need it for a few more calls.
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 477
Quoted 12 lines
drivers/cxl/core/resource.c +12
-0+ + dvsec =pci_find_dvsec_capability (pdev,PCI_VENDOR_ID_CXL , +PCI_DVSEC_CXL_DEVICE ); +if (!dvsec) +return -ENOTTY ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL , + &info->dvsec_ctrl); +if (rc) +return pcibios_err_to_errno (rc); + info->dvsec_ctrl_valid = true; +I'd make this read and allocate - having found the size so we don't allocate more than necessary. If any crazy device changes number of decoders on reset then we can just clamp it and print a rude message.
Quoted 9 lines
drivers/cxl/core/resource.c +9
-0+ rc =cxl_pci_hdm_read_info (pdev, &map, info); +if (rc) +return rc; + + pdev->hdm =no_free_ptr (info); + +return 0 ; +} +
Quoted 9 lines
drivers/cxl/core/resource.c int cxl_hdm_decode_decoder(struct cxl_decoder_settings *settings, int id, · line 494
Quoted 8 lines
drivers/cxl/core/resource.c +8
-0+void pci_cxl_hdm_init (struct pci_dev *pdev) +{ +int rc; + + rc =__pci_cxl_hdm_init (pdev); +if (rc && rc != -ENOTTY && rc != -ENODEV ) +pci_dbg (pdev,"CXL HDM cache init failed: %d\n" , rc); +}
These are now getting addressed in v13 patch 9:
- removed the resource-to-BAR reverse lookup and now obtain the BAR
and offset directly from the Register Locator path;
- removed the duplicate raw-register decoder cache and retain one
decoded settings representation;
- combined the per-decoder state into the flexible array and allocate
it with struct_size() after discovering the decoder count;
- replaced the unexplained 0x10 size check.
- protect cache publication and removal with cxl_rwsem.dpa; and
- simplified PCI_COMMAND restoration and its error path.
I chose to reject a changed decoder count rather than clamp it because a partial snapshot would not describe all current hardware state. Added a comment for this.
https://lore.kernel.org/linux-cxl/[email protected]/
Signature · 2 lines
Regards, Srirangan
PATCH v12 6/12 cxl: Add CXL Device Reset helper
In reply to Srirangan MadhavanAdd an internal CXL Device Reset helper for Type 2 functions that advertise CXL Reset and CXL Reset Memory Clear in the CXL Device DVSEC. The helper disables CXL.cache, performs cache writeback when supported, initiates reset with Memory Clear enabled, waits for completion, and re-enables CXL.cache on exit.
Leave the helper unregistered until range validation and reset-scope validation are in place.
--- drivers/cxl/core/resource.c | 251 ++++++++++++++++++++++++++++++++++ include/cxl/cxl.h | 7 + include/uapi/linux/pci_regs.h | 14 ++ 3 files changed, 272 insertions(+)
drivers/cxl/core/resource.c +251 -0
@@ -8,6 +8,8 @@#include <linux/export.h> #include <linux/io.h> #include <linux/ioport.h> +#include <linux/iommu.h> +#include <linux/jiffies.h> #include <linux/kernel.h> #include <linux/pci.h> #include <linux/slab.h>
@@ -492,3 +494,252 @@ void pci_cxl_hdm_init(struct pci_dev *pdev)if (rc && rc != -ENOTTY && rc != -ENODEV )pci_dbg (pdev,"CXL HDM cache init failed: %d\n" , rc); } +/* +* CXL r4.0 sec 9.7.2 defines the reset completion timeout encodings. +* Sec 9.7.3 leaves config-space access behavior undefined for 100 ms after +* initiating CXL Reset, then limits software to CXL Status2 access until +* reset completion, timeout, or error. +*/ +#define CXL_RESET_RRS_WAIT_MS 100 +#define CXL_RESET_STATUS_POLL_MS 20 +static const u32 cxl_reset_timeout_ms[] = { +10 ,100 ,1000 ,10000 ,100000 , +}; + +#define CXL_CACHE_WBI_TIMEOUT_US 100000 +#define CXL_CACHE_WBI_POLL_US 100 + +static int cxl_reset_get_dvsec (struct pci_dev *pdev,u16 *cap_out) +{ +int dvsec, rc; +u16 cap, ctrl; + + dvsec =cxl_pci_get_device_dvsec_cap (pdev,0 , &cap); +if (dvsec <0 ) +return dvsec; + +if (!(cap &PCI_DVSEC_CXL_CACHE_CAPABLE ) || + !(cap &PCI_DVSEC_CXL_MEM_CAPABLE )) +return -ENOTTY ; + +if (!(cap &PCI_DVSEC_CXL_RST_CAPABLE )) +return -ENOTTY ; +if (!(cap &PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE )) +return -ENOTTY ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL , &ctrl); +if (rc) +return pcibios_err_to_errno (rc); + +if (!(ctrl &PCI_DVSEC_CXL_CACHE_ENABLE ) || + !(ctrl &PCI_DVSEC_CXL_MEM_ENABLE )) +return -ENOTTY ; + + *cap_out = cap; +return dvsec; +} + +#define CXL_RESET_CTRL2_CMD_MASK \ + (PCI_DVSEC_CXL_INIT_CACHE_WBI | PCI_DVSEC_CXL_INIT_CXL_RST) + +static int cxl_reset_read_ctrl2 (struct pci_dev *pdev,int dvsec,u16 *ctrl2) +{ +int rc; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL2 , ctrl2); +if (rc) +return pcibios_err_to_errno (rc); + + *ctrl2 &= ~CXL_RESET_CTRL2_CMD_MASK ; +return 0 ; +} + +static int cxl_reset_write_ctrl2 (struct pci_dev *pdev,int dvsec,u16 ctrl2) +{ +int rc; + + rc =pci_write_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL2 , ctrl2); +if (rc) +return pcibios_err_to_errno (rc); + +return 0 ; +} + +static int cxl_reset_modify_ctrl2 (struct pci_dev *pdev,int dvsec,u16 set, +u16 clear) +{ +u16 ctrl2; +int rc; + + rc =cxl_reset_read_ctrl2 (pdev, dvsec, &ctrl2); +if (rc) +return rc; + + ctrl2 &= ~clear; + ctrl2 |= set; +return cxl_reset_write_ctrl2 (pdev, dvsec, ctrl2); +} + +static int cxl_reset_enable_cache (struct pci_dev *pdev,int dvsec) +{ +return cxl_reset_modify_ctrl2 (pdev, dvsec,0 , +PCI_DVSEC_CXL_DISABLE_CACHING ); +} + +static int cxl_reset_initiate (struct pci_dev *pdev,int dvsec) +{ +return cxl_reset_modify_ctrl2 (pdev, dvsec, +PCI_DVSEC_CXL_INIT_CXL_RST | +PCI_DVSEC_CXL_RST_MEM_CLR_EN ,0 ); +} + +static int cxl_reset_wait_cache_wbi (struct pci_dev *pdev,int dvsec) +{ +unsigned long deadline; +u16 status2; +int rc; + + rc =cxl_reset_modify_ctrl2 (pdev, dvsec, +PCI_DVSEC_CXL_INIT_CACHE_WBI ,0 ); +if (rc) +return rc; + + deadline = jiffies +usecs_to_jiffies (CXL_CACHE_WBI_TIMEOUT_US ); +do { +usleep_range (CXL_CACHE_WBI_POLL_US ,CXL_CACHE_WBI_POLL_US +1 ); + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_STATUS2 , + &status2); +if (rc) +return pcibios_err_to_errno (rc); +if (status2 !=U16_MAX && (status2 &PCI_DVSEC_CXL_CACHE_INV )) +return 0 ; + }while (time_before (jiffies, deadline)); + +return -ETIMEDOUT ; +} + +static int cxl_reset_disable_cache (struct pci_dev *pdev,int dvsec,u16 cap) +{ +int rc, rc2; + + rc =cxl_reset_modify_ctrl2 (pdev, dvsec, +PCI_DVSEC_CXL_DISABLE_CACHING ,0 ); +if (rc) +return rc; + +if (!(cap &PCI_DVSEC_CXL_CACHE_WBI_CAPABLE )) +return 0 ; + + rc =cxl_reset_wait_cache_wbi (pdev, dvsec); +if (!rc) +return 0 ; + +/* +* Restore the default CXL.cache policy after error paths that +* successfully set DISABLE_CACHING before reset completion. +*/ + rc2 =cxl_reset_enable_cache (pdev, dvsec); +if (rc2) +pci_warn (pdev,"failed to re-enable CXL caching: %d\n" , rc2); + +return rc; +} + +static int cxl_reset_wait_done (struct pci_dev *pdev,int dvsec,u16 cap) +{ +unsigned long deadline; +u32 timeout_ms; +u16 status2; +bool final = false; +int idx, rc; + + idx =FIELD_GET (PCI_DVSEC_CXL_RST_TIMEOUT , cap); +if (idx >=ARRAY_SIZE (cxl_reset_timeout_ms)) { +int last =ARRAY_SIZE (cxl_reset_timeout_ms) -1 ; + +pci_warn (pdev, +"unknown CXL reset timeout encoding %d; using %u ms\n" , + idx, cxl_reset_timeout_ms[last]); + idx = last; + } + + timeout_ms =max_t (u32, cxl_reset_timeout_ms[idx], +CXL_RESET_RRS_WAIT_MS ); +msleep (CXL_RESET_RRS_WAIT_MS ); + deadline = jiffies +msecs_to_jiffies (timeout_ms - +CXL_RESET_RRS_WAIT_MS ); + +do { + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_STATUS2 , + &status2); +if (rc) +return pcibios_err_to_errno (rc); + +if (status2 !=U16_MAX ) { +if (status2 &PCI_DVSEC_CXL_RST_ERR ) +return -EIO ; + +if (status2 &PCI_DVSEC_CXL_RST_DONE ) +return 0 ; + } + +if (time_after_eq (jiffies, deadline)) { +if (final) +return -ETIMEDOUT ; + final = true; +continue ; + } + +msleep (CXL_RESET_STATUS_POLL_MS ); + }while (true); +} + +static int cxl_reset_execute (struct pci_dev *pdev,int dvsec,u16 cap) +{ +bool target_prepared = false; +int rc, rc2; + + rc =cxl_reset_disable_cache (pdev, dvsec, cap); +if (rc) +return rc; + +if (!pci_wait_for_pending_transaction (pdev)) +pci_err (pdev,"timed out waiting for pending transactions\n" ); + + rc =pci_dev_reset_iommu_prepare (pdev); +if (rc) +pci_err (pdev,"failed to stop IOMMU for CXL reset: %d\n" , rc); +else + target_prepared = true; + +if (!rc) + rc =cxl_reset_initiate (pdev, dvsec); +if (!rc) + rc =cxl_reset_wait_done (pdev, dvsec, cap); + + rc2 =cxl_reset_enable_cache (pdev, dvsec); +if (rc2 && rc) +pci_warn (pdev,"failed to re-enable CXL caching: %d\n" , rc2); +else if (rc2) + rc = rc2; + +if (target_prepared) +pci_dev_reset_iommu_done (pdev); +return rc; +} + +int cxl_reset_function (struct pci_dev *pdev,bool probe) +{ +int dvsec; +u16 cap; + + dvsec =cxl_reset_get_dvsec (pdev, &cap); +if (dvsec <0 ) +return dvsec; + +if (probe) +return 0 ; + +return cxl_reset_execute (pdev, dvsec, cap); +}
include/cxl/cxl.h +7 -0
@@ -9,6 +9,7 @@#include <linux/node.h> #include <linux/ioport.h> #include <linux/range.h> +#include <linux/errno.h> #include <cxl/mailbox.h> /**
@@ -165,6 +166,7 @@ struct cxl_hdm_info {#ifdef CONFIG_CXL_RESET void pci_cxl_hdm_init (struct pci_dev *pdev);void pci_cxl_hdm_release (struct pci_dev *pdev); +int cxl_reset_function (struct pci_dev *pdev,bool probe);#else static inline void pci_cxl_hdm_init (struct pci_dev *pdev) {
@@ -173,6 +175,11 @@ static inline void pci_cxl_hdm_init(struct pci_dev *pdev)static inline void pci_cxl_hdm_release (struct pci_dev *pdev) { } + +static inline int cxl_reset_function (struct pci_dev *pdev,bool probe) +{ +return -ENOTTY ; +}#endif struct cxl_reg_map {
include/uapi/linux/pci_regs.h +14 -0
@@ -1352,8 +1352,22 @@#define PCI_DVSEC_CXL_CACHE_CAPABLE _BITUL(0)#define PCI_DVSEC_CXL_MEM_CAPABLE _BITUL(2)#define PCI_DVSEC_CXL_HDM_COUNT __GENMASK(5, 4) +#define PCI_DVSEC_CXL_CACHE_WBI_CAPABLE _BITUL(6) +#define PCI_DVSEC_CXL_RST_CAPABLE _BITUL(7) +#define PCI_DVSEC_CXL_RST_TIMEOUT __GENMASK(10, 8) +#define PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE _BITUL(11)#define PCI_DVSEC_CXL_CTRL 0xC +#define PCI_DVSEC_CXL_CACHE_ENABLE _BITUL(0)#define PCI_DVSEC_CXL_MEM_ENABLE _BITUL(2) +#define PCI_DVSEC_CXL_CTRL2 0x10 +#define PCI_DVSEC_CXL_DISABLE_CACHING _BITUL(0) +#define PCI_DVSEC_CXL_INIT_CACHE_WBI _BITUL(1) +#define PCI_DVSEC_CXL_INIT_CXL_RST _BITUL(2) +#define PCI_DVSEC_CXL_RST_MEM_CLR_EN _BITUL(3) +#define PCI_DVSEC_CXL_STATUS2 0x12 +#define PCI_DVSEC_CXL_CACHE_INV _BITUL(0) +#define PCI_DVSEC_CXL_RST_DONE _BITUL(1) +#define PCI_DVSEC_CXL_RST_ERR _BITUL(2)#define PCI_DVSEC_CXL_RANGE_SIZE_HIGH (i) (0x18 + (i * 0x10))#define PCI_DVSEC_CXL_RANGE_SIZE_LOW (i) (0x1C + (i * 0x10))#define PCI_DVSEC_CXL_MEM_INFO_VALID _BITUL(0)
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 06/12] cxl: Add CXL Device Reset helper
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] The PCI_DVSEC_CXL_RST_MEM_CLR_EN bit is not cleared on error paths, causing subsequent resets to…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] The PCI_DVSEC_CXL_RST_MEM_CLR_EN bit is not cleared on error paths, causing subsequent resets to inadvertently perform a full Memory Clear.
- [Medium] Missing final status check after timeout in cache WBI polling can cause a false timeout.
--
commit e65504bb1432889a2f2835801327341b00b21dfc
Author: Srirangan Madhavan <[email protected]>
cxl: Add CXL Device Reset helper
Add an internal CXL Device Reset helper for Type 2 functions that advertise CXL Reset and CXL Reset Memory Clear in the CXL Device DVSEC. The helper disables CXL.cache, performs cache writeback when supported, initiates reset with Memory Clear enabled, waits for completion, and re-enables CXL.cache on exit.
Quoted 4 lines
drivers/cxl/core/resource.c +0
-0
[ ... ]
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 553
Quoted 12 lines
drivers/cxl/core/resource.c +12
-0+#define CXL_RESET_CTRL2_CMD_MASK \ + (PCI_DVSEC_CXL_INIT_CACHE_WBI | PCI_DVSEC_CXL_INIT_CXL_RST) + +static int cxl_reset_read_ctrl2 (struct pci_dev *pdev,int dvsec,u16 *ctrl2) +{ +int rc; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL2 , ctrl2); +if (rc) +return pcibios_err_to_errno (rc); + + *ctrl2 &= ~CXL_RESET_CTRL2_CMD_MASK ;
[Severity: High]
Does this mask inadvertently exclude the memory clear enable bit, preserving it on error paths?
If cxl_reset_initiate() sets PCI_DVSEC_CXL_RST_MEM_CLR_EN and the reset subsequently fails, the error path calls cxl_reset_enable_cache(). That calls cxl_reset_modify_ctrl2(), which relies on cxl_reset_read_ctrl2() to fetch the current register state.
Because CXL_RESET_CTRL2_CMD_MASK does not include PCI_DVSEC_CXL_RST_MEM_CLR_EN, the stale bit is read, preserved, and written back. Could this cause subsequent resets to inadvertently perform a full Memory Clear?
[ ... ]
Quoted 14 lines
drivers/cxl/core/resource.c +14
-0+static int cxl_reset_wait_cache_wbi (struct pci_dev *pdev,int dvsec) +{ +unsigned long deadline; +u16 status2; +int rc; + + rc =cxl_reset_modify_ctrl2 (pdev, dvsec, +PCI_DVSEC_CXL_INIT_CACHE_WBI ,0 ); +if (rc) +return rc; + + deadline = jiffies +usecs_to_jiffies (CXL_CACHE_WBI_TIMEOUT_US ); + do { + usleep_range(CXL_CACHE_WBI_POLL_US, CXL_CACHE_WBI_POLL_US + 1);
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 619
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_STATUS2 , + &status2); +if (rc) +return pcibios_err_to_errno (rc); +if (status2 !=U16_MAX && (status2 &PCI_DVSEC_CXL_CACHE_INV )) +return 0 ; + }while (time_before (jiffies, deadline)); + +return -ETIMEDOUT ;
[Severity: Medium]
Does this loop miss a final status check after the timeout expires?
If the thread is preempted or delayed between the register read and the time check, time_before(jiffies, deadline) can fail even if the cache invalidation succeeded just as the timeout expired.
Standard kernel polling macros (like readx_poll_timeout) avoid false timeouts by evaluating the status one final time after the timeout expires.
Quoted 1 line
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Jonathan Cameron
maintainerRe: [PATCH v12 06/12] cxl: Add CXL Device Reset helper
In reply to Srirangan MadhavanQuoted 10 lines · On Thu, 10 Sep 2026 07:08:02 +0000 Srirangan Madhavan <[email protected]>…
Add an internal CXL Device Reset helper for Type 2 functions that advertise CXL Reset and CXL Reset Memory Clear in the CXL Device DVSEC. The helper disables CXL.cache, performs cache writeback when supported, initiates reset with Memory Clear enabled, waits for completion, and re-enables CXL.cache on exit.
Leave the helper unregistered until range validation and reset-scope validation are in place.
Signed-off-by Srirangan Madhavan <[email protected]>
A few things inline.
Quoted 34 lines
--- drivers/cxl/core/resource.c | 251 ++++++++++++++++++++++++++++++++++ include/cxl/cxl.h | 7 + include/uapi/linux/pci_regs.h | 14 ++ 3 files changed, 272 insertions(+)drivers/cxl/core/resource.c +13
-0@@ -8,6 +8,8 @@#include <linux/export.h> #include <linux/io.h> #include <linux/ioport.h> +#include <linux/iommu.h> +#include <linux/jiffies.h> #include <linux/kernel.h> #include <linux/pci.h> #include <linux/slab.h> @@ -492,3 +494,252 @@ void pci_cxl_hdm_init(struct pci_dev *pdev)if (rc && rc != -ENOTTY && rc != -ENODEV )pci_dbg (pdev,"CXL HDM cache init failed: %d\n" , rc); } +/* +* CXL r4.0 sec 9.7.2 defines the reset completion timeout encodings. +* Sec 9.7.3 leaves config-space access behavior undefined for 100 ms after +* initiating CXL Reset, then limits software to CXL Status2 access until +* reset completion, timeout, or error. +*/ +#define CXL_RESET_RRS_WAIT_MS 100 +#define CXL_RESET_STATUS_POLL_MS 20 +static const u32 cxl_reset_timeout_ms[] = { +10 ,100 ,1000 ,10000 ,100000 , +};
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 517
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ +#define CXL_CACHE_WBI_TIMEOUT_US 100000 +#define CXL_CACHE_WBI_POLL_US 100 + +static int cxl_reset_get_dvsec (struct pci_dev *pdev,u16 *cap_out) +{ +int dvsec, rc; +u16 cap, ctrl; + + dvsec =cxl_pci_get_device_dvsec_cap (pdev,0 , &cap);
That is an odd function given the control read is separate - why should we wrap up the cap read?
I'd just drop this helper and open code the search for the extended cap at the few callsites. A little more code, but less odd and all standard PCI stuff not needing an extra helper.
Probably a bit of code evolution caused this given you are up at v12!
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 522
Quoted 5 lines
drivers/cxl/core/resource.c +5
-0+if (dvsec <0 ) +return dvsec; + +if (!(cap &PCI_DVSEC_CXL_CACHE_CAPABLE ) || + !(cap &PCI_DVSEC_CXL_MEM_CAPABLE ))
Why do we need them both? Sure that's type 2, but a non class code matching type3 would I think need the same infrastructure you are building here. That would have cxl.mem but not cxl.cache
- I think some of the CXL SSD prototypes fit in this category.
Quoted 21 lines
drivers/cxl/core/resource.c +21
-0+return -ENOTTY ; + +if (!(cap &PCI_DVSEC_CXL_RST_CAPABLE )) +return -ENOTTY ; +if (!(cap &PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE )) +return -ENOTTY ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL , &ctrl); +if (rc) +return pcibios_err_to_errno (rc); + +if (!(ctrl &PCI_DVSEC_CXL_CACHE_ENABLE ) || + !(ctrl &PCI_DVSEC_CXL_MEM_ENABLE )) +return -ENOTTY ; + + *cap_out = cap; +return dvsec; +} + +#define CXL_RESET_CTRL2_CMD_MASK \ + (PCI_DVSEC_CXL_INIT_CACHE_WBI | PCI_DVSEC_CXL_INIT_CXL_RST)
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 553
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ +static int cxl_reset_read_ctrl2 (struct pci_dev *pdev,int dvsec,u16 *ctrl2) +{ +int rc; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL2 , ctrl2); +if (rc) +return pcibios_err_to_errno (rc); + + *ctrl2 &= ~CXL_RESET_CTRL2_CMD_MASK ;
I'd rename this. It is doing more than reading crl2. Or push the code inline and avoid need for any name bikeshedding.
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 563
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+return 0 ; +} + +static int cxl_reset_write_ctrl2 (struct pci_dev *pdev,int dvsec,u16 ctrl2) +{ +int rc; + + rc =pci_write_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL2 , ctrl2); +if (rc) +return pcibios_err_to_errno (rc);
This helper is also providing little value
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 569
Quoted 6 lines
drivers/cxl/core/resource.c +6
-0+ +return 0 ; +} + +static int cxl_reset_modify_ctrl2 (struct pci_dev *pdev,int dvsec,u16 set, +u16 clear)
This needs a rename because it has that subtle mask of previous state. What is it actually allowing you to modify?
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 580
Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+{ +u16 ctrl2; +int rc; + + rc =cxl_reset_read_ctrl2 (pdev, dvsec, &ctrl2); +if (rc) +return rc; + + ctrl2 &= ~clear; + ctrl2 |= set; +return cxl_reset_write_ctrl2 (pdev, dvsec, ctrl2);
So with them squashed inline this would just be
{
u16 ctrl2;
int rc;
rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2);
if (rc)
return pcibios_err_to_errno(rc);
/* Comment on why this is masked */
ctrl2 &= ~(PCI_DVSEC_CXL_INIT_CACHE_WBI | PCI_DVSEC_CXL_INIT_CXL_RST);
ctrl2 &= ~clear;
ctrl2 |= set;
rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2);
if (rc)
return pcibios_err_to_errno(rc);
return 0;
}which is if anything easier to read.
Quoted 23 lines
drivers/cxl/core/resource.c +20
-0+} + +static int cxl_reset_enable_cache (struct pci_dev *pdev,int dvsec) +{ +return cxl_reset_modify_ctrl2 (pdev, dvsec,0 , +PCI_DVSEC_CXL_DISABLE_CACHING ); +} + +static int cxl_reset_initiate (struct pci_dev *pdev,int dvsec) +{ +return cxl_reset_modify_ctrl2 (pdev, dvsec, +PCI_DVSEC_CXL_INIT_CXL_RST | +PCI_DVSEC_CXL_RST_MEM_CLR_EN ,0 ); +} + +static int cxl_reset_execute (struct pci_dev *pdev,int dvsec,u16 cap) +{ +bool target_prepared = false; +int rc, rc2; +
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 712
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ rc =cxl_reset_disable_cache (pdev, dvsec, cap); +if (rc) +return rc; + +if (!pci_wait_for_pending_transaction (pdev)) +pci_err (pdev,"timed out waiting for pending transactions\n" ); + + rc =pci_dev_reset_iommu_prepare (pdev); +if (rc) +pci_err (pdev,"failed to stop IOMMU for CXL reset: %d\n" , rc);
Maybe a comment on whether this is even remotely safe to continue. My gut feeling is this sort of thing happens, just give up...
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 721
Quoted 9 lines
drivers/cxl/core/resource.c +9
-0+else + target_prepared = true; + +if (!rc) + rc =cxl_reset_initiate (pdev, dvsec); +if (!rc) + rc =cxl_reset_wait_done (pdev, dvsec, cap); + + rc2 =cxl_reset_enable_cache (pdev, dvsec);
Can't we delay this until after the iommu is told things are back? I'm not keen on the sequence not being a clean tear down then a clean setup in the other order. I may well be missing some subtleties. This maybe needs some documentation.
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 723
Quoted 2 lines
drivers/cxl/core/resource.c +2
-0+if (rc2 && rc) +pci_warn (pdev,"failed to re-enable CXL caching: %d\n" , rc2);
What is logic about not printing if we have another failure sat in rc? Is it not true anyway?
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 725
Quoted 2 lines
drivers/cxl/core/resource.c +2
-0+else if (rc2) + rc = rc2;
This code flow is less than ideal I'd use some gotos rather than that if (!rc) dance.
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 729
Quoted 4 lines
drivers/cxl/core/resource.c +4
-0+ +if (target_prepared) +pci_dev_reset_iommu_done (pdev); +return rc;
return rc ? rc : rc2; and some of the complexity above goes away.
Quoted 16 lines
drivers/cxl/core/resource.c +16
-0+} + +int cxl_reset_function (struct pci_dev *pdev,bool probe) +{ +int dvsec; +u16 cap; + + dvsec =cxl_reset_get_dvsec (pdev, &cap); +if (dvsec <0 ) +return dvsec; + +if (probe) +return 0 ; + +return cxl_reset_execute (pdev, dvsec, cap); +}
Re: [PATCH v12 06/12] cxl: Add CXL Device Reset helper
Quoted 13 lines · Jonathan Cameron
Quoted 10 lines · On Thu, 10 Sep 2026 07:08:02 +0000 Srirangan Madhavan <[email protected]>…
Add an internal CXL Device Reset helper for Type 2 functions that advertise CXL Reset and CXL Reset Memory Clear in the CXL Device DVSEC. The helper disables CXL.cache, performs cache writeback when supported, initiates reset with Memory Clear enabled, waits for completion, and re-enables CXL.cache on exit.
Leave the helper unregistered until range validation and reset-scope validation are in place.
Signed-off-by Srirangan Madhavan <[email protected]>
<snip>
Quoted 9 lines
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 522
Quoted 5 lines
drivers/cxl/core/resource.c +5
-0+if (dvsec <0 ) +return dvsec; + +if (!(cap &PCI_DVSEC_CXL_CACHE_CAPABLE ) || + !(cap &PCI_DVSEC_CXL_MEM_CAPABLE ))Why do we need them both? Sure that's type 2, but a non class code matching type3 would I think need the same infrastructure you are building here. That would have cxl.mem but not cxl.cache
- I think some of the CXL SSD prototypes fit in this category.
Not sure if this reset path requires only Type2, but see my related comment below.
Quoted 6 lines
Quoted 6 lines
drivers/cxl/core/resource.c +6
-0+return -ENOTTY ; + +if (!(cap &PCI_DVSEC_CXL_RST_CAPABLE )) +return -ENOTTY ; +if (!(cap &PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE )) +return -ENOTTY ;
Quoted 8 lines
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 536
Quoted 8 lines
drivers/cxl/core/resource.c +8
-0+ + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL , &ctrl); +if (rc) +return pcibios_err_to_errno (rc); + +if (!(ctrl &PCI_DVSEC_CXL_CACHE_ENABLE ) || + !(ctrl &PCI_DVSEC_CXL_MEM_ENABLE )) +return -ENOTTY ;
Type2 cache support is coming and it could be CXL.cache forced to disable if IOMMU enabled and the device not supporting ATS nor belonging to IOMMU identity mapping (Benjamin Cheatham will send this support soon). In that case, I can not see a reason for not allowing a reset assuming CXL.mem would be still usable independently of CXL.cache functionality.
Re: [PATCH v12 06/12] cxl: Add CXL Device Reset helper
Quoted 9 lines · Alejandro
Quoted 7 lines
Quoted 7 lines
+ rc =
pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL, &ctrl); + if (rc)
+ returnpcibios_err_to_errno(rc);
+
+ if (!(ctrl &PCI_DVSEC_CXL_CACHE_ENABLE) ||
+ !(ctrl &PCI_DVSEC_CXL_MEM_ENABLE))
+ return-ENOTTY;
Quoted 6 lines
Type2 cache support is coming and it could be CXL.cache forced to disable if IOMMU enabled and the device not supporting ATS nor belonging to IOMMU identity mapping (Benjamin Cheatham will send this support soon). In that case, I can not see a reason for not allowing a reset assuming CXL.mem would be still usable independently of CXL.cache functionality.
Alejandro,
I forgot to fold this into my earlier reply to Jonathan.
Ack, on the requirement.
But before changing the eligibility check, I would like to confirm whether the existing reset policy around cache invalidation & memory handling otherwise applies unchanged to this configuration.
I left the v13 checks unchanged for now, I'll update based on the pointers.
Signature · 2 lines
Regards, Srirangan
Re: [PATCH v12 06/12] cxl: Add CXL Device Reset helper
In reply to Jonathan CameronQuoted 243 lines · Jonathan Cameron
External email: Use caution opening links or attachments
Quoted 10 lines · On Thu, 10 Sep 2026 07:08:02 +0000 Srirangan Madhavan <[email protected]>…
Add an internal CXL Device Reset helper for Type 2 functions that advertise CXL Reset and CXL Reset Memory Clear in the CXL Device DVSEC. The helper disables CXL.cache, performs cache writeback when supported, initiates reset with Memory Clear enabled, waits for completion, and re-enables CXL.cache on exit.
Leave the helper unregistered until range validation and reset-scope validation are in place.
Signed-off-by Srirangan Madhavan <[email protected]>A few things inline.
Quoted 34 lines
--- drivers/cxl/core/resource.c | 251 ++++++++++++++++++++++++++++++++++ include/cxl/cxl.h | 7 + include/uapi/linux/pci_regs.h | 14 ++ 3 files changed, 272 insertions(+)drivers/cxl/core/resource.c +13
-0@@ -8,6 +8,8 @@#include <linux/export.h> #include <linux/io.h> #include <linux/ioport.h> +#include <linux/iommu.h> +#include <linux/jiffies.h> #include <linux/kernel.h> #include <linux/pci.h> #include <linux/slab.h> @@ -492,3 +494,252 @@ void pci_cxl_hdm_init(struct pci_dev *pdev)if (rc && rc != -ENOTTY && rc != -ENODEV )pci_dbg (pdev,"CXL HDM cache init failed: %d\n" , rc); } +/* +* CXL r4.0 sec 9.7.2 defines the reset completion timeout encodings. +* Sec 9.7.3 leaves config-space access behavior undefined for 100 ms after +* initiating CXL Reset, then limits software to CXL Status2 access until +* reset completion, timeout, or error. +*/ +#define CXL_RESET_RRS_WAIT_MS 100 +#define CXL_RESET_STATUS_POLL_MS 20 +static const u32 cxl_reset_timeout_ms[] = { +10 ,100 ,1000 ,10000 ,100000 , +};drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 517
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ +#define CXL_CACHE_WBI_TIMEOUT_US 100000 +#define CXL_CACHE_WBI_POLL_US 100 + +static int cxl_reset_get_dvsec (struct pci_dev *pdev,u16 *cap_out) +{ +int dvsec, rc; +u16 cap, ctrl; + + dvsec =cxl_pci_get_device_dvsec_cap (pdev,0 , &cap);That is an odd function given the control read is separate - why should we wrap up the cap read?
I'd just drop this helper and open code the search for the extended cap at the few callsites. A little more code, but less odd and all standard PCI stuff not needing an extra helper.
Probably a bit of code evolution caused this given you are up at v12!
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 522
Quoted 5 lines
drivers/cxl/core/resource.c +5
-0+if (dvsec <0 ) +return dvsec; + +if (!(cap &PCI_DVSEC_CXL_CACHE_CAPABLE ) || + !(cap &PCI_DVSEC_CXL_MEM_CAPABLE ))Why do we need them both? Sure that's type 2, but a non class code matching type3 would I think need the same infrastructure you are building here. That would have cxl.mem but not cxl.cache
- I think some of the CXL SSD prototypes fit in this category.Quoted 21 lines
drivers/cxl/core/resource.c +21
-0+return -ENOTTY ; + +if (!(cap &PCI_DVSEC_CXL_RST_CAPABLE )) +return -ENOTTY ; +if (!(cap &PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE )) +return -ENOTTY ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL , &ctrl); +if (rc) +return pcibios_err_to_errno (rc); + +if (!(ctrl &PCI_DVSEC_CXL_CACHE_ENABLE ) || + !(ctrl &PCI_DVSEC_CXL_MEM_ENABLE )) +return -ENOTTY ; + + *cap_out = cap; +return dvsec; +} + +#define CXL_RESET_CTRL2_CMD_MASK \ + (PCI_DVSEC_CXL_INIT_CACHE_WBI | PCI_DVSEC_CXL_INIT_CXL_RST)drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 553
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ +static int cxl_reset_read_ctrl2 (struct pci_dev *pdev,int dvsec,u16 *ctrl2) +{ +int rc; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL2 , ctrl2); +if (rc) +return pcibios_err_to_errno (rc); + + *ctrl2 &= ~CXL_RESET_CTRL2_CMD_MASK ;I'd rename this. It is doing more than reading crl2. Or push the code inline and avoid need for any name bikeshedding.
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 563
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+return 0 ; +} + +static int cxl_reset_write_ctrl2 (struct pci_dev *pdev,int dvsec,u16 ctrl2) +{ +int rc; + + rc =pci_write_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL2 , ctrl2); +if (rc) +return pcibios_err_to_errno (rc);This helper is also providing little value
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 569
Quoted 6 lines
drivers/cxl/core/resource.c +6
-0+ +return 0 ; +} + +static int cxl_reset_modify_ctrl2 (struct pci_dev *pdev,int dvsec,u16 set, +u16 clear)This needs a rename because it has that subtle mask of previous state. What is it actually allowing you to modify?
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 580
Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+{ +u16 ctrl2; +int rc; + + rc =cxl_reset_read_ctrl2 (pdev, dvsec, &ctrl2); +if (rc) +return rc; + + ctrl2 &= ~clear; + ctrl2 |= set; +return cxl_reset_write_ctrl2 (pdev, dvsec, ctrl2);So with them squashed inline this would just be
{ u16 ctrl2; int rc; rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2); if (rc) return pcibios_err_to_errno(rc); /* Comment on why this is masked */ ctrl2 &= ~(PCI_DVSEC_CXL_INIT_CACHE_WBI | PCI_DVSEC_CXL_INIT_CXL_RST); ctrl2 &= ~clear; ctrl2 |= set; rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2); if (rc) return pcibios_err_to_errno(rc); return 0; }which is if anything easier to read.
Quoted 23 lines
drivers/cxl/core/resource.c +20
-0+} + +static int cxl_reset_enable_cache (struct pci_dev *pdev,int dvsec) +{ +return cxl_reset_modify_ctrl2 (pdev, dvsec,0 , +PCI_DVSEC_CXL_DISABLE_CACHING ); +} + +static int cxl_reset_initiate (struct pci_dev *pdev,int dvsec) +{ +return cxl_reset_modify_ctrl2 (pdev, dvsec, +PCI_DVSEC_CXL_INIT_CXL_RST | +PCI_DVSEC_CXL_RST_MEM_CLR_EN ,0 ); +} + +static int cxl_reset_execute (struct pci_dev *pdev,int dvsec,u16 cap) +{ +bool target_prepared = false; +int rc, rc2; +drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 712
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ rc =cxl_reset_disable_cache (pdev, dvsec, cap); +if (rc) +return rc; + +if (!pci_wait_for_pending_transaction (pdev)) +pci_err (pdev,"timed out waiting for pending transactions\n" ); + + rc =pci_dev_reset_iommu_prepare (pdev); +if (rc) +pci_err (pdev,"failed to stop IOMMU for CXL reset: %d\n" , rc);Maybe a comment on whether this is even remotely safe to continue. My gut feeling is this sort of thing happens, just give up...
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 721
Quoted 9 lines
drivers/cxl/core/resource.c +9
-0+else + target_prepared = true; + +if (!rc) + rc =cxl_reset_initiate (pdev, dvsec); +if (!rc) + rc =cxl_reset_wait_done (pdev, dvsec, cap); + + rc2 =cxl_reset_enable_cache (pdev, dvsec);Can't we delay this until after the iommu is told things are back? I'm not keen on the sequence not being a clean tear down then a clean setup in the other order. I may well be missing some subtleties. This maybe needs some documentation.
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 723
Quoted 2 lines
drivers/cxl/core/resource.c +2
-0+if (rc2 && rc) +pci_warn (pdev,"failed to re-enable CXL caching: %d\n" , rc2);What is logic about not printing if we have another failure sat in rc? Is it not true anyway?
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 725
Quoted 2 lines
drivers/cxl/core/resource.c +2
-0+else if (rc2) + rc = rc2;This code flow is less than ideal I'd use some gotos rather than that if (!rc) dance.
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 729
Quoted 4 lines
drivers/cxl/core/resource.c +4
-0+ +if (target_prepared) +pci_dev_reset_iommu_done (pdev); +return rc;return rc ? rc : rc2; and some of the complexity above goes away.
Quoted 17 lines
Quoted 6 lines
drivers/cxl/core/resource.c +6
-0+} + +int cxl_reset_function (struct pci_dev *pdev,bool probe) +{ +int dvsec; +u16 cap;drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 745
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ + dvsec =cxl_reset_get_dvsec (pdev, &cap); +if (dvsec <0 ) +return dvsec; + +if (probe) +return 0 ; + +return cxl_reset_execute (pdev, dvsec, cap); +}
The fixes for these now are in v13 patch 10:
- removed cxl_reset_get_dvsec() and open-coded the standard DVSEC lookup;
- removed the separate Control2 read and write wrappers;
- renamed the remaining update helper tocxl_reset_update_ctrl2_no_replay() and documented why command bits are masked;
- abort reset when IOMMU preparation fails;
- moved cache-policy restoration until after IOMMU exclusion is released
- replaced the if (!rc) flow with explicit unwind paths while
preserving the primary error.
https://lore.kernel.org/linux-cxl/[email protected]/
Signature · 2 lines
Regards, Srirangan
Re: [PATCH v12 06/12] cxl: Add CXL Device Reset helper
In reply to Jonathan CameronQuoted 9 lines · Jonathan Cameron
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 522
Quoted 5 lines
drivers/cxl/core/resource.c +5
-0+if (dvsec <0 ) +return dvsec; + +if (!(cap &PCI_DVSEC_CXL_CACHE_CAPABLE ) || + !(cap &PCI_DVSEC_CXL_MEM_CAPABLE ))Why do we need them both? Sure that's type 2, but a non class code matching type3 would I think need the same infrastructure you are building here. That would have cxl.mem but not cxl.cache
- I think some of the CXL SSD prototypes fit in this category.
I did not yet address this in v13 because I did not want to expand the series beyond its original Type 2 scope without settling the expected eligibility policy first.
The current check requires both CXL.cache and CXL.mem capability and both protocols to be enabled. The reset sequence currently assumes that CXL.cache is active and performs cache disable, optional writeback, and subsequent cache-policy restoration. This might need change.
Is the current policies (HDM snapshot, range synchronization, and restoration infrastrcture) adapted by these patches for cxl_reset also apply directly for type 3? If so, I can generalize this. But if Type 3 requires additional ownership or lifecycle handling, can we please address that separately?
Signature · 2 lines
Regards, Srirangan
Jonathan Cameron
maintainerRe: [PATCH v12 06/12] cxl: Add CXL Device Reset helper
Quoted 11 lines · On Tue, 22 Sep 2026 17:20:06 -0700 Srirangan Madhavan <[email protected]>…
Quoted 9 lines · Jonathan Cameron
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 522
Quoted 5 lines
drivers/cxl/core/resource.c +5
-0+if (dvsec <0 ) +return dvsec; + +if (!(cap &PCI_DVSEC_CXL_CACHE_CAPABLE ) || + !(cap &PCI_DVSEC_CXL_MEM_CAPABLE ))Why do we need them both? Sure that's type 2, but a non class code matching type3 would I think need the same infrastructure you are building here. That would have cxl.mem but not cxl.cache
- I think some of the CXL SSD prototypes fit in this category.
Quoted 8 lines
I did not yet address this in v13 because I did not want to expand the series beyond its original Type 2 scope without settling the expected eligibility policy first.
The current check requires both CXL.cache and CXL.mem capability and both protocols to be enabled. The reset sequence currently assumes that CXL.cache is active and performs cache disable, optional writeback, and subsequent cache-policy restoration. This might need change.
Yes, we'd need to sanity check if CXL.cache is active. Probably need to do that anyway because we might have a situation where the driver code isn't done for CXL.cache yet but CXL.mem is in use. I think that's the case for Alejandro's device?
Quoted 4 lines
Is the current policies (HDM snapshot, range synchronization, and restoration infrastrcture) adapted by these patches for
cxl_resetalso apply directly for type 3? If so, I can generalize this.
Yes. That should all apply as a type3+ is a just a type 2 that doesn't support CXL.cache.
Quoted 3 lines
But if Type 3
requires additional ownership or lifecycle handling, can we please address that separately?
Agreed for anything new that is needed but I'm not sure what that would be.
Jonathan
Quoted 1 line
Re: [PATCH v12 06/12] cxl: Add CXL Device Reset helper
Quoted 17 lines · Jonathan Cameron
Quoted 14 lines · On Tue, 22 Sep 2026 17:20:06 -0700 Srirangan Madhavan <[email protected]>…
Quoted 9 lines · Jonathan Cameron
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 522
Quoted 5 lines
drivers/cxl/core/resource.c +5
-0+if (dvsec <0 ) +return dvsec; + +if (!(cap &PCI_DVSEC_CXL_CACHE_CAPABLE ) || + !(cap &PCI_DVSEC_CXL_MEM_CAPABLE ))Why do we need them both? Sure that's type 2, but a non class code matching type3 would I think need the same infrastructure you are building here. That would have cxl.mem but not cxl.cache
- I think some of the CXL SSD prototypes fit in this category.I did not yet address this in v13 because I did not want to expand the series beyond its original Type 2 scope without settling the expected eligibility policy first.
Quoted 8 lines
Quoted 4 lines
The current check requires both CXL.cache and CXL.mem capability and both protocols to be enabled. The reset sequence currently assumes that CXL.cache is active and performs cache disable, optional writeback, and subsequent cache-policy restoration. This might need change.
Yes, we'd need to sanity check if CXL.cache is active. Probably need to do that anyway because we might have a situation where the driver code isn't done for CXL.cache yet but CXL.mem is in use. I think that's the case for Alejandro's device?
Well, not exactly. It is true the driver does not handle CXL.cache, and it does not make use of this capability, but the protocol is enabled and ready between the device and the upstream port.
I think this can be a bit more complicated since that upstream port could need to disable cache as well, what is what I did comment on discord yesterday. Current firmware/BIOS does enable CXL.cache between them, assuming here an endpoint connected to a root port, so the protocol (and the potential and uncontrollable path to host memory) open.
If I am right and we end up disabling CXL.cache in the endpoint and in the root port, I think the reset should mimic same paranoic measure implying to restore (and I guess to keep/cache beforehand) the upstream port cache state.
Quoted 14 lines
Quoted 3 lines
Is the current policies (HDM snapshot, range synchronization, and restoration infrastrcture) adapted by these patches for
cxl_resetalso apply directly for type 3? If so, I can generalize this.Yes. That should all apply as a type3+ is a just a type 2 that doesn't support CXL.cache.
Quoted 3 lines
But if Type 3
requires additional ownership or lifecycle handling, can we please address that separately?Agreed for anything new that is needed but I'm not sure what that would be.
Jonathan
PATCH v12 7/12 cxl: Validate HDM ranges before CXL reset
In reply to Srirangan MadhavanBefore reset, require cached HDM decoder state, collect enabled decoder ranges, and reserve them with request_mem_region(). This rejects reset while affected CXL memory is busy and keeps the validation stable through reset.
If CPU cache invalidation support is available, invalidate the affected ranges before reset. If the runtime backend is unavailable, continue after the range reservation succeeds.
Reject CXL Reset when no cached HDM decoder state is available. The reset path needs the cached address map to validate affected ranges and perform CPU cache invalidation. Also reject normalized-addressing decoders for now because the cached decoder range is not a system physical address.
--- drivers/cxl/core/resource.c | 255 +++++++++++++++++++++++++++++++++++- 1 file changed, 254 insertions(+), 1 deletion(-)
drivers/cxl/core/resource.c +254 -1
@@ -11,6 +11,8 @@#include <linux/iommu.h> #include <linux/jiffies.h> #include <linux/kernel.h> +#include <linux/list.h> +#include <linux/memregion.h> #include <linux/pci.h> #include <linux/slab.h>
@@ -509,6 +511,229 @@ static const u32 cxl_reset_timeout_ms[] = {#define CXL_CACHE_WBI_TIMEOUT_US 100000#define CXL_CACHE_WBI_POLL_US 100 +struct cxl_hdm_range { +struct list_head list; +struct pci_dev *pdev; +struct range hpa_range; +u64 len; +struct resource *res; +}; + +struct cxl_hdm_range_context { +struct list_head ranges; +}; + +static void cxl_hdm_range_context_init (struct cxl_hdm_range_context *ctx) +{ +INIT_LIST_HEAD (&ctx->ranges); +} + +static void cxl_hdm_range_context_destroy (struct cxl_hdm_range_context *ctx) +{ +struct cxl_hdm_range *range, *next; + +list_for_each_entry_safe (range, next, &ctx->ranges, list) { +list_del (&range->list); +if (range->res) +release_mem_region (range->hpa_range.start, +resource_size (range->res)); +kfree (range); + } +} + +/* +* Bound the range twice: request_mem_region() takes resource_size_t while +* cpu_cache_invalidate_memregion() takes size_t, and the two differ on +* 32-bit builds with CONFIG_PHYS_ADDR_T_64BIT. range_len() can also reach +* RESOURCE_SIZE_MAX + 1 for a full-width range, and wraps to zero when +* resource_size_t is 64-bit, which the !len test catches. +*/ +static int cxl_hdm_range_validate (struct pci_dev *pdev, +const struct range *hpa_range) +{ +u64 len =range_len (hpa_range); + +if (!len) +return -EINVAL ; + +if (hpa_range->end >RESOURCE_SIZE_MAX ) { +pci_err (pdev, +"CXL reset range [%#llx-%#llx] exceeds resource address size\n" , + hpa_range->start, hpa_range->end); +return -EOVERFLOW ; + } + +if (len >RESOURCE_SIZE_MAX ) { +pci_err (pdev, +"CXL reset range [%#llx-%#llx] exceeds resource size\n" , + hpa_range->start, hpa_range->end); +return -EOVERFLOW ; + } + +if (len >SIZE_MAX ) { +pci_err (pdev, +"CXL reset range [%#llx-%#llx] exceeds cache flush size\n" , + hpa_range->start, hpa_range->end); +return -EOVERFLOW ; + } + +return 0 ; +} + +static int cxl_hdm_range_add (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev,const struct range *hpa_range) +{ +struct cxl_hdm_range *range; +int rc; + + rc =cxl_hdm_range_validate (pdev, hpa_range); +if (rc) +return rc; + +list_for_each_entry (range, &ctx->ranges, list) +if (range->hpa_range.start == hpa_range->start && + range->hpa_range.end == hpa_range->end) +return 0 ; + + range =kzalloc_obj (*range); +if (!range) +return -ENOMEM ; + + range->pdev = pdev; + range->hpa_range = *hpa_range; + range->len =range_len (hpa_range); +list_add_tail (&range->list, &ctx->ranges); + +return 0 ; +} + +static int cxl_hdm_ranges_collect (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev) +{ +struct cxl_hdm_info *info; +int rc; + +guard (rwsem_read)(&cxl_rwsem.dpa); + info = pdev->hdm; +if (!info) { +pci_err (pdev,"CXL HDM decoder state unavailable\n" ); +return -ENXIO ; + } + +for (int i =0 ; i < info->decoder_count; i++) { +struct cxl_decoder_settings *settings = &info->settings[i]; + +if (!(settings->flags &CXL_DECODER_F_ENABLE )) +continue ; + +/* A committed zero-size decoder maps no HPA. */ +if (!range_len (&settings->hpa_range)) +continue ; + +if (settings->flags &CXL_DECODER_F_NORMALIZED_ADDRESSING ) { +pci_err (pdev, +"CXL reset does not support normalized address decoders\n" ); +return -EOPNOTSUPP ; + } + + rc =cxl_hdm_range_add (ctx, pdev, &settings->hpa_range); +if (rc) +return rc; + } + +return 0 ; +} + +static int cxl_hdm_range_request (struct cxl_hdm_range *range) +{ +struct pci_dev *pdev = range->pdev; +const struct range *hpa_range = &range->hpa_range; + + range->res =request_mem_region (hpa_range->start, range->len, +"cxl_reset" ); +if (!range->res) { +pci_err (pdev, +"cannot reset while CXL memory range is busy [%#llx-%#llx]\n" , + hpa_range->start, hpa_range->end); +return -EBUSY ; + } + +return 0 ; +} + +static int cxl_hdm_ranges_request (struct cxl_hdm_range_context *ctx) +{ +struct cxl_hdm_range *range; +int rc; + +lockdep_assert_held_write (&cxl_rwsem.region); + +list_for_each_entry (range, &ctx->ranges, list) { + rc =cxl_hdm_range_request (range); +if (rc) +return rc; + } + +return 0 ; +} + +static int cxl_hdm_range_flush_cache (struct cxl_hdm_range *range) +{ +struct pci_dev *pdev = range->pdev; +const struct range *hpa_range = &range->hpa_range; +int rc; + + rc =cpu_cache_invalidate_memregion (hpa_range->start, range->len); +if (rc) +pci_err (pdev, +"failed to invalidate CPU cache [%#llx-%#llx]: %d\n" , + hpa_range->start, hpa_range->end, rc); + +return rc; +} + +static int cxl_hdm_ranges_flush_cpu_caches (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev) +{ +struct cxl_hdm_range *range; +int rc; + +if (list_empty (&ctx->ranges)) +return 0 ; + +if (!cpu_cache_has_invalidate_memregion ()) { +pci_warn (pdev, +"CPU cache synchronization unavailable; continuing without cache invalidation\n" ); +return 0 ; + } + +list_for_each_entry (range, &ctx->ranges, list) { + rc =cxl_hdm_range_flush_cache (range); +if (rc) +return rc; + } + +return 0 ; +} + +static int cxl_hdm_ranges_prepare (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev) +{ +int rc; + +lockdep_assert_held_write (&cxl_rwsem.region); + + rc =cxl_hdm_ranges_collect (ctx, pdev); +if (rc) +return rc; + + rc =cxl_hdm_ranges_request (ctx); +if (rc) +return rc; + +return cxl_hdm_ranges_flush_cpu_caches (ctx, pdev); +} +static int cxl_reset_get_dvsec (struct pci_dev *pdev,u16 *cap_out) {int dvsec, rc;
@@ -539,6 +764,20 @@ static int cxl_reset_get_dvsec(struct pci_dev *pdev, u16 *cap_out)return dvsec; } +static bool cxl_reset_hdm_available (struct pci_dev *pdev) +{ +struct cxl_hdm_info *info; + +/* +* pdev->hdm is owned by the PCI device and released with pci_dev, so +* reset-method probes and reset requests can test availability without +* a CXL driver bound to the device. +*/ +guard (rwsem_read)(&cxl_rwsem.dpa); + info = pdev->hdm; +return info && info->hdm_size; +} +#define CXL_RESET_CTRL2_CMD_MASK \ (PCI_DVSEC_CXL_INIT_CACHE_WBI | PCI_DVSEC_CXL_INIT_CXL_RST)
@@ -731,7 +970,9 @@ static int cxl_reset_execute(struct pci_dev *pdev, int dvsec, u16 cap)int cxl_reset_function (struct pci_dev *pdev,bool probe) { +struct cxl_hdm_range_context range_ctx;int dvsec; +int rc;u16 cap; dvsec =cxl_reset_get_dvsec (pdev, &cap);
@@ -741,5 +982,17 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)if (probe)return 0 ; -return cxl_reset_execute (pdev, dvsec, cap); +if (!cxl_reset_hdm_available (pdev)) +return -ENOTTY ; + +cxl_hdm_range_context_init (&range_ctx); + + scoped_guard(rwsem_write, &cxl_rwsem.region) { + rc =cxl_hdm_ranges_prepare (&range_ctx, pdev); +if (!rc) + rc =cxl_reset_execute (pdev, dvsec, cap); +cxl_hdm_range_context_destroy (&range_ctx); + } + +return rc; }
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 07/12] cxl: Validate HDM ranges before CXL reset
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
Before reset, require cached HDM decoder state, collect enabled decoder ranges, and reserve them with
request_mem_region(). This rejects reset while affected CXL memory is busy and keeps the validation stable through reset.If CPU cache invalidation support is available, invalidate the affected ranges before reset. If the runtime backend is unavailable, continue after the range reservation succeeds.
Quoted 6 lines
Reject CXL Reset when no cached HDM decoder state is available. The reset path needs the cached address map to validate affected ranges and perform CPU cache invalidation. Also reject normalized-addressing decoders for now because the cached decoder range is not a system physical address.
Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Jonathan Cameron
maintainerRe: [PATCH v12 07/12] cxl: Validate HDM ranges before CXL reset
In reply to Srirangan MadhavanQuoted 7 lines · On Thu, 10 Sep 2026 07:08:03 +0000 Srirangan Madhavan <[email protected]>…
Before reset, require cached HDM decoder state, collect enabled decoder ranges, and reserve them with
request_mem_region(). This rejects reset while affected CXL memory is busy and keeps the validation stable through reset.If CPU cache invalidation support is available, invalidate the affected ranges before reset.
If it is not what happens? I'm thinking dead system as coherency just broke but maybe not. My gut is no reset without cache invalidation support.
In general document your cache invalidation flow in this commit message. I'd suspicious you don't have a pair of invalidations, one before to ensure all caches are clean and one after to evict prefetched lines if your device was mean enough to serve garbage during the reset
(which many will do!)
If more comes in later patches, but a breadcrumb in here to say so.
Quoted 13 lines
If the runtime backend is unavailable, continue
after the range reservation succeeds.Reject CXL Reset when no cached HDM decoder state is available. The reset path needs the cached address map to validate affected ranges and perform CPU cache invalidation. Also reject normalized-addressing decoders for now because the cached decoder range is not a system physical address.
Signed-off-by Srirangan Madhavan <[email protected]>drivers/cxl/core/resource.c +1
-0+static int cxl_hdm_range_flush_cache (struct cxl_hdm_range *range)
drivers/cxl/core/resource.c static const u32 cxl_reset_timeout_ms[] = { · line 690
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+{ +struct pci_dev *pdev = range->pdev; +const struct range *hpa_range = &range->hpa_range; +int rc; + + rc =cpu_cache_invalidate_memregion (hpa_range->start, range->len); +if (rc) +pci_err (pdev, +"failed to invalidate CPU cache [%#llx-%#llx]: %d\n" , + hpa_range->start, hpa_range->end, rc);
What does the wrapper really bring?
Quoted 6 lines
drivers/cxl/core/resource.c +6
-0+ +return rc; +} + +static int cxl_hdm_ranges_flush_cpu_caches (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev)
drivers/cxl/core/resource.c static const u32 cxl_reset_timeout_ms[] = { · line 706
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+{ +struct cxl_hdm_range *range; +int rc; + +if (list_empty (&ctx->ranges)) +return 0 ; + +if (!cpu_cache_has_invalidate_memregion ()) { +pci_warn (pdev, +"CPU cache synchronization unavailable; continuing without cache invalidation\n" );
boom. Unless you have very strong reasonsing for me this is a no. This introduces extremely subtle cache coherency corruption into your system and no one wants to debug results of that.
Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+return 0 ; + } + +list_for_each_entry (range, &ctx->ranges, list) { + rc =cxl_hdm_range_flush_cache (range); +if (rc) +return rc; + } + +return 0 ; +}
Re: [PATCH v12 07/12] cxl: Validate HDM ranges before CXL reset
Quoted 71 lines · Jonathan Cameron
External email: Use caution opening links or attachments
Quoted 7 lines · On Thu, 10 Sep 2026 07:08:03 +0000 Srirangan Madhavan <[email protected]>…
Before reset, require cached HDM decoder state, collect enabled decoder ranges, and reserve them with
request_mem_region(). This rejects reset while affected CXL memory is busy and keeps the validation stable through reset.If CPU cache invalidation support is available, invalidate the affected ranges before reset.
If it is not what happens? I'm thinking dead system as coherency just broke but maybe not. My gut is no reset without cache invalidation support.
In general document your cache invalidation flow in this commit message. I'd suspicious you don't have a pair of invalidations, one before to ensure all caches are clean and one after to evict prefetched lines if your device was mean enough to serve garbage during the reset
(which many will do!)If more comes in later patches, but a breadcrumb in here to say so.
Quoted 13 lines
If the runtime backend is unavailable, continue
after the range reservation succeeds.Reject CXL Reset when no cached HDM decoder state is available. The reset path needs the cached address map to validate affected ranges and perform CPU cache invalidation. Also reject normalized-addressing decoders for now because the cached decoder range is not a system physical address.
Signed-off-by Srirangan Madhavan <[email protected]>drivers/cxl/core/resource.c +1
-0+static int cxl_hdm_range_flush_cache (struct cxl_hdm_range *range)drivers/cxl/core/resource.c static const u32 cxl_reset_timeout_ms[] = { · line 690
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+{ +struct pci_dev *pdev = range->pdev; +const struct range *hpa_range = &range->hpa_range; +int rc; + + rc =cpu_cache_invalidate_memregion (hpa_range->start, range->len); +if (rc) +pci_err (pdev, +"failed to invalidate CPU cache [%#llx-%#llx]: %d\n" , + hpa_range->start, hpa_range->end, rc);What does the wrapper really bring?
Quoted 6 lines
drivers/cxl/core/resource.c +6
-0+ +return rc; +} + +static int cxl_hdm_ranges_flush_cpu_caches (struct cxl_hdm_range_context *ctx, +struct pci_dev *pdev)drivers/cxl/core/resource.c static const u32 cxl_reset_timeout_ms[] = { · line 706
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+{ +struct cxl_hdm_range *range; +int rc; + +if (list_empty (&ctx->ranges)) +return 0 ; + +if (!cpu_cache_has_invalidate_memregion ()) { +pci_warn (pdev, +"CPU cache synchronization unavailable; continuing without cache invalidation\n" );boom. Unless you have very strong reasonsing for me this is a no. This introduces extremely subtle cache coherency corruption into your system and no one wants to debug results of that.
Quoted 12 lines
drivers/cxl/core/resource.c static const u32 cxl_reset_timeout_ms[] = { · line 717
Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+return 0 ; + } + +list_for_each_entry (range, &ctx->ranges, list) { + rc =cxl_hdm_range_flush_cache (range); +if (rc) +return rc; + } + +return 0 ; +}
These go into the v13 patch 11 now. I have:
- reset is rejected when active HDM ranges exist and CPU cache
invalidation is unavailable;
- the affected ranges are invalidated before reset and again after
state restoration;
- range reservations and IOMMU exclusion remain active through the second invalidation; and
- added description for invalidation sequence in the commit message.
The previous wrapper around one invalidation call is now a range-context helper used by both invalidation passes.
https://lore.kernel.org/linux-cxl/[email protected]/
Signature · 2 lines
Regards, Srirangan
PATCH v12 8/12 PCI/CXL: Reject CXL Reset on multifunction devices
In reply to Srirangan MadhavanCXL Reset can be exposed as a PCI function reset method only when the reset is function-scoped. PCI reset-method probing runs during enumeration, so walking sibling functions there can race with functions that have not been added to the bus yet.
Reject multifunction devices and devices with enabled VFs instead of trying to infer reset scope from sibling state. Check both conditions on each invocation because VF enablement can change after reset-method probing. Systems that can prove a multifunction device has function-scoped CXL Reset can be enabled later.
--- drivers/cxl/core/resource.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/cxl/core/resource.c +3 -0
@@ -979,6 +979,9 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)if (dvsec <0 )return dvsec; +if (pdev->multifunction ||pci_num_vf (pdev)) +return -ENOTTY ; +if (probe)return 0 ;
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 08/12] PCI/CXL: Reject CXL Reset on multifunction devices
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 5 lines
CXL Reset can be exposed as a PCI function reset method only when the reset is function-scoped. PCI reset-method probing runs during enumeration, so walking sibling functions there can race with functions that have not been added to the bus yet.
Quoted 7 lines
Reject multifunction devices and devices with enabled VFs instead of trying to infer reset scope from sibling state. Check both conditions on each invocation because VF enablement can change after reset-method probing. Systems that can prove a multifunction device has function-scoped CXL Reset can be enabled later.
Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
PATCH v12 9/12 cxl: Restore CXL state after PCI reset
In reply to Srirangan MadhavanAfter CXL reset, restore PCI config state enough to reach HDM MMIO, restore cached global and per-decoder HDM state, restore the cached CXL.cache and CXL.mem enable bits, and then run the normal PCI restore callbacks.
If reset clears a previously locked decoder, restore and commit its programming before reapplying the cached lock. Leave a locked committed decoder unchanged when that state survives reset.
Keep the target IOMMU reset block active until CXL state restore completes so Bus Master Enable cannot reopen DMA before decoder state is valid.
--- drivers/cxl/core/resource.c | 411 ++++++++++++++++++++++++++++++++++-- 1 file changed, 396 insertions(+), 15 deletions(-)
drivers/cxl/core/resource.c +396 -15
@@ -13,6 +13,7 @@#include <linux/kernel.h> #include <linux/list.h> #include <linux/memregion.h> +#include <linux/overflow.h> #include <linux/pci.h> #include <linux/slab.h>
@@ -83,6 +84,26 @@ static int cxld_await_commit(void __iomem *hdm, int id)return -ETIMEDOUT ; } +static int cxld_await_uncommit (void __iomem *hdm,int id) +{ +u32 ctrl; +int i; + +for (i =0 ; i <COMMIT_TIMEOUT_MS ; i++) { + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); +if (FIELD_GET (CXL_HDM_DECODER0_CTRL_COMMIT_ERROR , ctrl)) { + ctrl &= ~CXL_HDM_DECODER0_CTRL_COMMIT ; +writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); +return -EIO ; + } +if (!FIELD_GET (CXL_HDM_DECODER0_CTRL_COMMITTED , ctrl)) +return 0 ; +fsleep (1000 ); + } + +return -ETIMEDOUT ; +} +static int setup_hw_decoder (void __iomem *hdm,struct cxl_decoder_settings *settings) {
@@ -272,6 +293,31 @@ static void __iomem *cxl_pci_hdm_map(struct pci_dev *pdev,return hdm; } +static void __iomem *cxl_pci_hdm_ioremap_current (struct pci_dev *pdev, +int bar, +resource_size_t offset, +resource_size_t size) +{ +resource_size_t hdm_start, bar_len; +void __iomem *hdm; + +if (bar <0 || bar >=PCI_STD_NUM_BARS || !size) +return ERR_PTR (-EINVAL ); + + bar_len =pci_resource_len (pdev, bar); +if (!bar_len || offset > bar_len || size > bar_len - offset) +return ERR_PTR (-ENODEV ); + + hdm_start =pci_resource_start (pdev, bar) + offset; + hdm =ioremap (hdm_start, size); +if (!hdm) { +pci_err (pdev,"failed to remap CXL HDM decoder registers\n" ); +return ERR_PTR (-ENOMEM ); + } + +return hdm; +} +static void cxl_pci_hdm_read_decoder_state (struct cxl_hdm_decoder_state *state,void __iomem *hdm,int id) {
@@ -284,6 +330,40 @@ static void cxl_pci_hdm_read_decoder_state(struct cxl_hdm_decoder_state *state, state->target_high =readl (hdm +CXL_HDM_DECODER0_TL_HIGH (id)); } +static int cxl_hdm_enable_mem (struct pci_dev *pdev,u16 *command, +bool *restore_command) +{ +int rc; + + *restore_command = false; + + rc =pci_read_config_word (pdev,PCI_COMMAND , command); +if (rc) +return pcibios_err_to_errno (rc); + +if (*command &PCI_COMMAND_MEMORY ) +return 0 ; + + rc =pci_write_config_word (pdev,PCI_COMMAND , + *command |PCI_COMMAND_MEMORY ); +if (rc) +return pcibios_err_to_errno (rc); + + *restore_command = true; +return 0 ; +} + +static int cxl_hdm_restore_command (struct pci_dev *pdev,u16 command) +{ +int rc; + + rc =pci_write_config_word (pdev,PCI_COMMAND , command); +if (rc) +return pcibios_err_to_errno (rc); + +return 0 ; +} + static int cxl_pci_hdm_read_decoder(struct pci_dev *pdev, struct cxl_hdm_decoder_state *state, struct cxl_decoder_settings *settings,
@@ -496,6 +576,269 @@ void pci_cxl_hdm_init(struct pci_dev *pdev)if (rc && rc != -ENOTTY && rc != -ENODEV )pci_dbg (pdev,"CXL HDM cache init failed: %d\n" , rc); } +static int cxl_hdm_decoder_uncommit (struct pci_dev *pdev,void __iomem *hdm, +int id) +{ +u32 ctrl; +int rc; + + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); +if (PCI_POSSIBLE_ERROR (ctrl)) +return -EIO ; + +if (ctrl &CXL_HDM_DECODER0_CTRL_LOCK ) { +if (ctrl &CXL_HDM_DECODER0_CTRL_COMMITTED ) { +pci_dbg (pdev, +"CXL HDM decoder %d retained locked committed state\n" , + id); +return -EBUSY ; + } + +pci_err (pdev,"CXL HDM decoder %d is locked and uncommitted\n" , + id); +return -EIO ; + } + +if (!(ctrl &CXL_HDM_DECODER0_CTRL_COMMITTED )) +return 0 ; + + ctrl &= ~CXL_HDM_DECODER0_CTRL_COMMIT ; +writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); + + rc =cxld_await_uncommit (hdm, id); +if (rc) +pci_err (pdev,"CXL HDM decoder %d uncommit failed: %d\n" , + id, rc); + +return rc; +} + +static void cxl_restore_hdm_decoder_state (struct cxl_hdm_decoder_state *state, +void __iomem *hdm,int id) +{ +u32 ctrl = state->ctrl; + + ctrl &= ~(CXL_HDM_DECODER0_CTRL_COMMIT | +CXL_HDM_DECODER0_CTRL_COMMITTED | +CXL_HDM_DECODER0_CTRL_COMMIT_ERROR | +CXL_HDM_DECODER0_CTRL_LOCK ); + +writel (state->base_high, hdm +CXL_HDM_DECODER0_BASE_HIGH_OFFSET (id)); +writel (state->base_low, hdm +CXL_HDM_DECODER0_BASE_LOW_OFFSET (id)); +writel (state->size_high, hdm +CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (id)); +writel (state->size_low, hdm +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (id)); +writel (state->target_high, hdm +CXL_HDM_DECODER0_TL_HIGH (id)); +writel (state->target_low, hdm +CXL_HDM_DECODER0_TL_LOW (id)); +writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (id)); +} + +static int cxl_restore_dvsec_ctrl (struct pci_dev *pdev,u16 saved_ctrl) +{ +u16 mask =PCI_DVSEC_CXL_CACHE_ENABLE | +PCI_DVSEC_CXL_MEM_ENABLE ; +u16 ctrl; +int dvsec; +int rc; + + dvsec =pci_find_dvsec_capability (pdev,PCI_VENDOR_ID_CXL , +PCI_DVSEC_CXL_DEVICE ); +if (!dvsec) +return -ENOTTY ; + + rc =pci_read_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL , &ctrl); +if (rc) +return pcibios_err_to_errno (rc); + + ctrl = (ctrl & ~mask) | (saved_ctrl & mask); + rc =pci_write_config_word (pdev, dvsec +PCI_DVSEC_CXL_CTRL , ctrl); +if (rc) +return pcibios_err_to_errno (rc); + +return 0 ; +} + +static int cxl_restore_hdm_decoder (struct pci_dev *pdev, +struct cxl_hdm_decoder_state *state, +struct cxl_decoder_settings *settings, +void __iomem *hdm) +{ +u32 ctrl; +int rc; + + rc =cxl_hdm_decoder_uncommit (pdev, hdm, settings->id); +if (rc == -EBUSY ) +return 0 ; +if (rc) +return rc; + +cxl_restore_hdm_decoder_state (state, hdm, settings->id); + +if (!(settings->flags &CXL_DECODER_F_ENABLE )) +return 0 ; + +scoped_guard (rwsem_read, &cxl_rwsem.dpa) + rc = cxl_commit_start(hdm, settings); +if (!rc) + rc =cxl_commit_wait (hdm, settings); +if (rc) +pci_err (pdev,"CXL HDM decoder %d restore failed: %d\n" , + settings->id, rc); +else if (settings->flags &CXL_DECODER_F_LOCK ) { + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); + ctrl |=CXL_HDM_DECODER0_CTRL_LOCK ; +writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); + + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); +if (PCI_POSSIBLE_ERROR (ctrl) || + !(ctrl &CXL_HDM_DECODER0_CTRL_LOCK )) { +pci_err (pdev, +"CXL HDM decoder %d failed to restore lock\n" , + settings->id); +return -EIO ; + } + } + +return rc; +} + +static struct cxl_hdm_info *cxl_snapshot_hdm (struct pci_dev *pdev) +{ +struct cxl_hdm_info *snap; +struct cxl_hdm_info *info; +size_t state_sz; + +guard (rwsem_read)(&cxl_rwsem.dpa); + + info = pdev->hdm; +if (!info) +return NULL ; +if (info->decoder_count <0 || + info->decoder_count >CXL_HDM_DECODER_MAX_COUNT || + (info->decoder_count && !info->decoder_state)) +return ERR_PTR (-EINVAL ); + + state_sz =array_size (info->decoder_count,sizeof (*info->decoder_state)); + snap =kzalloc (size_add (sizeof (*snap), state_sz),GFP_KERNEL ); +if (!snap) +return ERR_PTR (-ENOMEM ); + + *snap = *info; + snap->decoder_state = (void *)(snap +1 ); +memcpy (snap->decoder_state, info->decoder_state, state_sz); + +return snap; +} + +static void cxl_restore_pci_state_for_hdm_restore (struct pci_dev *pdev, +u16 *command) +{ +u32 saved_config = pdev->saved_config_space[PCI_COMMAND /4 ]; + + pdev->saved_config_space[PCI_COMMAND /4 ] &= ~PCI_COMMAND_MASTER ; + pdev->saved_config_space[PCI_COMMAND /4 ] |=PCI_COMMAND_INTX_DISABLE ; +pci_restore_state (pdev); + pdev->saved_config_space[PCI_COMMAND /4 ] = saved_config; + *command = saved_config &0xffff ; +} + +static int cxl_restore_state (struct pci_dev *pdev) +{ +struct cxl_hdm_info *snap =cxl_snapshot_hdm (pdev); +bool restore_command = false; +void __iomem *hdm; +int first_rc =0 ; +u16 command; +int rc; + +if (!snap) +return 0 ; +if (IS_ERR (snap)) +return PTR_ERR (snap); + + rc =cxl_hdm_enable_mem (pdev, &command, &restore_command); +if (rc) { +kfree (snap); +return rc; + } + + hdm =cxl_pci_hdm_ioremap_current (pdev, snap->hdm_bar, + snap->hdm_offset, snap->hdm_size); +if (IS_ERR (hdm)) { + first_rc =PTR_ERR (hdm); + }else { +/* +* Restore global HDM control before per-decoder commit. PCI +* config memory decoding is enabled for MMIO access, but bus +* mastering remains disabled until HDM restore completes. +*/ +writel (snap->global_ctrl, hdm +CXL_HDM_DECODER_CTRL_OFFSET ); + +for (int i =0 ; i < snap->decoder_count; i++) { + rc =cxl_restore_hdm_decoder (pdev, + &snap->decoder_state[i], + &snap->settings[i], hdm); +if (rc && !first_rc) + first_rc = rc; + } + +/* Flush posted HDM writes before PCI_COMMAND can restore BME. */ +readl (hdm +CXL_HDM_DECODER_CTRL_OFFSET ); +iounmap (hdm); + } + +if (!first_rc && snap->dvsec_ctrl_valid) { + rc =cxl_restore_dvsec_ctrl (pdev, snap->dvsec_ctrl); +if (rc) + first_rc = rc; + } + +if (restore_command) { + rc =cxl_hdm_restore_command (pdev, command); +if (rc && !first_rc) + first_rc = rc; + } + +kfree (snap); +return first_rc; +} + +static void cxl_reset_save_disabled_state (struct pci_dev *pdev) +{ +int rc; + + rc =pci_write_config_word (pdev,PCI_COMMAND ,PCI_COMMAND_INTX_DISABLE ); +if (rc) { +pci_warn (pdev,"failed to keep device disabled after CXL reset restore failure: %d\n" , +pcibios_err_to_errno (rc)); +return ; + } + + rc =pci_save_state (pdev); +if (rc) +pci_warn (pdev,"failed to save disabled state after CXL reset restore failure: %d\n" , + rc); +} + +static int cxl_reset_save_restored_state (struct pci_dev *pdev,u16 command) +{ +int rc; + + rc =cxl_hdm_restore_command (pdev, command); +if (rc) { +cxl_reset_save_disabled_state (pdev); +return rc; + } + + rc =pci_save_state (pdev); +if (rc) { +pci_warn (pdev,"failed to save restored state after CXL reset: %d\n" , + rc); +cxl_reset_save_disabled_state (pdev); + } + +return rc; +} +/* * CXL r4.0 sec 9.7.2 defines the reset completion timeout encodings. * Sec 9.7.3 leaves config-space access behavior undefined for 100 ms after
@@ -523,6 +866,34 @@ struct cxl_hdm_range_context {struct list_head ranges; }; +static void cxl_pci_target_reset_done (struct pci_dev *pdev, +bool *target_prepared) +{ +if (!*target_prepared) +return ; + +pci_dev_reset_iommu_done (pdev); + *target_prepared = false; +} + +static int cxl_pci_target_reset_prepare (struct pci_dev *pdev, +bool *target_prepared) +{ +int rc; + +if (!pci_wait_for_pending_transaction (pdev)) +pci_err (pdev,"timed out waiting for pending transactions\n" ); + + rc =pci_dev_reset_iommu_prepare (pdev); +if (rc) { +pci_err (pdev,"failed to stop IOMMU for CXL reset: %d\n" , rc); +return rc; + } + + *target_prepared = true; +return 0 ; +} +static void cxl_hdm_range_context_init (struct cxl_hdm_range_context *ctx) {INIT_LIST_HEAD (&ctx->ranges);
@@ -934,26 +1305,21 @@ static int cxl_reset_wait_done(struct pci_dev *pdev, int dvsec, u16 cap) }while (true); } -static int cxl_reset_execute (struct pci_dev *pdev,int dvsec,u16 cap) +static int cxl_reset_execute (struct pci_dev *pdev,bool *target_prepared, +bool *reset_initiated,int dvsec,u16 cap) { -bool target_prepared = false;int rc, rc2; rc =cxl_reset_disable_cache (pdev, dvsec, cap);if (rc)return rc; -if (!pci_wait_for_pending_transaction (pdev)) -pci_err (pdev,"timed out waiting for pending transactions\n" ); - - rc =pci_dev_reset_iommu_prepare (pdev); -if (rc) -pci_err (pdev,"failed to stop IOMMU for CXL reset: %d\n" , rc); -else - target_prepared = true; - -if (!rc) + rc =cxl_pci_target_reset_prepare (pdev, target_prepared); +if (!rc) { rc =cxl_reset_initiate (pdev, dvsec); +if (!rc) + *reset_initiated = true; + }if (!rc) rc =cxl_reset_wait_done (pdev, dvsec, cap);
@@ -963,14 +1329,14 @@ static int cxl_reset_execute(struct pci_dev *pdev, int dvsec, u16 cap)else if (rc2) rc = rc2; -if (target_prepared) -pci_dev_reset_iommu_done (pdev);return rc; }int cxl_reset_function (struct pci_dev *pdev,bool probe) {struct cxl_hdm_range_context range_ctx; +bool target_prepared = false; +bool reset_initiated = false;int dvsec;int rc;u16 cap;
@@ -993,9 +1359,24 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe) scoped_guard(rwsem_write, &cxl_rwsem.region) { rc =cxl_hdm_ranges_prepare (&range_ctx, pdev);if (!rc) - rc =cxl_reset_execute (pdev, dvsec, cap); + rc =cxl_reset_execute (pdev, &target_prepared, + &reset_initiated, dvsec, cap); +if (!rc) { +u16 command; + +cxl_restore_pci_state_for_hdm_restore (pdev, &command); + rc =cxl_restore_state (pdev); +if (rc) +cxl_reset_save_disabled_state (pdev); +else + rc =cxl_reset_save_restored_state (pdev, + command); + }else if (reset_initiated) { +cxl_reset_save_disabled_state (pdev); + }cxl_hdm_range_context_destroy (&range_ctx); } +cxl_pci_target_reset_done (pdev, &target_prepared);return rc; }
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 09/12] cxl: Restore CXL state after PCI reset
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 5 lines
After CXL reset, restore PCI config state enough to reach HDM MMIO, restore cached global and per-decoder HDM state, restore the cached CXL.cache and CXL.mem enable bits, and then run the normal PCI restore callbacks.
Quoted 8 lines
If reset clears a previously locked decoder, restore and commit its programming before reapplying the cached lock. Leave a locked committed decoder unchanged when that state survives reset.
Keep the target IOMMU reset block active until CXL state restore completes so Bus Master Enable cannot reopen DMA before decoder state is valid.
Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Jonathan Cameron
maintainerRe: [PATCH v12 09/12] cxl: Restore CXL state after PCI reset
In reply to Srirangan MadhavanQuoted 5 lines · On Thu, 10 Sep 2026 07:08:05 +0000 Srirangan Madhavan <[email protected]>…
After CXL reset, restore PCI config state enough to reach HDM MMIO, restore cached global and per-decoder HDM state, restore the cached CXL.cache and CXL.mem enable bits, and then run the normal PCI restore callbacks.
Quoted 8 lines
If reset clears a previously locked decoder, restore and commit its programming before reapplying the cached lock. Leave a locked committed decoder unchanged when that state survives reset.
Keep the target IOMMU reset block active until CXL state restore completes so Bus Master Enable cannot reopen DMA before decoder state is valid.
Signed-off-by Srirangan Madhavan <[email protected]>
I'm nearly out of time for today and have given quite a bit of feedback on earlier patches. So this is going to be a scan read at most. Make sure to take another look at this and incorporate the sort of changes I've asked for elsewhere for v13.
This seems to carry on papering over the cracks after errors that to me indicate a broken device. I don't see any reason to do that. If a device needs quirks to say don't run a particular restore do that but if they are real errors, just give up - your device is not going to be useable with only some stuff restored and you may be making an obvious bug a much more subtle one.
Jonathan
Quoted 17 lines
--- drivers/cxl/core/resource.c | 411 ++++++++++++++++++++++++++++++++++-- 1 file changed, 396 insertions(+), 15 deletions(-)drivers/cxl/core/resource.c +12
-0+static int cxl_restore_hdm_decoder (struct pci_dev *pdev, +struct cxl_hdm_decoder_state *state, +struct cxl_decoder_settings *settings, +void __iomem *hdm) +{ +u32 ctrl; +int rc; + + rc =cxl_hdm_decoder_uncommit (pdev, hdm, settings->id); +if (rc == -EBUSY ) +return 0 ; +if (rc)
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 681
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+return rc; + +cxl_restore_hdm_decoder_state (state, hdm, settings->id); + +if (!(settings->flags &CXL_DECODER_F_ENABLE )) +return 0 ; + +scoped_guard (rwsem_read, &cxl_rwsem.dpa) + rc = cxl_commit_start(hdm, settings); +if (!rc)
Use a helper function for the stuff in here so you can return directly.
Quoted 28 lines
drivers/cxl/core/resource.c +27
-0+ rc =cxl_commit_wait (hdm, settings); +if (rc) +pci_err (pdev,"CXL HDM decoder %d restore failed: %d\n" , + settings->id, rc); +else if (settings->flags &CXL_DECODER_F_LOCK ) { + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); + ctrl |=CXL_HDM_DECODER0_CTRL_LOCK ; +writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); + + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); +if (PCI_POSSIBLE_ERROR (ctrl) || + !(ctrl &CXL_HDM_DECODER0_CTRL_LOCK )) { +pci_err (pdev, +"CXL HDM decoder %d failed to restore lock\n" , + settings->id); +return -EIO ; + } + } + +return rc; +} + +static void cxl_restore_pci_state_for_hdm_restore (struct pci_dev *pdev, +u16 *command) +{ +u32 saved_config = pdev->saved_config_space[PCI_COMMAND /4 ]; +
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 746
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ pdev->saved_config_space[PCI_COMMAND /4 ] &= ~PCI_COMMAND_MASTER ; + pdev->saved_config_space[PCI_COMMAND /4 ] |=PCI_COMMAND_INTX_DISABLE ; +pci_restore_state (pdev); + pdev->saved_config_space[PCI_COMMAND /4 ] = saved_config; + *command = saved_config &0xffff ; +} + +static int cxl_restore_state (struct pci_dev *pdev) +{ +struct cxl_hdm_info *snap =cxl_snapshot_hdm (pdev);
__free(kfree); looks like it will make life simpler in here.
Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+bool restore_command = false; +void __iomem *hdm; +int first_rc =0 ; +u16 command; +int rc; + +if (!snap) +return 0 ; +if (IS_ERR (snap)) +return PTR_ERR (snap); +
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 767
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ rc =cxl_hdm_enable_mem (pdev, &command, &restore_command); +if (rc) { +kfree (snap); +return rc; + } + + hdm =cxl_pci_hdm_ioremap_current (pdev, snap->hdm_bar, + snap->hdm_offset, snap->hdm_size); +if (IS_ERR (hdm)) { + first_rc =PTR_ERR (hdm);
I'm not really understanding the keep trying and paper over the cracks going on here. If almost any of these fail it looks to me like we are in a bad place and it would be cleaner to report and give up.
I restored 'some state' isn't likely to be very useful to anyone beyond maybe making the breakage subtle rather than major and easy to find.
Quoted 36 lines
drivers/cxl/core/resource.c +36
-0+ }else { +/* +* Restore global HDM control before per-decoder commit. PCI +* config memory decoding is enabled for MMIO access, but bus +* mastering remains disabled until HDM restore completes. +*/ +writel (snap->global_ctrl, hdm +CXL_HDM_DECODER_CTRL_OFFSET ); + +for (int i =0 ; i < snap->decoder_count; i++) { + rc =cxl_restore_hdm_decoder (pdev, + &snap->decoder_state[i], + &snap->settings[i], hdm); +if (rc && !first_rc) + first_rc = rc; + } + +/* Flush posted HDM writes before PCI_COMMAND can restore BME. */ +readl (hdm +CXL_HDM_DECODER_CTRL_OFFSET ); +iounmap (hdm); + } + +if (!first_rc && snap->dvsec_ctrl_valid) { + rc =cxl_restore_dvsec_ctrl (pdev, snap->dvsec_ctrl); +if (rc) + first_rc = rc; + } + +if (restore_command) { + rc =cxl_hdm_restore_command (pdev, command); +if (rc && !first_rc) + first_rc = rc; + } + +kfree (snap); +return first_rc; +}
Re: [PATCH v12 09/12] cxl: Restore CXL state after PCI reset
Quoted 164 lines · Jonathan Cameron
External email: Use caution opening links or attachments
Quoted 5 lines · On Thu, 10 Sep 2026 07:08:05 +0000 Srirangan Madhavan <[email protected]>…
After CXL reset, restore PCI config state enough to reach HDM MMIO, restore cached global and per-decoder HDM state, restore the cached CXL.cache and CXL.mem enable bits, and then run the normal PCI restore callbacks.
Quoted 8 lines
If reset clears a previously locked decoder, restore and commit its programming before reapplying the cached lock. Leave a locked committed decoder unchanged when that state survives reset.
Keep the target IOMMU reset block active until CXL state restore completes so Bus Master Enable cannot reopen DMA before decoder state is valid.
Signed-off-by Srirangan Madhavan <[email protected]>I'm nearly out of time for today and have given quite a bit of feedback on earlier patches. So this is going to be a scan read at most. Make sure to take another look at this and incorporate the sort of changes I've asked for elsewhere for v13.
This seems to carry on papering over the cracks after errors that to me indicate a broken device. I don't see any reason to do that. If a device needs quirks to say don't run a particular restore do that but if they are real errors, just give up - your device is not going to be useable with only some stuff restored and you may be making an obvious bug a much more subtle one.
Jonathan
Quoted 17 lines
--- drivers/cxl/core/resource.c | 411 ++++++++++++++++++++++++++++++++++-- 1 file changed, 396 insertions(+), 15 deletions(-)drivers/cxl/core/resource.c +12
-0+static int cxl_restore_hdm_decoder (struct pci_dev *pdev, +struct cxl_hdm_decoder_state *state, +struct cxl_decoder_settings *settings, +void __iomem *hdm) +{ +u32 ctrl; +int rc; + + rc =cxl_hdm_decoder_uncommit (pdev, hdm, settings->id); +if (rc == -EBUSY ) +return 0 ; +if (rc)drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 681
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+return rc; + +cxl_restore_hdm_decoder_state (state, hdm, settings->id); + +if (!(settings->flags &CXL_DECODER_F_ENABLE )) +return 0 ; + +scoped_guard (rwsem_read, &cxl_rwsem.dpa) + rc = cxl_commit_start(hdm, settings); +if (!rc)Use a helper function for the stuff in here so you can return directly.
Quoted 28 lines
drivers/cxl/core/resource.c +27
-0+ rc =cxl_commit_wait (hdm, settings); +if (rc) +pci_err (pdev,"CXL HDM decoder %d restore failed: %d\n" , + settings->id, rc); +else if (settings->flags &CXL_DECODER_F_LOCK ) { + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); + ctrl |=CXL_HDM_DECODER0_CTRL_LOCK ; +writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); + + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); +if (PCI_POSSIBLE_ERROR (ctrl) || + !(ctrl &CXL_HDM_DECODER0_CTRL_LOCK )) { +pci_err (pdev, +"CXL HDM decoder %d failed to restore lock\n" , + settings->id); +return -EIO ; + } + } + +return rc; +} + +static void cxl_restore_pci_state_for_hdm_restore (struct pci_dev *pdev, +u16 *command) +{ +u32 saved_config = pdev->saved_config_space[PCI_COMMAND /4 ]; +drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 746
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ pdev->saved_config_space[PCI_COMMAND /4 ] &= ~PCI_COMMAND_MASTER ; + pdev->saved_config_space[PCI_COMMAND /4 ] |=PCI_COMMAND_INTX_DISABLE ; +pci_restore_state (pdev); + pdev->saved_config_space[PCI_COMMAND /4 ] = saved_config; + *command = saved_config &0xffff ; +} + +static int cxl_restore_state (struct pci_dev *pdev) +{ +struct cxl_hdm_info *snap =cxl_snapshot_hdm (pdev);
__free(kfree); looks like it will make life simpler in here.Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+bool restore_command = false; +void __iomem *hdm; +int first_rc =0 ; +u16 command; +int rc; + +if (!snap) +return 0 ; +if (IS_ERR (snap)) +return PTR_ERR (snap); +drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 767
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ rc =cxl_hdm_enable_mem (pdev, &command, &restore_command); +if (rc) { +kfree (snap); +return rc; + } + + hdm =cxl_pci_hdm_ioremap_current (pdev, snap->hdm_bar, + snap->hdm_offset, snap->hdm_size); +if (IS_ERR (hdm)) { + first_rc =PTR_ERR (hdm);I'm not really understanding the keep trying and paper over the cracks going on here. If almost any of these fail it looks to me like we are in a bad place and it would be cleaner to report and give up.
I restored 'some state' isn't likely to be very useful to anyone beyond maybe making the breakage subtle rather than major and easy to find.
Quoted 28 lines
drivers/cxl/core/resource.c +28
-0+ }else { +/* +* Restore global HDM control before per-decoder commit. PCI +* config memory decoding is enabled for MMIO access, but bus +* mastering remains disabled until HDM restore completes. +*/ +writel (snap->global_ctrl, hdm +CXL_HDM_DECODER_CTRL_OFFSET ); + +for (int i =0 ; i < snap->decoder_count; i++) { + rc =cxl_restore_hdm_decoder (pdev, + &snap->decoder_state[i], + &snap->settings[i], hdm); +if (rc && !first_rc) + first_rc = rc; + } + +/* Flush posted HDM writes before PCI_COMMAND can restore BME. */ +readl (hdm +CXL_HDM_DECODER_CTRL_OFFSET ); +iounmap (hdm); + } + +if (!first_rc && snap->dvsec_ctrl_valid) { + rc =cxl_restore_dvsec_ctrl (pdev, snap->dvsec_ctrl); +if (rc) + first_rc = rc; + } + +if (restore_command) {
Quoted 9 lines
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 803
Quoted 8 lines
drivers/cxl/core/resource.c +8
-0+ rc =cxl_hdm_restore_command (pdev, command); +if (rc && !first_rc) + first_rc = rc; + } + +kfree (snap); +return first_rc; +}
Hi Jonathan, this patch is now v13's patch 13:
- moved the commit-under-DPA-lock operation into a helper that returns
the result directly;
- stop restoration at the first error instead of continuing with partially restored state
- leave the PCI command state disabled when restoration fails,
preventing the outer PCI path from re-enabling an incompletely restored device.
The snapshot is now taken by the reset caller before reset rather than allocated inside cxl_restore_state(). I kept its cleanup explicit because the caller has staged goto-based cleanup for the snapshot, range
reservations, IOMMU state, and cache policy.
https://lore.kernel.org/linux-cxl/[email protected]/
Signature · 2 lines
Regards, Srirangan
Jonathan Cameron
maintainerRe: [PATCH v12 09/12] cxl: Restore CXL state after PCI reset
Quoted 185 lines · On Tue, 22 Sep 2026 17:10:07 -0700 Srirangan Madhavan <[email protected]>…
Quoted 164 lines · Jonathan Cameron
External email: Use caution opening links or attachments
Quoted 5 lines · On Thu, 10 Sep 2026 07:08:05 +0000 Srirangan Madhavan <[email protected]>…
After CXL reset, restore PCI config state enough to reach HDM MMIO, restore cached global and per-decoder HDM state, restore the cached CXL.cache and CXL.mem enable bits, and then run the normal PCI restore callbacks.
Quoted 8 lines
If reset clears a previously locked decoder, restore and commit its programming before reapplying the cached lock. Leave a locked committed decoder unchanged when that state survives reset.
Keep the target IOMMU reset block active until CXL state restore completes so Bus Master Enable cannot reopen DMA before decoder state is valid.
Signed-off-by Srirangan Madhavan <[email protected]>I'm nearly out of time for today and have given quite a bit of feedback on earlier patches. So this is going to be a scan read at most. Make sure to take another look at this and incorporate the sort of changes I've asked for elsewhere for v13.
This seems to carry on papering over the cracks after errors that to me indicate a broken device. I don't see any reason to do that. If a device needs quirks to say don't run a particular restore do that but if they are real errors, just give up - your device is not going to be useable with only some stuff restored and you may be making an obvious bug a much more subtle one.
Jonathan
Quoted 17 lines
--- drivers/cxl/core/resource.c | 411 ++++++++++++++++++++++++++++++++++-- 1 file changed, 396 insertions(+), 15 deletions(-)drivers/cxl/core/resource.c +12
-0+static int cxl_restore_hdm_decoder (struct pci_dev *pdev, +struct cxl_hdm_decoder_state *state, +struct cxl_decoder_settings *settings, +void __iomem *hdm) +{ +u32 ctrl; +int rc; + + rc =cxl_hdm_decoder_uncommit (pdev, hdm, settings->id); +if (rc == -EBUSY ) +return 0 ; +if (rc)drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 681
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+return rc; + +cxl_restore_hdm_decoder_state (state, hdm, settings->id); + +if (!(settings->flags &CXL_DECODER_F_ENABLE )) +return 0 ; + +scoped_guard (rwsem_read, &cxl_rwsem.dpa) + rc = cxl_commit_start(hdm, settings); +if (!rc)Use a helper function for the stuff in here so you can return directly.
Quoted 28 lines
drivers/cxl/core/resource.c +27
-0+ rc =cxl_commit_wait (hdm, settings); +if (rc) +pci_err (pdev,"CXL HDM decoder %d restore failed: %d\n" , + settings->id, rc); +else if (settings->flags &CXL_DECODER_F_LOCK ) { + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); + ctrl |=CXL_HDM_DECODER0_CTRL_LOCK ; +writel (ctrl, hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); + + ctrl =readl (hdm +CXL_HDM_DECODER0_CTRL_OFFSET (settings->id)); +if (PCI_POSSIBLE_ERROR (ctrl) || + !(ctrl &CXL_HDM_DECODER0_CTRL_LOCK )) { +pci_err (pdev, +"CXL HDM decoder %d failed to restore lock\n" , + settings->id); +return -EIO ; + } + } + +return rc; +} + +static void cxl_restore_pci_state_for_hdm_restore (struct pci_dev *pdev, +u16 *command) +{ +u32 saved_config = pdev->saved_config_space[PCI_COMMAND /4 ]; +drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 746
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ pdev->saved_config_space[PCI_COMMAND /4 ] &= ~PCI_COMMAND_MASTER ; + pdev->saved_config_space[PCI_COMMAND /4 ] |=PCI_COMMAND_INTX_DISABLE ; +pci_restore_state (pdev); + pdev->saved_config_space[PCI_COMMAND /4 ] = saved_config; + *command = saved_config &0xffff ; +} + +static int cxl_restore_state (struct pci_dev *pdev) +{ +struct cxl_hdm_info *snap =cxl_snapshot_hdm (pdev);
__free(kfree); looks like it will make life simpler in here.Quoted 11 lines
drivers/cxl/core/resource.c +11
-0+bool restore_command = false; +void __iomem *hdm; +int first_rc =0 ; +u16 command; +int rc; + +if (!snap) +return 0 ; +if (IS_ERR (snap)) +return PTR_ERR (snap); +drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 767
Quoted 10 lines
drivers/cxl/core/resource.c +10
-0+ rc =cxl_hdm_enable_mem (pdev, &command, &restore_command); +if (rc) { +kfree (snap); +return rc; + } + + hdm =cxl_pci_hdm_ioremap_current (pdev, snap->hdm_bar, + snap->hdm_offset, snap->hdm_size); +if (IS_ERR (hdm)) { + first_rc =PTR_ERR (hdm);I'm not really understanding the keep trying and paper over the cracks going on here. If almost any of these fail it looks to me like we are in a bad place and it would be cleaner to report and give up.
I restored 'some state' isn't likely to be very useful to anyone beyond maybe making the breakage subtle rather than major and easy to find.
Quoted 28 lines
drivers/cxl/core/resource.c +28
-0+ }else { +/* +* Restore global HDM control before per-decoder commit. PCI +* config memory decoding is enabled for MMIO access, but bus +* mastering remains disabled until HDM restore completes. +*/ +writel (snap->global_ctrl, hdm +CXL_HDM_DECODER_CTRL_OFFSET ); + +for (int i =0 ; i < snap->decoder_count; i++) { + rc =cxl_restore_hdm_decoder (pdev, + &snap->decoder_state[i], + &snap->settings[i], hdm); +if (rc && !first_rc) + first_rc = rc; + } + +/* Flush posted HDM writes before PCI_COMMAND can restore BME. */ +readl (hdm +CXL_HDM_DECODER_CTRL_OFFSET ); +iounmap (hdm); + } + +if (!first_rc && snap->dvsec_ctrl_valid) { + rc =cxl_restore_dvsec_ctrl (pdev, snap->dvsec_ctrl); +if (rc) + first_rc = rc; + } + +if (restore_command) {Quoted 9 lines
drivers/cxl/core/resource.c void pci_cxl_hdm_init(struct pci_dev *pdev) · line 803
Quoted 8 lines
drivers/cxl/core/resource.c +8
-0+ rc =cxl_hdm_restore_command (pdev, command); +if (rc && !first_rc) + first_rc = rc; + } + +kfree (snap); +return first_rc; +}Hi Jonathan, this patch is now v13's patch 13:
- moved the commit-under-DPA-lock operation into a helper that returns
the result directly;- stop restoration at the first error instead of continuing with partially restored state
- leave the PCI command state disabled when restoration fails,
preventing the outer PCI path from re-enabling an incompletely restored device.
Quoted 6 lines
The snapshot is now taken by the reset caller before reset rather than allocated inside
cxl_restore_state(). I kept its cleanup explicit because the caller has staged goto-based cleanup for the snapshot, range
reservations, IOMMU state, and cache policy.
Hi. For future replies, put the specific items inline with the feedback they are resolving and aggressively crop out anything that doesn't need to be in the reply. In particular any feedback that you've accepted and will role into next version.
That keeps the focus on the stuff you want to discuss and not on the stuff everyone is agreed on.
I'm not sure if I missed any discussion points in here :(
Jonathan
Quoted 1 line
PATCH v12 10/12 PCI/CXL: Expose CXL Reset as a PCI reset method
In reply to Srirangan MadhavanAdd the CXL Reset helper to the PCI reset-method table so userspace can select it through the existing reset_method ABI.
Advertise the method for Type 2 CXL devices that report CXL Reset and CXL Reset Memory Clear support in the CXL Device DVSEC and pass the function-scope check. Reset execution requires cached HDM decoder state for the target so affected ranges can be validated and HDM programming can be restored. If that state is unavailable at reset time, return -ENOTTY so PCI can try the next reset method.
--- drivers/pci/pci.c | 2 ++ include/linux/pci.h | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
drivers/pci/pci.c +2 -0
@@ -35,6 +35,7 @@#include <linux/aer.h> #include <linux/bitfield.h> #include <linux/suspend.h> +#include <cxl/cxl.h> #include "pci.h" DEFINE_MUTEX (pci_slot_mutex);
@@ -5145,6 +5146,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { }, { pci_dev_specific_reset, .name = "device_specific" }, { pci_dev_acpi_reset, .name = "acpi" }, + { cxl_reset_function, .name = "cxl_reset" }, { pcie_reset_flr, .name = "flr" }, { pci_af_flr, .name = "af_flr" }, { pci_pm_reset, .name = "pm" },
include/linux/pci.h +1 -1
@@ -52,7 +52,7 @@ PCI_STATUS_PARITY)/* Number of reset methods used in pci_reset_fn_methods array in pci.c */ -#define PCI_NUM_RESET_METHODS 8 +#define PCI_NUM_RESET_METHODS 9#define PCI_RESET_PROBE true#define PCI_RESET_DO_RESET false
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 10/12] PCI/CXL: Expose CXL Reset as a PCI reset method
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Add the CXL Reset helper to the PCI reset-method table so userspace can select it through the existing
reset_methodABI.Advertise the method for Type 2 CXL devices that report CXL Reset and CXL Reset Memory Clear support in the CXL Device DVSEC and pass the function-scope check. Reset execution requires cached HDM decoder state for the target so affected ranges can be validated and HDM programming can be restored. If that state is unavailable at reset time, return
-ENOTTYso PCI can try the next reset method.Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
PATCH v12 11/12 Documentation/ABI: Document CXL Reset PCI reset method
In reply to Srirangan MadhavanDocument the "cxl_reset" PCI reset_method value for Type 2 CXL devices and its reset-safety contract. Describe generic PCI driver quiescing, exclusive HDM range reservation, cache and IOMMU handling, function-scope restrictions, and state restoration. Clarify that the method does not offline memory or revoke mappings, and that it requests CXL Reset Memory Clear.
--- Documentation/ABI/testing/sysfs-bus-pci | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +25 -0
@@ -153,6 +153,31 @@ Description: "default" enables all supported reset methods in the default ordering. + If present, "cxl_reset" selects CXL Reset for CXL Type 2 + devices that advertise CXL Reset and CXL Reset Memory Clear + support. The method is only listed for single-function devices, + and reset is rejected if the device has enabled VFs, so CXL Reset + cannot affect another PCI function. + + When reset is triggered through the reset attribute, the generic + PCI reset path calls the device driver's reset_prepare() callback, + when provided, and disables PCI I/O, memory decoding, and bus + mastering before invoking this method. + The method then rejects the operation if cached HDM state is + unavailable or an affected HDM range cannot be exclusively + reserved. It does not offline memory or revoke mappings; CXL.mem + users are expected to retain conflicting iomem resources while + access to an affected range remains active. + + Before issuing CXL Reset, the kernel invalidates CPU caches for + affected HDM ranges where supported, disables CXL.cache, initiates + cache write-back where supported, and blocks IOMMU translations + where applicable. After reset, it restores PCI config state needed + to access HDM MMIO, cached HDM decoder state, and the CXL Device + DVSEC Cache and Mem Enable controls before completing PCI reset + recovery. "cxl_reset" requests CXL Reset Memory Clear so that a + successful generic PCI reset does not preserve tenant memory. + What: /sys/bus/pci/devices/.../reset Date: July 2009 Contact: Michael S. Tsirkin <[email protected]>
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 11/12] Documentation/ABI: Document CXL Reset PCI reset method
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 8 lines
Document the "
cxl_reset" PCIreset_methodvalue for Type 2 CXL devices and its reset-safety contract. Describe generic PCI driver quiescing, exclusive HDM range reservation, cache and IOMMU handling, function-scope restrictions, and state restoration. Clarify that the method does not offline memory or revoke mappings, and that it requests CXL Reset Memory Clear.Signed-off-by Srirangan Madhavan <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
PATCH v12 12/12 PCI/CXL: Restore CXL state after CXL bus reset
In reply to Srirangan MadhavanCXL bus reset can clear HDM decoder programming and CXL Device DVSEC protocol enables. Reuse the common CXL state restore path after a successful cxl_bus reset while the IOMMU reset block remains active.
Restore PCI config state first so BARs are valid before restoring cached CXL state. If restoration fails, save a disabled PCI command state so later PCI restore does not reopen bus mastering over invalid decoder state.
Reject cxl_bus reset for multifunction devices and PFs with enabled VFs, matching cxl_reset.
--- drivers/cxl/core/resource.c | 18 ++++++++++++++++++ drivers/pci/pci.c | 6 ++++++ include/cxl/cxl.h | 6 ++++++ 3 files changed, 30 insertions(+)
drivers/cxl/core/resource.c +18 -0
@@ -894,6 +894,24 @@ static int cxl_pci_target_reset_prepare(struct pci_dev *pdev,return 0 ; } +int cxl_restore_state_after_pci_reset (struct pci_dev *pdev) +{ +u16 command; +int rc; + +device_lock_assert (&pdev->dev); +guard (rwsem_write)(&cxl_rwsem.region); + +cxl_restore_pci_state_for_hdm_restore (pdev, &command); + rc =cxl_restore_state (pdev); +if (rc) { +cxl_reset_save_disabled_state (pdev); +return rc; + } + +return cxl_reset_save_restored_state (pdev, command); +} +static void cxl_hdm_range_context_init (struct cxl_hdm_range_context *ctx) {INIT_LIST_HEAD (&ctx->ranges);
drivers/pci/pci.c +6 -0
@@ -5027,6 +5027,9 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)if (!dvsec)return -ENOTTY ; +if (dev->multifunction ||pci_num_vf (dev)) +return -ENOTTY ; +if (probe)return 0 ;
@@ -5054,6 +5057,9 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)pci_write_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , reg); +if (!rc) + rc =cxl_restore_state_after_pci_reset (dev); +pci_dev_reset_iommu_done (dev);return rc; }
include/cxl/cxl.h +6 -0
@@ -166,6 +166,7 @@ struct cxl_hdm_info {#ifdef CONFIG_CXL_RESET void pci_cxl_hdm_init (struct pci_dev *pdev);void pci_cxl_hdm_release (struct pci_dev *pdev); +int cxl_restore_state_after_pci_reset (struct pci_dev *pdev);int cxl_reset_function (struct pci_dev *pdev,bool probe);#else static inline void pci_cxl_hdm_init (struct pci_dev *pdev)
@@ -176,6 +177,11 @@ static inline void pci_cxl_hdm_release(struct pci_dev *pdev) { } +static inline int cxl_restore_state_after_pci_reset (struct pci_dev *pdev) +{ +return 0 ; +} +static inline int cxl_reset_function (struct pci_dev *pdev,bool probe) {return -ENOTTY ;
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v12 12/12] PCI/CXL: Restore CXL state after CXL bus reset
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [Critical] cxl_reset_bus_function bypasses memory range safety checks and CPU cache invalidation before initiating a bus reset, risking MCEs and system crashes.
- [High] Priority inversion in pci_reset_fn_methods[] bypasses cxl_reset_bus_function, causing CXL state restoration to fail if SBR is already unmasked. --
commit a0905bda4417fb243d6c3399ed1032ddb3d39e35
Author: Srirangan Madhavan <[email protected]>
Subject: PCI/CXL: Restore CXL state after CXL bus reset
This commit updates the CXL bus reset path to restore CXL HDM decoder programming and CXL Device DVSEC protocol enables after a successful CXL bus reset. It introduces a state restoration function that restores PCI config state first, ensuring BARs are valid before restoring the cached CXL state. It also rejects CXL bus reset for multifunction devices and PFs with enabled VFs.
Quoted 4 lines
drivers/pci/pci.c +0
-0
[ ... ]
drivers/pci/pci.c static int cxl_reset_bus_function(struct pci_dev *dev, bool probe) · line 5062
Quoted 7 lines
drivers/pci/pci.c +3
-0@@ -5054,6 +5057,9 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe)pci_write_config_word (bridge, dvsec +PCI_DVSEC_CXL_PORT_CTL , reg); +if (!rc) + rc =cxl_restore_state_after_pci_reset (dev); +
[Severity: Critical]
This is a pre-existing issue, but does cxl_reset_bus_function() bypass memory range safety checks and CPU cache invalidation before initiating a bus reset?
The hardware bus reset is executed in this function (just prior to this hunk via rc = pci_reset_bus_function()) without any preceding memory range safety checks or cache invalidation. This bypasses the safety mechanisms present in the traditional CXL reset path.
While cxl_reset_function() properly invokes cxl_hdm_ranges_prepare() to verify no memory regions are busy and flush caches, cxl_reset_bus_function() entirely skips these safety checks before issuing the reset.
A hardware PCI bus reset instantly clears CXL HDM decoders. If the CXL backing memory is in active use by the host (e.g., DAX or system RAM), pulling the memory away can cause Machine Check Exceptions (MCE).
Could this allow a VFIO-assigned VM to crash the host via the sysfs reset attribute?
[Severity: High]
This isn't a bug introduced by this patch, but does the priority inversion in pci_reset_fn_methods[] bypass cxl_reset_bus_function() entirely?
The pci_reset_fn_methods[] array defines the fallback priority of reset mechanisms. The cxl_bus entry is placed after bus:
drivers/pci/pci.c:pci_reset_fn_methods[] = {
...
{ pci_reset_bus_function, .name = "bus" },
{ cxl_reset_bus_function, .name = "cxl_bus" },
};Placing cxl_bus after bus ensures that if the standard bus reset is physically possible and unmasked, the CXL-specific wrapper with the state restoration logic will never run.
If a CXL device's Secondary Bus Reset (SBR) is already unmasked, pci_reset_bus_function() will successfully probe and execute the reset. Because it returns 0 on success, the PCI reset core stops iterating and completely bypasses cxl_reset_bus_function().
As a result, the newly added cxl_restore_state_after_pci_reset() is never called, leaving the device with unprogrammed CXL HDM decoders.
Quoted 3 lines
drivers/pci/pci.c +0
-0pci_dev_reset_iommu_done (dev);return rc; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Quoted 8 lines · Srirangan Madhavan
v12 changes from v11 [2]:
- Rejectcxl_resetandcxl_busreset when the PF has enabled VFs, including VFs enabled after reset-method probing.
- Require CXL Reset Memory Clear support and request memory clearing so the generic PCI reset does not preserve tenant memory.
- Cache the CXL Device DVSEC control register during early PCI discovery and restore its CXL.cache and CXL.mem enable bits after CXL Reset and CXL bus reset.
Quoted 8 lines
- Initialize the early PCI HDM cache only from
pci_bus_add_device(), removing the later CXL core retry and its lifecycle locking.
- Reapply and verify a cached HDM decoder lock after decoder restoration and commit, while leaving surviving locked committed state unchanged.
- Convert temporary early HDM-cache ownership to scoped cleanup.
- Document the reset-safety contract, including the iomem ownership requirement and the limits around memory offlining and mapping revocation.
Hi Bjorn, Dave, and CXL maintainers
Based on the review feedback so far, I believe the overall design is in reasonable shape. I have addressed the feedback received so far in v12. The series passes the kernel/module build checks and both cxl_reset and cxl_bus testing on a target system.
I would appreciate any guidance on remaining issues or additional changes needed to move the series further toward acceptance.
Signature · 2 lines
Regards, Srirangan